Commit Graph

9409 Commits (c493c3da28ceefd72139e768cf425a2fd3ff7ff9)

Author SHA1 Message Date
dependabot[bot] c493c3da28
chore(deps): Bump comfy-table from 6.1.0 to 6.1.1 (#5903)
Bumps [comfy-table](https://github.com/nukesor/comfy-table) from 6.1.0 to 6.1.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases)
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nukesor/comfy-table/compare/v6.1.0...v6.1.1)

---
updated-dependencies:
- dependency-name: comfy-table
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-19 09:26:27 +00:00
Marco Neumann e1b50227f8
refactor: avoid some clones while caching ns schema (#5896)
Found while reviewing the code.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-19 06:28:15 +00:00
Andrew Lamb d706f8221d
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 (#5900)
* chore: Update datafusion and  `arrow` / `parquet` / `arrow-flight` 25.0.0

* chore: Update for structure changes

* chore: Update for new projection pushdown

* chore: Run cargo hakari tasks

* fix: fmt

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
Carol (Nichols || Goulding) c28ac4a3c3
fix: Return an error for unsupported SQL queries (#5876)
* test: Failing tests for unsupported queries

* fix: Catch unsupported SQL operations and error rather than return nothing

* test: Document a few more error messages that come through DataFusion

* refactor: Extract a Step to make query error tests nicer to read and write

* fix: update tests for new error codes

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
Andrew Lamb 76d3fbbe53
refactor: remove old `packers` crate (#5893)
* refactor: remove old packers code

* refactor: Remove from workspace
2022-10-18 19:00:28 +00:00
Dom e6700fc249
Merge pull request #5895 from influxdata/dom/ttbr-histogram
refactor: use histogram to record TTBR
2022-10-18 16:05:20 +01:00
Dom Dwyer c63312ce12 refactor: use histogram to record TTBR
Changes the TTBR metric from a gauge to a histogram so that observations
maintain a time dimension.
2022-10-18 16:29:09 +02:00
Nga Tran 6672404711
docs: part 2 of compactor documents with best practice and guidelines (#5880)
* docs: part 2 of compactor

* fix: typos

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: address review comments

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 13:07:48 +00:00
Marco Neumann 9b7a7b5e40
refactor: remove read buffer (#5891) 2022-10-18 12:46:17 +00:00
Marco Neumann 9310d26b92
refactor: remove querier dual chunk stage (#5890) 2022-10-18 12:38:30 +00:00
Marco Neumann d89aae88eb
refactor: remove querier read buffer cache (#5889)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 10:55:04 +00:00
Andrew Lamb e16b04891e
fix: improve tonic status codes returned for query errors (#5864)
* fix: improve tonic status codes returned for query errors

* fix: update tests

* fix: Add rationale for categorization and avoid recurson handling DataFusionError::Context

* fix: Apply suggestions from code review

Thanks @carols10cents!

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Update tests

* fix: add future TOOD

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 10:47:52 +00:00
Andrew Lamb d78591f52e
feat: add more logging to SQL execution via flight (#5877)
* refactor: Extract do_get implementation into its own function

* feat: add more logging to SQL execution

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 10:36:11 +00:00
Marco Neumann 819dbe9e0c
refactor: remove querier chunk load settings (#5888)
We no longer use dual-state ReadBuffer/Parquet chunks.
2022-10-18 10:22:46 +00:00
Andrew Lamb 6f931411f3
feat: read from parquet and only parquet (#5879)
* feat: query only from parquet

* Revert "feat: query only from parquet"

This reverts commit 5ce3c3449c0b9c90154c8c6ece4a40a9c083b7ba.

* Revert "revert: disable read buffer usage in querier (#5579) (#5603)"

This reverts commit df5ef875b4.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 10:09:48 +00:00
dependabot[bot] 235abf65c5
chore(deps): Bump clap from 4.0.15 to 4.0.16 (#5883)
Bumps [clap](https://github.com/clap-rs/clap) from 4.0.15 to 4.0.16.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.15...v4.0.16)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-18 09:19:06 +00:00
dependabot[bot] f3c27c5c71
chore(deps): Bump dotenvy from 0.15.5 to 0.15.6 (#5881)
Bumps [dotenvy](https://github.com/allan2/dotenvy) from 0.15.5 to 0.15.6.
- [Release notes](https://github.com/allan2/dotenvy/releases)
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.5...v0.15.6)

---
updated-dependencies:
- dependency-name: dotenvy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-18 07:06:40 +00:00
Marco Neumann b3b22cff37
feat: harden and polish backoff system (#5873)
* refactor: `impl Iterator for Backoff`

* feat: `Backoff::fade_to`

* feat: impl `PartialEq` for `BackoffConfig`

* fix: harden backoff logic

* fix: harden backoff base handling

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-17 15:35:26 +00:00
Nga Tran 6ebad08036
docs: compactor document and how to resolve OOMs (#5868)
* docs: compactor document

* chore: Apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* docs: fix indent and add an explanation why OOMs won't go away by themselves

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-17 14:25:41 +00:00
Andrew Lamb ece0ce440d
chore: update datafusion (#5869)
* chore: update datafusion

* chore: Update split conjunction api

* chore: remove code that was upstreamed

* chore: simplification is better upstream

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-17 13:22:16 +00:00
kodiakhq[bot] 1e33715a9d
Merge pull request #5871 from influxdata/dom/use-rskafka
revert: rdkafka/rskafka swapping (#5844)
2022-10-17 10:53:50 +00:00
Dom Dwyer 5d835d5047 revert: rdkafka/rskafka swapping (#5844)
This reverts commit 442a7ff2a4.

This commit restores rskafka as the producer Kafka client, effectively
undoing the change made (and follow-up PRs) in:

    https://github.com/influxdata/influxdb_iox/pull/5800
2022-10-17 12:34:28 +02:00
Marco Neumann eec6bf40ce
chore: update deps that dependabot cannot handle (#5870)
* chore: update `tracing-subscriber`

* chore: update `crossbeam-utils`

* chore: update `tokio`

* chore: update `uuid`
2022-10-17 07:05:45 +00:00
Stuart Carnie 58e4c2697f
fix: Address InfluxQL parsing issues (#5857)
* feat: Reimplement keywords as BTreeSet for more efficient lookup

Added a `Token` type to perform case-insensitive comparisons to avoid
allocations when performing lookups in the set.

Also introduced new `keyword` combinator to replace use of
`tag_no_case`. This is necessary to prevent eagerly recognising tags
without the appropriate separator. For example, `tag_no_case("OR")`
will recognise `ORDER` in a conditional expression.

The new combinator a `keyword` follows the pattern:

```
keyword ::= (a..z | A..Z)* keyword_separator
keyword_separator ::= ' '  | '('  | ')' | ';' | ',' | '=' |
                      '\n' | '\t'
```

* fix: Allow quoted time; using more efficient case-insensitive comparison

* fix: Digits to right of decimal point are optional

* fix: More idiomatic use of zip

* fix: Use `keyword` combinator and adjust whitespace handling

* fix: <> is a valid alias for the != operator

* fix: Special handling of DISTINCT identifier for function call name

* chore: Add tests

* chore: Feedback to remove unnecessary comments

* chore: Switch from BTreeSet to HashSet – thanks @domodwyer!
2022-10-16 23:33:09 +00:00
kodiakhq[bot] 2a1089576b
Merge pull request #5862 from influxdata/dom/service-limit-http-code
refactor: return HTTP 400 for service limits
2022-10-14 18:42:53 +00:00
kodiakhq[bot] 0d23c0013c
Merge branch 'main' into dom/service-limit-http-code 2022-10-14 18:34:59 +00:00
Andrew Lamb ff7b571fae
feat: Log object store configuration on startup (#5865) 2022-10-14 18:10:20 +00:00
kodiakhq[bot] 9628cb8ff1
Merge pull request #5856 from influxdata/cn/l2-file-size
feat: Limit L1 -> L2 compaction based on file size
2022-10-14 15:33:17 +00:00
kodiakhq[bot] e6fd0bc621
Merge branch 'main' into cn/l2-file-size 2022-10-14 15:26:13 +00:00
Carol (Nichols || Goulding) 475db15a51
fix: Compact one L1 file even if it is over the file size limit 2022-10-14 10:55:34 -04:00
Andrew Lamb 8021b8be0b
fix: Use Display rather than Debug when logging errors (#5859)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-14 14:43:11 +00:00
kodiakhq[bot] 0044cd0f81
Merge pull request #5860 from influxdata/dom/reduce-column-limit
refactor(iox_catalog): reduce default column limit
2022-10-14 14:14:29 +00:00
Dom Dwyer 895ac97d75 refactor: return HTTP 400 for service limits
Changes the HTTP response code from 429 to 400 for requests that exceed
a service protection limit (table count, per-table column count).
2022-10-14 16:12:56 +02:00
Dom Dwyer afdc008855 fix: correct default limits 2022-10-14 16:05:56 +02:00
Dom Dwyer e4179605df test: assert default service limit values
Adds tests that assert the default service limit values.
2022-10-14 14:46:34 +02:00
Dom Dwyer 46bbee5423 refactor: reduce default column limit
Reduces the default number of columns allowed per-table, from 1,000 to
200.
2022-10-14 14:45:48 +02:00
Luke Bond 475c8a0704
fix: only emit ttbr metric for applied ops (#5854)
* fix: only emit ttbr metric for applied ops

* fix: move DmlApplyAction to s/w accessible

* chore: test for skipped ingest; comments and log improvements

* fix: fixed ingester test re skipping write

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-14 12:06:49 +00:00
Carol (Nichols || Goulding) efb964c390
feat: Enforce table column limits from the schema cache (#5819)
* fix: Avoid some allocations by collecting instead of inserting into a vec

* refactor: Encode that adding columns is for one table at a time

* test: Add another test of column limits

* test: Add below/above limit tests for create_or_get_many

* fix: Explicitly DO NOT check column limits when inserting many columns

* feat: Cache the max_columns_per_table on the NamespaceSchema

* feat: Add a function to validate column limits in-memory

* fix: Provide more useful information when over column limits

* fix: Swap types to remove intermediate allocation

* docs: Explain the interactions of the cache and the column limits

* test: Actually set up test that showcases column limit race condition

* fix: Allow writing to existing columns even if table is over column limit

Co-authored-by: Dom <dom@itsallbroken.com>
2022-10-14 11:34:17 +00:00
Stuart Carnie 81722dc19b
feat: AST traversal using Visitor pattern (#5796)
* feat: Partition implementation of Visitable for InfluxQL AST

* feat: Added consistent structures for each clause to simplify visitor

Continued to expand `accept` and `pre` / `post` visit implementations.

* feat: Added insta and tests using snapshots (thanks @crepererum)

The insta crate simplifies the process of validating the combination of
visitor and accept implementations are called and in the correct order.

* chore: Run cargo hakari tasks

* feat: Added remaining snapshot tests

Some tests are failing as some minor type changes must be added along
with the addition of related visitor functions.

* feat: Add types to represent each clause in numerous statements

These clauses permit distinct visit functions on the `Visitor` type.

* chore: Reformat `SELECT`

* chore: Explicitly specify access to export selected types only

This required completing all the missing documentation for the exported
types.

* chore: Update Cargo.lock

* chore: macro to implement common traits and hide 0th tuple element

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-10-13 22:37:49 +00:00
Carol (Nichols || Goulding) b2df492558
feat: Limit L1 -> L2 compaction based on file size 2022-10-13 16:20:22 -04:00
Andrew Lamb 9134ccd6c3
chore: Update datafusion again (#5855)
* chore: Update datafusion

* chore: Updates for changes in datafusion

* chore: more updates

* fix: update doc example

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-13 19:18:57 +00:00
kodiakhq[bot] 20095c16c9
Merge pull request #5849 from influxdata/cn/l2-limits
test: Add verification that aspects of L1 -> L2 compaction are working as expected
2022-10-13 18:32:51 +00:00
Carol (Nichols || Goulding) 082d045633
fix: Update test compactor limit values 2022-10-13 14:25:10 -04:00
Carol (Nichols || Goulding) cdd01eb3fc
test: Verify L1 files chosen for compaction are limited by the memory budget 2022-10-13 14:15:39 -04:00
Carol (Nichols || Goulding) 3cdf2556ec
test: Verify L1 files in a group by themselves get upgraded to L2 2022-10-13 14:15:39 -04:00
kodiakhq[bot] 1ebf2bcc3c
Merge pull request #5852 from influxdata/dom/no-persist-lookups
perf(ingester): reduce catalog queries during persist()
2022-10-13 15:21:08 +00:00
kodiakhq[bot] 3039b5877b
Merge branch 'main' into dom/no-persist-lookups 2022-10-13 15:13:36 +00:00
Dom Dwyer 86d28d3359 fix: update cached sort key
Once persist() has successfully updated the sort key in the catalog, set
the partition sort key cache to reflect the new value.
2022-10-13 17:12:07 +02:00
Nga Tran fab3cd845c
feat: add memory need for output streams into our estimation (#5847)
* feat: add memory need for output streams into our estimation

* test: modify tests to have better coverage

* refactor: use constants isntead of numbers

* chore: address review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-13 14:31:19 +00:00
kodiakhq[bot] b3786e4280
Merge pull request #5853 from influxdata/dom/consistent-shard
refactor(ingester): log shard_id in op result
2022-10-13 13:52:42 +00:00