bf6ab7fd88
fix: error message typo
...
columns -> tables for table limit error.
2023-02-06 18:03:26 +01:00
3f2eb54bce
test(router): catalog service limit errors
...
Assert the service limit error messages from the catalog.
2023-02-06 17:55:09 +01:00
3881e11734
test(router): service limit error messages
...
Assert the user-facing service limit error messages.
2023-02-06 17:43:37 +01:00
114bafe9a1
perf(router): cached table limit enforcement
...
Use the namespace schema cache in the router to enforce the
per-namespace table limit (service protection limit), adding O(1)
overhead to the existing column limit evaluation logic.
Prior to this commit, each request that would breach the table limit
would be (potentially partially) applied to the catalog and return an
error. Every subsequent request creating a new table continued to cause
a catalog query, unnecessarily adding load proportional to request
counts.
After this commit, catalog requests are sent when the router instance
can determine (to the best of it's ability, see below) that the request
will not cause the namespace to exceed the table limit.
Because this uses cached schemas, the actual state set of tables may
have changed - this will cause inconsistent enforcement and spurious
errors in the same way it currently does for the column limit. For more
details (and to track a resolution) see:
https://github.com/influxdata/influxdb_iox/issues/5957
2023-02-06 17:43:26 +01:00
dfa4ab2585
perf(router): fast-path column limit for new table
...
When validating column limits for new tables, skip the column set
generation and union operations against the empty existing column set.
2023-02-06 17:33:56 +01:00
a1764ee7cb
refactor: ExactSizeIterator for columns iter
...
Adds ExactSizeIterator bounds to the MutableBatch::column() iter,
allowing O(1) length discovery / pre-allocation optimisations for
container collection.
2023-02-06 17:33:56 +01:00
a633964f2b
feat(catalog): return max table limit in schema
...
The maximum number of tables is part of the Namespace, which is already
loaded in its entirety. This commit copies the value into the
NamespaceSchema, making it available for the router to utilise.
2023-02-06 17:33:55 +01:00
639964ec15
Merge pull request #6869 from influxdata/dom/http-error-ct
...
fix: correct http content-type header for error responses
2023-02-06 16:20:17 +00:00
b8ec022ea6
test(e2e): HTTP error response
...
Ensure a HTTP error response contains a well-formed JSON structure
containing "code" and "message" fields (for backwards compatibility with
existing InfluxDB versions) and a correct "content-type" header.
2023-02-06 17:10:30 +01:00
4aa5826300
fix: return content-type for HTTP errors
...
No "Content-Type" header has ever been returned, even though the
response body has always been hard-coded to a JSON string.
This commit returns a content type of "application/json" for all
JSON-encoded HTTP errors.
2023-02-06 17:08:21 +01:00
5b6b7cbf86
feat: rework commit log+metrics ( #6871 )
...
* feat: add more information to commit logging
* feat: rework commit metrics
- more consistent metric names
- histograms per file and per job
- more histogram types (number of files, rows, bytes)
2023-02-06 15:56:24 +00:00
ae10b33f0c
chore: Update datafusion ( #6868 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update error message
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-06 14:22:45 +00:00
f83db0a52e
refactor: pass more information to `commit` ( #6867 )
...
This is NOT used yet but will greatly help w/ logging and metrics. E.g.
it allows us to count rows and bytes of in/out-flow, create per-file
histograms of bytes/rows, and more.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 12:29:32 +00:00
d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` ( #6756 )
...
* chore: update DataFusion
* fix: test
* chore: format
* chore: clippy
* chore: update arrow
* chore: arrow upgrade fallout
* chore: Run cargo hakari tasks
* chore: remove failing warm compaction test
* fix: flight error propagation
* chore: update parquet size
* fix: Update error message
* chore: Update parquet metadata test
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:35:39 +00:00
52b43c40bc
refactor: use "endless" stream for compactor work ( #6803 )
...
Instead of looping and polling a fresh set of partitions and
constructing a stream from that, use an endless stream instead. This
helps w/ efficiency during roll-overs since we can already start to
process the next set of partitions while the last ones from the previous
round are still in-progress.
Closes #6750 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:11:39 +00:00
18287dc30e
fix: do not silently ignore errors ( #6866 )
...
Fixes https://github.com/influxdata/idpe/issues/17085 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 11:01:53 +00:00
4fb1b1948a
feat: throttle partitions that do not receive commits ( #6831 )
...
* feat: throttle partitions that do not receive commits
* test: add failing test
* fix: partition ID in "unique" combo
* fix: partition ID in "throttle" combo
* docs: improve
Co-authored-by: Dom <dom@itsallbroken.com>
---------
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 09:59:33 +00:00
9f3a068d22
chore(deps): Bump tikv-jemalloc-sys ( #6865 )
...
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator ) from 0.5.2+5.3.0-patched to 0.5.3+5.3.0-patched.
- [Release notes](https://github.com/tikv/jemallocator/releases )
- [Changelog](https://github.com/tikv/jemallocator/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tikv/jemallocator/compare/tikv-jemalloc-sys-0.5.2...tikv-jemalloc-sys-0.5.3 )
---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 09:28:19 +00:00
362a18343f
chore(deps): Bump proptest from 1.0.0 to 1.1.0 ( #6863 )
...
Bumps [proptest](https://github.com/proptest-rs/proptest ) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/proptest-rs/proptest/releases )
- [Changelog](https://github.com/proptest-rs/proptest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/proptest-rs/proptest/compare/1.0.0...v1.1.0 )
---
updated-dependencies:
- dependency-name: proptest
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 09:11:44 +00:00
a0c551618b
chore(deps): Bump rustix from 0.36.7 to 0.36.8 ( #6862 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.7 to 0.36.8.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.7...v0.36.8 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 08:57:17 +00:00
d122f0d087
chore(deps): Bump is-terminal from 0.4.2 to 0.4.3 ( #6861 )
...
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/sunfishcode/is-terminal/releases )
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.2...v0.4.3 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-06 08:46:17 +00:00
cd6d69c9b1
docs: high-level documentation for `compactor2` ( #6818 )
...
* docs: high-level documentation for `comapctor2`
* docs: typos & clarification
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* docs: improve wording
* fix: typo
---------
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-06 08:37:57 +00:00
6f4e287a3a
chore(deps): Bump serde_json from 1.0.91 to 1.0.92 ( #6860 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.91 to 1.0.92.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.91...v1.0.92 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 08:27:41 +00:00
e85de74a5d
feat: partition filters for TargetLevel version and a complete test ( #6858 )
...
* feat: partition filters for TargetLevel version and a complete test
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: run fmt after applying review suggestions in git
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-04 19:54:41 +00:00
f1561676de
chore: refactor splitting files by level ( #6857 )
...
* chore: refactor splitting files by level
* fix: docs
2023-02-04 16:05:04 +00:00
2b36692488
refactor: reduce more code duplication ( #6856 )
2023-02-04 15:12:06 +00:00
77b2f53cc4
refactor: reduce code duplication in compactor ( #6855 )
2023-02-04 15:00:55 +00:00
edacc21e8e
refactor: extract timerange calculations ( #6854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-04 13:40:55 +00:00
f46881496d
feat: split files to upgrade and compact ( #6852 )
...
* feat: first step to split upgrade
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: address review comments
* chore: rename per review request
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-04 13:17:29 +00:00
670b811a2e
feat: split non-overlapping files and not compact them ( #6849 )
...
* chore: start the overlap filter module
* feat: split non-overlapping files and not compact them
* chore: remove unused file
* refactor: address review comment
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-03 23:25:34 +00:00
0dba335e94
Merge pull request #6826 from influxdata/ntran/c2-algo2-target-level
...
feat: compaction target level detection
2023-02-03 17:46:40 -05:00
db3ae70366
chore: merge main to branch
2023-02-03 17:38:05 -05:00
f071a3674e
refactor: rename TargetLevelDetection to TargetLevelChooser
2023-02-03 17:28:33 -05:00
cf929d0535
Merge pull request #6847 from influxdata/cn/port-more-query-tests
...
test: Port more query tests to kafkaless
2023-02-03 20:27:35 +00:00
ca8558e636
chore: Unnest test code out of the module and clean up long lines
2023-02-03 14:20:30 -05:00
b17d1c0862
test: Port more query tests to kafkaless
2023-02-03 13:56:01 -05:00
8ab71d08ba
Merge pull request #6808 from influxdata/cn/upgrade-rust
...
chore: Upgrade to Rust 1.67
2023-02-03 18:38:42 +00:00
87c9dcbd80
Merge branch 'main' into cn/upgrade-rust
2023-02-03 18:30:40 +00:00
f1d0568272
refactor: Break snapshot_comparison into smaller modules ( #6846 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 18:17:21 +00:00
b0226e9baf
test: Update expected spans in query tracing
2023-02-03 13:06:20 -05:00
38b204c604
fix: Update test expectation, need to investigate
2023-02-03 13:06:20 -05:00
b3aa52700b
fix: Box an error containing an error to reduce size
...
As suggested by
<https://rust-lang.github.io/rust-clippy/master/index.html#result_large_err >.
2023-02-03 13:06:20 -05:00
2cdfaba4e2
fix: Remove unnecessary lifetime annotations. Thanks clippy!
...
These lifetime annotations are the same as what the lifetime elision
rules would produce, so they're not needed.
See <https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes >
2023-02-03 13:06:19 -05:00
ae944668c1
fix: Remove unneeded let underscore for awaited future
...
See <https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_future >
This might be a false positive of the lint because we are awaiting the
future, but it's not needed as nothing is must_use here, so we can avoid
the lint by removing this.
2023-02-03 13:06:19 -05:00
11b014f64f
fix: Allow large error variants where it likely doesn't matter
2023-02-03 13:06:19 -05:00
30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
fbfbe1adb4
fix: Remove track_caller from async fns as it's a no-op
...
Rust 1.67 now says:
warning: `#[track_caller]` on async functions is a no-op
= note: see issue #87417 <https://github.com/rust-lang/rust/issues/87417 > for more information
= note: `#[warn(ungated_async_fn_track_caller)]` on by default
2023-02-03 13:06:01 -05:00
a82f7916d0
chore: Upgrade to Rust 1.67
2023-02-03 13:06:01 -05:00
0ed9754c4a
feat: split higher target level files from compact ones ( #6829 )
...
* feat: split upgrade & compact
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* chore: merge main to branch
* refactor: address review comments
* fix: doc comment fo []
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-03 17:57:38 +00:00
a9433302dd
Revert "chore: preserve `query_tests2` output in CI ( #6782 )" ( #6844 )
...
This reverts commit d7d8c84203
2023-02-03 17:49:33 +00:00
Dom Dwyer
Marco Neumann
Andrew Lamb
Raphael Taylor-Davies
dependabot[bot]
Nga Tran
kodiakhq[bot]
Carol (Nichols || Goulding)