b4b50d7dc1
feat: Collect the ingester UUIDs and persistence counts in the table
...
And pass them to the parquet file cache, which doesn't use them yet.
2022-12-12 15:52:56 -05:00
b0ba171742
feat: Keep track of ingester UUIDs and counts in IngesterPartition
2022-12-12 15:52:08 -05:00
9c8b55c5be
docs: Fix some wrapping/typos in comments
2022-12-12 14:30:52 -05:00
1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
...
Per partition and starting over each time the ingester restarts.
Fixes #6334 .
2022-12-12 11:45:00 -05:00
33886970ef
refactor: Extract a helper fn for test messages
...
Reduces duplication, makes it easier to see what's different between the
tests, will make it easier to add another field in the next commit
2022-12-12 11:45:00 -05:00
e91d8998a8
Merge pull request #6357 from influxdata/cn/ingester2-uuid
...
feat: Identify each run of an ingester with a Uuid
2022-12-12 16:29:04 +00:00
727efcbdee
Merge branch 'main' into cn/ingester2-uuid
2022-12-12 16:21:15 +00:00
e49ffc02f8
refactor: faster sort key calculation ( #6375 )
...
Avoid nasty string lookups to dermine which columns make a parquet's
sort key.
For #6358 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 15:32:04 +00:00
336ca761a3
chore: Update datafusion pin (to get sqlparser update) ( #6378 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-12 14:39:42 +00:00
6b1c43f01e
refactor: use column IDs for partition cache invalidation ( #6374 )
...
This shall avoid a bunch of string hashing during query planning.
For #6358 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 14:22:28 +00:00
7c28a30d1b
test(ingester2): WAL replay benchmark
...
This adds a simple WAL replay benchmark to ingester2 that executes a
replay of a single line of LP.
Unfortunately each file in the benches directory is compiled as it's own
binary/crate, and as such is restricted to importing only "pub" types.
This sucks, as it requires you to either benchmark at a high level
(macro, not microbenchmarks - i.e. benchmarking the ingester startup,
not just the WAL replay) or you are forced to mark the reliant types &
functions as "pub", as well as all the other types/traits they reference
in their signatures. Because the performance sensitive code is usually
towards the lower end of the call stack, this can quickly lead to an
explosion of "pub" types causing a large amount of internal code to be
exported.
Instead this commit uses a middle-ground; benchmarked types & fns are
conditionally marked as "pub" iff the "benches" feature is enabled. This
prevents them from being visible by default, but allows the benchmark
function to call them.
The benchmark itself is also restricted to only run when this feature is
enabled.
2022-12-12 15:02:36 +01:00
e5322b24b9
feat: Add --token CLI argument, improve update docs about writing ( #6356 )
...
* feat: Add --token CLI argument, improve update docs about writing
* fix: support environment tokens too
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 13:43:15 +00:00
e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) ( #6363 )
...
* chore: Update DataFusion pin to get median fix
* chore: Update for new Expr node
* test: add test for median
* test: add test for coercion of strings to timestamps
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 12:14:00 +00:00
95969ad24f
chore(deps): Bump base64 from 0.13.1 to 0.20.0 ( #6371 )
...
* chore(deps): Bump base64 from 0.13.1 to 0.20.0
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.13.1 to 0.20.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases )
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.13.1...v0.20.0 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 07:07:19 +00:00
9305e4c566
chore(deps): Bump insta from 1.22.0 to 1.23.0 ( #6370 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.22.0...1.23.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-12 06:48:35 +00:00
a66895ecdc
chore(deps): Bump serde from 1.0.149 to 1.0.150 ( #6369 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.149 to 1.0.150.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.149...v1.0.150 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-12 06:35:16 +00:00
061d582a9b
chore: patch object_store to get apache#3274 ( #6362 )
...
* chore: patch object_store to get apache#3274
* chore: Run cargo hakari tasks
* fix: add issue breadcrumb
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 14:52:25 +00:00
db933c44b6
refactor: store reverse column ID map for cached tables ( #6360 )
2022-12-09 11:58:24 +00:00
450b452148
refactor: avoid string-hashing of parquet file column names ( #6359 )
2022-12-09 11:51:18 +00:00
0221820123
feat: rate-limit Jaeger UDP messages ( #6354 )
...
* feat: rate-limit Jaeger UDP messages
The Jaeger UDP protocol provides no way to signal backpressure /
overload. In certain situations, we are emitting that many tracing spans
in a short period of time that the OS, the network, or Jaeger drop them.
While a rate limit is not a perfect solution, it for sure helps a lot
(tested locally).
Note that the limiter does NOT lead to unlimited buffering because we
already have a limited outbox queue in place (see
`trace_exporters::export::CHANNEL_SIZE`).
Fixes #5446 .
* fix: only warn ones when the tracing channel is full
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 09:46:07 +00:00
c3a7575d46
feat: Enable rpc_write on the inner command if it's enabled for tests
...
And only run rpc_write specific tests if the feature is enabled when
running the tests.
2022-12-08 17:45:30 -05:00
0a4df1f3fb
chore: Run tests in CI in both RPC write mode and not
2022-12-08 17:40:04 -05:00
5141cba1db
fix: Only switch into querier RPC write path if ingester addresses specified
...
This enables testing of the querier using the old path with the
rpc_write feature turned on.
2022-12-08 17:40:04 -05:00
b85130cb7c
fix: Make --ingester-addresses optional for the querier in RPC write mode
2022-12-08 17:22:52 -05:00
2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
...
And send that UUID in the Flight response for queries to that ingester
run.
Fixes #6333 .
2022-12-08 17:22:52 -05:00
6014c10866
test: Enable running ingester2/router RPC write servers in e2e tests
...
Add configuration and server types to be able to create server fixtures
for them.
2022-12-08 17:22:52 -05:00
62db312a8f
feat: Switch to escargot to get more control over running Cargo bins
2022-12-08 15:29:44 -05:00
619a2d0856
fix: Remove conflicting arguments from the RouterRpcWriteConfig ( #6355 )
...
These were added in
https://github.com/influxdata/influxdb_iox/pull/6346 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 20:21:37 +00:00
4ded68de62
test: "not found" end2end tests for querier ( #6352 )
...
I couldn't find any end2end tests for these cases and I was kinda
worried that our error codes were wrong. Turns out they are correct, but
let's have some nice tests for this behavior.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 18:17:53 +00:00
64aae97ce7
Merge pull request #6337 from influxdata/cn/ingester2-querier
...
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-08 14:07:36 +00:00
6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
d4e321a2bd
refactor: add additional span around chunk spans ( #6353 )
...
* refactor: add additional span around chunk spans
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-12-08 13:57:32 +00:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
c25afda6cc
fix: `GroupGenerator`/`Converter` panic ( #6351 )
...
Do not poll a ready future.
2022-12-08 11:08:21 +00:00
080aff8f71
fix: account for memory allocations in InfluxRPC group outputs ( #6345 )
...
* fix: account for memory allocations in InfluxRPC group outputs
This should prevent the querier from OOMing.
See https://github.com/influxdata/idpe/issues/16614 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* refactor: pull out constant
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-12-08 09:55:31 +00:00
8c5c6aee0b
chore(deps): Bump filetime from 0.2.18 to 0.2.19 ( #6350 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.18 to 0.2.19.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.18...0.2.19 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-08 09:26:52 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
b1c5ec4dee
fix: Correct compiler errors in places I missed while running crate tests
2022-12-07 10:25:36 -05:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
c97b025751
chore(deps): Bump zstd from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2 ( #6344 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/compare/v0.12.0...v0.12.1 )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-07 09:19:35 +00:00
7e43fe57fa
fix: support InfluxRPC OR-chains w/ arbitrary child nodes ( #6343 )
...
* fix: support InfluxRPC OR-chains w/ arbitrary child nodes
Also convert another assertion regarding child nodes of Eq-nodes into a
proper error.
See https://github.com/influxdata/idpe/issues/16582 .
* test: more tests
2022-12-06 18:04:27 +00:00
abe60ee0dc
chore(deps): Bump serde from 1.0.148 to 1.0.149 ( #6341 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.148 to 1.0.149.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.148...v1.0.149 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 17:10:26 +00:00
687493e8e6
chore(deps): Bump sqlparser from 0.27.0 to 0.28.0 ( #6342 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.27.0...v0.28.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 12:02:01 +00:00
04e9253ade
chore(deps): Bump rustix from 0.36.4 to 0.36.5 ( #6340 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.4 to 0.36.5.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.4...v0.36.5 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 09:47:00 +00:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
2c771ea9f2
chore(deps): Bump mockito from 0.31.0 to 0.31.1 ( #6338 )
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.31.0 to 0.31.1.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.31.0...0.31.1 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:44:07 +00:00
f62b270852
fix: gRPC errors regarding group cols ( #6314 )
...
* fix: gRPC errors regarding group cols
- missing group col prev. produced an "internal error" but should be
"invalid argument"
- duplicate group cols produced a panic but should also be "invalid
argument"
* docs: clarify
2022-12-06 07:36:32 +00:00
cc9d4d302f
Merge pull request #6328 from influxdata/dom/fix-wal-panic
...
fix: do not panic when writer disconnects
2022-12-05 14:56:52 +00:00
609e23988b
Merge branch 'main' into dom/fix-wal-panic
2022-12-05 14:50:07 +00:00
Carol (Nichols || Goulding)
kodiakhq[bot]
Marco Neumann
Andrew Lamb
Dom Dwyer
dependabot[bot]
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Luke Bond