6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
d4e321a2bd
refactor: add additional span around chunk spans ( #6353 )
...
* refactor: add additional span around chunk spans
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-12-08 13:57:32 +00:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
c25afda6cc
fix: `GroupGenerator`/`Converter` panic ( #6351 )
...
Do not poll a ready future.
2022-12-08 11:08:21 +00:00
080aff8f71
fix: account for memory allocations in InfluxRPC group outputs ( #6345 )
...
* fix: account for memory allocations in InfluxRPC group outputs
This should prevent the querier from OOMing.
See https://github.com/influxdata/idpe/issues/16614 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* refactor: pull out constant
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-12-08 09:55:31 +00:00
8c5c6aee0b
chore(deps): Bump filetime from 0.2.18 to 0.2.19 ( #6350 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.18 to 0.2.19.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.18...0.2.19 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-08 09:26:52 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
b1c5ec4dee
fix: Correct compiler errors in places I missed while running crate tests
2022-12-07 10:25:36 -05:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
c97b025751
chore(deps): Bump zstd from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2 ( #6344 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/compare/v0.12.0...v0.12.1 )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-07 09:19:35 +00:00
7e43fe57fa
fix: support InfluxRPC OR-chains w/ arbitrary child nodes ( #6343 )
...
* fix: support InfluxRPC OR-chains w/ arbitrary child nodes
Also convert another assertion regarding child nodes of Eq-nodes into a
proper error.
See https://github.com/influxdata/idpe/issues/16582 .
* test: more tests
2022-12-06 18:04:27 +00:00
abe60ee0dc
chore(deps): Bump serde from 1.0.148 to 1.0.149 ( #6341 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.148 to 1.0.149.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.148...v1.0.149 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 17:10:26 +00:00
687493e8e6
chore(deps): Bump sqlparser from 0.27.0 to 0.28.0 ( #6342 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.27.0...v0.28.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 12:02:01 +00:00
04e9253ade
chore(deps): Bump rustix from 0.36.4 to 0.36.5 ( #6340 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.4 to 0.36.5.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.4...v0.36.5 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 09:47:00 +00:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
2c771ea9f2
chore(deps): Bump mockito from 0.31.0 to 0.31.1 ( #6338 )
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.31.0 to 0.31.1.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.31.0...0.31.1 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:44:07 +00:00
f62b270852
fix: gRPC errors regarding group cols ( #6314 )
...
* fix: gRPC errors regarding group cols
- missing group col prev. produced an "internal error" but should be
"invalid argument"
- duplicate group cols produced a panic but should also be "invalid
argument"
* docs: clarify
2022-12-06 07:36:32 +00:00
cc9d4d302f
Merge pull request #6328 from influxdata/dom/fix-wal-panic
...
fix: do not panic when writer disconnects
2022-12-05 14:56:52 +00:00
609e23988b
Merge branch 'main' into dom/fix-wal-panic
2022-12-05 14:50:07 +00:00
cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics ( #6313 )
...
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics
Closes #6310 .
* refactor: rename and tune default exec mem limits
* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
c3a2ac99aa
chore(deps): Bump insta from 1.21.2 to 1.22.0 ( #6332 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.21.2...1.22.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 10:18:59 +00:00
05cc735849
chore(deps): Bump tower-http from 0.3.4 to 0.3.5 ( #6331 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.4 to 0.3.5.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.4...tower-http-0.3.5 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:18:12 +00:00
fc9431f6ae
chore(deps): Bump assert_cmd from 2.0.6 to 2.0.7 ( #6330 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.6...v2.0.7 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:05:56 +00:00
3740110cb5
chore(deps): Bump libc from 0.2.137 to 0.2.138 ( #6329 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.137 to 0.2.138.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.137...0.2.138 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 08:49:09 +00:00
7cb1636c64
fix: do not panic when writer disconnects
...
When the RPC write disconnects without waiting for completion, the WAL
panics as there is no longer a consumer of the "committed" ACK.
2022-12-03 17:26:42 +01:00
ce75b6b239
Merge pull request #6327 from influxdata/dom/remove-comment
...
chore: remove old comment
2022-12-02 18:05:09 +00:00
228c81c6fb
Merge branch 'main' into dom/remove-comment
2022-12-02 17:58:33 +00:00
0dd84526ac
Merge pull request #6326 from influxdata/dom/drop-empty-wal
...
feat(ingester2): drop empty WAL files
2022-12-02 17:56:31 +00:00
fff6cdf951
Merge branch 'main' into dom/drop-empty-wal
2022-12-02 17:49:47 +00:00
c00a31db6e
Merge pull request #6325 from influxdata/dom/track-caller
...
test: track_caller for make_write_op
2022-12-02 17:49:34 +00:00
de52a24751
chore: remove old comment
...
The ingester will replay the ops, and then immediately trigger a
persist! This comment is done :)
Outstanding is actually deleting the replayed files, which will come
with the ref-counted WAL segment change later.
2022-12-02 18:44:45 +01:00
a8f99d4593
feat(ingester2): drop empty WAL files
...
A shutdown of an ingester that has received no writes leaves an empty
WAL file - these can be deleted at startup.
2022-12-02 18:43:32 +01:00
c3a88f105d
test: track_caller for make_write_op
...
Changes the stack trace to point at the call site when a panic occurs
within make_write_op, and adds a unique message per panic explaining the
issue.
2022-12-02 18:42:28 +01:00
533a6581be
Merge pull request #6324 from influxdata/dom/drop-wal-segments
...
feat(ingester2): drop WAL segments after persist
2022-12-02 17:42:09 +00:00
26bf54d041
feat(ingester2): drop WAL segments after persist
...
Changes the WAL rotation task to drop the WAL segment after the
partition data has completely persisted.
This logic contains a race condition outlined (at length) in the code
comments, along with a plan to resolve it once I'm back from holiday.
For now, the extremely low likelihood and minor impact of the race
occurring is likely acceptable for testing purposes.
2022-12-02 18:29:35 +01:00
464fcba98f
Merge pull request #6323 from influxdata/dom/wal-rotate-persist
...
feat(ingester2): persist data when rotating WAL segments
2022-12-02 17:29:00 +00:00
f145d6415f
feat(ingester2): persist partitions on WAL rotate
...
Changes the WAL rotation code to cause the current set of partitions to
be submitted for persistence.
Because rotating the WAL segment and triggering persistence is not
atomic (not under an exclusive lock preventing writes) the persisted
buffers MAY contain writes that appear in the new segment file.
In the happy/non-crash path, this will have no effect, however if the
ingester crashes and replays the WAL files, these writes will be
duplicated into object storage (where compaction will resolve the issue)
- this seems like a good trade-off, allowing us to avoid blocking write
requests for as long as it takes to rotate the WAL and mark all
partitions as persisting.
(though currently WAL files are not dropped and thus everything is
replayed all the time.)
2022-12-02 17:18:39 +01:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
e234187a94
Merge pull request #6322 from influxdata/dom/buffer-tree-iter
...
refactor(ingester2): BufferTree partition iterator
2022-12-02 16:17:06 +00:00
1137f2fc7e
Merge branch 'main' into dom/buffer-tree-iter
2022-12-02 16:10:38 +00:00
15e1a3dd29
Merge pull request #6318 from influxdata/dom/persist
...
feat(ingester2): parallel partition persistence
2022-12-02 16:06:02 +00:00
85a41e34df
refactor(ingester2): BufferTree partition iterator
...
Expose an iterator of PartitionData within a BufferTree.
2022-12-02 17:01:34 +01:00
346337ad9f
Merge branch 'main' into dom/persist
2022-12-02 15:59:18 +00:00
a51848b361
fix: Use client_util GrpcConnection instead of tonic Channel ( #6320 )
...
* fix: Use client_util GrpcConnection instead of tonic Channel
* refactor: include server addr in error
Co-authored-by: Dom <dom@itsallbroken.com>
2022-12-02 15:57:42 +00:00
f524687602
feat(ingester2): parallel partition persistence
...
Implements actor-based, parallel persistence in ingester2 with
controllable fan-out parallelism and queue depths.
This implementation encapsulates the complexity of persistence, queuing
and parallelism - the caller simply uses the handle to persist a
partition, while the actor handles fan-out to a set of persistence
workers, compaction in a separate thread-pool, and optional completion
notifications.
By consistently hashing persist jobs onto workers, parallelism is
achieved across partitions, but serialisation of partition persists is
enforced so that the sort key update is correctly serialised.
2022-12-02 16:34:03 +01:00
85abd09be0
Merge pull request #6316 from influxdata/dom/partition-queue
...
perf(ingester2): partition persist queue
2022-12-02 15:25:25 +00:00
e25920c7c0
Merge branch 'main' into dom/partition-queue
2022-12-02 15:18:44 +00:00
0e21f3dcfe
Merge pull request #6303 from influxdata/cn/ingester2
...
feat: Add an ingester2 command behind the rpc_write feature flag
2022-12-02 13:46:30 +00:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
kodiakhq[bot]
Marco Neumann
Andrew Lamb
dependabot[bot]
Carol (Nichols || Goulding)
Luke Bond
Dom Dwyer
Carol (Nichols || Goulding)