dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
90d53ae383
chore(deps): Bump sqlparser from 0.28.0 to 0.30.0 ( #6479 )
...
* chore(deps): Bump sqlparser from 0.28.0 to 0.30.0
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.28.0 to 0.30.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.28.0...v0.30.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: Update for new API
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-01-03 14:28:15 +00:00
4add55d39e
feat: InfluxQL planner learns how to normalise InfluxQL AST ( #6236 )
...
* chore: Move logic to context, in line with DataFusion SQL
* chore: Add ordering for InfluxQL data types
Ordering is used to determine automatic casting operations. If two
field columns are present in an expression, one float and one integer,
the integer should be cast to a float, such that the final expression
will be a float.
* chore: Add DerefMut trait to collection types
Will allow these collections to be mutated when traversing the InfluxQL
AST.
* chore: Add influxql module with initial AST normalisation implementation
* chore: Add more unit tests and docs
* chore: Run cargo hakari tasks
* chore: Fix link
* chore: Support regular expression expansion and Call expressions
* chore: Add tests for walk_expr functions
* chore: Add insta snapshot files
* chore: Add docs and make API accessible to the crate
* chore: Move to Arc<dyn SchemaProvider> for use in influxql planner
* chore: Move code back; it is better encapsulated here
* chore: Remove redundant attribute
* chore: Improve regex compatibility with InfluxQL / Go
* chore: Style improvement.
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-02 23:48:21 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
ec869d2384
chore(deps): Bump comfy-table from 6.1.3 to 6.1.4 ( #6478 )
...
Bumps [comfy-table](https://github.com/nukesor/comfy-table ) from 6.1.3 to 6.1.4.
- [Release notes](https://github.com/nukesor/comfy-table/releases )
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nukesor/comfy-table/compare/v6.1.3...v6.1.4 )
---
updated-dependencies:
- dependency-name: comfy-table
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-02 15:16:11 +00:00
3d7b4e29b9
chore(deps): Bump nom from 7.1.1 to 7.1.2 ( #6477 )
...
Bumps [nom](https://github.com/Geal/nom ) from 7.1.1 to 7.1.2.
- [Release notes](https://github.com/Geal/nom/releases )
- [Changelog](https://github.com/rust-bakery/nom/blob/main/CHANGELOG.md )
- [Commits](https://github.com/Geal/nom/compare/7.1.1...7.1.2 )
---
updated-dependencies:
- dependency-name: nom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 15:09:27 +00:00
0bc4f43319
chore(deps): Bump rustix from 0.36.5 to 0.36.6 ( #6470 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.5 to 0.36.6.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.5...v0.36.6 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 15:00:42 +00:00
f56c792f79
chore(deps): Bump serde from 1.0.151 to 1.0.152 ( #6468 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.151 to 1.0.152.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.151...v1.0.152 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 10:27:01 +00:00
febaf3c737
chore(deps): Bump clap from 4.0.31 to 4.0.32 ( #6466 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.31 to 4.0.32.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.31...v4.0.32 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 10:14:23 +00:00
7279b9597a
chore(deps): Bump libc from 0.2.138 to 0.2.139 ( #6465 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.138 to 0.2.139.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.138...0.2.139 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-02 09:04:29 +00:00
cd1a604df0
fix: make estimate memory the a query needs higher due to recent observation ( #6476 )
2022-12-30 21:15:39 +00:00
0c944346e5
chore: debug info of estimate file size ( #6475 )
2022-12-30 20:13:21 +00:00
d27e137c39
chore: add debug info for the investigation ( #6472 )
2022-12-29 23:49:29 +00:00
5b9e5b4a92
chore(deps): Bump clap from 4.0.29 to 4.0.31 ( #6464 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.29 to 4.0.31.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.29...v4.0.31 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-22 20:00:23 +00:00
858a41c733
chore(deps): Bump handlebars from 4.3.5 to 4.3.6 ( #6458 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.3.5 to 4.3.6.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.3.6/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.5...v4.3.6 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-22 18:53:45 +00:00
e874fef24a
Merge pull request #6463 from influxdata/dom/drop-replayed-wal-segments
...
fix: drop WAL segments after replay
2022-12-22 16:26:53 +00:00
2636eefe03
fix: only replay non-empty segments
...
If the WAL file was dropped because it is empty, do not retry dropping
it.
2022-12-22 17:18:19 +01:00
66f1628238
fix: drop WAL segments after replay
...
Changes the WAL replay logic to:
* Replay a segment file
* Persist all replayed data
* Drop segment file
* ...repeat...
This ensures old WAL segments are removed once their contents have been
made durable, fixing #6461 .
2022-12-22 16:56:47 +01:00
fcb46a3fa7
Merge pull request #6462 from influxdata/dom/decouple-persist
...
refactor(ingester2): decouple persistence subsystem
2022-12-22 15:50:39 +00:00
456368f71d
refactor(persist): no PersistQueue Clone bound
...
Removes the Clone bound from PersistQueue, also removing the Clone impl
from the PersistHandle.
Instead of wrapping all internal PersistHandle state in Arcs, this
commit changes the system to use a single Arc wrapping the PersistHandle
which is shared.
2022-12-22 15:04:52 +01:00
26eea6078d
refactor(ingester2): decouple persist subsystem
...
Multiple components of the ingester depend on being able to enqueue a
partition's data for persistence. This commit decouples those components
from the concrete PersistHandle by introducing a PersistQueue trait that
defines the desired behaviour, on which the components depend.
This is a much needed clean-up of something I knowingly punted on for
the MVP, and I feel much better about the situation now!
2022-12-22 15:04:51 +01:00
dee9743e52
refactor(persist): decoupled PartitionIter
...
The persist_buffer() fn iterates over all the partitions in a BufferTree
and persists them - however it only depends on one behaviour; getting an
iterator of partitions.
This commit introduces the PartitionIter, an abstraction over anything
that can produce an iterator of PartitionData, decoupling the
persist_buffer() helper (and the callers!) from the concrete BufferTree
type.
2022-12-22 14:53:30 +01:00
e54896e5f8
refactor: extract BufferTree persist helper
...
Extract an existing function for re-use (from the WAL rotation task)
that marks & enqueues all non-empty partitions in a BufferTree for
persistence.
2022-12-22 11:58:41 +01:00
2899f65ad1
Merge pull request #6457 from influxdata/cn/update-rust
...
chore: Update to Rust 1.66
2022-12-21 21:57:36 +00:00
46ff8854ec
fix: Use code backticks around invalid HTML tags in doc strings
2022-12-21 16:36:17 -05:00
5f06ab4609
fix: Allow a clippy lint in a case where I don't think it's an improvement
2022-12-21 14:32:35 -05:00
f121d395cc
refactor: Extract a constructor for PolicyBackend using a HashMap
2022-12-21 14:32:35 -05:00
a5c93ec907
chore: Wrap comments at 100 cols, copyedit
2022-12-21 14:32:35 -05:00
7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy!
2022-12-21 14:32:35 -05:00
bfc74db94c
fix: Use into_values function. Thanks clippy!
2022-12-21 14:32:35 -05:00
a082ae85f2
fix: Use clamp function. Thanks clippy!
2022-12-21 14:32:34 -05:00
39acfc4f0d
fix: Remove needless casts. Thanks clippy!
2022-12-21 14:32:34 -05:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
a672fea405
chore: Update to Rust 1.66
2022-12-21 13:31:31 -05:00
b0cf20eb93
Merge pull request #6454 from influxdata/dom/persist-saturation-metric
...
feat(ingester2): persist saturation metric
2022-12-21 17:21:05 +00:00
6df3c1d4ca
Merge branch 'main' into dom/persist-saturation-metric
2022-12-21 17:07:42 +00:00
ff9a05d157
Merge pull request #6456 from influxdata/dom/fix-sort-key-retry
...
fix(persist): invalidate cached sort key
2022-12-21 17:07:23 +00:00
c85c25e963
Merge branch 'main' into dom/persist-saturation-metric
2022-12-21 17:05:29 +00:00
23dc2c4e06
refactor: consistent metric naming
...
Removes _ns (and incorrect _ms) suffix.
2022-12-21 18:04:20 +01:00
7f01911c81
Merge branch 'main' into dom/fix-sort-key-retry
2022-12-21 17:00:22 +00:00
38d7102710
Merge pull request #6455 from influxdata/cn/why-querier-why
...
fix: Allow partitions from ingesters to overlap in RPC write mode
2022-12-21 16:56:57 +00:00
9cdf512ed1
Merge branch 'main' into dom/fix-sort-key-retry
2022-12-21 16:50:14 +00:00
125228eb8e
Merge branch 'main' into cn/why-querier-why
2022-12-21 16:50:11 +00:00
23b781f274
fix(persist): invalidate cached sort key
...
The sort-key conflict path invalidated the cached sort key in the
PartitionData, but not the cached sort key in the persist's Context. Now
both are invalidated.
2022-12-21 17:45:48 +01:00
56ba3b17de
fix: Allow partitions from ingesters to overlap in RPC write mode
...
This was added in c82d0d8ca6dc02dcdd40a4c656a1ee51f3f9bfee with the
comment:
> Right now this would clearly indicate a bug and before I am trying to
> understand some prod issues, I wanna rule that one out.
In the RPC write path, this isn't a bug, it's quite expected.
2022-12-21 11:32:58 -05:00
ae296886c2
chore(deps): Bump prost-types from 0.11.2 to 0.11.5 ( #6452 )
...
* chore(deps): Bump prost-types from 0.11.2 to 0.11.5
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.11.2 to 0.11.5.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.2...v0.11.5 )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 16:31:35 +00:00
257c155d1e
chore: Line wrapping at 100 cols
2022-12-21 11:18:47 -05:00
679c6a7896
feat(ingester2): persist saturation metric
...
Expose a metric ("ingester_persist_saturated_duration_ns") that records
the cumulative duration of time the persist system has spent in the
"saturated" state.
2022-12-21 17:01:22 +01:00
ed17069087
chore(deps): Bump num_cpus from 1.14.0 to 1.15.0 ( #6451 )
...
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 15:17:55 +00:00
7abef9b7bd
Merge pull request #6453 from influxdata/dom/persist-worker
...
refactor(persist): explicit worker module
2022-12-21 14:18:11 +00:00
Andrew Lamb
dependabot[bot]
Stuart Carnie
Nga Tran
Dom
kodiakhq[bot]
Carol (Nichols || Goulding)