c733f27eea
chore: cleanup
2023-01-04 16:00:42 -05:00
72977bf250
feat: catalog query to select partitions with recently created files
2023-01-04 15:54:46 -05:00
1088baea3d
chore: index for selecting partitions with parquet files created after a given time ( #6496 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 18:07:07 +00:00
25f275f1b0
refactor: improve influxRPC warning logging ( #6493 )
...
The current version is barely readable because the logged schema w/ all
it's metadata is soooo long.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 16:52:32 +00:00
91680854ce
feat(replication): define replication RPC API
...
Defines the rough outline of an replication RPC API. More details/docs
to follow.
2023-01-04 17:37:32 +01:00
d4e890d01f
Merge pull request #6480 from influxdata/cn/groundwork
...
test: Lay the groundwork for porting query_tests to the RPC write path
2023-01-04 15:22:05 +00:00
afaaedc758
test: Reproduce the test for querier caching
2023-01-04 10:15:35 -05:00
9f7dede433
refactor: Reorganize tests to make deleting half the tests easier
...
When we switch to RPC write mode, the with_kafka module can be deleted
and the kafkaless_rpc_write module can be unwrapped.
2023-01-04 10:15:35 -05:00
c464487bb2
test: Ability to set up a querier2 that doesn't connect to any ingesters
2023-01-04 10:06:57 -05:00
08ceb4ee48
test: Check catalog for new Parquet files to know when data is persisted
2023-01-04 10:06:57 -05:00
e49bee0c26
test: Make test ingester2 instances either persist very quickly or not at all
2023-01-04 10:06:57 -05:00
afd09d96be
test: Create a Querier2 test server type
2023-01-04 10:06:56 -05:00
96029654ab
test: Add a shared MiniCluster for version 2 services
2023-01-04 10:06:56 -05:00
bc5e37c1bf
fix: Correct RecordBatch splitting logic ( #6484 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 14:04:22 +00:00
17239d5feb
Merge pull request #6415 from influxdata/feat/delete-namespace-api
...
feat: delete namespace api
2023-01-04 13:50:58 +00:00
69c82e590c
Merge branch 'main' into feat/delete-namespace-api
2023-01-04 13:44:08 +00:00
6843eee1d2
feat: Extract encoding from `RecordBatch` --> `FlightData` from flight implementations ( #6460 )
...
* feat: Extract encoding from `RecordBatch` --> `FlightData` from flight implementations
Refactor existing flight server impl
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fixup code review comments
* fix: update for more details
* fix: Update names / types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 13:36:16 +00:00
fb8ead8515
Merge branch 'main' into feat/delete-namespace-api
2023-01-04 11:55:01 +00:00
b39ee78160
chore(deps): Bump arrow-ord from 29.0.0 to 30.0.0 ( #6488 )
...
* chore(deps): Bump arrow-ord from 29.0.0 to 30.0.0
Bumps [arrow-ord](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.0 )
---
updated-dependencies:
- dependency-name: arrow-ord
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-04 11:19:44 +00:00
1b44f4c1c6
chore(deps): Bump arrow-string from 29.0.0 to 30.0.0 ( #6490 )
...
* chore(deps): Bump arrow-string from 29.0.0 to 30.0.0
Bumps [arrow-string](https://github.com/apache/arrow-rs ) from 29.0.0 to 30.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/29.0.0...30.0.0 )
---
updated-dependencies:
- dependency-name: arrow-string
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-04 11:13:04 +00:00
8bb56f6ef5
chore(deps): Bump insta from 1.23.0 to 1.24.0 ( #6491 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.23.0 to 1.24.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.23.0...1.24.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-04 07:11:08 +00:00
aacd91db94
feat: Teach InfluxQL rewriter how to name columns ( #6481 )
...
* feat: Add timestamp data type
* feat: Add with_quiet API to suppress output to STDOUT
* fix: Field name resolution to match InfluxQL
* refactor: Allow TestChunks to be directly accessed
This will be useful when testing the InfluxQL planner.
* fix: Add Timestamp case to var_ref module
* feat: Add InfluxQL compatible column naming
* chore: Add doc comment.
* fix: keywords may be followed by a `!` such as `!=`
* fix: field_name improvements
* No longer clones expressions
* Explicitly handle all Expr enumerated items
* more tests
* fix: collision with explicitly aliased column
Fixes case where column is explicitly aliased to an auto-named variant.
Test case added to validate.
2023-01-04 00:55:18 +00:00
dbe52f1ca1
chore: Upgrade datafusion ( #6467 )
...
* chore: Update datafusion
* fix: Update for new apis
* chore: Update expected plan
* fix: Update for new config construction
* chore: update clippy
* fix: Fix error codes
* fix: update another test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-03 15:29:11 +00:00
90d53ae383
chore(deps): Bump sqlparser from 0.28.0 to 0.30.0 ( #6479 )
...
* chore(deps): Bump sqlparser from 0.28.0 to 0.30.0
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.28.0 to 0.30.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.28.0...v0.30.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: Update for new API
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2023-01-03 14:28:15 +00:00
4add55d39e
feat: InfluxQL planner learns how to normalise InfluxQL AST ( #6236 )
...
* chore: Move logic to context, in line with DataFusion SQL
* chore: Add ordering for InfluxQL data types
Ordering is used to determine automatic casting operations. If two
field columns are present in an expression, one float and one integer,
the integer should be cast to a float, such that the final expression
will be a float.
* chore: Add DerefMut trait to collection types
Will allow these collections to be mutated when traversing the InfluxQL
AST.
* chore: Add influxql module with initial AST normalisation implementation
* chore: Add more unit tests and docs
* chore: Run cargo hakari tasks
* chore: Fix link
* chore: Support regular expression expansion and Call expressions
* chore: Add tests for walk_expr functions
* chore: Add insta snapshot files
* chore: Add docs and make API accessible to the crate
* chore: Move to Arc<dyn SchemaProvider> for use in influxql planner
* chore: Move code back; it is better encapsulated here
* chore: Remove redundant attribute
* chore: Improve regex compatibility with InfluxQL / Go
* chore: Style improvement.
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-02 23:48:21 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
ec869d2384
chore(deps): Bump comfy-table from 6.1.3 to 6.1.4 ( #6478 )
...
Bumps [comfy-table](https://github.com/nukesor/comfy-table ) from 6.1.3 to 6.1.4.
- [Release notes](https://github.com/nukesor/comfy-table/releases )
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nukesor/comfy-table/compare/v6.1.3...v6.1.4 )
---
updated-dependencies:
- dependency-name: comfy-table
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-02 15:16:11 +00:00
3d7b4e29b9
chore(deps): Bump nom from 7.1.1 to 7.1.2 ( #6477 )
...
Bumps [nom](https://github.com/Geal/nom ) from 7.1.1 to 7.1.2.
- [Release notes](https://github.com/Geal/nom/releases )
- [Changelog](https://github.com/rust-bakery/nom/blob/main/CHANGELOG.md )
- [Commits](https://github.com/Geal/nom/compare/7.1.1...7.1.2 )
---
updated-dependencies:
- dependency-name: nom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 15:09:27 +00:00
0bc4f43319
chore(deps): Bump rustix from 0.36.5 to 0.36.6 ( #6470 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.5 to 0.36.6.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.5...v0.36.6 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 15:00:42 +00:00
f56c792f79
chore(deps): Bump serde from 1.0.151 to 1.0.152 ( #6468 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.151 to 1.0.152.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.151...v1.0.152 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 10:27:01 +00:00
febaf3c737
chore(deps): Bump clap from 4.0.31 to 4.0.32 ( #6466 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.31 to 4.0.32.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.31...v4.0.32 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-02 10:14:23 +00:00
7279b9597a
chore(deps): Bump libc from 0.2.138 to 0.2.139 ( #6465 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.138 to 0.2.139.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.138...0.2.139 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-02 09:04:29 +00:00
cd1a604df0
fix: make estimate memory the a query needs higher due to recent observation ( #6476 )
2022-12-30 21:15:39 +00:00
0c944346e5
chore: debug info of estimate file size ( #6475 )
2022-12-30 20:13:21 +00:00
d27e137c39
chore: add debug info for the investigation ( #6472 )
2022-12-29 23:49:29 +00:00
5b9e5b4a92
chore(deps): Bump clap from 4.0.29 to 4.0.31 ( #6464 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.29 to 4.0.31.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.29...v4.0.31 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-22 20:00:23 +00:00
858a41c733
chore(deps): Bump handlebars from 4.3.5 to 4.3.6 ( #6458 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.3.5 to 4.3.6.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.3.6/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.5...v4.3.6 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-22 18:53:45 +00:00
e874fef24a
Merge pull request #6463 from influxdata/dom/drop-replayed-wal-segments
...
fix: drop WAL segments after replay
2022-12-22 16:26:53 +00:00
2636eefe03
fix: only replay non-empty segments
...
If the WAL file was dropped because it is empty, do not retry dropping
it.
2022-12-22 17:18:19 +01:00
66f1628238
fix: drop WAL segments after replay
...
Changes the WAL replay logic to:
* Replay a segment file
* Persist all replayed data
* Drop segment file
* ...repeat...
This ensures old WAL segments are removed once their contents have been
made durable, fixing #6461 .
2022-12-22 16:56:47 +01:00
fcb46a3fa7
Merge pull request #6462 from influxdata/dom/decouple-persist
...
refactor(ingester2): decouple persistence subsystem
2022-12-22 15:50:39 +00:00
456368f71d
refactor(persist): no PersistQueue Clone bound
...
Removes the Clone bound from PersistQueue, also removing the Clone impl
from the PersistHandle.
Instead of wrapping all internal PersistHandle state in Arcs, this
commit changes the system to use a single Arc wrapping the PersistHandle
which is shared.
2022-12-22 15:04:52 +01:00
26eea6078d
refactor(ingester2): decouple persist subsystem
...
Multiple components of the ingester depend on being able to enqueue a
partition's data for persistence. This commit decouples those components
from the concrete PersistHandle by introducing a PersistQueue trait that
defines the desired behaviour, on which the components depend.
This is a much needed clean-up of something I knowingly punted on for
the MVP, and I feel much better about the situation now!
2022-12-22 15:04:51 +01:00
dee9743e52
refactor(persist): decoupled PartitionIter
...
The persist_buffer() fn iterates over all the partitions in a BufferTree
and persists them - however it only depends on one behaviour; getting an
iterator of partitions.
This commit introduces the PartitionIter, an abstraction over anything
that can produce an iterator of PartitionData, decoupling the
persist_buffer() helper (and the callers!) from the concrete BufferTree
type.
2022-12-22 14:53:30 +01:00
e54896e5f8
refactor: extract BufferTree persist helper
...
Extract an existing function for re-use (from the WAL rotation task)
that marks & enqueues all non-empty partitions in a BufferTree for
persistence.
2022-12-22 11:58:41 +01:00
2899f65ad1
Merge pull request #6457 from influxdata/cn/update-rust
...
chore: Update to Rust 1.66
2022-12-21 21:57:36 +00:00
46ff8854ec
fix: Use code backticks around invalid HTML tags in doc strings
2022-12-21 16:36:17 -05:00
5f06ab4609
fix: Allow a clippy lint in a case where I don't think it's an improvement
2022-12-21 14:32:35 -05:00
f121d395cc
refactor: Extract a constructor for PolicyBackend using a HashMap
2022-12-21 14:32:35 -05:00
a5c93ec907
chore: Wrap comments at 100 cols, copyedit
2022-12-21 14:32:35 -05:00
NGA-TRAN
Marco Neumann
Dom Dwyer
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Luke Bond
dependabot[bot]
Stuart Carnie