Commit Graph

9521 Commits (6369d88633ca1a170925e2a0974b6679579109f8)

Author SHA1 Message Date
Carol (Nichols || Goulding) 73d7105f20 fix: Update from clap ArgEnum to ValueEnum
See <https://github.com/clap-rs/clap/pull/4127>
2022-09-30 16:59:03 -04:00
Carol (Nichols || Goulding) 3c11d3640f fix: Update use of clap::StructOpt to clap::Parser
StructOpt is now fully part of Clap.

https://docs.rs/clap/latest/clap/_faq/index.html#how-does-clap-compare-to-structopt
2022-09-30 16:59:03 -04:00
Carol (Nichols || Goulding) e76b93d47b fix: Remove deprecated takes_value attribute
See <https://github.com/clap-rs/clap/issues/2688>
2022-09-30 16:48:26 -04:00
dependabot[bot] 199e47721a chore(deps): Bump clap from 3.2.22 to 4.0.7
Bumps [clap](https://github.com/clap-rs/clap) from 3.2.22 to 4.0.7.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...v4.0.7)
2022-09-30 16:46:56 -04:00
kodiakhq[bot] fc0634792b
Merge pull request #5780 from influxdata/dom/test-cleanup
test: refactor ingester helpers
2022-09-30 16:43:18 +00:00
kodiakhq[bot] 27d98479f1
Merge branch 'main' into dom/test-cleanup 2022-09-30 16:35:16 +00:00
Nga Tran 2f08a64f16
feat: not split output files in the first step of cold compaction (#5781)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 16:08:03 +00:00
Nga Tran d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… (#5772)
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently

* fix: comment

* fix: test output

* refactor: using var instead of literal

* fix: consider deleted L0s for recent writes

* chore: cleanup

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
Dom Dwyer 7dd28f4230 test: simplify PartitionProvider mock
The PartitionKey is now part of the PartitionData, so there is no need
to specify the redundant ID when configuring the mock.
2022-09-30 16:32:39 +02:00
Dom Dwyer c33499764d test: share populate_catalog() across tests
Parametrises test_util::populate_catalog() and exports for re-use in
ingester tests.
2022-09-30 16:32:37 +02:00
Dom Dwyer fc47f6ab8f test: re-use test_utils::make_op
Share the make_op helper across all tests in the Ingester.
2022-09-30 16:32:36 +02:00
Dom Dwyer f0885612e9 test: shared mock LifecycleHandle impl
Moves the NoopLifecycleHandle to the Ingester's test_utils to share it
across multiple components.
2022-09-30 16:32:34 +02:00
kodiakhq[bot] d7677c1b1d
Merge pull request #5778 from influxdata/dom/lifecycle-ids
refactor: LifecycleStats tracks Namespace/TableId
2022-09-30 13:56:21 +00:00
Dom 4c3697c5c7
Merge branch 'main' into dom/lifecycle-ids 2022-09-30 14:31:07 +01:00
Dom Dwyer e84186763f refactor: LifecycleStats tracks Namespace/TableId
Changes the lifecycle handle to also track the namespace + table ID in
addition to the existing shard ID.

Adds asserts to ensure the values never vary for a given partition.
2022-09-30 15:29:39 +02:00
dependabot[bot] b1390368fb
chore(deps): Bump tikv-jemalloc-sys (#5774)
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator) from 0.5.1+5.3.0-patched to 0.5.2+5.3.0-patched.
- [Release notes](https://github.com/tikv/jemallocator/releases)
- [Changelog](https://github.com/tikv/jemallocator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tikv/jemallocator/commits)

---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 11:22:38 +00:00
dependabot[bot] c187392be6
chore(deps): Bump libc from 0.2.133 to 0.2.134 (#5773)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.133 to 0.2.134.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.133...0.2.134)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-30 11:14:18 +00:00
Andrew Lamb 04ae0aee80
refactor: Remove protobuf based write service (#5750)
* refactor: Remove grpc WriteService

* fix: update end to end test

* fix: Update generated_types/protos/influxdata/pbdata/v1/influxdb_pb_data_protocol.proto

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 10:55:03 +00:00
Stuart Carnie 9feb27b3b0
feat: Add `SELECT` statement parsing (#5692)
* feat: Identifier → VarRef, which represents a tag or field reference

InfluxQL calls these variables, so the Rust parser will follow
precedent.

* chore: Add docs

* chore: Separate arithmetic and conditional expression parsing

This is necessary to ensure conditional expressions are only supported
in a `WHERE` clause and arithmetic expressions in `SELECT` field
projection list

* feat: Add a `verify` API to transform combinator functions to errors

This is useful to preform additional validation on a valid result.

* feat: Add customisation for operand parsing to arithmetic expressions

This permits narrowing of valid function calls in `WHERE` clauses
and to customise valid operands in a `SELECT` projection, such as a
wildcard (`*`)

* chore: move logic to appropriate modules.

* feat: Add wildcard discriminator to Expr enum required for `SELECT`

* chore: Remove dead_code attribute

* feat: Add `DISTINCT` operator

* chore: Add `distinct` macro; simplify existing macros

* chore: add assert_matches; parse Field and Field list; refactor literal

* feat: Add a new type to parse a separated_list1 into a specialised type

* chore: Refactor types using `separated_list1`

* chore: Ensure items in a list can be preceded by a space

* chore: Refactor IN clause to use OneOrMore

* feat: Parse `GROUP BY` clause

* chore: appease clippy

* chore: refactor number parsing to share code; add signed Number type

* feat: completed `SELECT` statement parser.

* chore: Consistent error messages when expecting tokens

* chore: Add recommended derive implementations

* feat: Add `SELECT` statement to `statement` combinator

* chore: Refactor OneOrMore::separated_list1 to a single error message

* chore: More tests and cleanup 🧹

* chore: More tests

* chore: Appease broken doc links checker

* chore: remove remaining `dead_code` attributes 🥳

* chore: Address PR feedback

* https://github.com/influxdata/influxdb_iox/pull/5692#discussion_r982559420
* https://github.com/influxdata/influxdb_iox/pull/5692#discussion_r982527476

* chore: Address PR feedback

* https://github.com/influxdata/influxdb_iox/pull/5692#discussion_r982626857
* https://github.com/influxdata/influxdb_iox/pull/5692#discussion_r982569760

* chore: Address PR feedback for Display implementations

Also removed duplicate test case
2022-09-29 23:10:18 +00:00
kodiakhq[bot] d2e645603c
Merge pull request #5771 from influxdata/cn/generate-parquet-in-data-generator
feat: Generate parquet files instead of line protocol
2022-09-29 20:58:06 +00:00
Carol (Nichols || Goulding) 39a810ec3a
feat: Generate parquet files instead of line protocol 2022-09-29 16:22:37 -04:00
Carol (Nichols || Goulding) 4fbf04d850
fix: Make the convenient ParquetFileReader pub 2022-09-29 16:03:34 -04:00
Carol (Nichols || Goulding) be6519d4f2
fix: Add filename to the data generator read file error 2022-09-29 16:03:33 -04:00
Jake Goulding 7389fbe528
feat: Write data to Parquet files from the data generator 2022-09-29 16:03:33 -04:00
Carol (Nichols || Goulding) 79afb6ef54
fix: Don't attempt to write 0 lines 2022-09-29 13:07:42 -04:00
Jake Goulding 8187b003c3
refactor: improve panic message 2022-09-29 12:33:26 -04:00
Dom c213feb60c
Merge pull request #5770 from influxdata/dom/partitions-know-their-ids
refactor: PartitionData carries parent IDs
2022-09-29 15:53:39 +01:00
Dom Dwyer 726b1d1d3b refactor: PartitionData carries parent IDs
This commit changes the PartitionData buffer structure to carry the IDs
of all its parents - the table, namespace, and shard. Previously only
the table & shard were carried.
2022-09-29 15:07:03 +02:00
Marko Mikulicic 0287a58118
chore: use 'error' label in log instead of 'e' (#5769)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 12:26:18 +00:00
Dom fc8e98a24d
Merge pull request #5768 from influxdata/dom/partition-contains-key
refactor: PartitionData carries PartitionKey
2022-09-29 13:11:10 +01:00
Dom e9bd03b77c
Merge branch 'main' into dom/partition-contains-key 2022-09-29 12:32:35 +01:00
Dom 268b99265b
Merge pull request #5766 from influxdata/dom/remove-dbg
ci: add dbg!() lint
2022-09-29 12:25:51 +01:00
Dom Dwyer f5a7fbf8e2 refactor: PartitionData carries PartitionKey
Changes the PartitionData to carry the derived PartitionKey for which it
is buffering ops for. This is used at persist time.
2022-09-29 13:22:50 +02:00
Dom 8dbeef25ce
Merge branch 'main' into dom/remove-dbg 2022-09-29 12:14:19 +01:00
Andrew Lamb 56a1c579a1
refactor: Change influxdb_iox client to use http rather than grpc for write (#5756)
* refactor: Change influxdb_iox client to use http rather than grpc for write

* refactor: remove custom variants

* refactor: consolidate more
2022-09-29 11:12:51 +00:00
Dom Dwyer 5a5f9d104b refactor(read_buffer): todo!() -> unimplemented!()
todo!() is useful for incremental work (telling the compiler to shut up
during dev work).

unimplemented!() should be used for prod code that is not yet
implemented but expected to be deployed. It signifies "this isn't
implemented but doesn't need doing now".
2022-09-29 13:10:07 +02:00
Dom Dwyer cd4087e00d style: add no todo!() or dbg!() lints
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
Dom Dwyer a5e5dfda6f refactor: remove dbg!
Left over from debugging.
2022-09-29 13:08:47 +02:00
dependabot[bot] 227dde1dfc
chore(deps): Bump thiserror from 1.0.36 to 1.0.37 (#5753)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.36 to 1.0.37.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.36...1.0.37)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 10:37:14 +00:00
dependabot[bot] ab49f0c6d0
chore(deps): Bump sqlparser from 0.23.0 to 0.24.0 (#5760)
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.23.0 to 0.24.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.23.0...v0.24.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-29 10:29:14 +00:00
dependabot[bot] 6a734e62a9
chore(deps): Bump tonic-build from 0.8.0 to 0.8.2 (#5759)
Bumps [tonic-build](https://github.com/hyperium/tonic) from 0.8.0 to 0.8.2.
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.8.0...v0.8.2)

---
updated-dependencies:
- dependency-name: tonic-build
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 10:18:22 +00:00
dependabot[bot] 354d0f1e0e
chore(deps): Bump tonic from 0.8.1 to 0.8.2 (#5762)
Bumps [tonic](https://github.com/hyperium/tonic) from 0.8.1 to 0.8.2.
- [Release notes](https://github.com/hyperium/tonic/releases)
- [Changelog](https://github.com/hyperium/tonic/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/tonic/compare/v0.8.1...v0.8.2)

---
updated-dependencies:
- dependency-name: tonic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-29 10:10:40 +00:00
Dom 71cab5a1a6
Merge pull request #5764 from influxdata/dom/remove-after-watermark-check
fix: remove future offset read check
2022-09-29 10:53:25 +01:00
Dom Dwyer 5f49c568c9 fix: remove future offset read check
In https://github.com/influxdata/influxdb_iox/pull/5754 I added code at
seek() time to check if the offset exists, and refuse to seek if that's
not the case, effectively making this check redundant - I left it in on
the assumption that some cases previously added would work!

Unfortunately this doesn't seem to be the case -
performing a read-ahead-of-data and read-behind-data seems to cause the
high_watermark to be returned as -1, meaning this code never worked?!
This new read-ahead-of-data match arm took priority over the
SequenceNumberNoLongerExists arm, effectively preventing the ingester
from taking the desired remediation (skipping to most recent write, or
erroring, depending on configuration).
2022-09-29 11:39:57 +02:00
kodiakhq[bot] 58d1b9f4e3
Merge pull request #5743 from influxdata/cn/hot-cold-generation
feat: Generate the requested num rows and compaction type
2022-09-28 18:51:05 +00:00
Carol (Nichols || Goulding) 8d90181e67
fix: Make sure generated level 1 files don't overlap
The data generator is inclusive in start and end times, so don't use the
same end time as the next level 1 file's start time to prevent overlap.
2022-09-28 14:41:00 -04:00
Carol (Nichols || Goulding) 72203b3b31
feat: Generate the number of rows across the cold time range 2022-09-28 14:34:27 -04:00
Carol (Nichols || Goulding) 21488ac360
feat: Generate the number of rows across the "hot" time range 2022-09-28 14:34:27 -04:00
Carol (Nichols || Goulding) 2c70cd0cd0
refactor: Only pass along the one relevant config field 2022-09-28 14:34:27 -04:00
kodiakhq[bot] 03940d0750
Merge pull request #5747 from influxdata/dom/partition-cache
perf(ingester): partition ID + persist marker cache
2022-09-28 15:29:59 +00:00