a227366432
refactor: do not project chunks in `TestDatabase::chunks` ( #5960 )
...
Databases are NOT required to project chunks (in practice this is only
done by the querier for ingester-based chunks). Instead `iox_query`
should (and already does) add the right stream adapters to project
chunks or to create NULL-columns. Removing the special handling from the
test setup makes it easier to understand and also less likely that
`iox_query` starts to rely on this behavior.
Helps with #5897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 16:36:08 +00:00
3e4db81bc6
refactor: make `SchemaBuilder::field` fallible
...
It would be nice if the IOx data type would not be optional and this is
a prep clean-up to achieve that.
2022-10-24 18:12:42 +02:00
c9b1066b89
refactor: simplify `iox_query::provider::overlap` ( #5961 )
...
- remove generic that is basically unused (`group_potential_duplicates`
is always called w/ `Arc<dyn QueryChunk>`)
- remove half-baked `impl` that is unused
Helps w/ #5897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 15:45:41 +00:00
84e5c2a0ee
fix: cardinality of each batch should use row count of the batch ( #5946 )
...
* fix: cardinality of each batch should use row count of the batch
* chore: cleanup
* fix: auto-merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 15:36:33 +00:00
1d440ddb2d
refactor: `IOxReadFilterNode` can always accumulate statistics ( #5954 )
...
* refactor: `IOxReadFilterNode` can always accumulate statistics
`IOxReadFilterNode` used to not emit statistics if one chunk has
duplicates or delete predicates. This is wrong (or at least overly
conservative), because the node itself (or the chunks themselves) do NOT
perform dedup or delete predicate filtering. Instead this is done is
done by parent nodes (`DeduplicateExec` and `FilterExec`) and its their
job to propagate statistics correctly.
Helps w/ #5897 .
* test: explain setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-24 13:34:22 +00:00
0d4d7e266d
Merge pull request #5955 from influxdata/dom/revert-ttbr-histogram
...
revert: use histogram to record TTBR
2022-10-24 13:12:15 +00:00
60d413bcf9
Merge branch 'main' into dom/revert-ttbr-histogram
2022-10-24 13:05:39 +00:00
284f253846
refactor: remove unused constant ( #5956 )
...
Now that we read throw `ParquetExec`, `ROW_GROUP_READ_SIZE` is no longer
used.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 11:08:44 +00:00
6cbdc749df
chore(deps): Bump prometheus from 0.13.2 to 0.13.3 ( #5952 )
...
Bumps [prometheus](https://github.com/tikv/rust-prometheus ) from 0.13.2 to 0.13.3.
- [Release notes](https://github.com/tikv/rust-prometheus/releases )
- [Changelog](https://github.com/tikv/rust-prometheus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/rust-prometheus/compare/v0.13.2...v0.13.3 )
---
updated-dependencies:
- dependency-name: prometheus
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 09:24:21 +00:00
5fb560dbff
chore(deps): Bump snafu from 0.7.2 to 0.7.3 ( #5951 )
...
Bumps [snafu](https://github.com/shepmaster/snafu ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/shepmaster/snafu/releases )
- [Changelog](https://github.com/shepmaster/snafu/blob/main/CHANGELOG.md )
- [Commits](https://github.com/shepmaster/snafu/compare/0.7.2...0.7.3 )
---
updated-dependencies:
- dependency-name: snafu
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 08:49:15 +00:00
8e5e63f2a6
chore(deps): Bump base64 from 0.13.0 to 0.13.1 ( #5950 )
...
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.13.0 to 0.13.1.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases )
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.13.0...v0.13.1 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 08:39:29 +00:00
ee6e023d4e
chore(deps): Bump serde from 1.0.145 to 1.0.147 ( #5949 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.145 to 1.0.147.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.145...v1.0.147 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 08:29:32 +00:00
39f826518b
revert: use histogram to record TTBR
...
This reverts commit c63312ce12
2022-10-24 10:27:22 +02:00
e0062f2d40
refactor: do NOT use fake DF context for parquet reading ( #5942 )
...
Use the proper top-level DataFusion context and register the object
store there.
Note that we still hide the `ParquetExec` behind an opaque record batch
stream. Fixing that is next on my list.
Helps with #5897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 08:20:26 +00:00
4ca869fcd9
fix: textwrap 0.15.1 was yanked ( #5953 )
...
Need to pull an unreleased clap v3 so we no longer depend on a yanked
version of textwrap. Also see https://github.com/clap-rs/clap/issues/4418
2022-10-24 07:19:51 +00:00
dbcee1f74f
Merge pull request #5948 from influxdata/cn/fix-skipped-at-display
...
fix: Compactor skipped_at is in seconds, not nanoseconds oops
2022-10-21 21:11:27 +00:00
87191d21ea
Merge branch 'main' into cn/fix-skipped-at-display
2022-10-21 21:04:34 +00:00
e26a3edebc
Merge pull request #5947 from influxdata/cn/document-skipped-compaction-cli
...
docs: Add skipped-compaction CLI to the compactor documentation
2022-10-21 20:55:28 +00:00
fe98e7a65c
fix: Compactor skipped_at is in seconds, not nanoseconds oops
2022-10-21 16:41:55 -04:00
3143843078
docs: Explain the new debug skipped-compactions CLI
2022-10-21 16:34:54 -04:00
f07e59e778
fix: Clean up typos, markdown/code formatting, inconsistencies
2022-10-21 16:32:59 -04:00
95853aed1a
Merge pull request #5945 from influxdata/jpg/delete-skipped-compactions
...
feat: Add gRPC and CLI APIs to delete skipped compactions
2022-10-21 19:38:11 +00:00
fa7fe2e9cf
feat: Add a gRPC endpoint to delete a skipped compaction
...
Also add a CLI usage of it for convenience
2022-10-21 15:12:20 -04:00
df2ba85661
feat: add the ability to delete a skipped compaction
2022-10-21 15:12:20 -04:00
1ba21d1d66
Merge pull request #5931 from influxdata/cn/show-skipped-compactions
...
feat: Add gRPC and CLI APIs to list skipped compactions
2022-10-21 18:05:47 +00:00
68e310f45d
feat: Display skipped compactions in a table instead of JSON
2022-10-21 13:59:19 -04:00
b8a9fe4222
docs: Explain the meanings of skipped compaction's fields
2022-10-21 13:40:38 -04:00
0132a33946
fix: Rename SkippedCompactionService to CompactionService
...
To make a good place for other compactor-related gRPC actions in the
future.
2022-10-21 13:40:37 -04:00
699332fd6b
fix: Actually implement the error conversion, oops
2022-10-21 13:40:31 -04:00
ba25300b01
feat: Create compactor service to list skipped compactions
2022-10-21 13:40:31 -04:00
5f04acc9d5
refactor: Alphabetize module declarations
2022-10-21 13:40:30 -04:00
14c77c6748
Merge pull request #5944 from influxdata/dom/disable-incremental-snapshots
...
refactor: disable incremental snapshot generation
2022-10-21 16:17:41 +00:00
7b3fa43209
refactor: disable incremental snapshot generation
...
This commit removes the on-demand, incremental snapshot generation
driven by queries.
This functionality is "on hold" due to concerns documented in:
https://github.com/influxdata/influxdb_iox/issues/5805
Incremental snapshots will be introduced alongside incremental
compactions of those same snapshots.
2022-10-21 17:41:43 +02:00
2173f0a400
Merge pull request #5943 from influxdata/dom/buffer-fsm
...
feat(ingester): partition buffer state machine
2022-10-21 16:39:57 +01:00
db83053be7
Merge branch 'main' into dom/buffer-fsm
2022-10-21 16:32:54 +01:00
8ca72ceff1
docs: fix state mod comments
2022-10-21 17:32:19 +02:00
b3cfb7d5a6
Merge pull request #5920 from influxdata/cn/ingester-tracing
...
feat: Pass trace id through Flight requests from querier to ingester
2022-10-21 13:19:58 +00:00
9b67db3c06
Merge branch 'main' into cn/ingester-tracing
2022-10-21 13:13:13 +00:00
0d8a6395ad
chore(deps): Bump filetime from 0.2.17 to 0.2.18 ( #5939 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.17 to 0.2.18.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.17...0.2.18 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 12:53:31 +00:00
effe4ca1c5
Merge pull request #5941 from influxdata/dependabot/cargo/getrandom-0.2.8
...
chore(deps): Bump getrandom from 0.2.7 to 0.2.8
2022-10-21 13:45:31 +01:00
c8fdd76033
feat(ingester): partition buffer state machine
...
This commit introduces code that is intended to replace the current
implicit state machine used by PartitionData. The existing code is still
in use, the new code is NOT used in this commit. A follow-up commit will
switch over to minimise the diff.
This change has two main goals;
* encapsulation & simplification for callers
* robust implementation so developing correct additions is easier
This is a significant refactor of the partition buffering logic to
encapsulate the various states of data (buffering, snapshot, persisting
and the mixed states between them) within the Partition. This alleviates
the rest of the system from having to be concerned with the differences
between "buffering" data, and "unpersisted data", "snapshot data",
"persisting data", "persisting with snapshots" etc - callers now invoke
a method called get_query_data() and they are provided with all the
relevant data for a partition. This abstraction change alone
significantly reduces code and test complexity in the rest of the
ingester.
For the second goal, the new implementation leverages an explicit state
machine, encoded using typestates. Typestate ensures compile-time
correctness of transitions and method calls, and the explicit FSM itself
helps ensure the system progresses in the desired manner - this fixes
and helps prevent bugs caused by implicit states such as:
https://github.com/influxdata/influxdb_iox/issues/5805
This state machine makes the system states explicit and
self-descriptive, helping to reduce the cost of developer on-boarding
(no prior knowledge of "how this bit works") and reduces ongoing
developer burden. This explicit nature also de-risks adding new
functionality - it should be relatively easy to add concurrent snapshot
generation or incremental compaction without introducing bugs. The state
transition logic is abstracted away from callers, minimising the
overhead of this strategy.
2022-10-21 14:25:51 +02:00
bf34e6923f
chore(deps): Bump getrandom from 0.2.7 to 0.2.8
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.7...v0.2.8 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-21 11:05:13 +00:00
9fb1433317
chore(deps): Bump futures from 0.3.24 to 0.3.25 ( #5938 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-21 11:03:20 +00:00
6e6e180aad
chore(deps): Bump assert_cmd from 2.0.4 to 2.0.5 ( #5937 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.4...v2.0.5 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:50:06 +00:00
1a7776dee6
chore(deps): Bump futures-sink from 0.3.24 to 0.3.25 ( #5936 )
...
Bumps [futures-sink](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-sink
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:32:32 +00:00
240bbe2c53
chore(deps): Bump clap from 4.0.17 to 4.0.18 ( #5935 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.17 to 4.0.18.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.17...v4.0.18 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:11:21 +00:00
7f2033ab4c
chore(deps): Bump futures-core from 0.3.24 to 0.3.25 ( #5934 )
...
Bumps [futures-core](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:43:10 +00:00
8dd3f84a9d
chore(deps): Bump futures-task from 0.3.24 to 0.3.25 ( #5933 )
...
Bumps [futures-task](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-task
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:10:03 +00:00
c763f8b17d
chore(deps): Bump syn from 1.0.102 to 1.0.103 ( #5932 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.102 to 1.0.103.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.102...1.0.103 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:01:41 +00:00
7781ed0455
chore: Update datafusion ( #5928 )
...
* chore: Upgrade datafusion
* chore: Update for new API
* chore: Update expected output
* fix: Update comment
* fix: compilation
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 14:37:49 +00:00
Marco Neumann
Nga Tran
kodiakhq[bot]
dependabot[bot]
Dom Dwyer
Carol (Nichols || Goulding)
Jake Goulding
Andrew Lamb