fa7fe2e9cf
feat: Add a gRPC endpoint to delete a skipped compaction
...
Also add a CLI usage of it for convenience
2022-10-21 15:12:20 -04:00
df2ba85661
feat: add the ability to delete a skipped compaction
2022-10-21 15:12:20 -04:00
1ba21d1d66
Merge pull request #5931 from influxdata/cn/show-skipped-compactions
...
feat: Add gRPC and CLI APIs to list skipped compactions
2022-10-21 18:05:47 +00:00
68e310f45d
feat: Display skipped compactions in a table instead of JSON
2022-10-21 13:59:19 -04:00
b8a9fe4222
docs: Explain the meanings of skipped compaction's fields
2022-10-21 13:40:38 -04:00
0132a33946
fix: Rename SkippedCompactionService to CompactionService
...
To make a good place for other compactor-related gRPC actions in the
future.
2022-10-21 13:40:37 -04:00
699332fd6b
fix: Actually implement the error conversion, oops
2022-10-21 13:40:31 -04:00
ba25300b01
feat: Create compactor service to list skipped compactions
2022-10-21 13:40:31 -04:00
5f04acc9d5
refactor: Alphabetize module declarations
2022-10-21 13:40:30 -04:00
14c77c6748
Merge pull request #5944 from influxdata/dom/disable-incremental-snapshots
...
refactor: disable incremental snapshot generation
2022-10-21 16:17:41 +00:00
7b3fa43209
refactor: disable incremental snapshot generation
...
This commit removes the on-demand, incremental snapshot generation
driven by queries.
This functionality is "on hold" due to concerns documented in:
https://github.com/influxdata/influxdb_iox/issues/5805
Incremental snapshots will be introduced alongside incremental
compactions of those same snapshots.
2022-10-21 17:41:43 +02:00
2173f0a400
Merge pull request #5943 from influxdata/dom/buffer-fsm
...
feat(ingester): partition buffer state machine
2022-10-21 16:39:57 +01:00
db83053be7
Merge branch 'main' into dom/buffer-fsm
2022-10-21 16:32:54 +01:00
8ca72ceff1
docs: fix state mod comments
2022-10-21 17:32:19 +02:00
b3cfb7d5a6
Merge pull request #5920 from influxdata/cn/ingester-tracing
...
feat: Pass trace id through Flight requests from querier to ingester
2022-10-21 13:19:58 +00:00
9b67db3c06
Merge branch 'main' into cn/ingester-tracing
2022-10-21 13:13:13 +00:00
0d8a6395ad
chore(deps): Bump filetime from 0.2.17 to 0.2.18 ( #5939 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.17 to 0.2.18.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.17...0.2.18 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 12:53:31 +00:00
effe4ca1c5
Merge pull request #5941 from influxdata/dependabot/cargo/getrandom-0.2.8
...
chore(deps): Bump getrandom from 0.2.7 to 0.2.8
2022-10-21 13:45:31 +01:00
c8fdd76033
feat(ingester): partition buffer state machine
...
This commit introduces code that is intended to replace the current
implicit state machine used by PartitionData. The existing code is still
in use, the new code is NOT used in this commit. A follow-up commit will
switch over to minimise the diff.
This change has two main goals;
* encapsulation & simplification for callers
* robust implementation so developing correct additions is easier
This is a significant refactor of the partition buffering logic to
encapsulate the various states of data (buffering, snapshot, persisting
and the mixed states between them) within the Partition. This alleviates
the rest of the system from having to be concerned with the differences
between "buffering" data, and "unpersisted data", "snapshot data",
"persisting data", "persisting with snapshots" etc - callers now invoke
a method called get_query_data() and they are provided with all the
relevant data for a partition. This abstraction change alone
significantly reduces code and test complexity in the rest of the
ingester.
For the second goal, the new implementation leverages an explicit state
machine, encoded using typestates. Typestate ensures compile-time
correctness of transitions and method calls, and the explicit FSM itself
helps ensure the system progresses in the desired manner - this fixes
and helps prevent bugs caused by implicit states such as:
https://github.com/influxdata/influxdb_iox/issues/5805
This state machine makes the system states explicit and
self-descriptive, helping to reduce the cost of developer on-boarding
(no prior knowledge of "how this bit works") and reduces ongoing
developer burden. This explicit nature also de-risks adding new
functionality - it should be relatively easy to add concurrent snapshot
generation or incremental compaction without introducing bugs. The state
transition logic is abstracted away from callers, minimising the
overhead of this strategy.
2022-10-21 14:25:51 +02:00
bf34e6923f
chore(deps): Bump getrandom from 0.2.7 to 0.2.8
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.7...v0.2.8 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-21 11:05:13 +00:00
9fb1433317
chore(deps): Bump futures from 0.3.24 to 0.3.25 ( #5938 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-21 11:03:20 +00:00
6e6e180aad
chore(deps): Bump assert_cmd from 2.0.4 to 2.0.5 ( #5937 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.4...v2.0.5 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:50:06 +00:00
1a7776dee6
chore(deps): Bump futures-sink from 0.3.24 to 0.3.25 ( #5936 )
...
Bumps [futures-sink](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-sink
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:32:32 +00:00
240bbe2c53
chore(deps): Bump clap from 4.0.17 to 4.0.18 ( #5935 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.17 to 4.0.18.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.17...v4.0.18 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:11:21 +00:00
7f2033ab4c
chore(deps): Bump futures-core from 0.3.24 to 0.3.25 ( #5934 )
...
Bumps [futures-core](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:43:10 +00:00
8dd3f84a9d
chore(deps): Bump futures-task from 0.3.24 to 0.3.25 ( #5933 )
...
Bumps [futures-task](https://github.com/rust-lang/futures-rs ) from 0.3.24 to 0.3.25.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.24...0.3.25 )
---
updated-dependencies:
- dependency-name: futures-task
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:10:03 +00:00
c763f8b17d
chore(deps): Bump syn from 1.0.102 to 1.0.103 ( #5932 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.102 to 1.0.103.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.102...1.0.103 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 09:01:41 +00:00
7781ed0455
chore: Update datafusion ( #5928 )
...
* chore: Upgrade datafusion
* chore: Update for new API
* chore: Update expected output
* fix: Update comment
* fix: compilation
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 14:37:49 +00:00
444eaec319
fix: Propagate Span correctly in MockIngester
2022-10-20 09:18:09 -04:00
712cfc3f38
fix: Use span rather than child_span
2022-10-20 09:14:28 -04:00
2179aeb785
fix: Allow a clippy false positive for mutable_key_type
2022-10-20 08:55:30 -04:00
59e1c1d5b9
feat: Pass trace id through Flight requests from querier to ingester
...
Fixes #5723 .
2022-10-20 08:55:30 -04:00
48fa18f346
fix: Alphabetize deps in client Cargo.toml whether they're in the workspace or not
2022-10-20 08:55:30 -04:00
5c15c93fc2
fix: Clean up query test dependencies
...
Alphabetize dependencies; remove dev deps that are already regular deps
2022-10-20 08:55:29 -04:00
e1d37b52b2
refactor: arrow API usage cleanup ( #5927 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 12:50:16 +00:00
c8f2a5772e
fix: Add comments about rationale and context for headers ( #5926 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 11:08:10 +00:00
04320aced1
refactor: replace `croaring` with `arrow` ( #5910 )
...
* refactor: replace `croaring` with `roaring`
With the read buffer gone, roaring bitmaps are only used to calculate
series sets and these calculations are pretty much possible with the
pure-Rust version. Also I don't deem that that performance-critical
(compared to the roaring bitmaps in the read buffer core).
This removes a bunch of dependencies, mostly because `bindgen` is gone.
This also removes our "croaring architecture detection" hack.
* refactor: replace manual roaring sets with arrow
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 10:45:41 +00:00
d76d787630
fix: use structured errors for logging influxrpc errors ( #5912 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:38:30 +00:00
3fd0c5e4c2
fix: improve error message for storage read group command ( #5915 )
...
* fix: improve error message for storage read group command
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:31:50 +00:00
487f4a420f
fix: add `storage-type=iox` header to influxrpc responses ( #5917 )
...
* refactor: Move response creation into a single location
* fix: add storage-type=iox header to influxrpc responses
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:24:55 +00:00
5a63cf1f33
feat: Add debug/tracing logging into the influxrpc_storage_client ( #5916 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:18:00 +00:00
fa220589bd
chore(deps): Bump sqlparser from 0.25.0 to 0.26.0 ( #5923 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.25.0...v0.26.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:11:02 +00:00
83e3a96c19
fix: improve ttbr histogram metric description ( #5909 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:03:58 +00:00
99808ca52e
Merge pull request #5925 from influxdata/dependabot/cargo/serde_json-1.0.87
...
chore(deps): Bump serde_json from 1.0.86 to 1.0.87
2022-10-20 09:57:10 +01:00
bebb15d30f
chore(deps): Bump serde_json from 1.0.86 to 1.0.87
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.86 to 1.0.87.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.86...v1.0.87 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-20 07:52:33 +00:00
5f97f0c8a6
chore(deps): Bump ordered-float from 3.2.0 to 3.3.0 ( #5924 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.2.0...v3.3.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-20 07:50:41 +00:00
82d6fc3bda
feat: support queries via influxrpc with periods in field names ( #5919 )
...
* feat: support queries via influxrpc with periods in field names
* fix: update comments
* fix: more tests
* fix: more tests
2022-10-19 20:09:55 +00:00
1df7a0d4fb
refactor: remove outdated observer sql repl mode ( #5918 )
...
* refactor: remove Observer mode from repl
* chore: remove outdated SQL docs
* fix: more update of sql docs
2022-10-19 18:39:05 +00:00
755fb029f9
docs: replace database with namespace ( #5914 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-19 16:17:31 +00:00
21e8fcad25
feat: rework cache refresh logic ( #5886 )
...
* feat: rework cache refresh logic
Instead of issuing a single refresh when a GET request for a cached key
comes in, start a background job (using some efficient logic to not
overload tokio) per key that refreshes the key using some exponential
backoff. The timer is reset a new GET request comes in. This has the
following advantages:
- our backoff logic decorrelates the requests
- the longer a key was not used, the less often it will be updated
All test (esp. integration tests) as adjusted accordingly, mostly to
account for the fact that no extra GET is required to start the refresh
timer.
Closes #5720 .
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* refactor: simplify rng overwrite
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-19 16:01:39 +00:00
Jake Goulding
kodiakhq[bot]
Carol (Nichols || Goulding)
Dom Dwyer
dependabot[bot]
Andrew Lamb
Marco Neumann
Nga Tran