3bdaafe36a
refactor: optimize hash-collection constructions ( #8707 )
...
* refactor: optimize `SortKey` construction
* refactor: optimize column set construction
* refactor: optimize "should cover" calculcation for partitions
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-11 09:46:47 +00:00
e440740aed
chore(deps): Bump croaring from 0.9.0 to 1.0.0 ( #8703 )
...
* chore(deps): Bump croaring from 0.9.0 to 1.0.0
Bumps [croaring](https://github.com/RoaringBitmap/croaring-rs ) from 0.9.0 to 1.0.0.
- [Release notes](https://github.com/RoaringBitmap/croaring-rs/releases )
- [Commits](https://github.com/RoaringBitmap/croaring-rs/commits/1.0.0 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* refactor: fix croaring upgrade and remove dead code
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2023-09-11 09:41:16 +00:00
b741dca8b8
refactor: wrap `CachedPartition` into `Arc` ( #8706 )
...
Even though all subfields of `CachedPartition` are `Arc`ed, the size of
this structure grows and copying more and more fields around for every
cache access gets quite expensive. `Arc` the whole thing and simplify
management a bit.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-11 09:22:15 +00:00
9a0332912b
chore(deps): Bump toml from 0.7.6 to 0.7.8 ( #8704 )
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.7.6 to 0.7.8.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.6...toml-v0.7.8 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-11 08:44:31 +00:00
3e7fe33c56
feat: add easy-to-use compositions of layers ( #8686 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-11 08:34:24 +00:00
87bfe52f6b
chore(deps): Bump serde_json from 1.0.105 to 1.0.106 ( #8702 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.105 to 1.0.106.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.105...v1.0.106 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-11 08:28:31 +00:00
806124b700
feat: reconnect layer ( #8695 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-11 08:21:24 +00:00
5cd9c37519
chore(deps): Bump base64 from 0.21.3 to 0.21.4 ( #8701 )
...
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.21.3 to 0.21.4.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.21.3...v0.21.4 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-11 08:14:43 +00:00
50799e1f66
chore: Add ticket reference to comment ( #8697 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-10 10:32:29 +00:00
90fc0370ae
chore: adjust compactor catalog query rate limiter for small clusters ( #8699 )
2023-09-08 18:53:08 +00:00
c332029d38
fix(k8s-idpe-28726): use backoff on startup authz probe() ( #8683 )
...
* Note that an error returned from authz is still a valid response,
therefore we are using a conditional retry based on communication errors.
2023-09-08 10:19:26 -07:00
cb2d6d1d25
chore(deps): Bump chrono from 0.4.29 to 0.4.30 ( #8693 )
...
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.29 to 0.4.30.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.29...v0.4.30 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-08 13:03:27 +00:00
45c6bfea9c
chore: Update datafusion, arrow/flight/parquet to `46.0.0` , object_store to `0.7.0` ( #8577 )
...
* chore: Update DataFusion pin
* chore: Update for new API
* fix: Update for API
* fix: update compactor test
* fix: Update to patched version of arrow 46.0.0
* fix: map `DataFusionError::Configuration` to an internal error
* fix: do not use deprecated API
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-08 12:49:57 +00:00
7f20b0faa0
chore(deps): Bump bytes from 1.4.0 to 1.5.0 ( #8692 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.4.0...v1.5.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-08 12:17:12 +00:00
9b7697e0d0
feat: backoff & retry for i->q V2 client ( #8688 )
...
* feat: error classifiers for retries etc.
* feat: backoff-based retries
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-08 08:14:28 +00:00
7f06f524eb
feat: i->q V2 metrics integration ( #8687 )
...
Same metric names as V1, so dashboards don't need any migration.
2023-09-08 08:07:22 +00:00
6f1c6fa44a
feat: i->q V2 tracing integration ( #8680 )
...
* refactor: improve `TestLayer`
* feat: tracing layer
2023-09-07 09:13:17 +00:00
581276f974
chore(deps): Bump sysinfo from 0.29.9 to 0.29.10 ( #8685 )
...
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) from 0.29.9 to 0.29.10.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits )
---
updated-dependencies:
- dependency-name: sysinfo
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-07 08:57:26 +00:00
d87c2ae821
chore: change loglevel to info on No compaction job found ( #8684 )
2023-09-06 14:49:47 -07:00
98960a353c
fix(compactor): prevent sort order mismatches from creating overlapping chains ( #8675 )
...
* fix(compactor): prevent sort order mismatches from creating overlapping regions
* chore: test additions for incorrectly created regions
* fix(compactor): more sort order mismatch fixes
* chore: insta updates
* chore: insta updates after merge
2023-09-06 14:53:09 +00:00
fdffa871c3
feat: Optionally specify a table name to get just its schema ( #8650 )
...
Rather than always having to request all of a namespace's schema then
filtering to the one you want. Will make this more consistent with
upserting schema by namespace+table.
Fixes #4997 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 13:33:25 +00:00
de6f710c31
chore: reland have ingester's SortKeyState include sort_key_ids ( #8678 )
2023-09-06 12:01:07 +00:00
d05380db09
test(ingester): Cover more WAL reader errors using wal replay test macro
2023-09-06 12:01:16 +01:00
36604d30ae
refactor(ingester): Only automatically recover when encountering `IncompleteEntry` during wal replay
2023-09-06 12:01:15 +01:00
27a106ba8b
Merge pull request #8670 from influxdata/dom/gossip
...
feat(compactor): gossip compaction completion events
2023-09-06 11:30:16 +01:00
b5a9a6c141
Merge branch 'main' into dom/gossip
2023-09-06 11:24:56 +01:00
73533a71fa
refactor(wal): Disambiguate between `UnexpectedEof` and other errors during entry read
2023-09-06 11:10:08 +01:00
6056571e74
fix(influxql): FILL(linear) for selectors ( #8396 )
...
* fix(influxql): FILL(linear) for selectors
Ensure that selector functions such as FIRST, LAST, MIN and MAX can
use LINEAR filling in the same way as influxdb 1.8.
* chore: review suggestions
Apply suggestions from the review. This adds more tests and support
for interpolation in SQL.
* fix: lint
* fix: lint
* chore: buffered input for struct arrays
Ensure that for linear interpolation the buffered input of a struct
field ensures that buffering only stops when there is a non-null
struct containing a non-null value.
* fix: integration test
* fix(iox_query): make clippy happy
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 09:44:28 +00:00
4f6864c0b9
chore(deps): Bump chrono from 0.4.28 to 0.4.29 ( #8677 )
...
* chore(deps): Bump chrono from 0.4.28 to 0.4.29
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.28 to 0.4.29.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.28...v0.4.29 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: deprecations
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 09:20:58 +00:00
720bdc22c8
feat: add "deserialize" layer for i->q V2 client ( #8639 )
...
Adds the code that deserializes the gRPC response into proper high level
types. Uses mostly the low level code added in #8347 .
For #8349 .
2023-09-06 09:15:23 +00:00
377d2a8215
feat: network layer for i->q V2 client ( #8640 )
...
Adds the actual network IO layer for #8349 .
This is a rather simple layer for now, but we may want to tune some
connection settings in the future.
2023-09-06 09:07:17 +00:00
4d49be9777
feat: add "serialize" layer for i->q V2 client ( #8638 )
...
The layer that serializes our requests. This also contains the logic to
leave out non-serialiable filters like the V1 version (same tests, just
slightly differently arranged).
For #8349 .
2023-09-06 08:36:33 +00:00
260aa0d64c
feat: "logging" layer for i->q V2 client ( #8641 )
...
* feat: more `TestResponse` constructors
* feat: "logging" layer for i->q V2 client
Logging layer for #8349 . This mostly logs in debug mode but emits errors
to the log. Simple implementation that can be extended later.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 08:16:17 +00:00
d0d355ba4d
refactor: unpack record batches later during query ( #8663 )
...
For #8350 we want to be able to stream record batches from the ingester
instead of waiting to buffer them fully before the query starts. Hence
we can no longer inspect the batches in the "display" implementation of
the plan.
This change mostly contains the display change, not the actual streaming
part. I'll do that in a follow-up.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 08:08:54 +00:00
ecc3a2c416
Merge pull request #8676 from influxdata/ntran/revert-jic
...
chore: prepare for revert just in case
2023-09-05 23:53:07 -04:00
7380d76993
chore: prepare for revert just in case
2023-09-05 17:40:24 -04:00
9af06dee9e
feat: have ingester's SortKeyState include sort_key_ids ( #8556 )
...
* feat: have ingester's SortKeyState include sort_key_ids
* fix: test failures
* chore: address review comments
* chore: address review comments by asding asserts to catch bugs if any
* chore: fix typo
* test: get column IDs for the tests
* refactor: reuse function
* chore: address review comments
2023-09-05 20:41:15 +00:00
2a71fcbc76
feat: reland compactor consumes sort_key_ids ( #8674 )
2023-09-05 18:45:49 +00:00
5c4ec830c5
Merge pull request #8673 from influxdata/ntran/compact-revert-jic
...
chore: prepare a revert PR just in case
2023-09-05 13:47:27 -04:00
399c0e257d
chore: prepare a revert PR just in case
2023-09-05 13:26:18 -04:00
fb453ede1e
chore: reland 'teach compactor to use sortkey_ids' after catalog migration is fixed ( #8575 )
2023-09-05 17:05:13 +00:00
6e6970cfe1
test(ingester): Cover precedence of `IngestState::read_with_exceptions()`
2023-09-05 17:49:53 +01:00
c15dfb25b4
test(ingester): Expand `IngestState::read_with_exceptions()` testing
...
This covers multiple error states and multiple exceptions return the
expected results.
2023-09-05 17:49:51 +01:00
be9064c75f
feat(ingester): Allow read of `IngestState` with exceptions
...
This will enable some subsystems to trivially respect any `IngestStateError`
set while ignoring specific errors which they may be responsible for
resolving (such as WAL replay needing to ingest from disk when `DiskFull`
is set).
2023-09-05 16:49:41 +01:00
91939decbd
Merge pull request #8668 from influxdata/dom/memory-cache-arc
...
refactor(router): remove unnecessary Arc wrapper in NamespaceCache stack
2023-09-05 14:42:33 +01:00
f5bb59ded9
Merge branch 'main' into dom/memory-cache-arc
2023-09-05 14:35:01 +01:00
7e6e865bea
Merge pull request #8669 from influxdata/dom/namespace-cache-docs
...
docs: remove outdated cache race warning
2023-09-05 14:31:39 +01:00
021e22b5bf
refactor: remove Arc wrap from ReadThroughCache
...
This Arc was unnecessary in most uses.
2023-09-05 14:15:35 +02:00
529f11e85d
refactor: remove Arc wrap from InstrumentedCache
...
This Arc is unnecessary in most calls.
2023-09-05 14:11:17 +02:00
78d40ba59a
feat(compactor): gossip compaction complete events
...
Add post-compaction calls to send a "compaction complete" gossip event
containing the set of upgraded, deleted, and newly created parquet
files.
2023-09-05 14:01:09 +02:00
Marco Neumann
dependabot[bot]
Andrew Lamb
Joe-Blount
wiedld
Michael Angerman
Carol (Nichols || Goulding)
Nga Tran
Fraser Savage
Dom
Martin Hilton