4d49be9777
feat: add "serialize" layer for i->q V2 client ( #8638 )
...
The layer that serializes our requests. This also contains the logic to
leave out non-serialiable filters like the V1 version (same tests, just
slightly differently arranged).
For #8349 .
2023-09-06 08:36:33 +00:00
260aa0d64c
feat: "logging" layer for i->q V2 client ( #8641 )
...
* feat: more `TestResponse` constructors
* feat: "logging" layer for i->q V2 client
Logging layer for #8349 . This mostly logs in debug mode but emits errors
to the log. Simple implementation that can be extended later.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 08:16:17 +00:00
d0d355ba4d
refactor: unpack record batches later during query ( #8663 )
...
For #8350 we want to be able to stream record batches from the ingester
instead of waiting to buffer them fully before the query starts. Hence
we can no longer inspect the batches in the "display" implementation of
the plan.
This change mostly contains the display change, not the actual streaming
part. I'll do that in a follow-up.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-06 08:08:54 +00:00
ecc3a2c416
Merge pull request #8676 from influxdata/ntran/revert-jic
...
chore: prepare for revert just in case
2023-09-05 23:53:07 -04:00
7380d76993
chore: prepare for revert just in case
2023-09-05 17:40:24 -04:00
9af06dee9e
feat: have ingester's SortKeyState include sort_key_ids ( #8556 )
...
* feat: have ingester's SortKeyState include sort_key_ids
* fix: test failures
* chore: address review comments
* chore: address review comments by asding asserts to catch bugs if any
* chore: fix typo
* test: get column IDs for the tests
* refactor: reuse function
* chore: address review comments
2023-09-05 20:41:15 +00:00
2a71fcbc76
feat: reland compactor consumes sort_key_ids ( #8674 )
2023-09-05 18:45:49 +00:00
5c4ec830c5
Merge pull request #8673 from influxdata/ntran/compact-revert-jic
...
chore: prepare a revert PR just in case
2023-09-05 13:47:27 -04:00
399c0e257d
chore: prepare a revert PR just in case
2023-09-05 13:26:18 -04:00
fb453ede1e
chore: reland 'teach compactor to use sortkey_ids' after catalog migration is fixed ( #8575 )
2023-09-05 17:05:13 +00:00
6e6970cfe1
test(ingester): Cover precedence of `IngestState::read_with_exceptions()`
2023-09-05 17:49:53 +01:00
c15dfb25b4
test(ingester): Expand `IngestState::read_with_exceptions()` testing
...
This covers multiple error states and multiple exceptions return the
expected results.
2023-09-05 17:49:51 +01:00
be9064c75f
feat(ingester): Allow read of `IngestState` with exceptions
...
This will enable some subsystems to trivially respect any `IngestStateError`
set while ignoring specific errors which they may be responsible for
resolving (such as WAL replay needing to ingest from disk when `DiskFull`
is set).
2023-09-05 16:49:41 +01:00
91939decbd
Merge pull request #8668 from influxdata/dom/memory-cache-arc
...
refactor(router): remove unnecessary Arc wrapper in NamespaceCache stack
2023-09-05 14:42:33 +01:00
f5bb59ded9
Merge branch 'main' into dom/memory-cache-arc
2023-09-05 14:35:01 +01:00
7e6e865bea
Merge pull request #8669 from influxdata/dom/namespace-cache-docs
...
docs: remove outdated cache race warning
2023-09-05 14:31:39 +01:00
021e22b5bf
refactor: remove Arc wrap from ReadThroughCache
...
This Arc was unnecessary in most uses.
2023-09-05 14:15:35 +02:00
529f11e85d
refactor: remove Arc wrap from InstrumentedCache
...
This Arc is unnecessary in most calls.
2023-09-05 14:11:17 +02:00
78d40ba59a
feat(compactor): gossip compaction complete events
...
Add post-compaction calls to send a "compaction complete" gossip event
containing the set of upgraded, deleted, and newly created parquet
files.
2023-09-05 14:01:09 +02:00
5aee376766
feat(compactor): initialise gossip subsystem
...
Optionally initialise the gossip subsystem in the compactor.
This will cause the compactor to perform PEX and join the cluster, but
as it registers no topic interests, it will not receive any
application-level payloads.
No messages are currently sent (in fact, gossip shuts down immediately).
2023-09-05 13:58:19 +02:00
871f9b6807
refactor(compactor): sort Cargo.toml dependencies
...
Alphabetically sort the dependencies to avoid diff noise.
2023-09-05 13:58:02 +02:00
b200d82d0f
docs: remove outdated cache race warning
...
Concurrent writes to the cache no longer overwrite each other - entries
are now merged.
2023-09-05 13:55:56 +02:00
bcdafa5f25
refactor: remove Arc wrapper from ShardedCache
...
This Arc wrapper is unnecessary.
2023-09-05 13:49:57 +02:00
51096119be
refactor: remove Arc from MemoryNamespaceCache
...
Prior to this commit, the NamespaceCache was only implemented for
Arc<MemoryNamespaceCache> instead of the cache type itself.
In the vast majority of cases, this Arc wrapper is completely
unnecessary - it adds both runtime overhead, and code/type complexity.
This commit impls NamespaceCache for any Arc-wrapped NamespaceCache, and
removes all unnecessary Arc wrapping of the MemoryNamespaceCache.
2023-09-05 13:47:00 +02:00
790e797e6c
chore(deps): Bump regex from 1.9.4 to 1.9.5 ( #8667 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.9.4 to 1.9.5.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.9.4...1.9.5 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-05 08:26:38 +00:00
2e689f250b
Merge pull request #8664 from influxdata/dom/gossip-compaction-crate
...
feat(gossip): compaction events crate
2023-09-04 15:04:11 +01:00
79ed787d3a
chore: fmt
2023-09-04 15:53:23 +02:00
c4a9a46b3e
Merge branch 'main' into dom/gossip-compaction-crate
2023-09-04 14:52:10 +01:00
784e20ffd7
Merge pull request #8665 from influxdata/dom/merkle-consistency-probe-rename
...
refactor(proto): SyncMessage -> ConsistencyProbe
2023-09-04 14:52:01 +01:00
a4b75fb917
docs: copy/paste doc fixes
...
Co-authored-by: Fraser Savage <fsavage@influxdata.com>
2023-09-04 14:45:36 +01:00
992bae0c56
Merge branch 'main' into dom/merkle-consistency-probe-rename
2023-09-04 14:43:40 +01:00
8f6b416d3f
Merge pull request #8649 from influxdata/savage/cover-wal-replay-of-empty-wal-file
...
test(ingester): Cover WAL replay of empty files
2023-09-04 13:43:16 +00:00
51c2865276
Merge branch 'main' into savage/cover-wal-replay-of-empty-wal-file
2023-09-04 13:37:59 +00:00
96d099a295
refactor(proto): SyncMessage -> ConsistencyProbe
...
Rename "SyncMessage" to "ConsistencyProbe" - it's a clearer name for the
use - the message does no syncing!
2023-09-04 15:12:19 +02:00
c0b4a10874
feat(gossip): add gossip_compaction crate
...
Adds a crate that layers compaction-specific gossip types and
abstractions over the underlying gossip transport for a nicer (and
decoupled!) internal API.
2023-09-04 14:05:39 +02:00
044d5bfdcf
refactor(gossip): add Topic::CompactionEvents
...
This adds a separate compaction event gossip topic.
2023-09-04 13:54:05 +02:00
438d975571
chore(deps): Bump ouroboros from 0.17.2 to 0.18.0 ( #8656 )
...
Bumps [ouroboros](https://github.com/joshua-maros/ouroboros ) from 0.17.2 to 0.18.0.
- [Commits](https://github.com/joshua-maros/ouroboros/commits )
---
updated-dependencies:
- dependency-name: ouroboros
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 10:38:31 +00:00
755ee4111e
chore(deps): Bump tower-http from 0.4.3 to 0.4.4 ( #8657 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.3...tower-http-0.4.4 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:59:07 +00:00
77f340f964
chore(deps): Bump thiserror from 1.0.47 to 1.0.48 ( #8658 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.47 to 1.0.48.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:21:22 +00:00
64cc947996
chore(deps): Bump regex-automata from 0.3.7 to 0.3.8 ( #8661 )
...
Bumps [regex-automata](https://github.com/rust-lang/regex ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.7...regex-automata-0.3.8 )
---
updated-dependencies:
- dependency-name: regex-automata
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:10:42 +00:00
af75d2f53b
chore(deps): Bump handlebars from 4.3.7 to 4.4.0 ( #8660 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.3.7 to 4.4.0.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.4.0/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.7...v4.4.0 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:03:13 +00:00
843c497fdd
chore(deps): Bump memchr from 2.6.2 to 2.6.3 ( #8653 )
...
Bumps [memchr](https://github.com/BurntSushi/memchr ) from 2.6.2 to 2.6.3.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.2...2.6.3 )
---
updated-dependencies:
- dependency-name: memchr
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 08:54:38 +00:00
9b57cb17e6
test(ingester): Cover WAL replay of empty files
...
Clearing out the TODO
2023-09-01 16:35:50 +01:00
7a6de3d422
feat: use recurring L0 end time as hint for split times ( #8635 )
...
* chore: add test case for L0 added after vertical splitting
* feat: use recurring L0 end time as hint for split times
* chore: insta test updates
* chore: add split time verification to simulator
2023-09-01 15:34:26 +00:00
296418d60f
Merge pull request #8646 from influxdata/dom/gossip-compaction-proto
...
feat(gossip): define CompactionEvent protobuf
2023-09-01 16:27:38 +01:00
035aa8e6ea
Merge branch 'main' into dom/gossip-compaction-proto
2023-09-01 16:20:56 +01:00
77bfd57579
docs: typo entire
...
Fix typo.
2023-09-01 17:20:28 +02:00
ac1449c5c7
Merge pull request #8647 from influxdata/savage/fix-wal-replay-debug-log
...
fix(ingester): Correctly compute min sequence number for WAL replay debug log
2023-09-01 14:51:47 +00:00
4cf1031fbc
Merge branch 'main' into savage/fix-wal-replay-debug-log
2023-09-01 14:46:01 +00:00
93f3ec6999
feat: teach querier to use sort_key_ids ( #8604 )
...
* feat: teach querier to use sort_key_ids
* chore: add an assert to capture bugs
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-01 14:45:42 +00:00
Marco Neumann
Nga Tran
Fraser Savage
Dom
dependabot[bot]
kodiakhq[bot]
Joe-Blount