be9064c75f
feat(ingester): Allow read of `IngestState` with exceptions
...
This will enable some subsystems to trivially respect any `IngestStateError`
set while ignoring specific errors which they may be responsible for
resolving (such as WAL replay needing to ingest from disk when `DiskFull`
is set).
2023-09-05 16:49:41 +01:00
91939decbd
Merge pull request #8668 from influxdata/dom/memory-cache-arc
...
refactor(router): remove unnecessary Arc wrapper in NamespaceCache stack
2023-09-05 14:42:33 +01:00
f5bb59ded9
Merge branch 'main' into dom/memory-cache-arc
2023-09-05 14:35:01 +01:00
7e6e865bea
Merge pull request #8669 from influxdata/dom/namespace-cache-docs
...
docs: remove outdated cache race warning
2023-09-05 14:31:39 +01:00
021e22b5bf
refactor: remove Arc wrap from ReadThroughCache
...
This Arc was unnecessary in most uses.
2023-09-05 14:15:35 +02:00
529f11e85d
refactor: remove Arc wrap from InstrumentedCache
...
This Arc is unnecessary in most calls.
2023-09-05 14:11:17 +02:00
78d40ba59a
feat(compactor): gossip compaction complete events
...
Add post-compaction calls to send a "compaction complete" gossip event
containing the set of upgraded, deleted, and newly created parquet
files.
2023-09-05 14:01:09 +02:00
5aee376766
feat(compactor): initialise gossip subsystem
...
Optionally initialise the gossip subsystem in the compactor.
This will cause the compactor to perform PEX and join the cluster, but
as it registers no topic interests, it will not receive any
application-level payloads.
No messages are currently sent (in fact, gossip shuts down immediately).
2023-09-05 13:58:19 +02:00
871f9b6807
refactor(compactor): sort Cargo.toml dependencies
...
Alphabetically sort the dependencies to avoid diff noise.
2023-09-05 13:58:02 +02:00
b200d82d0f
docs: remove outdated cache race warning
...
Concurrent writes to the cache no longer overwrite each other - entries
are now merged.
2023-09-05 13:55:56 +02:00
bcdafa5f25
refactor: remove Arc wrapper from ShardedCache
...
This Arc wrapper is unnecessary.
2023-09-05 13:49:57 +02:00
51096119be
refactor: remove Arc from MemoryNamespaceCache
...
Prior to this commit, the NamespaceCache was only implemented for
Arc<MemoryNamespaceCache> instead of the cache type itself.
In the vast majority of cases, this Arc wrapper is completely
unnecessary - it adds both runtime overhead, and code/type complexity.
This commit impls NamespaceCache for any Arc-wrapped NamespaceCache, and
removes all unnecessary Arc wrapping of the MemoryNamespaceCache.
2023-09-05 13:47:00 +02:00
790e797e6c
chore(deps): Bump regex from 1.9.4 to 1.9.5 ( #8667 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.9.4 to 1.9.5.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.9.4...1.9.5 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-05 08:26:38 +00:00
350d993774
feat(ingester): WAL replay automatic recovery from incomplete WAL write
...
When we come across an `UnableToReadNextOps` error during Ingester
start-up it means that one of two things has happened:
1. Ingester crashed mid-flush to WAL, so the write entry is incomplete
and the user received an error.
2. The WAL segment file has been corrupted in a way which cannot be
recovered from. The format does not allow for recovery of a bad
entry.
In either case we can safely drop the WAL file after all good entries
have been replayed and persisted so as not to block recovery.
See https://github.com/influxdata/influxdb_iox/issues/8613 for more
details.
2023-09-04 18:29:20 +01:00
c72b2412cd
refactor(ingester): Tie `SegmentId` to WAL op batch iter during WAL replay
2023-09-04 17:03:17 +01:00
7d8cc83f26
refactor(ingester): Use generic `SequencedWalOp` batched reader for WAL replay
...
This decouples the implementation of the `ClosedSegmentFileReader` from
the ability to replay batches of `Vec<SequencedWalOp>`s, making it easy
to test.
2023-09-04 16:38:38 +01:00
b6e8b9cc5e
docs(wal): Clearly document why `UnableToReadNextOps` is returned
...
Nothing currently relies on this public error variant, but for automatic
recovery we need the WAL to provide a contract that this error is
returned only when the next operation in the WAL is unsalvageable.
2023-09-04 16:18:21 +01:00
2e689f250b
Merge pull request #8664 from influxdata/dom/gossip-compaction-crate
...
feat(gossip): compaction events crate
2023-09-04 15:04:11 +01:00
79ed787d3a
chore: fmt
2023-09-04 15:53:23 +02:00
c4a9a46b3e
Merge branch 'main' into dom/gossip-compaction-crate
2023-09-04 14:52:10 +01:00
784e20ffd7
Merge pull request #8665 from influxdata/dom/merkle-consistency-probe-rename
...
refactor(proto): SyncMessage -> ConsistencyProbe
2023-09-04 14:52:01 +01:00
a4b75fb917
docs: copy/paste doc fixes
...
Co-authored-by: Fraser Savage <fsavage@influxdata.com>
2023-09-04 14:45:36 +01:00
992bae0c56
Merge branch 'main' into dom/merkle-consistency-probe-rename
2023-09-04 14:43:40 +01:00
8f6b416d3f
Merge pull request #8649 from influxdata/savage/cover-wal-replay-of-empty-wal-file
...
test(ingester): Cover WAL replay of empty files
2023-09-04 13:43:16 +00:00
51c2865276
Merge branch 'main' into savage/cover-wal-replay-of-empty-wal-file
2023-09-04 13:37:59 +00:00
96d099a295
refactor(proto): SyncMessage -> ConsistencyProbe
...
Rename "SyncMessage" to "ConsistencyProbe" - it's a clearer name for the
use - the message does no syncing!
2023-09-04 15:12:19 +02:00
c0b4a10874
feat(gossip): add gossip_compaction crate
...
Adds a crate that layers compaction-specific gossip types and
abstractions over the underlying gossip transport for a nicer (and
decoupled!) internal API.
2023-09-04 14:05:39 +02:00
044d5bfdcf
refactor(gossip): add Topic::CompactionEvents
...
This adds a separate compaction event gossip topic.
2023-09-04 13:54:05 +02:00
438d975571
chore(deps): Bump ouroboros from 0.17.2 to 0.18.0 ( #8656 )
...
Bumps [ouroboros](https://github.com/joshua-maros/ouroboros ) from 0.17.2 to 0.18.0.
- [Commits](https://github.com/joshua-maros/ouroboros/commits )
---
updated-dependencies:
- dependency-name: ouroboros
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 10:38:31 +00:00
755ee4111e
chore(deps): Bump tower-http from 0.4.3 to 0.4.4 ( #8657 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.4.3...tower-http-0.4.4 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:59:07 +00:00
77f340f964
chore(deps): Bump thiserror from 1.0.47 to 1.0.48 ( #8658 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.47 to 1.0.48.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:21:22 +00:00
64cc947996
chore(deps): Bump regex-automata from 0.3.7 to 0.3.8 ( #8661 )
...
Bumps [regex-automata](https://github.com/rust-lang/regex ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.7...regex-automata-0.3.8 )
---
updated-dependencies:
- dependency-name: regex-automata
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:10:42 +00:00
af75d2f53b
chore(deps): Bump handlebars from 4.3.7 to 4.4.0 ( #8660 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.3.7 to 4.4.0.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.4.0/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.7...v4.4.0 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:03:13 +00:00
843c497fdd
chore(deps): Bump memchr from 2.6.2 to 2.6.3 ( #8653 )
...
Bumps [memchr](https://github.com/BurntSushi/memchr ) from 2.6.2 to 2.6.3.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.2...2.6.3 )
---
updated-dependencies:
- dependency-name: memchr
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 08:54:38 +00:00
9b57cb17e6
test(ingester): Cover WAL replay of empty files
...
Clearing out the TODO
2023-09-01 16:35:50 +01:00
7a6de3d422
feat: use recurring L0 end time as hint for split times ( #8635 )
...
* chore: add test case for L0 added after vertical splitting
* feat: use recurring L0 end time as hint for split times
* chore: insta test updates
* chore: add split time verification to simulator
2023-09-01 15:34:26 +00:00
296418d60f
Merge pull request #8646 from influxdata/dom/gossip-compaction-proto
...
feat(gossip): define CompactionEvent protobuf
2023-09-01 16:27:38 +01:00
035aa8e6ea
Merge branch 'main' into dom/gossip-compaction-proto
2023-09-01 16:20:56 +01:00
77bfd57579
docs: typo entire
...
Fix typo.
2023-09-01 17:20:28 +02:00
ac1449c5c7
Merge pull request #8647 from influxdata/savage/fix-wal-replay-debug-log
...
fix(ingester): Correctly compute min sequence number for WAL replay debug log
2023-09-01 14:51:47 +00:00
4cf1031fbc
Merge branch 'main' into savage/fix-wal-replay-debug-log
2023-09-01 14:46:01 +00:00
93f3ec6999
feat: teach querier to use sort_key_ids ( #8604 )
...
* feat: teach querier to use sort_key_ids
* chore: add an assert to capture bugs
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-09-01 14:45:42 +00:00
168a9e37fc
fix(ingester): Correctly compute min sequence number for WAL replay debug logging
2023-09-01 15:34:22 +01:00
855e8bb34c
feat(gossip): define CompactionEvent protobuf
...
This commit adds the CompactionEvent protobuf type that'll be gossiped
between peers to provide a (compact) description of completed
compactions.
2023-09-01 15:16:34 +02:00
0f138dc2a5
Merge pull request #8645 from influxdata/dom/reuse-sort-key
...
refactor(persist): load sort key once
2023-09-01 14:11:08 +01:00
e5e12ebb1a
refactor: remove Result from infallible return
...
The compact_persisting_batch() call is infallible, but prior to this
commit, would return a Result with a () unit type as the error type.
This is misleading - it's never going to return an error, so call sites
checking for errors are misleading.
2023-09-01 14:20:06 +02:00
8853296bef
refactor(persist): load sort key once
...
Passes the already-loaded sort key into update_catalog_sort_key()
This makes the function inputs clear, and avoids loading the same key
twice (which should be a no-op now, but not in the future!).
2023-09-01 14:15:11 +02:00
edf47e907e
Merge pull request #8644 from influxdata/savage/crc-reader-checksum-consumes-self
...
refactor(wal): Consume `CrcReader` when invoking `checksum()`
2023-09-01 11:24:50 +00:00
b27060ea23
Merge branch 'main' into savage/crc-reader-checksum-consumes-self
2023-09-01 11:19:33 +00:00
c93045bb60
Merge pull request #8634 from influxdata/savage/cli-table-list
...
feat(cli): Add `table list <DATABASE>` command
2023-09-01 11:17:27 +00:00
Fraser Savage
Dom
dependabot[bot]
kodiakhq[bot]
Joe-Blount
Nga Tran