1737ce0450
test(gossip): sender identity
...
Ensure the correct Identity is provided to the payload Dispatcher.
2023-08-30 14:18:00 +02:00
507739f9fe
refactor(gossip): Identity in API, not UUID
...
Swap all uses of UUID in the gossip API for the Identity, allowing the
caller to parse into the UUID format if needed, and not forcing it on
them.
2023-08-30 14:18:00 +02:00
20d63b8aa6
feat(gossip): expose sender identity in Dispatcher
...
When the gossip subsystem hands an application payload off to the
Dispatcher impl, include the sender's Identity.
2023-08-30 14:17:59 +02:00
f81f6efb2c
refactor(gossip): remove Uuid from Identity
...
The identity contains both a UUID, and the byte representation of this
UUID. This change retains the latter, generating the former on-demand
(which is rarely needed).
This halves the size of the identity, and the underlying Bytes storage
is a ref-counted container, making clones very cheap.
2023-08-30 14:17:54 +02:00
5925440d6a
Merge pull request #8609 from influxdata/dom/gossip-subset
...
feat(gossip): broadcast to randomised peer subset
2023-08-30 13:07:42 +01:00
c983d1a096
Merge branch 'main' into dom/gossip-subset
2023-08-30 12:56:59 +01:00
1ef6e79aed
Merge pull request #8600 from influxdata/dom/merkle-actor
...
perf: asynchronously maintain merkle search tree
2023-08-30 11:40:13 +01:00
e120935a61
fix(bench): reuse same namespace cache
...
And add another thread for the hashing to happen on now there's two
async tasks to run.
2023-08-30 12:07:15 +02:00
38f67ae736
perf(anti_entropy): add AntiEntropyActor for MST
...
Separate the management of the Merkle Search Tree state into an actor
to manage concurrent access.
This moves hashing and tree updates off of the hot request path, and
into an asynchronous background process, practically eliminating
overhead for maintaining the MST structure.
This decoupling will allow convergence runs between peers to proceed
without causing contention on the lock in the request hot path.
2023-08-30 12:07:15 +02:00
934e4bc9c6
test(bench): MerkleTree overhead
...
Layers the MerkleTree over the NamespaceCache stack in benchmarks to
measure the overhead.
2023-08-30 12:07:14 +02:00
53d7b23624
Merge pull request #8598 from influxdata/dom/gossip-parquet-proto
...
refactor: protobuf serialisation primitives for ParquetFile
2023-08-30 11:01:35 +01:00
b68d108baf
Merge branch 'main' into dom/gossip-parquet-proto
2023-08-30 10:55:46 +01:00
43db19b7db
feat(gossip): broadcast to randomised peer subset
...
Provide a new broadcast_subset() method to gossip users, enabling a
message to be sent to a random subset of peers that have a registered
interest in the message being dispatched.
2023-08-30 11:54:09 +02:00
74f6de78fd
Merge pull request #8608 from influxdata/dependabot/cargo/memchr-2.6.1
...
chore(deps): Bump memchr from 2.6.0 to 2.6.1
2023-08-30 10:52:47 +01:00
0d8ba84923
chore(deps): Bump memchr from 2.6.0 to 2.6.1
...
Bumps [memchr](https://github.com/BurntSushi/memchr ) from 2.6.0 to 2.6.1.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.0...2.6.1 )
---
updated-dependencies:
- dependency-name: memchr
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-08-30 01:25:04 +00:00
68a9019768
chore: remove compactor concurrency scaling ( #8606 )
...
* chore: remove compactor concurrency scaling
* chore: comment update to retrigger tests
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 23:42:53 +00:00
5011f7fe22
chore: add partition_id into panic message ( #8605 )
2023-08-29 16:37:21 -07:00
0996a95630
chore: enable more ManySmallFiles compactions ( #8603 )
...
* chore: enable more ManySmallFiles compactions
* chore: insta churn
2023-08-29 20:42:03 +00:00
72c48d34f8
chore: add partition id to compactor panics ( #8602 )
2023-08-29 12:40:11 -07:00
6c6a3507d1
Merge pull request #8601 from influxdata/dom/no-compiler-err
...
ci: lint failures as warnings
2023-08-29 16:26:19 +01:00
9055bc1754
ci: lint failures as warnings
...
Don't fail to compile / run tests because of an unreachable pub, or
missing debug impl - just emit a compiler warning.
This lets the compilation complete, but isn't accepted in PRs as CI runs
with "deny warnings".
2023-08-29 16:29:31 +02:00
674a9e3eff
Merge pull request #8597 from influxdata/dom/merkle-cache
...
feat(router): merkle tree cache observer for anti-entropy
2023-08-29 13:40:02 +00:00
bcc4003222
Merge branch 'main' into dom/merkle-cache
2023-08-29 14:34:34 +01:00
38147ec25e
chore: compact available regions before splitting everything ( #8586 )
2023-08-29 13:29:11 +00:00
15da02b59f
Merge branch 'main' into dom/merkle-cache
2023-08-29 14:21:26 +01:00
c1ba3918a4
test: cache content hash fixture
...
Assert the hash representing static cache content does not change.
2023-08-29 13:57:39 +02:00
19e7c90fc1
test: compose prop::Strategy for schema generation
...
Accept an arbitrary ID generation strategy, composing it into a
NamespaceSchema generation strategy to simplify the call args / usage.
2023-08-29 13:47:22 +02:00
b1555621cb
Merge pull request #8599 from influxdata/dom/fix-bench
...
fix(bench): schema validator bench panic
2023-08-29 11:49:37 +01:00
932532c3e3
fix(bench): schema validator bench panic
...
The benchmark code was completely broken - running the benches would
immediately panic.
2023-08-29 12:41:47 +02:00
10396e6986
refactor(service_grpc_catalog): use conversion fns
...
Use well-tested From impls for converting partition IDs & ParquetFile
instances to their proto representation.
2023-08-29 12:35:46 +02:00
4c2945719a
feat: proto serialisation of ParquetFile
...
Adds conversion functions to serialise a ParquetFile into a protobuf
representation, and back again.
Adds randomised testing to assert round-trip equality.
2023-08-29 12:35:45 +02:00
ae3f73f65e
refactor(proto): optional ParquetFile::to_delete
...
This field is nullable, so lets model it as nullable.
2023-08-29 12:35:44 +02:00
124b3d2b42
test: MerkleTree cache decorator
...
Adds an integration test asserting the derived MST content hashes
accurately track updates to an underlying cache entry merge
implementation.
This ensures the merge implementation, and content hashes do not become
out-of-sync.
2023-08-29 12:26:42 +02:00
b694b9f494
feat(router): Merkle tree content hash for cache
...
Adds a (currently unused) NamespaceCache decorator that observes the
post-merge content of the cache to maintain a content hash.
This makes use of a Merkle Search Tree
(https://inria.hal.science/hal-02303490 ) to track the CRDT content of
the cache in a deterministic structure that allows for synchronisation
between peers (itself a CRDT).
The hash of two routers' NamespaceCache will be equal iff their cache
contents are equal - this can be used to (very cheaply) identify
out-of-sync routers, and trigger convergence. The MST structure used
here provides functionality to compare two compact MST representations,
and identify subsets of the cache that are out-of-sync, allowing for
cheap convergence.
Note this content hash only covers the tables, their set of columns, and
those column schemas - this reflects the fact that only these values may
currently be converged by gossip. Future work will enable full
convergence of all fields.
2023-08-29 12:19:43 +02:00
bd4a3fbbb8
refactor: impl Hash for NamespaceSchema
...
Allow the NamespaceSchema to be hashed (including the underlying proto
types it contains).
2023-08-29 12:19:41 +02:00
fc694effda
refactor: AsRef bytes for NamespaceName
...
The NamespaceName is a wrapper over a str (conceptually), which allows
for cheap use of the underlying bytes.
2023-08-29 12:18:33 +02:00
74212e90e7
Merge pull request #8499 from influxdata/savage/disk-full-protection
...
feat(ingester): Disk full protection
2023-08-29 10:17:44 +00:00
2319f6bdf1
fix(ingester): Do not await empty file notification disk protection
...
It's possible that this could never happen (or be very slow), while
the evaluation and enforcement of disk protection happens in another
thread. Blocking further clean-up on this condition might wedge the
ingest state!
2023-08-29 11:08:32 +01:00
3d88c9047c
Merge branch 'main' into savage/disk-full-protection
2023-08-29 10:56:02 +01:00
338483a232
chore: Make disk log statement less terriblé!
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-29 10:54:59 +01:00
281c20cd1a
chore(deps): Bump memchr from 2.5.0 to 2.6.0 ( #8592 )
...
Bumps [memchr](https://github.com/BurntSushi/memchr ) from 2.5.0 to 2.6.0.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.5.0...2.6.0 )
---
updated-dependencies:
- dependency-name: memchr
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:45:43 +00:00
e9bc7215a1
chore(deps): Bump sysinfo from 0.29.8 to 0.29.9 ( #8591 )
...
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) from 0.29.8 to 0.29.9.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits )
---
updated-dependencies:
- dependency-name: sysinfo
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:39:58 +00:00
83abd6404d
chore(deps): Bump nix from 0.27.0 to 0.27.1 ( #8593 )
...
Bumps [nix](https://github.com/nix-rust/nix ) from 0.27.0 to 0.27.1.
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.27.0...v0.27.1 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:33:42 +00:00
673b6493fa
chore(deps): Bump rustls from 0.21.6 to 0.21.7 ( #8594 )
...
Bumps [rustls](https://github.com/rustls/rustls ) from 0.21.6 to 0.21.7.
- [Release notes](https://github.com/rustls/rustls/releases )
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.6...v/0.21.7 )
---
updated-dependencies:
- dependency-name: rustls
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:27:53 +00:00
191aaf3208
chore(deps): Bump url from 2.4.0 to 2.4.1 ( #8595 )
...
Bumps [url](https://github.com/servo/rust-url ) from 2.4.0 to 2.4.1.
- [Release notes](https://github.com/servo/rust-url/releases )
- [Commits](https://github.com/servo/rust-url/compare/v2.4.0...v2.4.1 )
---
updated-dependencies:
- dependency-name: url
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:22:21 +00:00
eaabac74d0
chore(deps): Bump clap from 4.4.0 to 4.4.1 ( #8596 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.4.0...v4.4.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-29 09:16:46 +00:00
12b8095c46
feat: Upgrade to Rust 1.72.0 ( #8589 )
...
* feat: Upgrade to Rust 1.72.0
* fix: Allow a warning about an error we're intentionally creating
This is a test for an error. This lint warns that this code will cause
an error. Thanks lint, that's what we wanted!
* chore: rustfmt 1.72
* fix: Remove unnecessary hashes in raw string literals
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/needless_raw_string_hashes
Note that there are a number of false negatives with this lint; see
https://github.com/rust-lang/rust-clippy/issues/11420
* fix: Remove unnecessary explicit iteration
Looks like clippy::explicit_iter_loop was improved.
https://rust-lang.github.io/rust-clippy/master/index.html#/explicit_iter_loop
* fix: Allow clippy::manual_try_fold in a few places
Some of these might not be possible to rewrite with try_fold, or at
least not trivially. I don't feel confident enough to change these, in
any case. I think the lint is good to have on for future code though, so
that new code can be written with try_fold.
* fix: Remove useless creation of vectors when an array will do
Mostly in tests. Also fix some long lines.
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_vec
* fix: Allow a single range in a vec init, which is actually what we want
Looks like Clippy's trying to catch a common mistake here, but for realz
we actually want `Vec<Range<usize>>` not `Vec<usize>`
https://rust-lang.github.io/rust-clippy/master/index.html#/single_range_in_vec_init
* fix: Remove a useless conversion
This looks like removing explicit iteration, but it's actually caught by
useless_conversion.
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_conversion
* fix: Remove redundant pattern matching
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/redundant_pat
* fix: Allow an unwrap on a literal None in a test
This matches with the other tests better, and also when I tried to
remove the `unwrap_or_default` it changed the JSON sent from something
with an empty value to `null`, so I think the `or_default` part is
actually changing from one `None` to another `None`.
https://rust-lang.github.io/rust-clippy/master/index.html#/unnecessary_literal_unwrap
2023-08-29 05:57:38 +00:00
468b3ac7f7
fix: Update to a non-yanked version of nix ( #8590 )
...
This looks like why it was yanked:
https://github.com/nix-rust/nix/issues/2112
2023-08-28 21:36:09 +00:00
02c338ba70
fix: don't add empty compact branches ( #8587 )
2023-08-28 19:02:19 +00:00
2572bea8d7
chore(deps): Bump nix from 0.26.2 to 0.27.0 ( #8582 )
...
* chore(deps): Bump nix from 0.26.2 to 0.27.0
Bumps [nix](https://github.com/nix-rust/nix ) from 0.26.2 to 0.27.0.
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.26.2...v0.27.0 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* chore: fix `nix` features
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-28 13:48:56 +00:00
Dom Dwyer
dependabot[bot]
Joe-Blount
wiedld
kodiakhq[bot]
Fraser Savage
Carol (Nichols || Goulding)