932532c3e3
fix(bench): schema validator bench panic
...
The benchmark code was completely broken - running the benches would
immediately panic.
2023-08-29 12:41:47 +02:00
10396e6986
refactor(service_grpc_catalog): use conversion fns
...
Use well-tested From impls for converting partition IDs & ParquetFile
instances to their proto representation.
2023-08-29 12:35:46 +02:00
4c2945719a
feat: proto serialisation of ParquetFile
...
Adds conversion functions to serialise a ParquetFile into a protobuf
representation, and back again.
Adds randomised testing to assert round-trip equality.
2023-08-29 12:35:45 +02:00
ae3f73f65e
refactor(proto): optional ParquetFile::to_delete
...
This field is nullable, so lets model it as nullable.
2023-08-29 12:35:44 +02:00
124b3d2b42
test: MerkleTree cache decorator
...
Adds an integration test asserting the derived MST content hashes
accurately track updates to an underlying cache entry merge
implementation.
This ensures the merge implementation, and content hashes do not become
out-of-sync.
2023-08-29 12:26:42 +02:00
b694b9f494
feat(router): Merkle tree content hash for cache
...
Adds a (currently unused) NamespaceCache decorator that observes the
post-merge content of the cache to maintain a content hash.
This makes use of a Merkle Search Tree
(https://inria.hal.science/hal-02303490 ) to track the CRDT content of
the cache in a deterministic structure that allows for synchronisation
between peers (itself a CRDT).
The hash of two routers' NamespaceCache will be equal iff their cache
contents are equal - this can be used to (very cheaply) identify
out-of-sync routers, and trigger convergence. The MST structure used
here provides functionality to compare two compact MST representations,
and identify subsets of the cache that are out-of-sync, allowing for
cheap convergence.
Note this content hash only covers the tables, their set of columns, and
those column schemas - this reflects the fact that only these values may
currently be converged by gossip. Future work will enable full
convergence of all fields.
2023-08-29 12:19:43 +02:00
bd4a3fbbb8
refactor: impl Hash for NamespaceSchema
...
Allow the NamespaceSchema to be hashed (including the underlying proto
types it contains).
2023-08-29 12:19:41 +02:00
fc694effda
refactor: AsRef bytes for NamespaceName
...
The NamespaceName is a wrapper over a str (conceptually), which allows
for cheap use of the underlying bytes.
2023-08-29 12:18:33 +02:00
74212e90e7
Merge pull request #8499 from influxdata/savage/disk-full-protection
...
feat(ingester): Disk full protection
2023-08-29 10:17:44 +00:00
2319f6bdf1
fix(ingester): Do not await empty file notification disk protection
...
It's possible that this could never happen (or be very slow), while
the evaluation and enforcement of disk protection happens in another
thread. Blocking further clean-up on this condition might wedge the
ingest state!
2023-08-29 11:08:32 +01:00
3d88c9047c
Merge branch 'main' into savage/disk-full-protection
2023-08-29 10:56:02 +01:00
338483a232
chore: Make disk log statement less terriblé!
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-29 10:54:59 +01:00
281c20cd1a
chore(deps): Bump memchr from 2.5.0 to 2.6.0 ( #8592 )
...
Bumps [memchr](https://github.com/BurntSushi/memchr ) from 2.5.0 to 2.6.0.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.5.0...2.6.0 )
---
updated-dependencies:
- dependency-name: memchr
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:45:43 +00:00
e9bc7215a1
chore(deps): Bump sysinfo from 0.29.8 to 0.29.9 ( #8591 )
...
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) from 0.29.8 to 0.29.9.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits )
---
updated-dependencies:
- dependency-name: sysinfo
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:39:58 +00:00
83abd6404d
chore(deps): Bump nix from 0.27.0 to 0.27.1 ( #8593 )
...
Bumps [nix](https://github.com/nix-rust/nix ) from 0.27.0 to 0.27.1.
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.27.0...v0.27.1 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:33:42 +00:00
673b6493fa
chore(deps): Bump rustls from 0.21.6 to 0.21.7 ( #8594 )
...
Bumps [rustls](https://github.com/rustls/rustls ) from 0.21.6 to 0.21.7.
- [Release notes](https://github.com/rustls/rustls/releases )
- [Commits](https://github.com/rustls/rustls/compare/v/0.21.6...v/0.21.7 )
---
updated-dependencies:
- dependency-name: rustls
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:27:53 +00:00
191aaf3208
chore(deps): Bump url from 2.4.0 to 2.4.1 ( #8595 )
...
Bumps [url](https://github.com/servo/rust-url ) from 2.4.0 to 2.4.1.
- [Release notes](https://github.com/servo/rust-url/releases )
- [Commits](https://github.com/servo/rust-url/compare/v2.4.0...v2.4.1 )
---
updated-dependencies:
- dependency-name: url
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 09:22:21 +00:00
eaabac74d0
chore(deps): Bump clap from 4.4.0 to 4.4.1 ( #8596 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.4.0...v4.4.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-29 09:16:46 +00:00
12b8095c46
feat: Upgrade to Rust 1.72.0 ( #8589 )
...
* feat: Upgrade to Rust 1.72.0
* fix: Allow a warning about an error we're intentionally creating
This is a test for an error. This lint warns that this code will cause
an error. Thanks lint, that's what we wanted!
* chore: rustfmt 1.72
* fix: Remove unnecessary hashes in raw string literals
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/needless_raw_string_hashes
Note that there are a number of false negatives with this lint; see
https://github.com/rust-lang/rust-clippy/issues/11420
* fix: Remove unnecessary explicit iteration
Looks like clippy::explicit_iter_loop was improved.
https://rust-lang.github.io/rust-clippy/master/index.html#/explicit_iter_loop
* fix: Allow clippy::manual_try_fold in a few places
Some of these might not be possible to rewrite with try_fold, or at
least not trivially. I don't feel confident enough to change these, in
any case. I think the lint is good to have on for future code though, so
that new code can be written with try_fold.
* fix: Remove useless creation of vectors when an array will do
Mostly in tests. Also fix some long lines.
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_vec
* fix: Allow a single range in a vec init, which is actually what we want
Looks like Clippy's trying to catch a common mistake here, but for realz
we actually want `Vec<Range<usize>>` not `Vec<usize>`
https://rust-lang.github.io/rust-clippy/master/index.html#/single_range_in_vec_init
* fix: Remove a useless conversion
This looks like removing explicit iteration, but it's actually caught by
useless_conversion.
https://rust-lang.github.io/rust-clippy/master/index.html#/useless_conversion
* fix: Remove redundant pattern matching
Thanks Clippy!
https://rust-lang.github.io/rust-clippy/master/index.html#/redundant_pat
* fix: Allow an unwrap on a literal None in a test
This matches with the other tests better, and also when I tried to
remove the `unwrap_or_default` it changed the JSON sent from something
with an empty value to `null`, so I think the `or_default` part is
actually changing from one `None` to another `None`.
https://rust-lang.github.io/rust-clippy/master/index.html#/unnecessary_literal_unwrap
2023-08-29 05:57:38 +00:00
468b3ac7f7
fix: Update to a non-yanked version of nix ( #8590 )
...
This looks like why it was yanked:
https://github.com/nix-rust/nix/issues/2112
2023-08-28 21:36:09 +00:00
02c338ba70
fix: don't add empty compact branches ( #8587 )
2023-08-28 19:02:19 +00:00
2572bea8d7
chore(deps): Bump nix from 0.26.2 to 0.27.0 ( #8582 )
...
* chore(deps): Bump nix from 0.26.2 to 0.27.0
Bumps [nix](https://github.com/nix-rust/nix ) from 0.26.2 to 0.27.0.
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.26.2...v0.27.0 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* chore: fix `nix` features
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-28 13:48:56 +00:00
1df5948c97
feat: Add Compaction Regions ( #8559 )
...
* feat: add CompactRanges RoundInfo type
* chore: insta test updates for adding CompactRange
* feat: simplify/improve ManySmallFiles logic, now that its problem set is simpler
* chore: insta test updates for ManySmallFiles improvement
* chore: upgrade files more aggressively
* chore: insta updates from more aggressive file upgrades
* chore: addressing review comments
2023-08-28 12:59:12 +00:00
8739f30451
chore(deps): Bump serde from 1.0.186 to 1.0.188 ( #8583 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.186 to 1.0.188.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.186...v1.0.188 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-08-28 09:25:33 +00:00
0f32b1d235
Merge pull request #8584 from influxdata/dependabot/cargo/base64-0.21.3
...
chore(deps): Bump base64 from 0.21.2 to 0.21.3
2023-08-28 10:19:41 +01:00
aae478d0f5
chore(deps): Bump base64 from 0.21.2 to 0.21.3
...
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.21.2 to 0.21.3.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.21.2...v0.21.3 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-08-28 09:05:14 +00:00
9ae072d3f8
chore(deps): Bump regex from 1.9.3 to 1.9.4 ( #8581 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.9.3 to 1.9.4.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.9.3...1.9.4 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-28 09:00:52 +00:00
57be618413
chore(deps): Bump regex-automata from 0.3.6 to 0.3.7 ( #8580 )
...
Bumps [regex-automata](https://github.com/rust-lang/regex ) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/regex-automata-0.3.6...regex-syntax-0.3.7 )
---
updated-dependencies:
- dependency-name: regex-automata
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-28 08:49:21 +00:00
323af3d679
Merge pull request #8578 from influxdata/jrb_88_cap_region_count
...
chore: limit compactor region count computations
2023-08-25 15:56:28 -05:00
46179bf164
chore: limit compactor region count computations
2023-08-25 14:41:24 -05:00
2eb74ddb87
chore: revert teaching compactor to use sort_key_ids ( #8574 )
2023-08-25 13:21:12 +00:00
d029265696
Merge pull request #8565 from influxdata/dom/gossip-parquet-crate
...
feat(gossip): new parquet file gossip crate
2023-08-25 13:19:19 +01:00
fa738cfec3
Merge branch 'main' into dom/gossip-parquet-crate
2023-08-25 13:07:07 +01:00
ed94b56c6e
docs: copy/paste fix
...
Co-authored-by: Fraser Savage <fsavage@influxdata.com>
2023-08-25 13:06:58 +01:00
196c589ef6
refactor: `is_terminal` no longer requires an external crate ( #8572 )
...
* refactor: `is_terminal` no longer requires an external crate
This is now part of the stdlib.
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-08-25 09:01:36 +00:00
af8f98e35f
chore(deps): Bump clap from 4.3.24 to 4.4.0 ( #8571 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.24 to 4.4.0.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.24...clap_complete-v4.4.0 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-25 07:54:14 +00:00
e4505912a1
chore: Update DataFusion pin ( #8544 )
...
* chore: Update DataFusion pin
* refactor: Use upstream check
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-24 18:31:33 +00:00
246918feb6
feat: teach compactor to use sort_key_ids instead of sort_key ( #8560 )
...
* feat: teach compactor to use sort_key_ids instead of sort_key
* test: update the test output after chatting with Joe and know the reason of the chnanges
2023-08-24 16:16:12 +00:00
36e66158a2
chore: add log message before compacting partition ( #8569 )
2023-08-24 14:13:23 +00:00
c3f402a0f8
Merge branch 'main' into dom/gossip-parquet-crate
2023-08-24 14:11:09 +01:00
544ff8c3ef
Merge pull request #8567 from influxdata/dom/gossip-docs
...
docs(gossip): move crate docs to README
2023-08-24 13:27:17 +01:00
43bb149ddc
docs(gossip): move crate docs to README
...
Move the docs to a README to be linked to.
2023-08-24 14:15:56 +02:00
180d4affc7
docs: fix diagram padding
...
Super important!
2023-08-24 12:59:55 +02:00
b2e93c97f1
docs: clarify tx input type
...
The ParquetFileTx type accepts any type that converts into a
ParquetFile, not just a ParquetFile.
This allows applications to use whatever application-level type they
have at hand so long as it's convertible.
2023-08-24 12:29:01 +02:00
6a0c54758b
feat(gossip): new parquet file gossip crate
...
Adds a reusable "gossip_parquet_file" crate that provides a use-case
specific wrapper over the underlying gossip transport.
This crate deals with the encoding and decoding of parquet gossip
messages, handling them off to the application, and decoupling latency
of handlers from the gossip reactor.
2023-08-24 11:23:17 +02:00
04051aea4c
refactor(gossip): define NewParquetFiles topic
...
Separate the parquet file gossiping onto a new topic to allow specific
interest subscriptions and minimise cluster traffic.
2023-08-24 11:23:17 +02:00
cdd20a2337
Merge pull request #8553 from influxdata/dom/partition-id-proto
...
feat: proto serialisation of TransitionPartitionId
2023-08-24 10:21:49 +01:00
c350873bf6
Merge branch 'main' into dom/partition-id-proto
2023-08-24 10:14:41 +01:00
7b37479cd1
chore(deps): Bump siphasher from 0.3.10 to 1.0.0 ( #8561 )
...
Bumps [siphasher](https://github.com/jedisct1/rust-siphash ) from 0.3.10 to 1.0.0.
- [Commits](https://github.com/jedisct1/rust-siphash/compare/0.3.10...1.0.0 )
---
updated-dependencies:
- dependency-name: siphasher
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-24 09:13:14 +00:00
aed214afa8
chore(deps): Bump reqwest from 0.11.19 to 0.11.20 ( #8562 )
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.19 to 0.11.20.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.19...v0.11.20 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-24 09:07:49 +00:00
Dom Dwyer
kodiakhq[bot]
Fraser Savage
dependabot[bot]
Carol (Nichols || Goulding)
Joe-Blount
Joe-Blount
Nga Tran
Marco Neumann
Andrew Lamb