Commit Graph

49187 Commits (3c9e6ed836fcd607af90589e438cb35ca83efeb6)

Author SHA1 Message Date
Dom 45079c0bc1
Merge branch 'main' into dom/ingester-gossip-persist 2023-08-31 11:03:11 +01:00
dependabot[bot] f631b13fb0
chore(deps): Bump chrono from 0.4.27 to 0.4.28 (#8622)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.27 to 0.4.28.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.27...v0.4.28)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 09:49:55 +00:00
dependabot[bot] 6c770004af
chore(deps): Bump memchr from 2.6.1 to 2.6.2 (#8621)
Bumps [memchr](https://github.com/BurntSushi/memchr) from 2.6.1 to 2.6.2.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.1...2.6.2)

---
updated-dependencies:
- dependency-name: memchr
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 09:30:55 +00:00
dependabot[bot] 4ce11fd9f2
chore(deps): Bump chrono from 0.4.26 to 0.4.27 (#8607)
* chore(deps): Bump chrono from 0.4.26 to 0.4.27

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.26...v0.4.27)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

* fix: Update deprecated chrono methods to their now-recommended versions

`chrono::DateTime::<Tz>::from_utc` has been deprecated and it's now
recommended to use `chrono::DateTime::from_naive_utc_and_offset`
instead.

<https://github.com/chronotope/chrono/pull/1175>

Note that the `Timestamp` type in `influxdb_influxql_parser` is an alias
for `chrono::DateTime<FixedOffset>`.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 09:18:25 +00:00
wiedld a4567a80e6
chore: more cleanup of compactor panics (#8620) 2023-08-30 14:41:42 -07:00
Dom 44348cdbbb
Merge pull request #8612 from influxdata/dom/merkle-move-mst
refactor: move MST et al into module
2023-08-30 18:11:05 +01:00
Dom bb93cbd3aa
Merge branch 'main' into dom/merkle-move-mst 2023-08-30 17:03:51 +01:00
kodiakhq[bot] 0e5bc6dbb8
Merge pull request #8614 from influxdata/savage/cli-should-display-partition-template
feat(cli): Queries for namespaces and tables should return their `PartitionTemplate`
2023-08-30 15:34:36 +00:00
kodiakhq[bot] c46b754529
Merge branch 'main' into savage/cli-should-display-partition-template 2023-08-30 15:29:07 +00:00
Fraser Savage c0fdd97212
test(e2e): Add minimal coverage that CLI reports custom partition templates 2023-08-30 16:27:03 +01:00
Dom 2f8edd316f
Merge pull request #8617 from influxdata/dom/sync-gossip
feat(gossip): schema sync proto message & topic
2023-08-30 16:10:24 +01:00
Dom a5a3996e1c
Merge branch 'main' into dom/sync-gossip 2023-08-30 16:01:51 +01:00
Dom Dwyer c2d50c0a95
feat(gossip): schema cache convergence topic
Adds a new topic to be used for schema cache convergence messages.
2023-08-30 16:58:44 +02:00
Fraser Savage 1c80c853b4
feat(table): Return PartitionTemplate in table create proto response
If tables can be created with a custom partition template (or the namespace
has a custom partition template) then this should be exposed to the user
interacting with it through the CLI too!
2023-08-30 15:53:54 +01:00
Fraser Savage e602f067ad
feat(namespace): Return namespace custom partition templates to API
This exposes the custom partitionining scheme of a namespace (if any)
in the response to namespace create and list calls.
2023-08-30 15:53:43 +01:00
Dom Dwyer 8751720f1f
feat(gossip): sync message proto
Defines the message one peer will send to another to solicit a
consistency check.
2023-08-30 16:44:26 +02:00
Dom be3c8220f5
Merge pull request #8610 from influxdata/dom/gossip-ident
feat(gossip): expose peer identity & socket address
2023-08-30 15:42:03 +01:00
Dom Dwyer 4414c6940b
refactor: move MST et al into module
Adds a "mst" (merkle search tree) submodule in anti_entropy, and moves
all the MST code into it.

This makes space for a gossip-based sync primitive to live here too.
2023-08-30 14:47:15 +02:00
Dom Dwyer 8df7eb0ee5
perf: gossip new parquet file persistence events
Configures the ingesters to gossip parquet file persistence
notifications to listening peers.

Because there's no current listeners, this will be a no-op - the gossip
subsystem will not send the messages as no peer will be registered as
interested in them, so the frame rate metrics, etc, will not change
until something registers as interested in the new topic.
2023-08-30 14:38:39 +02:00
Dom Dwyer 98dcabc689
feat(ingester): gossip parquet persist observer
Adds a PersistCompletionObserver that gossips the ParquetFile records of
persisted parquet files to listening peers (if any).
2023-08-30 14:38:00 +02:00
Dom Dwyer cba9dc7b12
refactor: expose ParquetFile in persist note
Allow consumers of persistence notifications to immutably access the
underlying ParquetFile metadata.
2023-08-30 14:33:39 +02:00
Dom Dwyer 76dafa5116
feat(gossip): peer socket address lookup
This allows a caller to obtain the socket address of a peer identity,
and therefore the address from which a message came from.

I think the use cases of this are pretty slim (which is why I purposely
kept the concept of peers out of the handle) but it definitely is useful
as a primitive service discovery directory.
2023-08-30 14:18:02 +02:00
Dom Dwyer 1737ce0450
test(gossip): sender identity
Ensure the correct Identity is provided to the payload Dispatcher.
2023-08-30 14:18:00 +02:00
Dom Dwyer 507739f9fe
refactor(gossip): Identity in API, not UUID
Swap all uses of UUID in the gossip API for the Identity, allowing the
caller to parse into the UUID format if needed, and not forcing it on
them.
2023-08-30 14:18:00 +02:00
Dom Dwyer 20d63b8aa6
feat(gossip): expose sender identity in Dispatcher
When the gossip subsystem hands an application payload off to the
Dispatcher impl, include the sender's Identity.
2023-08-30 14:17:59 +02:00
Dom Dwyer f81f6efb2c
refactor(gossip): remove Uuid from Identity
The identity contains both a UUID, and the byte representation of this
UUID. This change retains the latter, generating the former on-demand
(which is rarely needed).

This halves the size of the identity, and the underlying Bytes storage
is a ref-counted container, making clones very cheap.
2023-08-30 14:17:54 +02:00
Dom 5925440d6a
Merge pull request #8609 from influxdata/dom/gossip-subset
feat(gossip): broadcast to randomised peer subset
2023-08-30 13:07:42 +01:00
Dom c983d1a096
Merge branch 'main' into dom/gossip-subset 2023-08-30 12:56:59 +01:00
Dom 1ef6e79aed
Merge pull request #8600 from influxdata/dom/merkle-actor
perf: asynchronously maintain merkle search tree
2023-08-30 11:40:13 +01:00
Dom Dwyer e120935a61
fix(bench): reuse same namespace cache
And add another thread for the hashing to happen on now there's two
async tasks to run.
2023-08-30 12:07:15 +02:00
Dom Dwyer 38f67ae736
perf(anti_entropy): add AntiEntropyActor for MST
Separate the management of the Merkle Search Tree state into an actor
to manage concurrent access.

This moves hashing and tree updates off of the hot request path, and
into an asynchronous background process, practically eliminating
overhead for maintaining the MST structure.

This decoupling will allow convergence runs between peers to proceed
without causing contention on the lock in the request hot path.
2023-08-30 12:07:15 +02:00
Dom Dwyer 934e4bc9c6
test(bench): MerkleTree overhead
Layers the MerkleTree over the NamespaceCache stack in benchmarks to
measure the overhead.
2023-08-30 12:07:14 +02:00
Dom 53d7b23624
Merge pull request #8598 from influxdata/dom/gossip-parquet-proto
refactor: protobuf serialisation primitives for ParquetFile
2023-08-30 11:01:35 +01:00
Dom b68d108baf
Merge branch 'main' into dom/gossip-parquet-proto 2023-08-30 10:55:46 +01:00
Dom Dwyer 43db19b7db
feat(gossip): broadcast to randomised peer subset
Provide a new broadcast_subset() method to gossip users, enabling a
message to be sent to a random subset of peers that have a registered
interest in the message being dispatched.
2023-08-30 11:54:09 +02:00
Dom 74f6de78fd
Merge pull request #8608 from influxdata/dependabot/cargo/memchr-2.6.1
chore(deps): Bump memchr from 2.6.0 to 2.6.1
2023-08-30 10:52:47 +01:00
dependabot[bot] 0d8ba84923
chore(deps): Bump memchr from 2.6.0 to 2.6.1
Bumps [memchr](https://github.com/BurntSushi/memchr) from 2.6.0 to 2.6.1.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.0...2.6.1)

---
updated-dependencies:
- dependency-name: memchr
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-08-30 01:25:04 +00:00
Joe-Blount 68a9019768
chore: remove compactor concurrency scaling (#8606)
* chore: remove compactor concurrency scaling

* chore: comment update to retrigger tests

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-29 23:42:53 +00:00
wiedld 5011f7fe22
chore: add partition_id into panic message (#8605) 2023-08-29 16:37:21 -07:00
Joe-Blount 0996a95630
chore: enable more ManySmallFiles compactions (#8603)
* chore: enable more ManySmallFiles compactions

* chore: insta churn
2023-08-29 20:42:03 +00:00
wiedld 72c48d34f8
chore: add partition id to compactor panics (#8602) 2023-08-29 12:40:11 -07:00
Dom 6c6a3507d1
Merge pull request #8601 from influxdata/dom/no-compiler-err
ci: lint failures as warnings
2023-08-29 16:26:19 +01:00
Dom Dwyer 9055bc1754
ci: lint failures as warnings
Don't fail to compile / run tests because of an unreachable pub, or
missing debug impl - just emit a compiler warning.

This lets the compilation complete, but isn't accepted in PRs as CI runs
with "deny warnings".
2023-08-29 16:29:31 +02:00
kodiakhq[bot] 674a9e3eff
Merge pull request #8597 from influxdata/dom/merkle-cache
feat(router): merkle tree cache observer for anti-entropy
2023-08-29 13:40:02 +00:00
Dom bcc4003222
Merge branch 'main' into dom/merkle-cache 2023-08-29 14:34:34 +01:00
Joe-Blount 38147ec25e
chore: compact available regions before splitting everything (#8586) 2023-08-29 13:29:11 +00:00
Dom 15da02b59f
Merge branch 'main' into dom/merkle-cache 2023-08-29 14:21:26 +01:00
Dom Dwyer c1ba3918a4
test: cache content hash fixture
Assert the hash representing static cache content does not change.
2023-08-29 13:57:39 +02:00
Dom Dwyer 19e7c90fc1
test: compose prop::Strategy for schema generation
Accept an arbitrary ID generation strategy, composing it into a
NamespaceSchema generation strategy to simplify the call args / usage.
2023-08-29 13:47:22 +02:00
Dom b1555621cb
Merge pull request #8599 from influxdata/dom/fix-bench
fix(bench): schema validator bench panic
2023-08-29 11:49:37 +01:00