Commit Graph

49662 Commits (pbarnett/update-script-to-dockerhub)

Author SHA1 Message Date
kodiakhq[bot] e23c6840cf
Merge branch 'main' into savage/cli-table-list 2023-09-01 11:12:05 +00:00
Dom fbe2723fba
Merge pull request #8643 from influxdata/crepererum/proto_check_iq_proto
chore: apply proto lints to ingester->query gRPC
2023-09-01 12:10:54 +01:00
Fraser Savage b969d07bc7
refactor(wal): Consume `CrcReader` when invoking `checksum()` 2023-09-01 12:06:10 +01:00
Marco Neumann 3498ded991
chore: apply proto lints to ingester->query gRPC
We should catch breaking changes for the ingester->querier protobufs as well.
2023-09-01 13:03:11 +02:00
Marco Neumann 12f2716180
feat: scaffolding for ingester->querier V2 client (#8632)
Adds basic structure for #8349. This will be filled in using separate
PRs for easier review.

The layer structure was chosen to simplify testing and allow composition
of features (like retries, circuit breaking, metrics, etc.). In contrast
to the V1 client (`querier::ingester`) a client here addresses exactly 1
ingester, not multiple (via an `addr` parameter). The tracking around
mutiple states in the V1 version is not really nice and overly
complicated.
2023-09-01 07:58:26 +00:00
dependabot[bot] b337ce91d6
chore(deps): Bump clap from 4.4.1 to 4.4.2 (#8636)
Bumps [clap](https://github.com/clap-rs/clap) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.4.1...v4.4.2)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-01 07:52:28 +00:00
Marco Neumann 1f732595d1
feat: re-introduce sqlx pool "used" counter (#8624)
This was removed in #8336 because the tracking was broken. However
having an "almost right" metric is still helpful because it helps with
dashboarding (see code comment).

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 16:18:36 +00:00
Marco Neumann 1f3ee8bf91
refactor: prep work for #8349 (#8626)
* refactor: make projection masks unsigned

* fix: buffer alignment

* feat: more precise serialization error

* refactor: make `client_util` tower helper public

This can be used for #8349 to set tracing headers.

* fix: impl `Eq` for `TimestampMinMax`

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 16:12:50 +00:00
Fraser Savage 4ff3998c2c
docs(cli): Table create should have help text match arg for database 2023-08-31 16:47:11 +01:00
Fraser Savage 00873fac09
test(cli): Add e2e test for `table list` command 2023-08-31 16:47:10 +01:00
Fraser Savage 2a396c1b73
feat(cli): Add `table list <DATABASE>` command
This commit hooks the influxdb_iox_client and CLI up to the new
`GetTables` gRPC endpoint on the `TableService`, allowing users to
query a list of tables within a database and consequently see any
custom partitioning schemes.
2023-08-31 16:47:09 +01:00
Marco Neumann 5af22cefe6
fix: add assertion migration for #8616 (#8623)
We don't think that there are any dirty clusters left, but better be
safe than sorry.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 15:07:18 +00:00
Dom 083b516b9d
Merge pull request #8630 from influxdata/dom/merkle-unsigned-port
refactor(proto): unsigned port numbers
2023-08-31 15:29:01 +01:00
Dom 2531cbbad2
Merge branch 'main' into dom/merkle-unsigned-port 2023-08-31 15:10:06 +01:00
kodiakhq[bot] ae4c3e1c2a
Merge pull request #8629 from influxdata/savage/add-get-tables-request-rpc
feat(table): `GetTables` gRPC endpoint implementation
2023-08-31 14:00:31 +00:00
Dom Dwyer 3a2e70157e
refactor(proto): unsigned port numbers
I was trying to remember what the notation for integer types was in
proto - -1 is not a port number...
2023-08-31 15:57:41 +02:00
kodiakhq[bot] b2222bb827
Merge branch 'main' into savage/add-get-tables-request-rpc 2023-08-31 13:55:12 +00:00
Marco Neumann 0b2c883440
fix: clippy (#8625)
I think by default we don't check all tests, but this one annoyed me
because my IDE complained about it.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 12:32:27 +00:00
Fraser Savage f110aee262
feat(table): GetTables gRPC endpoint implementation 2023-08-31 13:19:11 +01:00
Fraser Savage 8d47317345
chore(table): Add missing assertion for None partition template 2023-08-31 13:19:10 +01:00
Fraser Savage 1007989cd8
refactor(proto): Impl From<Table> to proto Table 2023-08-31 13:19:09 +01:00
kodiakhq[bot] 5b48b4983b
Merge pull request #8627 from influxdata/savage/define-get-tables-request-rpc
feat(proto): Define GetTables RPC for TableService
2023-08-31 11:54:27 +00:00
kodiakhq[bot] 0c21079884
Merge branch 'main' into savage/define-get-tables-request-rpc 2023-08-31 11:49:12 +00:00
Fraser Savage 683b513e40
feat(proto): Define GetTables RPC for TableService 2023-08-31 12:35:34 +01:00
Dom eb6a4a9dc3
Merge pull request #8611 from influxdata/dom/ingester-gossip-persist
feat(ingester): gossip new parquet file persistence
2023-08-31 11:26:04 +01:00
Dom 45079c0bc1
Merge branch 'main' into dom/ingester-gossip-persist 2023-08-31 11:03:11 +01:00
dependabot[bot] f631b13fb0
chore(deps): Bump chrono from 0.4.27 to 0.4.28 (#8622)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.27 to 0.4.28.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.27...v0.4.28)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 09:49:55 +00:00
dependabot[bot] 6c770004af
chore(deps): Bump memchr from 2.6.1 to 2.6.2 (#8621)
Bumps [memchr](https://github.com/BurntSushi/memchr) from 2.6.1 to 2.6.2.
- [Commits](https://github.com/BurntSushi/memchr/compare/2.6.1...2.6.2)

---
updated-dependencies:
- dependency-name: memchr
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 09:30:55 +00:00
dependabot[bot] 4ce11fd9f2
chore(deps): Bump chrono from 0.4.26 to 0.4.27 (#8607)
* chore(deps): Bump chrono from 0.4.26 to 0.4.27

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.26 to 0.4.27.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.26...v0.4.27)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

* fix: Update deprecated chrono methods to their now-recommended versions

`chrono::DateTime::<Tz>::from_utc` has been deprecated and it's now
recommended to use `chrono::DateTime::from_naive_utc_and_offset`
instead.

<https://github.com/chronotope/chrono/pull/1175>

Note that the `Timestamp` type in `influxdb_influxql_parser` is an alias
for `chrono::DateTime<FixedOffset>`.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-08-31 09:18:25 +00:00
wiedld a4567a80e6
chore: more cleanup of compactor panics (#8620) 2023-08-30 14:41:42 -07:00
Dom 44348cdbbb
Merge pull request #8612 from influxdata/dom/merkle-move-mst
refactor: move MST et al into module
2023-08-30 18:11:05 +01:00
Dom bb93cbd3aa
Merge branch 'main' into dom/merkle-move-mst 2023-08-30 17:03:51 +01:00
kodiakhq[bot] 0e5bc6dbb8
Merge pull request #8614 from influxdata/savage/cli-should-display-partition-template
feat(cli): Queries for namespaces and tables should return their `PartitionTemplate`
2023-08-30 15:34:36 +00:00
kodiakhq[bot] c46b754529
Merge branch 'main' into savage/cli-should-display-partition-template 2023-08-30 15:29:07 +00:00
Fraser Savage c0fdd97212
test(e2e): Add minimal coverage that CLI reports custom partition templates 2023-08-30 16:27:03 +01:00
Dom 2f8edd316f
Merge pull request #8617 from influxdata/dom/sync-gossip
feat(gossip): schema sync proto message & topic
2023-08-30 16:10:24 +01:00
Dom a5a3996e1c
Merge branch 'main' into dom/sync-gossip 2023-08-30 16:01:51 +01:00
Dom Dwyer c2d50c0a95
feat(gossip): schema cache convergence topic
Adds a new topic to be used for schema cache convergence messages.
2023-08-30 16:58:44 +02:00
Fraser Savage 1c80c853b4
feat(table): Return PartitionTemplate in table create proto response
If tables can be created with a custom partition template (or the namespace
has a custom partition template) then this should be exposed to the user
interacting with it through the CLI too!
2023-08-30 15:53:54 +01:00
Fraser Savage e602f067ad
feat(namespace): Return namespace custom partition templates to API
This exposes the custom partitionining scheme of a namespace (if any)
in the response to namespace create and list calls.
2023-08-30 15:53:43 +01:00
Dom Dwyer 8751720f1f
feat(gossip): sync message proto
Defines the message one peer will send to another to solicit a
consistency check.
2023-08-30 16:44:26 +02:00
Dom be3c8220f5
Merge pull request #8610 from influxdata/dom/gossip-ident
feat(gossip): expose peer identity & socket address
2023-08-30 15:42:03 +01:00
Dom Dwyer 4414c6940b
refactor: move MST et al into module
Adds a "mst" (merkle search tree) submodule in anti_entropy, and moves
all the MST code into it.

This makes space for a gossip-based sync primitive to live here too.
2023-08-30 14:47:15 +02:00
Dom Dwyer 8df7eb0ee5
perf: gossip new parquet file persistence events
Configures the ingesters to gossip parquet file persistence
notifications to listening peers.

Because there's no current listeners, this will be a no-op - the gossip
subsystem will not send the messages as no peer will be registered as
interested in them, so the frame rate metrics, etc, will not change
until something registers as interested in the new topic.
2023-08-30 14:38:39 +02:00
Dom Dwyer 98dcabc689
feat(ingester): gossip parquet persist observer
Adds a PersistCompletionObserver that gossips the ParquetFile records of
persisted parquet files to listening peers (if any).
2023-08-30 14:38:00 +02:00
Dom Dwyer cba9dc7b12
refactor: expose ParquetFile in persist note
Allow consumers of persistence notifications to immutably access the
underlying ParquetFile metadata.
2023-08-30 14:33:39 +02:00
Dom Dwyer 76dafa5116
feat(gossip): peer socket address lookup
This allows a caller to obtain the socket address of a peer identity,
and therefore the address from which a message came from.

I think the use cases of this are pretty slim (which is why I purposely
kept the concept of peers out of the handle) but it definitely is useful
as a primitive service discovery directory.
2023-08-30 14:18:02 +02:00
Dom Dwyer 1737ce0450
test(gossip): sender identity
Ensure the correct Identity is provided to the payload Dispatcher.
2023-08-30 14:18:00 +02:00
Dom Dwyer 507739f9fe
refactor(gossip): Identity in API, not UUID
Swap all uses of UUID in the gossip API for the Identity, allowing the
caller to parse into the UUID format if needed, and not forcing it on
them.
2023-08-30 14:18:00 +02:00
Dom Dwyer 20d63b8aa6
feat(gossip): expose sender identity in Dispatcher
When the gossip subsystem hands an application payload off to the
Dispatcher impl, include the sender's Identity.
2023-08-30 14:17:59 +02:00