bf031641c5
feat(cli): Add measurement name lookup to `wal regenerate-lp` command
...
This commit adds support for the CLI to query the namespace and schema
APIs to retrieve database and table names from the IDs found in WAL
entries being regenerated.
2023-05-26 17:31:19 +01:00
51d59f8216
refactor(`wal_inspect`): Make `LineProtoWriter` namespace unaware
...
Instead, the type responsible for initialising it handles namespaced
`Write` initialisation and management, as well as the failure paths that
may need handling. This commit introduces a `NamespaceDemultiplexer`
type with a generic implementation allowing fallible `async` lazy init
of any type from a given `NamespaceId`. This paves the way for catalog-aware
initialisation of `LineProtoWriter`s.
2023-05-26 17:12:35 +01:00
0b9b775fbd
Merge pull request #7877 from influxdata/alamb/object_store_dep
...
chore: use workspace dependencies for `object_store`
2023-05-26 13:34:27 +01:00
1ff76b7bf2
chore: use workspace dependencies for `object_store`
2023-05-26 07:03:42 -04:00
c1a448e930
feat: Add decoded payload type and size to querier <--> ingester tracing ( #7870 )
...
* feat: Add decoded payload type and size to querier <--> ingester tracing
* feat: add aggregate sizes
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-26 10:05:14 +00:00
7bcde3c544
chore(7618): trace ingester response encoding v2 ( #7820 )
...
* test: integration test for tracing of queries to the ingester
* chore: add FlightFrameEncodeRecorder to record spans per each polling result
* refactor(trace): impl TraceCollector for Arc
Allow any Arc-wrapped TraceCollector implementation to be used as a
TraceCollector. This avoids needing to as_any() and downcast later.
* test: assert FlightFrameEncodeRecorder trace spans
This test exercises the FlightDataEncoder wrapped with the trace
decorator (FlightFrameEncodeRecorder) when executing against a data
source that yields data after varying numbers of Stream polls.
This test passing will validate the FlightFrameEncodeRecorder correctly
instruments the amount of time a client spends waiting on the
FlightDataEncoder to acquire or encode a protocol frame, but also
ensures the decorator correctly accounts for varying behaviours allowed
through the Stream abstraction. It does this by simulating a data source
that is not always immediately ready to provide data, such as a buffer
wrapped in a contended async mutex.
* refactor: move tracing decorator into separate mod
* fix: record spans
* refactor(test): update test
The frame encoder is not one-to-one - it emits two frames for the first
data payload, a schema and a payload. This commit updates the test to
account for it!
* refactor: remove unneeded mut ref, and use enum state method which panics when in a (should be unreachable) state
* chore: add more docs to FlightFrameEncodeRecorder and related
---------
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-26 09:40:16 +00:00
eb0d77f354
chore(deps): Bump regex from 1.8.2 to 1.8.3 ( #7873 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.8.2...1.8.3 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-26 09:34:25 +00:00
ececd0ada7
chore(deps): Bump base64 from 0.21.1 to 0.21.2 ( #7874 )
...
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.21.1 to 0.21.2.
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.21.1...v0.21.2 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-26 09:28:43 +00:00
8a0062b3b5
Merge pull request #7875 from influxdata/sgc/issue/influxql_cleanup
...
chore: Minor cleanup
2023-05-26 07:17:25 +00:00
6c8cab1073
chore: Separate scope from context, as it is a separate concern
2023-05-26 12:12:11 +10:00
377e108152
chore: No need to be exported from current module
2023-05-26 12:12:10 +10:00
928731767e
Merge pull request #7826 from influxdata/cn/table-create-grpc-api
...
feat: table creation gRPC API with optional custom table template
2023-05-25 18:59:00 +00:00
c4eca5fecf
Merge branch 'main' into cn/table-create-grpc-api
2023-05-25 18:53:12 +00:00
27e700f54c
docs: Flag race condition possibility as a known issue
2023-05-25 14:15:18 -04:00
c2e19b3826
docs: Mention tag column creation in the table creation service description
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-25 14:02:37 -04:00
d68a399a7b
fix: fix span name ( #7868 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-25 17:40:43 +00:00
138b14e0db
chore: Update DataFusion and arrow to `40.0.0` ( #7864 )
...
* chore: Update DataFusion and arrow to `40.0.0`
* chore: Run cargo hakari tasks
* fix: update for API
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-25 17:34:51 +00:00
5d457212d4
feat: Log the table's partition template on successful creation
2023-05-25 13:19:33 -04:00
7662b12dd9
fix: Return 'invalid argument' error if a column already exists and it's not a tag
...
No test because I think this is only possible with a race condition.
2023-05-25 13:19:33 -04:00
c3117e7eb8
fix: Return 'already exists' errors from namespace and table gRPC APIs
...
When appropriate, rather than internal errors.
2023-05-25 13:19:33 -04:00
de243ad823
test: Verify default template usage
2023-05-25 10:55:51 -04:00
fe07e34714
test: Add router tests that set templates and verify writes
2023-05-25 10:44:57 -04:00
17219d71fe
feat: Use the table service in the router
2023-05-25 10:44:57 -04:00
e1a93252c5
feat: Add a new table service crate
2023-05-25 10:44:57 -04:00
32195748a3
feat: Add proto definitions for a table create gRPC API
2023-05-25 10:44:57 -04:00
cdd519424d
feat(cli): Automatically send influx-trace-id, and improve help text ( #7830 )
...
* feat(cli): Automatically send influx-trace-id, and improve tracing CLI help
* fix: remove random ra
* fix: clippy
* fix: Update influxdb_iox/src/main.rs
Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com>
* refactor: Use Vec<String>
* fix: clippy
---------
Co-authored-by: Chunchun Ye <14298407+appletreeisyellow@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-25 09:43:55 +00:00
89d8207784
chore(deps): Bump io-lifetimes from 1.0.10 to 1.0.11 ( #7865 )
...
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes ) from 1.0.10 to 1.0.11.
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/v1.0.10...v1.0.11 )
---
updated-dependencies:
- dependency-name: io-lifetimes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-25 07:49:53 +00:00
deb0c52fed
Merge pull request #7862 from influxdata/dom/remove-catalog-metric-accessor
...
refactor(catalog): mark metrics() as test only
2023-05-24 16:48:47 +01:00
2094b45c10
refactor(catalog): mark metrics() as test only
...
This method is used to enable tests - it's never intended to be used in
production code to access the underlying metric registry. The Catalog
trait is responsible for Catalog things, not acting as a dependency
injection for metrics.
The only current use of this is in test code, so no changes needed.
2023-05-24 17:38:10 +02:00
2eebf2a0a7
Merge pull request #7790 from influxdata/cn/store
...
feat: Set, store, and use custom namespace and table partitions on write
2023-05-24 14:43:10 +00:00
d91b75526f
fix: Clarify that the expect is on the Option, not the Result
2023-05-24 10:36:52 -04:00
e67e336a88
docs: Explain why the partition template types are implemented the way they are
2023-05-24 10:36:52 -04:00
efc817c2a8
fix: Remove From impl, leaving TablePartitionTemplateOverride::new as only creation mechanism
...
This makes it clearer that you do or do not have a custom table override
(in the first argument to `new`).
2023-05-24 10:36:52 -04:00
46f7e3e48a
fix: Handle potential for data race in catalog table insertion by re-fetching if detected
2023-05-24 10:36:52 -04:00
90cb4b6ed9
refactor: Extract a function for handling a table missing from the namespace cache
2023-05-24 10:36:52 -04:00
73b09d895f
feat: Store and handle NULL partition_template database values
...
Treat them as the default partition template in the application, but
save space and avoid having to backfill the tables by having the
database values be NULL when no custom template has been specified.
2023-05-24 10:36:52 -04:00
c8712bbc90
fix: Add a fixture test encoding and documenting default partition template assumptions
2023-05-24 10:36:52 -04:00
fb53faaa2f
refactor: Only use Partitioner::default and derive it
2023-05-24 10:34:31 -04:00
aab0acc16a
fix: Panic if attempting to partition on a non-tag column
2023-05-24 10:34:31 -04:00
42804a20bc
fix: Switch to using Sqlite when encoding so there's no extra 1 in the JSON
2023-05-24 10:34:30 -04:00
d713ba935a
refactor: Reduce duplication of encode/decode implementations
...
This is much less gobbledygook.
2023-05-24 10:34:30 -04:00
c479ed184d
refactor: Rearrange definitions in the partition_template module
...
Move the application types to the top, which puts all the sqlx
conversion gobbledygook at the end because it's an internal
implementation detail I'm about to refactor
Git probably isn't going to display this in a super obvious way, but
this commit is only moving code around, not changing any of it
2023-05-24 10:34:30 -04:00
a22d809cdf
test: Create an overridden namespace, and create a table from it (no override), read it back and assert the expected partitioning scheme is derived
2023-05-24 10:34:30 -04:00
2ab3ea03b8
test: Create a default (not overridden) namespace, read it back, assert the expected partitioning scheme is derived
2023-05-24 10:34:30 -04:00
9c0faa66f0
feat: Set a table partition template explicitly or from the namespace
...
And use the table partition template when partitioning writes to that
table.
2023-05-24 10:34:30 -04:00
604bab9508
fix: Make Table create_or_get be only create
2023-05-24 10:34:30 -04:00
afb3838437
feat: Optionally supply the namespace partition template when creating a namespace
2023-05-24 10:10:34 -04:00
47157015d9
feat: Add columns to store the partition templates
2023-05-24 10:10:34 -04:00
6f92bccc99
feat: Use protobuf for PartitionTemplate in CreateNamespace gRPC API
...
The service implementation doesn't use this field yet.
2023-05-24 10:10:34 -04:00
29dccdc61a
Merge pull request #7859 from influxdata/crepererum/clean_up_parquet_indices
...
refactor: remove ununused `parquet_file` indices
2023-05-24 13:51:28 +02:00
Fraser Savage
Dom
Andrew Lamb
Andrew Lamb
wiedld
dependabot[bot]
kodiakhq[bot]
Stuart Carnie
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Marco Neumann