This commit changes the iox_catalog test harness so that each test is
run in an independent, randomly generated schema to avoid concurrent
tests interfering with each other.
Each test creates a randomly-named schema, grants permissions to the new
schema, runs the full migration stack against the new schema, and then
executes the code under test.
Allow the caller to set the Postgres schema a migration should be
applied to, rather than restricting the migration to a specific,
hard-coded schema.
BREAKING CHANGE: manually adds a new migration that precedes the
existing migration to ensure the iox_catalog schema exists before
applying the migration. You'll probably have to drop any existing
databases and migrate from scratch:
sqlx database drop; sqlx database create;
This should have been taken care of with the update to Rust 2021
Edition, but rustc had a bug [1]. The bug will be fixed in Rust 1.59, which
means this code will stop compiling.
[1] https://github.com/rust-lang/rust/pull/93179
Wraps the sharded write buffer, schema validator, partitioner and
overall request handler in instrumentation to record call latencies and
export them via the /metrics endpoint.
This adds persistence into the ingester with a lifecycle manager. The persist operation must still be updated to keep track of the min_unpersisted_sequence_number for each sequencer.
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: initial implementaion the Query Plan that query QueryableBatch with filters
* fix: read_filter of QueryableBatch should provide the shema of the columns/projection it needs
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: address review comment
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Changes the NamespaceAutocreation handler to be generic over any
WriteInput.
This allows the NamespaceAutocreation layer to be placed anywhere in the
handler stack, without needing a prior transformation or specific write
type.
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
Allow the MockDmlHandler to capture any input type given to the write()
method. This lets us reuse the mock across all handler implementations,
regardless of their expected write input type.
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
* feat: allow catalog access w/o a transaction
Now the caller has the full control if they want to use a transaction or
not.
* fix: remove non-transaction-safe `create_many`
* fix: remove unnecessary transactions
For sparse data the PB-encoded data (our Kafka wire format) is way
smaller than the MutableBatch (up to a factor 20). So lets use this one
to estimate the size during batching.
Andrew Lamb
kodiakhq[bot]
Dom Dwyer
Marco Neumann
Edd Robinson
Carol (Nichols || Goulding)
Marko Mikulicic
Nga Tran
Paul Dix
jaymebrd
Luke Bond
dependabot[bot]
Raphael Taylor-Davies