Allows the DmlHandler to return different types for each method.
This enables a DmlHandler implementation decorating an inner handler to
return the inner handler's error directly, avoiding any "wrapper"
errors.
* refactor: remove InfluxColumnType::IOx
Remove unused column variant - see #3554 for context.
* refactor: reserve SEMANTIC_TYPE_IOX name in proto
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: Debug bounds on Catalog trait
* feat: validate MutableBatch schema
Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.
* refactor: Send bound on boxed errors
* refactor: clippy
Allow assert_eq!(bool, bool) for readability.
* refactor: no PartialEq<MB Column> for ColumnSchema
Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: move chunk metrics into module
* refactor: use Metrics as internal name
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* build: don't pull in all of tokio
We already specify the tokio features we need so "full" (all features)
is not necessary.
* build: remove chrono dependency
Appears unused.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: Implement a snapshot method on DataBuffer
Fixes#3510.
* test: Add a test snapshotting batches with different but compatible schemas
* fix: Simplify min/max sequencer number collection
The first batch should always have the min sequencer number. The last
batch should always have the max sequencer number. The min should always
be less than (or equal to, in case there's only one batch) the max.
Initialises a ShardedWriteBuffer for the hard-coded "iox_shared" topic.
Adds the following CLI flags:
* --write-buffer: type of buffer [kafka, rskafka, file]
* --write-buffer-addr: write buffer endpoint address
The server uses these config options to initialise the appropriate write
buffer backend, and configure the TableNamespaceSharder to shard
operations over the set of sequencers exposed by the write buffer.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
- Use a more standard way to setup the tracing subsystem (as described
in tracing-subscriber docs)
- Also capture content from `log` crate
- Play nice w/ Rust's libtest message capture
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This was likely broken since #3313 and leads to IOx reporting `UKNOWN`
instead of a proper GIT revision. Having the latter one available can be
very useful for debugging a binary or if you look through log files (we
print the IOx version and revision during server startup).
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: have the deduplicate work without chunk statistics
* test: more tests for duplicates data on different combinations of record batches
* refactor: address review comments
* feat: Sequencer wrapper
This type wraps an underlying WriteBufferWriter implementation, tagging
it with a sequencer ID it should use when enqueuing operations to the
buffer.
* feat: mock sharder
Implements a mock Sharder impl that returns pre-configured responses to
shard(), and captures the input to the call.
* feat: sharded write buffer
Implements sharding of ops into an underlying WriteBuffer.
Writes are sharded by some abstract Sharder impl, collated per shard to
maximise the size of each op (and therefore compression efficiency),
converted into a DML operation and then enqueued in parallel to the
underlying WriteBuffer implementation.
Deletes are modelled as being mapped to a single write buffer shard,
which is the case while we support sharding based on the table &
namespace only. Deletes will be extended to support (potentially)
multiple shards when column overrides are implemented.
* refactor: runtime write buffers
Switch from using static dispatch, to using a runtime specified
WriteBufferWriting implementation.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: support line protocol precision parameter (#3522)
* chore: format imports
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This updates the catalog API to make it easier to work with for consumers. I also found a bug in the MemCatalog implementation while refactoring the tests to work with the new API definition. Consumers will now be able to Arc wrap the catalog and use it across awaits.
Dom Dwyer
Marco Neumann
Andrew Lamb
Edd Robinson
Luke Bond
Raphael Taylor-Davies
Nga Tran
Paul Dix
Carol (Nichols || Goulding)
kodiakhq[bot]
Marko Mikulicic
NGA-TRAN
Paul Dix