The WAL reference tracker's inactive segment empty notification only fires
when dropping to 0, not if it is already 0 (it may do this multiple times
over the lifetime of the ingester). This makes sure that the graceful
shutdown notifier listener is set up before the WAL is rotated and the
file is enqueued in the tracker for deletion.
The ingester can project arbitrary columns at query time, and has no
special requirement that the "time" column be part of that projection.
Because the timestamp summary generation explicitly requires the time
column to exist, it panics when there's no "time" column in the
projection - this is a bit of a modelling mismatch more than anything.
As pointed out, use of the turbofish for the `MockPersistQueue` default
constructor can be avoided by a specialised `Default` implementation
on the type. The WAL reference actor handle is internally refcounted,
so this commit also stops wrapping it in an `Arc`.
Co-authored-by: Dom <dom@itsallbroken.com>
When partition pruning is possible, it skips sending the data for
partitions that have no affect on the query outcome.
This commit does the same for the partition metadata - these frames can
form a significant portion of the query response when the row count is
low, and for pruned partitions have no bearing on the query result.
Benchmark the performance of concurrent queries against a single
partition, varying the number of concurrent queries and size of buffered
data in the partition.
This change integrates the WAL reference actor with the graceful
shutdown buffer drain & persist behaviour, relying on its knowledge of
partial persistence for deletion and shutdown timing.
Instead of refreshing every metric in the System every 10 seconds,
refresh only the disk statistics for the disk we're interested in.
Additionally resolve the parent disk for the directory path once,
instead of each loop.
This adds an integration test that writes some data to the ingester,
waits for the WAL to be rotated and then ensures that the segment file
has been dropped.
* the metric attributes are hardcoded to the path
* the duration (frequency) of the background task is hardcoded
* the tick.await now occurs after the first metric recording, such that the test doesn't have to wait 15 seconds.
This implements `PersistCompletionObserver` for the `WalReferenceHandle`
so that it can be given to the persist handle and notified of persist
completions in order to drop WAL segments once all writes are
persistent.
This commit updates the DML sink for the write-ahead log to notify the
reference tracker of writes that have been committed to the log, but
failed to be applied to the buffer tree.
This is the first commit in line to connect the WAL segment reference
tracker actor up to the rest of the ingester. It removes the segment file
deletion and hacky sleep from the rotate task, deferring to the actor
for deletion tracking.
Writes now contain multiple sequence numbers, so the WAL reference
actor must be notified of *all* sequence numbers contained for a write
that failed to be applied to the buffer.
This adds extra test coverage for the ingester's WAL replay & RPC write
paths, as well as the WAL E2E tests, to ensure that all sequence numbers
present in a WriteOperation/WalOperation are encoded and present when
decoded.
This commit asks the oracle for a new sequence number for each table
batch of a write operation (and thus each partition of a write) when
handling an RPC write operation before appending the operation to the
WAL. The ingester now honours the sequence numbers per-partition when
WAL replay is performed.
This commit removes the op-level sequence number from the proto
definition, now reading and writing solely to the per table (and thus
per partition) sequence number map. Tables/partitions within the same
write op are still assigned the same number for now, so there should be
no semantic different
dependabot[bot]
Carol (Nichols || Goulding)
Fraser Savage
kodiakhq[bot]
Dom Dwyer
wiedld
wiedld