This is the first commit in line to connect the WAL segment reference
tracker actor up to the rest of the ingester. It removes the segment file
deletion and hacky sleep from the rotate task, deferring to the actor
for deletion tracking.
Writes now contain multiple sequence numbers, so the WAL reference
actor must be notified of *all* sequence numbers contained for a write
that failed to be applied to the buffer.
This adds extra test coverage for the ingester's WAL replay & RPC write
paths, as well as the WAL E2E tests, to ensure that all sequence numbers
present in a WriteOperation/WalOperation are encoded and present when
decoded.
This commit asks the oracle for a new sequence number for each table
batch of a write operation (and thus each partition of a write) when
handling an RPC write operation before appending the operation to the
WAL. The ingester now honours the sequence numbers per-partition when
WAL replay is performed.
This commit removes the op-level sequence number from the proto
definition, now reading and writing solely to the per table (and thus
per partition) sequence number map. Tables/partitions within the same
write op are still assigned the same number for now, so there should be
no semantic different
Configure the partition pruning test to use a partition template that
partitions on the "region" field. This will allow it to be used for
pruning at query time.
Similar to #8109.
This was once implemented by the RUB but as it stands right now, no
chunk implements this anymore.
If we ever want to bring this back, we should use the output of
`QueryChunk::data` instead (i.e. use a data-based implementation instead
of a per-chunk one).
Closes#8096.
This interface was once specially implemented by the RUB. The only
actual implementation of it is within the querier that just forwards it
to a simple schema scan. Lift this semantic to `iox_query_influxrpc`
instead so all the chunks can use it.
If we ever want to optimize this again, we should use `QueryChunk::data`
instead (i.e. instead of implementing it within the chunk it should use
the data method and do something smart based on that).
First half of #8096.
Do not (ab)use per-chunk delete predicates for the retention policy.
Instead use a per-table predicate.
This makes the code way cleaner, since the scoping is correct (i.e.
delete predicates are a table-wide attribute, not a chunk-based one) and
it is consistent time predicates that the user providers (e.g. via
`WHERE time > x`).
It also allows us to remove delete predicates (in their current,
non-scalable form) from the query path. A potential future version would
likely not use per chunk predicates (and "is processed" markers) but use
the timestamp / chunk order to determine to which data the predicate
should be applied.
Note that the lowering of the retention policy changed slightly from
```text
(time > (now() - retention)) AND (time < MAX)
```
to
```text
time > (now() - retention)
```
Since the `MAX` cut is just an artifact of the lowering and was unnecessary.
Closes#7409.
Closes#7410.
Moved TABLE2_ID and TABLE2_NAME to the top of the test module, even
though TABLE2_NAME is only used in one spot, to encourage use of the
constants if new tests are added to this file that need a table that's
different from the arbitrary table.
Replaced all occurrences of TableId::new(1234) with TABLE2_ID even
though TABLE2_ID is 1234321; the exact value doesn't matter, the
important property is that it does not equal ARBITRARY_TABLE_ID (which
is 4).
Replace:
- PartitionId::new(0) with ARBITRARY_PARTITION_ID (which is actually 1)
- PartitionId::new(1) with PARTITION2_ID (actually 2)
- PartitionId::new(2) with PARTITION3_ID (actually 3)
So while adding one is a bit confusing in this diff, in the long run,
this will make the test more understandable and consistent with other
tests.
So that when I change the type of PartitionIds to TransitionPartitionId,
I don't have to update all these places that just need an arbitrary
partition ID or related values.
These test constants probably didn't exist when these tests
were created.
Now that sequence numbers are internal to the ingester and the WAL,
there's no need for them to be a signed integer. As noted by
[#7260](https://github.com/influxdata/influxdb_iox/issues/7260) this was
a quirk related to the kafka-based IOx and Postgres only supported
signed integers.
This will hold the deterministic ID for partitions.
Until all existing partitions have this value, this is optional/nullable.
The row ID still exists and is used as the main foreign key in the
parquet_file and skipped_compaction tables.
The hash_id has a unique index so that we can look up records based on
it (if it's available).
If the parquet file record has a partition_hash_id value, use that to
generate the object storage path instead of the partition_id.
This enables the whole of the RPC write path to use the new
`dml_payload::IngestOp` instead of the `DmlOperation` type. The
implementation of `From<&WriteOperation> for DmlWrite` has been removed
to complete the switch.
Removes one of the temporary conversion traits and adds a test helper
method `encode_batch(NamespaceId, WriteOperation)` for removal of the
`DmlOperation` from WAL replay and the RPC write handler.
This change replaces the test_util equality and write generation code
to use the new `IngestOp::Write(WriteOperation)` type, removing many
pointless conversions in tests.
This commit switches over the trait definition to take `IngestOp`
instead of `DmlOperation`. This commit is not enough to complete the
switch, as the conversion from `DmlOperation` and back is a performance
regression.
This commit implements the `From` trait to allow quick conversion from
a `&WriteOperation` back to an owned `DmlWrite`. This conversion
copies all the record batches and should be removed once WAL encoding
can be done from `&WriteOperation`.
This allows accessing temporarily agnostic values without matching on
the kind of `IngestOp`. There is only one type at the moment, so this
just provides a bridge for users of the `DmlOperation` getters.
Fraser Savage
kodiakhq[bot]
dependabot[bot]
Dom Dwyer
Marco Neumann
Carol (Nichols || Goulding)