This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
kodiakhq[bot]
Marco Neumann
Jacob Marble
Marko Mikulicic
Andrew Lamb
Carol (Nichols || Goulding)
Paul Dix
Carol (Nichols || Goulding)
Raphael Taylor-Davies