Adds the server id to replicated writes. Changes type from u8 to u16 for more potential write servers. Also updates the server struct in Delorean to no longer have locks. Locking will happen higher up, likely with the server being arc_swap'd out on any configuration update.
This commit refactors the flatbuffers data types from the wal to a new crate where they can be used by storage, write buffer, and cluster. It also refactors cluster to move the configuration types out to the data types crate so they can be used across storage and elsewhere.
Finally, it adds a new method to store replicated writes on a database in the database trait and implements it.
* chore: Refactor write buffer WAL
This commit refactors the WAL to remove partition events and to collapse rows into a single write buffer entry.
This further simplifies the WAL by removing WriteBufferBatch.
Finally, this removes the concept of a partition generation as that is currently not used.
* refactor: WriteBuffer database and WAL Flatbuffers
This refactor updates the WriteBuffer write path signficantly. At the public API it takes parsed lines, but then immediately converts them over to a built Flatbuffer byte array, which has also been signficantly refactored.
The Flatbuffer structure has been updated so that a WriteBufferBatch contains a vec of WriteBufferEntry. Each of those entries corresponds to a collection of data that is bound for a single partition. The generated partition key is now kept as part of this entry.
Within the WriteBufferEntry you now have a vec of TableWriteBatch which have the table name and a vec of Row. This pulls the table name out of the row, elminating redundancy for writes that have multiple rows being written into the same table.
The database now has methods to accept the Flatbuffer WriteBufferEntry with updates down the line to Partition and Table.
This also has a nice little performance bump for WAL restore:
wal-restoration/restore_single_entry_single_partition
time: [684.51 us 688.45 us 692.53 us]
thrpt: [1.4440 Melem/s 1.4525 Melem/s 1.4609 Melem/s]
change:
time: [-55.913% -55.351% -54.800%] (p = 0.00 < 0.05)
thrpt: [+121.24% +123.97% +126.82%]
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe
wal-restoration/restore_multiple_entry_multiple_partition
time: [8.7483 ms 8.8964 ms 9.0815 ms]
thrpt: [1.3214 Melem/s 1.3489 Melem/s 1.3717 Melem/s]
change:
time: [-55.952% -55.166% -54.213%] (p = 0.00 < 0.05)
thrpt: [+118.40% +123.04% +127.02%]
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
5 (5.00%) high mild
4 (4.00%) high severe
* fix: fmt
Co-authored-by: alamb <andrew@nerdnetworks.org>
* feat: write_database support for predicates
* fix: temporarily pull in arrow fork to pick up fix for ARROW-10136
* fix: Update mutex usage based on PR feedback
* fix: more mutex polish and use OptionExt
* fix: update comments
* fix: rust-fu the table lookup
* fix: update docs
* fix: more idomatic rust types
* fix: better usage of reference types
Previously, we needed to add an entry to the WAL when we added a new
string in the dictionary; now that the WAL entries are self-describing,
we only need to do one lookup here.
* refactor: Extract column count to be a method on ParsedLine
* refactor: Simplify parsing nanoseconds as datetime
* refactor: Extract a constructor for WalEntryBuilder
* refactor: Extract a method on WalEntryBuilder to get bytes
* refactor: Avoid an iteration and a vec allocation
* refactor: Organize and alphabetize dependencies and imports
* fix: Propagate warning config to new crates
By removing the database name argument (which was only used for errors)
and making a separate error type for restoration errors.
This makes it easier to benchmark the restore_partitions_from_wal
function independently. This also has a nice side effect of splitting
errors that can happen during restoration from errors that can happen at
other times.
Switched some fields that were named `_id` to not have a suffix to set
up easier switching of those from u32 to String in a future commit.
Andrew Lamb
Paul Dix
Carol (Nichols || Goulding)
Paul Dix
Jake Goulding
Carol (Nichols || Goulding)