30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
d9c72bb93f
feat: optimize wal with batching ( #6399 )
...
* feat: optimize wal with batching
Simplified the wal writer so that it batches up write operations. Currently it waits 10ms between fsync calls. We can pull this out to a config variable later if we want, but I think this is good enough for now.
Also updated the reader to be a more simple blocking reader without the extra tasks and channels as that wasn't really getting us anything that I know of.
* chore: cleanup wal code for PR feedback
2022-12-14 16:07:20 +00:00
f40885d4ca
refactor(wal): remove needless async/await
...
Obtaining a rotation handle isn't async.
2022-12-01 16:03:52 +01:00
b6b8e6ac10
Merge remote-tracking branch 'origin/main' into dom/wal-write
2022-11-30 13:27:28 -05:00
096d850fd5
fix: Maintain WAL segment file ordering ( #6287 )
...
Rather than naming WAL files with a UUID, give them a number that
indicates the order they were created in so that they can be read back
in order.
Fixes #6227 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 17:37:37 +00:00
f326baa5d0
test: Update to correctly expect old open files are closed on replay
2022-11-30 12:09:36 -05:00
c48a3b49fb
refactor(wal): rename next_ops -> next_op
...
It only returns one op, so remove the plural.
2022-11-30 16:37:05 +01:00
3029146f5a
refactor(wal): remove SequenceNumberNg
...
This actually starts getting more confusing than passing the bare u64
around.
2022-11-30 16:37:00 +01:00
eafc0ea131
fix: Get the file stem rather than file name for the UUID ( #6284 )
...
Oops. Stupid mistake, behavior that should have had a test but didn't.
Fixes #6270 .
2022-11-30 15:22:03 +00:00
edd606aa3b
feat: Serialize using protobuf instead of json
2022-11-23 17:07:49 -05:00
a85abadfd5
fix: Define and use a sequence number represented as u64 instead
2022-11-23 17:07:49 -05:00
d5b439732d
feat: Implement delete WAL file
2022-11-23 17:07:48 -05:00
cdec449963
fix: Return single SequencedWalOps for now, will batch later
2022-11-23 17:07:48 -05:00
db9a43f1b3
fix: Create types for 3 different tasks of a WAL
2022-11-23 17:07:48 -05:00
ee99b93d09
fix: Make checksum private
2022-11-23 17:07:47 -05:00
70b12a8112
fix: Clarify and simplify the public API
2022-11-23 17:07:47 -05:00
1f5fd43829
fix: cross ps and dot qs
2022-11-23 17:07:47 -05:00
0ae40ba025
fix: move writing to blocking and wire up some things
2022-11-23 17:07:47 -05:00
49633842a5
test: Rotate and reading of closed segments
2022-11-23 17:07:46 -05:00
48e5000ec6
fix: Make test data deterministic
2022-11-23 17:07:46 -05:00
4bb8d3a408
fix: Update to DmlWrite only containing IDs
2022-11-23 17:07:46 -05:00
42ed02ea31
feat: Actually write entries to the open segment file
2022-11-23 17:07:46 -05:00
e81a84dd40
fix: don't require a reference just to make an owned value
2022-11-23 17:07:45 -05:00
3eda17ea8b
fix: Make SegmentId into a newtype; test the open segment file is created
2022-11-23 17:07:44 -05:00
eea7f113fb
fix: Start implementing traits on Wal to enable outside use
2022-11-23 17:07:44 -05:00
f0b93c5c8c
refactor: Rename the wal crate to write_buffer
2021-04-21 17:43:03 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
6f473984d0
style: wrap comments
...
Runs rustfmt with the new config.
2020-12-11 18:22:26 +00:00
a52e0001c5
refactor: rename all crates that start with`delorean_` in preparation for rename ( #415 )
...
* refactor: rename delorean_cluster --> cluster
* refactor: rebane delorean_generated_types --> generated_types
* refactor: rename delorean_write_buffer --> write_buffer
* refactor: rename delorean_ingest --> ingest
* refactor: rename delorean_storage --> storage
* refactor: rename delorean_tsm --> tsm
* refactor: rename delorean_test_helpers --> test_helpers
* refactor: rename delorean_arrow --> arrow_deps
* refactor: rename delorean_line_parser --> influxdb_line_protocol
2020-11-05 13:44:36 -05:00
ff824a5477
refactor: rename delorean_wal --> wal, conslidate wal_writer ( #411 )
2020-11-05 09:25:29 -05:00
Carol (Nichols || Goulding)
Paul Dix
Dom Dwyer
Carol (Nichols || Goulding)
Jake Goulding
Carol (Nichols || Goulding)
Andrew Lamb