7cb1636c64
fix: do not panic when writer disconnects
...
When the RPC write disconnects without waiting for completion, the WAL
panics as there is no longer a consumer of the "committed" ACK.
2022-12-03 17:26:42 +01:00
f40885d4ca
refactor(wal): remove needless async/await
...
Obtaining a rotation handle isn't async.
2022-12-01 16:03:52 +01:00
b6b8e6ac10
Merge remote-tracking branch 'origin/main' into dom/wal-write
2022-11-30 13:27:28 -05:00
096d850fd5
fix: Maintain WAL segment file ordering ( #6287 )
...
Rather than naming WAL files with a UUID, give them a number that
indicates the order they were created in so that they can be read back
in order.
Fixes #6227 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 17:37:37 +00:00
f326baa5d0
test: Update to correctly expect old open files are closed on replay
2022-11-30 12:09:36 -05:00
c48a3b49fb
refactor(wal): rename next_ops -> next_op
...
It only returns one op, so remove the plural.
2022-11-30 16:37:05 +01:00
3029146f5a
refactor(wal): remove SequenceNumberNg
...
This actually starts getting more confusing than passing the bare u64
around.
2022-11-30 16:37:00 +01:00
eafc0ea131
fix: Get the file stem rather than file name for the UUID ( #6284 )
...
Oops. Stupid mistake, behavior that should have had a test but didn't.
Fixes #6270 .
2022-11-30 15:22:03 +00:00
727dd864d2
test: Add tests exercising writing of delete and persist ops
2022-11-23 17:15:57 -05:00
47173e53f0
test: Rotating without writing anything is fine
2022-11-23 17:15:57 -05:00
fe837245e7
test: Add some WAL reader unit-ey tests
2022-11-23 17:15:56 -05:00
be445b5057
fix: Remove API to get multiple SegmentEntry records at once; it's not quite right
...
See #6219 for batching reads.
2022-11-23 17:07:49 -05:00
edd606aa3b
feat: Serialize using protobuf instead of json
2022-11-23 17:07:49 -05:00
a85abadfd5
fix: Define and use a sequence number represented as u64 instead
2022-11-23 17:07:49 -05:00
d7ffa65918
feat: Add an id method to OpenSegmentFileWriter
2022-11-23 17:07:49 -05:00
35576b26b5
refactor: Rename placeholder fnamex to build_segment_path
2022-11-23 17:07:49 -05:00
40dd575de9
feat: Add segment_id metadata to WriteSummary
2022-11-23 17:07:48 -05:00
8618264fe6
fix: Remove unused created_at field
2022-11-23 17:07:48 -05:00
1ba0f193a9
fix: fsync parent directory before returning that new segment file was created
2022-11-23 17:07:48 -05:00
d5b439732d
feat: Implement delete WAL file
2022-11-23 17:07:48 -05:00
a2c25f5191
docs: Improve some descriptions
2022-11-23 17:07:48 -05:00
cdec449963
fix: Return single SequencedWalOps for now, will batch later
2022-11-23 17:07:48 -05:00
db9a43f1b3
fix: Create types for 3 different tasks of a WAL
2022-11-23 17:07:48 -05:00
de9c02caad
fix: Use bytes_written rather than file metadata
2022-11-23 17:07:48 -05:00
1354136b93
fix: fsync and panic if it fails
2022-11-23 17:07:47 -05:00
ee99b93d09
fix: Make checksum private
2022-11-23 17:07:47 -05:00
be2207bcd3
refactor: Rename to clarify closed segment types
2022-11-23 17:07:47 -05:00
4335edc6d1
refactor: Rename to clarify open segment types
2022-11-23 17:07:47 -05:00
70b12a8112
fix: Clarify and simplify the public API
2022-11-23 17:07:47 -05:00
1f5fd43829
fix: cross ps and dot qs
2022-11-23 17:07:47 -05:00
0ae40ba025
fix: move writing to blocking and wire up some things
2022-11-23 17:07:47 -05:00
2e6f269840
refactor: move blocking code to new file
2022-11-23 17:07:47 -05:00
49633842a5
test: Rotate and reading of closed segments
2022-11-23 17:07:46 -05:00
58d9ada90c
feat: Stream the reading
2022-11-23 17:07:46 -05:00
51aa2a118b
feat: Add buffering to File reader
2022-11-23 17:07:46 -05:00
5be0315db9
fix: Move blocking read code to separate task
2022-11-23 17:07:46 -05:00
48e5000ec6
fix: Make test data deterministic
2022-11-23 17:07:46 -05:00
4bb8d3a408
fix: Update to DmlWrite only containing IDs
2022-11-23 17:07:46 -05:00
42ed02ea31
feat: Actually write entries to the open segment file
2022-11-23 17:07:46 -05:00
ebe605a246
fix: Notes and questions
2022-11-23 17:07:46 -05:00
497afc5c91
fix: Change name from e to clarify this covers the Ok case too
2022-11-23 17:07:45 -05:00
2df915b189
fix: Notes and questions
2022-11-23 17:07:45 -05:00
1ee14155a5
refactor: associate functions with SegmentFileReader
2022-11-23 17:07:45 -05:00
41aa4f09fc
fix: remove unneeded usage of trait object
2022-11-23 17:07:45 -05:00
e81a84dd40
fix: don't require a reference just to make an owned value
2022-11-23 17:07:45 -05:00
5104dda6fb
fix: Notes and questions
2022-11-23 17:07:45 -05:00
2852d1ae89
fix: simplify error check
2022-11-23 17:07:45 -05:00
05ae070da2
fix: remove impossible failure case
2022-11-23 17:07:45 -05:00
eb6abb5d67
fix: avoid unneeded allocation
2022-11-23 17:07:45 -05:00
651b7a1ce6
fix: use write_all instead of write
2022-11-23 17:07:44 -05:00
Dom Dwyer
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Jake Goulding