c68049c37a
chore(deps): Bump regex from 1.7.0 to 1.7.1 ( #6546 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.0...1.7.1 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:55:41 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
d9c72bb93f
feat: optimize wal with batching ( #6399 )
...
* feat: optimize wal with batching
Simplified the wal writer so that it batches up write operations. Currently it waits 10ms between fsync calls. We can pull this out to a config variable later if we want, but I think this is good enough for now.
Also updated the reader to be a more simple blocking reader without the extra tasks and channels as that wasn't really getting us anything that I know of.
* chore: cleanup wal code for PR feedback
2022-12-14 16:07:20 +00:00
7cb1636c64
fix: do not panic when writer disconnects
...
When the RPC write disconnects without waiting for completion, the WAL
panics as there is no longer a consumer of the "committed" ACK.
2022-12-03 17:26:42 +01:00
f40885d4ca
refactor(wal): remove needless async/await
...
Obtaining a rotation handle isn't async.
2022-12-01 16:03:52 +01:00
b6b8e6ac10
Merge remote-tracking branch 'origin/main' into dom/wal-write
2022-11-30 13:27:28 -05:00
096d850fd5
fix: Maintain WAL segment file ordering ( #6287 )
...
Rather than naming WAL files with a UUID, give them a number that
indicates the order they were created in so that they can be read back
in order.
Fixes #6227 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 17:37:37 +00:00
f326baa5d0
test: Update to correctly expect old open files are closed on replay
2022-11-30 12:09:36 -05:00
c48a3b49fb
refactor(wal): rename next_ops -> next_op
...
It only returns one op, so remove the plural.
2022-11-30 16:37:05 +01:00
3029146f5a
refactor(wal): remove SequenceNumberNg
...
This actually starts getting more confusing than passing the bare u64
around.
2022-11-30 16:37:00 +01:00
eafc0ea131
fix: Get the file stem rather than file name for the UUID ( #6284 )
...
Oops. Stupid mistake, behavior that should have had a test but didn't.
Fixes #6270 .
2022-11-30 15:22:03 +00:00
727dd864d2
test: Add tests exercising writing of delete and persist ops
2022-11-23 17:15:57 -05:00
47173e53f0
test: Rotating without writing anything is fine
2022-11-23 17:15:57 -05:00
fe837245e7
test: Add some WAL reader unit-ey tests
2022-11-23 17:15:56 -05:00
be445b5057
fix: Remove API to get multiple SegmentEntry records at once; it's not quite right
...
See #6219 for batching reads.
2022-11-23 17:07:49 -05:00
edd606aa3b
feat: Serialize using protobuf instead of json
2022-11-23 17:07:49 -05:00
a85abadfd5
fix: Define and use a sequence number represented as u64 instead
2022-11-23 17:07:49 -05:00
d7ffa65918
feat: Add an id method to OpenSegmentFileWriter
2022-11-23 17:07:49 -05:00
35576b26b5
refactor: Rename placeholder fnamex to build_segment_path
2022-11-23 17:07:49 -05:00
40dd575de9
feat: Add segment_id metadata to WriteSummary
2022-11-23 17:07:48 -05:00
8618264fe6
fix: Remove unused created_at field
2022-11-23 17:07:48 -05:00
1ba0f193a9
fix: fsync parent directory before returning that new segment file was created
2022-11-23 17:07:48 -05:00
d5b439732d
feat: Implement delete WAL file
2022-11-23 17:07:48 -05:00
a2c25f5191
docs: Improve some descriptions
2022-11-23 17:07:48 -05:00
cdec449963
fix: Return single SequencedWalOps for now, will batch later
2022-11-23 17:07:48 -05:00
db9a43f1b3
fix: Create types for 3 different tasks of a WAL
2022-11-23 17:07:48 -05:00
de9c02caad
fix: Use bytes_written rather than file metadata
2022-11-23 17:07:48 -05:00
1354136b93
fix: fsync and panic if it fails
2022-11-23 17:07:47 -05:00
ee99b93d09
fix: Make checksum private
2022-11-23 17:07:47 -05:00
be2207bcd3
refactor: Rename to clarify closed segment types
2022-11-23 17:07:47 -05:00
4335edc6d1
refactor: Rename to clarify open segment types
2022-11-23 17:07:47 -05:00
70b12a8112
fix: Clarify and simplify the public API
2022-11-23 17:07:47 -05:00
1f5fd43829
fix: cross ps and dot qs
2022-11-23 17:07:47 -05:00
0ae40ba025
fix: move writing to blocking and wire up some things
2022-11-23 17:07:47 -05:00
2e6f269840
refactor: move blocking code to new file
2022-11-23 17:07:47 -05:00
49633842a5
test: Rotate and reading of closed segments
2022-11-23 17:07:46 -05:00
58d9ada90c
feat: Stream the reading
2022-11-23 17:07:46 -05:00
51aa2a118b
feat: Add buffering to File reader
2022-11-23 17:07:46 -05:00
5be0315db9
fix: Move blocking read code to separate task
2022-11-23 17:07:46 -05:00
48e5000ec6
fix: Make test data deterministic
2022-11-23 17:07:46 -05:00
4bb8d3a408
fix: Update to DmlWrite only containing IDs
2022-11-23 17:07:46 -05:00
42ed02ea31
feat: Actually write entries to the open segment file
2022-11-23 17:07:46 -05:00
ebe605a246
fix: Notes and questions
2022-11-23 17:07:46 -05:00
497afc5c91
fix: Change name from e to clarify this covers the Ok case too
2022-11-23 17:07:45 -05:00
2df915b189
fix: Notes and questions
2022-11-23 17:07:45 -05:00
1ee14155a5
refactor: associate functions with SegmentFileReader
2022-11-23 17:07:45 -05:00
41aa4f09fc
fix: remove unneeded usage of trait object
2022-11-23 17:07:45 -05:00
e81a84dd40
fix: don't require a reference just to make an owned value
2022-11-23 17:07:45 -05:00
dependabot[bot]
Carol (Nichols || Goulding)
Paul Dix
Dom Dwyer
Carol (Nichols || Goulding)
Jake Goulding