44eeab7e2b
chore: Run cargo hakari tasks
2022-11-24 14:51:21 +00:00
a66fc0b645
feat(ingester): ingester2 init
...
Adds an ingester2 crate to hold the MVP of the Kafkaless project.
This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.
This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.
This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
2022-11-24 15:34:02 +01:00
92719f5b2b
Merge pull request #6181 from influxdata/pauldix/wal
...
feat: Add a WAL crate for ingester2 to use
2022-11-24 09:21:55 +00:00
b86eb763b1
fix: Tell buf to fetch to a depth of 100 commits when looking for breaking changes
...
Per <https://docs.buf.build/reference/inputs#other-options >, this is by
default 50 (and I have 51 commits on this PR 😰 )
100 commits should be enough for anyone.
2022-11-23 17:24:50 -05:00
727dd864d2
test: Add tests exercising writing of delete and persist ops
2022-11-23 17:15:57 -05:00
47173e53f0
test: Rotating without writing anything is fine
2022-11-23 17:15:57 -05:00
fe837245e7
test: Add some WAL reader unit-ey tests
2022-11-23 17:15:56 -05:00
be445b5057
fix: Remove API to get multiple SegmentEntry records at once; it's not quite right
...
See #6219 for batching reads.
2022-11-23 17:07:49 -05:00
edd606aa3b
feat: Serialize using protobuf instead of json
2022-11-23 17:07:49 -05:00
a85abadfd5
fix: Define and use a sequence number represented as u64 instead
2022-11-23 17:07:49 -05:00
d7ffa65918
feat: Add an id method to OpenSegmentFileWriter
2022-11-23 17:07:49 -05:00
35576b26b5
refactor: Rename placeholder fnamex to build_segment_path
2022-11-23 17:07:49 -05:00
40dd575de9
feat: Add segment_id metadata to WriteSummary
2022-11-23 17:07:48 -05:00
8618264fe6
fix: Remove unused created_at field
2022-11-23 17:07:48 -05:00
1ba0f193a9
fix: fsync parent directory before returning that new segment file was created
2022-11-23 17:07:48 -05:00
d5b439732d
feat: Implement delete WAL file
2022-11-23 17:07:48 -05:00
a2c25f5191
docs: Improve some descriptions
2022-11-23 17:07:48 -05:00
cdec449963
fix: Return single SequencedWalOps for now, will batch later
2022-11-23 17:07:48 -05:00
db9a43f1b3
fix: Create types for 3 different tasks of a WAL
2022-11-23 17:07:48 -05:00
de9c02caad
fix: Use bytes_written rather than file metadata
2022-11-23 17:07:48 -05:00
1354136b93
fix: fsync and panic if it fails
2022-11-23 17:07:47 -05:00
ee99b93d09
fix: Make checksum private
2022-11-23 17:07:47 -05:00
be2207bcd3
refactor: Rename to clarify closed segment types
2022-11-23 17:07:47 -05:00
4335edc6d1
refactor: Rename to clarify open segment types
2022-11-23 17:07:47 -05:00
70b12a8112
fix: Clarify and simplify the public API
2022-11-23 17:07:47 -05:00
1f5fd43829
fix: cross ps and dot qs
2022-11-23 17:07:47 -05:00
0ae40ba025
fix: move writing to blocking and wire up some things
2022-11-23 17:07:47 -05:00
2e6f269840
refactor: move blocking code to new file
2022-11-23 17:07:47 -05:00
49633842a5
test: Rotate and reading of closed segments
2022-11-23 17:07:46 -05:00
58d9ada90c
feat: Stream the reading
2022-11-23 17:07:46 -05:00
51aa2a118b
feat: Add buffering to File reader
2022-11-23 17:07:46 -05:00
5be0315db9
fix: Move blocking read code to separate task
2022-11-23 17:07:46 -05:00
48e5000ec6
fix: Make test data deterministic
2022-11-23 17:07:46 -05:00
4bb8d3a408
fix: Update to DmlWrite only containing IDs
2022-11-23 17:07:46 -05:00
42ed02ea31
feat: Actually write entries to the open segment file
2022-11-23 17:07:46 -05:00
ebe605a246
fix: Notes and questions
2022-11-23 17:07:46 -05:00
497afc5c91
fix: Change name from e to clarify this covers the Ok case too
2022-11-23 17:07:45 -05:00
2df915b189
fix: Notes and questions
2022-11-23 17:07:45 -05:00
1ee14155a5
refactor: associate functions with SegmentFileReader
2022-11-23 17:07:45 -05:00
41aa4f09fc
fix: remove unneeded usage of trait object
2022-11-23 17:07:45 -05:00
e81a84dd40
fix: don't require a reference just to make an owned value
2022-11-23 17:07:45 -05:00
5104dda6fb
fix: Notes and questions
2022-11-23 17:07:45 -05:00
2852d1ae89
fix: simplify error check
2022-11-23 17:07:45 -05:00
05ae070da2
fix: remove impossible failure case
2022-11-23 17:07:45 -05:00
eb6abb5d67
fix: avoid unneeded allocation
2022-11-23 17:07:45 -05:00
651b7a1ce6
fix: use write_all instead of write
2022-11-23 17:07:44 -05:00
b96de5983d
fix: don't require a reference just to make an owned value
2022-11-23 17:07:44 -05:00
3eda17ea8b
fix: Make SegmentId into a newtype; test the open segment file is created
2022-11-23 17:07:44 -05:00
eea7f113fb
fix: Start implementing traits on Wal to enable outside use
2022-11-23 17:07:44 -05:00
d7134d4846
fix: Cleanups, notes, and clarifications noticed while reading
2022-11-23 17:07:44 -05:00
CircleCI[bot]
Dom Dwyer
Carol (Nichols || Goulding)
Jake Goulding