fcd80060be
feat(wal): Make `wal` WriteOpEntryDecoder an interator
...
Also, implement drop on `wal_inspect`'s LineProtoWriter and
bubble up flush to the caller.
2023-05-15 20:46:56 +01:00
6cdc95e49d
refactor(wal): Use a separate DecodeError type for WriteOpEntryDecoder
...
Having a ginormous error enum returned for this method means that
the catch-all behaviour gets leaked into the error naming and
semantics of callers. The decoder is a new type and could benefit from
not adding to the existing error enum.
2023-05-04 12:36:23 +01:00
b2e5ea2266
refactor(wal): Add test & docs for WriteOpEntryDecoder
...
Adds some documentation for the WriteOpEntryDecoder and
a unit test that asserts it skips over non write entries
and can continue to be consumed from.
2023-05-03 11:51:00 +01:00
f6dea224e8
feat(wal): Add wal_inspect crate & a write op entry decoder
...
This adds a new crate with a type capable of converting
decoded WAL Write Op entries to line protocol and writing
the result to a namespaced destination. The wal crate
now exports a type which reads the sequenced wal ops and
decodes them as namespaced table batch writes.
2023-05-03 11:50:59 +01:00
bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 ( #7667 )
...
* chore(deps): Bump tokio from 1.27.0 to 1.28.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
850d7f7011
chore(deps): Bump regex from 1.8.0 to 1.8.1 ( #7627 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/commits/1.8.1 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-24 08:25:26 +00:00
91ecf3e820
chore(deps): Bump regex from 1.7.3 to 1.8.0 ( #7616 )
...
* chore(deps): Bump regex from 1.7.3 to 1.8.0
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.3 to 1.8.0.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/commits )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-21 14:58:46 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
44551e7519
chore(deps): Bump regex from 1.7.2 to 1.7.3 ( #7338 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.2 to 1.7.3.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.2...1.7.3 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 09:20:35 +00:00
6172e6c513
chore(deps): Bump regex from 1.7.1 to 1.7.2 ( #7299 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.1 to 1.7.2.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.1...1.7.2 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-22 14:45:20 +00:00
d9ca8f948a
refactor(wal): const SegmentId constructor
...
Allow a SegmentId to be constructed in a const context.
2023-03-14 17:29:15 +01:00
0c530aa9f7
Merge branch 'main' into dom/wal-flusher-task-leak
2023-03-02 20:44:02 +00:00
8fe874a7f0
Merge branch 'main' into dom/record-wal-seqnum-sets
2023-03-02 15:13:36 +00:00
160e93ea48
Merge branch 'main' into dom/perf-batch-buffer-reuse
2023-03-02 10:43:06 +00:00
f3caf604b5
refactor(wal): last batch length for preallocation
...
There's no need to sub 1 from the batch length to shrink the buffer over
time - the capacity of the new batch will be the length of the last. A
large batch followed by a small batch will cause the pre-allocated next
batch to be small too.
2023-03-02 11:40:38 +01:00
0b40e0d17c
feat(wal): SequenceNumberSet for rotated file
...
Changes Wal::rotate() to return the SequenceNumberSet containing the IDs
of all writes in the segment file that is rotated out.
2023-03-02 10:58:03 +01:00
b22643350f
refactor(wal): track segment sequence numbers
...
Changes the WAL to maintain a SequenceNumberSet containing every ID
wrote to the currently open segment file.
The sets are derived from batched data for efficiency, rather than
recorded per write, to prevent any overhead in the hot path. The batch
set is merged with the file set off the hot path, in a separate I/O
thread (not the async runtime).
2023-03-02 10:58:02 +01:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
a55bbebbee
perf(wal): avoid batch buffer reallocations
...
This change causes the WAL to pre-allocate the write batch buffer,
reducing the reallocations & copies that occur in the hot path (this
buffer can grow to be moderately large).
This should automatically size to the correct capacity and (slowly)
reduce buffer overrun.
2023-03-01 15:58:45 +01:00
79f9411e11
fix: wal flusher task / memory leak
...
Although not a problem in conventional usage, leaking this task prevents
the memory used by the wal (which can be substantial) from ever being
deallocated. In turn, this prevents the WAL writer I/O thread from
stopping too.
2023-03-01 15:32:33 +01:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
b9f7f12c0c
perf(wal): avoid buffer allocation in writer
...
Eliminate buffer allocation (& growing) in the WAL file writer by
reusing a single buffer each time.
This implementation shrinks the buffer size down to 128KiB if it grows
above that amount to prevent one large write from consuming memory
forever more (128KiB should be plenty more than the common write size).
2023-02-23 18:05:06 +01:00
c180d3d8ac
perf(wal): reduce I/O syscall count
...
Each WAL entry is prepended by a two field header, followed by the
payload bytes. Previously a syscall was made for each header field, and
then another to write the payload bytes (or in reality, at least one
call is made).
This commit reduces the syscalls down to a single write call by building
the entire record in memory before calling write(). This adds 8 bytes to
the in-memory buffer size compared to prior to this commit.
This is effectively a reimplementation of a BufWriter but optimised for
our expected memory usage and (more importantly) capable of issuing the
fsync calls necessary for WAL durability.
2023-02-23 18:05:06 +01:00
6d147ec008
refactor: warn! -> error! and spelling
...
Fix a typo, use "error" level instead of "warn".
2023-02-23 11:13:57 +01:00
e3498e3925
perf(wal): use dedicated writer I/O thread
...
Change the WAL buffer flusher to use a dedicated I/O thread instead of
performing serialisation & blocking file I/O on the async runtime
threads. This should reduce runtime blocking / latency variance on the
async threads.
The added overhead is 1 channel send, but this is per WAL batch of
writes (not per DML write, or worse, per file write). This impl also
amortises allocation of the serialisation buffer, rather than growing
one incrementally for each batch.
2023-02-23 11:13:56 +01:00
c72c9d2dba
refactor: derive Debug on WAL types
...
Deriving debug is highly encouraged so that Result::unwrap() and friends
can print the state of an object if it is causing a panic (it's
impossible to call unwrap() otherwise!)
2023-02-23 11:13:56 +01:00
30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
b5ce0e4c4d
refactor: remove test-only checksum
...
The correctness of data checksumming is validated by the tests as a
reader property (corrupt checksum -> error), the actual value of the
checksum is irrelevant.
2023-02-03 14:26:32 +01:00
6e6a439ef6
refactor: remove unused checksum field
...
This unreachable checksum is meaningless outside of the WAL
reader/writer implementations.
2023-02-03 14:23:04 +01:00
4e650110cb
chore: reduce scope of allow_deadcode in wal ( #6822 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-03 10:33:41 +00:00
63d0a77daf
feat: Updating to new services for all-in-one ( #6811 )
...
* feat: Updating to new services for all-in-one
* fix: Use correct shard id for ingester2
* fix: clippy
* fix: use wal directory
* fix: end to end tests
* fix: Update tracing cases for new ingest reality
* fix: update metrics test
* fix: Use rpc mode
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-02 20:42:29 +00:00
d0e6b16450
chore(deps): Bump bytes from 1.3.0 to 1.4.0
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.3.0...v1.4.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 00:30:56 +00:00
ed7d02a225
chore(deps): Bump tokio from 1.24.2 to 1.25.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.2 to 1.25.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits/tokio-1.25.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-30 01:57:27 +00:00
c68049c37a
chore(deps): Bump regex from 1.7.0 to 1.7.1 ( #6546 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.0...1.7.1 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:55:41 +00:00
b49cc2e35e
chore(deps): Bump tokio from 1.24.0 to 1.24.1 ( #6545 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.24.0 to 1.24.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.24.0...tokio-1.24.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-10 09:48:44 +00:00
0aacef3c59
chore(deps): Bump once_cell from 1.16.0 to 1.17.0 ( #6473 )
...
* chore(deps): Bump once_cell from 1.16.0 to 1.17.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.16.0...v1.17.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Change once_cell version specifier to major.minor for less churn
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-01-02 17:07:15 +00:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
d9c72bb93f
feat: optimize wal with batching ( #6399 )
...
* feat: optimize wal with batching
Simplified the wal writer so that it batches up write operations. Currently it waits 10ms between fsync calls. We can pull this out to a config variable later if we want, but I think this is good enough for now.
Also updated the reader to be a more simple blocking reader without the extra tasks and channels as that wasn't really getting us anything that I know of.
* chore: cleanup wal code for PR feedback
2022-12-14 16:07:20 +00:00
7cb1636c64
fix: do not panic when writer disconnects
...
When the RPC write disconnects without waiting for completion, the WAL
panics as there is no longer a consumer of the "committed" ACK.
2022-12-03 17:26:42 +01:00
f40885d4ca
refactor(wal): remove needless async/await
...
Obtaining a rotation handle isn't async.
2022-12-01 16:03:52 +01:00
b6b8e6ac10
Merge remote-tracking branch 'origin/main' into dom/wal-write
2022-11-30 13:27:28 -05:00
096d850fd5
fix: Maintain WAL segment file ordering ( #6287 )
...
Rather than naming WAL files with a UUID, give them a number that
indicates the order they were created in so that they can be read back
in order.
Fixes #6227 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 17:37:37 +00:00
f326baa5d0
test: Update to correctly expect old open files are closed on replay
2022-11-30 12:09:36 -05:00
c48a3b49fb
refactor(wal): rename next_ops -> next_op
...
It only returns one op, so remove the plural.
2022-11-30 16:37:05 +01:00
3029146f5a
refactor(wal): remove SequenceNumberNg
...
This actually starts getting more confusing than passing the bare u64
around.
2022-11-30 16:37:00 +01:00
eafc0ea131
fix: Get the file stem rather than file name for the UUID ( #6284 )
...
Oops. Stupid mistake, behavior that should have had a test but didn't.
Fixes #6270 .
2022-11-30 15:22:03 +00:00
727dd864d2
test: Add tests exercising writing of delete and persist ops
2022-11-23 17:15:57 -05:00
47173e53f0
test: Rotating without writing anything is fine
2022-11-23 17:15:57 -05:00
fe837245e7
test: Add some WAL reader unit-ey tests
2022-11-23 17:15:56 -05:00
be445b5057
fix: Remove API to get multiple SegmentEntry records at once; it's not quite right
...
See #6219 for batching reads.
2022-11-23 17:07:49 -05:00
Fraser Savage
dependabot[bot]
Dom Dwyer
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Stuart Carnie
Paul Dix
Carol (Nichols || Goulding)