77f340f964
chore(deps): Bump thiserror from 1.0.47 to 1.0.48 ( #8658 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.47 to 1.0.48.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04 09:21:22 +00:00
d2c71bfe67
chore(deps): Bump thiserror from 1.0.46 to 1.0.47 ( #8519 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.46 to 1.0.47.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.46...1.0.47 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18 09:02:48 +00:00
7094189004
chore(deps): Bump tokio from 1.31.0 to 1.32.0 ( #8507 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.31.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.31.0...tokio-1.32.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:06:29 +00:00
fff313b80c
chore(deps): Bump thiserror from 1.0.44 to 1.0.46 ( #8496 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.44 to 1.0.46.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.44...1.0.46 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-16 10:54:47 +00:00
34b8585931
chore(deps): Bump tokio from 1.30.0 to 1.31.0 ( #8482 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.30.0 to 1.31.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.30.0...tokio-1.31.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:32:34 +00:00
3675043585
chore(deps): Bump tokio from 1.29.1 to 1.30.0 ( #8464 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.29.1 to 1.30.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.1...tokio-1.30.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 07:50:18 +00:00
faa8d44492
chore(deps): Bump thiserror from 1.0.43 to 1.0.44 ( #8315 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.43 to 1.0.44.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.43...1.0.44 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-24 10:18:44 +00:00
e06b6987f0
Merge branch 'main' into savage/remove-op-level-sequence-number-for-writes
2023-07-07 10:12:04 +00:00
057ee40cb9
chore(deps): Bump thiserror from 1.0.41 to 1.0.43 ( #8181 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.41 to 1.0.43.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.41...1.0.43 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:25:12 +00:00
30939cfe96
refactor(wal): Remove op-level `sequence_number`, use per table map
...
This commit removes the op-level sequence number from the proto
definition, now reading and writing solely to the per table (and thus
per partition) sequence number map. Tables/partitions within the same
write op are still assigned the same number for now, so there should be
no semantic different
2023-07-05 14:20:43 +01:00
3827257f94
chore(deps): Bump thiserror from 1.0.40 to 1.0.41 ( #8149 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.40...1.0.41 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-05 09:25:14 +00:00
b15c6062a9
chore(deps): Bump tokio from 1.28.2 to 1.29.0 ( #8100 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 13:18:08 +00:00
fad34c375e
refactor(wal): Use TableId type for look-aside map key
...
This adds a little extra layer of type safety and should be optimised
by the compiler. This commit also makes sure the ingester's WAL sink
tests assert the behaviour for partitioned sequence numbering on an
operation that hits multiple tables & thus partitions.
2023-06-08 11:39:23 +01:00
7de98a6f11
refactor(wal): Associate sequence numbers to table ID in `SequencedWalOp`s
...
Writes are partitioned before being placed in the buffer tree. This
has the effect of splitting up the persistence of a DmlWrite's contents
and thus the persistence of data referred to by write operations placed
into a single WAL entry for a write op.
This change associates the currently assigned sequence number
with every `TableId` in the write, so that persist events for a single
write can be tracked on a per table/partition level.
Making this partial change enables a transition period where changes
can be rolled back and WAL files can still be processed.
A future change will produce a new sequence number per table
ID.
2023-06-06 17:49:09 +01:00
1d6fd83a9a
Merge branch 'main' into savage/wal-regenerate-lp-catalog-support
2023-06-02 14:23:55 +00:00
e0720db138
chore(deps): Bump tokio from 1.28.1 to 1.28.2 ( #7885 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.1 to 1.28.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.1...tokio-1.28.2 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-29 09:11:24 +00:00
bf031641c5
feat(cli): Add measurement name lookup to `wal regenerate-lp` command
...
This commit adds support for the CLI to query the namespace and schema
APIs to retrieve database and table names from the IDs found in WAL
entries being regenerated.
2023-05-26 17:31:19 +01:00
51d59f8216
refactor(`wal_inspect`): Make `LineProtoWriter` namespace unaware
...
Instead, the type responsible for initialising it handles namespaced
`Write` initialisation and management, as well as the failure paths that
may need handling. This commit introduces a `NamespaceDemultiplexer`
type with a generic implementation allowing fallible `async` lazy init
of any type from a given `NamespaceId`. This paves the way for catalog-aware
initialisation of `LineProtoWriter`s.
2023-05-26 17:12:35 +01:00
928a4d163e
build: remove unused dependencies from crates
...
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).
I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!
https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html
This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
2023-05-23 14:55:43 +02:00
b9bcaf1aa0
Merge branch 'main' into savage/wal-regenerate-lp-cli-command
2023-05-22 16:18:44 +00:00
6344fe8c3f
chore: Add rationale for `clippy::future_not_send` ( #7822 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:58:56 +00:00
fd5d5e0758
fix(wal): Assert WriteOpDecoder handles tail-corrupt WAL files correctly
...
WAL read errors encountered by the new WAL WriteOpDecoder were being
discarded and presented as a "happy path" end of file to callers due
to a bug in handling a nested result type. This moves the test for
decoding a tail-corrupt WAL into the crate itself and asserts the
error is reported correctly.
2023-05-18 17:18:19 +01:00
21d6460453
refactor(wal_inspect): Put the ref inside the option
2023-05-17 11:37:49 +01:00
a56eb6efa6
refactor(wal_inspect): Take generic iterator rather than map as table_batch parameter
...
This allows for callers to use whatever collection implementation they
need, so we can ordered iterate in tests.
2023-05-17 11:31:05 +01:00
d530b728d4
feat(wal_inspect): Make table_name_index optional for line proto writing
...
This will allow for decoding a WAL into line protocol without connecting
to a catalog service (or having an equivalent name lookup mechanism).
2023-05-17 10:48:14 +01:00
e7a0ed4b24
refactor(wal_inspect): Make namespaced table batch writing a trait
...
This change allows callers to treat LineProtoWriter instances with
different write sink generators the same using generic parameters.
2023-05-17 10:48:13 +01:00
cfb6abf05b
fix(wal_inspect): Make LineProtoWriter flush best effort
2023-05-16 12:26:59 +01:00
a4a22b2732
refactor(wal): Tidy up WriteOpEntryDecoder next() body
...
Refactor out op comparison for `wal` decode tests to be
more general.
2023-05-16 12:06:54 +01:00
fcd80060be
feat(wal): Make `wal` WriteOpEntryDecoder an interator
...
Also, implement drop on `wal_inspect`'s LineProtoWriter and
bubble up flush to the caller.
2023-05-15 20:46:56 +01:00
6cdc95e49d
refactor(wal): Use a separate DecodeError type for WriteOpEntryDecoder
...
Having a ginormous error enum returned for this method means that
the catch-all behaviour gets leaked into the error naming and
semantics of callers. The decoder is a new type and could benefit from
not adding to the existing error enum.
2023-05-04 12:36:23 +01:00
b2e5ea2266
refactor(wal): Add test & docs for WriteOpEntryDecoder
...
Adds some documentation for the WriteOpEntryDecoder and
a unit test that asserts it skips over non write entries
and can continue to be consumed from.
2023-05-03 11:51:00 +01:00
f6dea224e8
feat(wal): Add wal_inspect crate & a write op entry decoder
...
This adds a new crate with a type capable of converting
decoded WAL Write Op entries to line protocol and writing
the result to a namespaced destination. The wal crate
now exports a type which reads the sequenced wal ops and
decodes them as namespaced table batch writes.
2023-05-03 11:50:59 +01:00
dependabot[bot]
kodiakhq[bot]
Fraser Savage
Dom Dwyer
Andrew Lamb