78926490f9
docs: Update dml_handlers module docs ( #7194 )
...
Update the module diagram to reflect kafkaless
changes.
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-13 13:28:29 +00:00
3a9ca8879b
chore(deps): Bump futures from 0.3.26 to 0.3.27 ( #7193 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 10:53:59 +00:00
535f3d92a2
chore(deps): Bump serde_json from 1.0.93 to 1.0.94 ( #7132 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.93 to 1.0.94.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.93...v1.0.94 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 12:12:23 +00:00
3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 ( #7130 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
3256fcc72e
chore(deps): Bump object_store from 0.5.4 to 0.5.5
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.4 to 0.5.5.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.4...object_store_0.5.5 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-03 02:00:51 +00:00
9ed7aa7257
Merge branch 'main' into dom/router-docs
2023-03-01 10:28:46 +00:00
90ceb2c896
chore(deps): Bump crossbeam-utils from 0.8.14 to 0.8.15 ( #7089 )
...
Bumps [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam ) from 0.8.14 to 0.8.15.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases )
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md )
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-utils-0.8.14...crossbeam-utils-0.8.15 )
---
updated-dependencies:
- dependency-name: crossbeam-utils
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-01 10:16:09 +00:00
32ce2564dc
refactor: consistent validator/validation
...
Use a consistent name for the retention validation (matching the schema
validation module).
2023-02-28 14:45:45 +01:00
f6c9f9b0e9
docs: update router diagram / overview docs
...
Updates the router documentation to reflect the "kafkaless"
implementation we'll use going forward.
2023-02-28 14:45:32 +01:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
aa7d458a81
chore(deps): Bump tokio-stream from 0.1.11 to 0.1.12 ( #7035 )
...
Bumps [tokio-stream](https://github.com/tokio-rs/tokio ) from 0.1.11 to 0.1.12.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.11...tokio-stream-0.1.12 )
---
updated-dependencies:
- dependency-name: tokio-stream
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-21 16:31:55 +00:00
5ca165b76e
docs: fix two typos
...
Comments in test code.
2023-02-15 10:49:39 +01:00
61fb92b85c
feat(router): soft-delete RPC handler
...
This implements the RPC "delete_namespace" handler, allowing a namespace
to be soft-deleted.
Adds unit coverage for all handlers & e2e test coverage for the new
handler (the rest were already covered).
The tests also highlight the caching issue documented here:
https://github.com/influxdata/influxdb_iox/issues/6175
2023-02-15 10:49:38 +01:00
4a0149a418
test(router): retention change after cache reset
...
Assert that the new retention period value is used once a router is
restarted and the cache converged.
2023-02-13 14:40:14 +01:00
ce12daa96e
refactor(e2e): simulate router restart
...
Allow a router to be "restarted" within an e2e test.
2023-02-13 14:40:11 +01:00
0e10561ce0
fix: Port router unit tests to the RPC write path ( #6956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:28:06 +00:00
2d46a364dc
feat: namespace soft-delete support
...
This commit adds initial support for "soft" namespace deletion, where
the actual records & data remain, but are no longer queryable /
writeable.
Soft deletion is eventually consistent - users can expect to continue
writing to and reading from a bucket after issuing a soft delete call,
until the various components either restart, or have their caches
flushed.
The components treat soft-deleted namespaces differently:
* router: ignore soft deleted namespaces
* ingester: accept soft deleted namespaces
* compactor: accept soft deleted namespaces
* querier: ignore soft deleted namespaces
* various gRPC services: ignore soft deleted namespaces
This ensures that the ingester & compactor do not see rows "vanishing"
from the database, and continue to make forward progress.
Writes for the deleted namespace that are buffered in the ingester will
be persisted as normal, allowing us to support "un-delete" operations
where the system is restored to a the state at which the delete was
issued (rather than loosing the buffered data).
Follow-on work is required to ensure GC drops the orphaned parquet files
after the configured GC time, and optimisations such as not compacting
parquet from soft-deleted namespaces seems like a trivial win.
2023-02-13 12:01:35 +01:00
779fb93ce7
refactor: move test builders out of compactor2 code ( #6953 )
...
* refactor: move test builders out of compactor2 code
* fix: docs
2023-02-10 18:28:09 +00:00
a85dcd745b
refactor(catalog): expose deleted_at on Namespace
...
Add the new catalog column to the Namespace representation/model.
2023-02-10 14:15:01 +01:00
6327e3d9c0
chore(deps): Bump serde_json from 1.0.92 to 1.0.93 ( #6918 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.92 to 1.0.93.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.92...v1.0.93 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-09 10:39:33 +00:00
0ecde75af5
chore(deps): Bump object_store from 0.5.3 to 0.5.4 ( #6900 )
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.3 to 0.5.4.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.3...object_store_0.5.4 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-08 09:40:11 +00:00
bf6ab7fd88
fix: error message typo
...
columns -> tables for table limit error.
2023-02-06 18:03:26 +01:00
3f2eb54bce
test(router): catalog service limit errors
...
Assert the service limit error messages from the catalog.
2023-02-06 17:55:09 +01:00
3881e11734
test(router): service limit error messages
...
Assert the user-facing service limit error messages.
2023-02-06 17:43:37 +01:00
114bafe9a1
perf(router): cached table limit enforcement
...
Use the namespace schema cache in the router to enforce the
per-namespace table limit (service protection limit), adding O(1)
overhead to the existing column limit evaluation logic.
Prior to this commit, each request that would breach the table limit
would be (potentially partially) applied to the catalog and return an
error. Every subsequent request creating a new table continued to cause
a catalog query, unnecessarily adding load proportional to request
counts.
After this commit, catalog requests are sent when the router instance
can determine (to the best of it's ability, see below) that the request
will not cause the namespace to exceed the table limit.
Because this uses cached schemas, the actual state set of tables may
have changed - this will cause inconsistent enforcement and spurious
errors in the same way it currently does for the column limit. For more
details (and to track a resolution) see:
https://github.com/influxdata/influxdb_iox/issues/5957
2023-02-06 17:43:26 +01:00
dfa4ab2585
perf(router): fast-path column limit for new table
...
When validating column limits for new tables, skip the column set
generation and union operations against the empty existing column set.
2023-02-06 17:33:56 +01:00
a633964f2b
feat(catalog): return max table limit in schema
...
The maximum number of tables is part of the Namespace, which is already
loaded in its entirety. This commit copies the value into the
NamespaceSchema, making it available for the router to utilise.
2023-02-06 17:33:55 +01:00
6f4e287a3a
chore(deps): Bump serde_json from 1.0.91 to 1.0.92 ( #6860 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.91 to 1.0.92.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.91...v1.0.92 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-06 08:27:41 +00:00
30fea67701
fix: Move variables within format strings. Thanks clippy!
...
Changes made automatically using `cargo clippy --fix`.
2023-02-03 13:06:17 -05:00
7f363b55df
test(router): e2e namespace retention coverage
...
Assert the correct handling of 0 and negative retention periods when
interacting with the namespace create & update gRPC handlers.
2023-02-01 11:49:53 +01:00
d0e6b16450
chore(deps): Bump bytes from 1.3.0 to 1.4.0
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.3.0...v1.4.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-01 00:30:56 +00:00
875b6a3e99
chore(deps): Bump futures from 0.3.25 to 0.3.26 ( #6766 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.25...0.3.26 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-31 11:33:50 +00:00
0ddef54b09
fix(router): envoy network error translation
...
Envoy will connect to an endpoint on demand, and return an
application-level error if it fails with a gRPC status code of
"Unavailable".
It also embeds a metadata entry of {"server": "envoy"} - this commit
uses the two signals (error status code + metadata entry) to drive an
immediate reconnection when observed, assuming the connection is bad.
2023-01-30 12:15:38 +01:00
353b1ad575
feat: configurable RPC write request timeout
...
Allows the user to configure the timeout used for a single RPC write
request, and changes the default to a more sensible value (30 -> 3
seconds).
2023-01-27 14:53:48 +01:00
5757674d5e
Merge branch 'main' into dom/service-limit-metric-labels
2023-01-27 10:04:56 +00:00
8140313775
test: drive catalog in ns rejection test
...
Use the actual catalog resolver, not the mock to assert the correct
behaviour with a populated catalog.
2023-01-26 17:55:39 +01:00
0aa5469ac6
test(e2e): explicit namespace creation
...
Adds an end-to-end test of the router's gRPC NamespaceService covering
creation and reading of new namespaces.
2023-01-26 17:32:12 +01:00
7eaa8f59b0
fix: explicit namespace creation w/ existing ns
...
Prior to this commit, namespaces that had been created on one router
could not be used on another router until the latter was restarted.
Effectively, newly created namespaces couldn't be used.
After this commit, the catalog is also checked when a cache miss occurs,
ensuring the router discovers new, not-yet-cached namespaces.
2023-01-26 17:32:12 +01:00
105e354299
refactor: clean up namespace errors
...
The namespace error was poorly refactored and duplicated the prefix
string. The "rejected" case is now also tested.
2023-01-26 17:32:11 +01:00
1a7679bcee
refactor: expose underlying gRPC implementations
...
Changes the gRPC delegate to return the underlying service (type erased)
implementations instead of the RPC service wrappers.
2023-01-26 17:32:11 +01:00
ac8fa293cb
refactor(test): TestContext::write_lp() helper
...
Adds a helper method to construct the HTTP write request.
2023-01-26 17:32:10 +01:00
6f1869f9dc
test(router): initialise gRPC delegate in e2e
...
Initialise the "rpc mode" gRPC handlers in the router e2e TestContext.
2023-01-26 17:32:10 +01:00
3efc42baac
refactor(test): dedicated e2e TestContext module
...
Moves the router's TestContext to its own file/module.
2023-01-26 17:32:10 +01:00
c66f4a3d92
fix(router): restore NamespaceService
...
This was removed in the RPC variant of the router - no idea why, we
definitely should have it!
2023-01-26 15:10:22 +01:00
b6018e1c39
feat(metrics): separate service limit counters
...
Service limits are enforced on two values:
* Number of tables in a namespace
* Number of columns in a table
This commit labels the existing service limit hit metric with the type
of limit reached, and adds this information to the log lines emitted.
2023-01-26 14:48:33 +01:00
f0d7ee59c3
Merge branch 'main' into dom/circuit-fuzz
2023-01-25 12:42:43 +00:00
6eb1773ec0
perf(router): faster balancer node recovery
...
Ensure a "probe" node is always returned as the first candidate, driving
it to recovery faster.
This also includes a fix for the balancer metrics that would report
probe candidate nodes as healthy nodes.
2023-01-25 13:18:24 +01:00
f5d4171be0
test: CircuitBreaker recovery property fuzz test
...
Adds a multi-threaded fuzz test that ensures a circuit breaker can
always transition to the healthy state, regardless of the sequence of
events prior.
2023-01-25 11:53:57 +01:00
caea42665b
Merge branch 'main' into dom/rpc-endpoint-metrics
2023-01-25 10:44:18 +11:00
442e8a8b79
Merge branch 'main' into dom/ingester-rediscovery
2023-01-24 19:13:02 +00:00
Fraser Savage
dependabot[bot]
Dom
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Andrew Lamb
Luke Bond