509fab2de0
test: add test demonstrating the behavior when a token is invalid
2023-05-12 15:31:14 -05:00
518d10d4c0
refactor: set explicit boundaries btwn the AuthzServer communication failure, versus an invalid token response from the server
2023-05-12 15:31:14 -05:00
62d83b9219
test: add tests for IoxAuthorizer contract
2023-05-12 15:31:12 -05:00
ea3029ee3f
refactor: break out Authorizer trait versus IoxAuthorizer struct.
...
* make the boundaries more evident
* also make explicit what actions are tied to the IoxAuthorizer client (a.k.a. the client connection & request)
2023-05-12 15:28:58 -05:00
4c30e7e04d
refactor: Authorizer trait should have a single interface for requested permissions()
...
* returns an intersection of requested_perms and actual perms_on_token
* returns ok if any of the requested_perms is within the actual perms_on_token
2023-05-12 15:28:58 -05:00
14007808bd
fix: Move remaining conversions between data types and proto into data_types
...
And have data_types depend on generated_types rather than vice versa.
2023-05-12 13:31:04 -04:00
1770d0f4d8
fix: Move ingester-querier gRPC communication to its own crate
2023-05-12 13:28:30 -04:00
4c7f96ead8
fix: Remove unused delete predicate proto conversion code
2023-05-12 11:27:46 -04:00
dfe1a7dec8
perf(router): parallel write replication
...
This commit changes the write replication loop to concurrently write to
N distinct upstream ingesters, instead of the previous sequential logic.
2023-05-12 17:04:32 +02:00
bf93014bb7
feat: concurrent lending iterator
...
Changes the UpstreamSnapshot to be suitable for concurrent use. This
type contains the core logic to enable a caller to uphold the
responsibility of ensuring replicated writes land on distinct ingesters
in the presence of concurrent replication.
The clients within the snapshot are returned to at most one concurrent
caller at a time, by tracking the state of each client as a FSM:
┌────────────────┐
┌─▶│ Available │
│ └────────────────┘
│ │
drop next()
│ │
│ ▼
│ ┌────────────────┐
└──│ Yielded │
└────────────────┘
│
remove
│
▼
┌────────────────┐
│ Used │
└────────────────┘
Once a client has been yielded it will not be yielded again until it is
dropped (transitioning the FSM from "yielded" to "available" again,
returning it to the candidate pool of clients) or removed (transitioning
to "used", permanently preventing it from being yielded to another
caller).
2023-05-12 17:04:32 +02:00
cdaf99268c
refactor: owned client in UpstreamSnapshot
...
Changes then UpstreamSnapshot to return owned clients, instead of
references to those clients.
This will allow the snapshot to have a 'static lifetime, suitable for
use across tasks.
2023-05-12 16:59:49 +02:00
dc27ae5fbf
refactor: eliminate impossible error
...
Because the number of candidate upstreams is checked to exceed the
number of desired data copies before starting the write loop, and
because the parallelism of the write loop matches the number of desired
data copies, it's not possible for any thread to observe an empty
snapshot.
This commit removes the unreachable error condition for clarity.
2023-05-12 16:59:49 +02:00
465158e08e
test(router): replication prop/invariant fuzzing
...
Adds a property-based test of the RPC write handler's replication logic,
ensuring:
1. If the number of healthy upstreams is 0, NoHealthyUpstreams is
returned and no requests are attempted.
2. Given N healthy upstreams (> 0) and a replication factor of R:
if N < R, "not enough replicas" is returned and no requests are
attempted.
3. Upstreams that return an error are retried until the entire
write succeeds or times out.
4. Writes are replicated to R distinct upstreams successfully, or
an error is returned.
5. One an upstream write is ack'd as successful, it is never
requested again.
6. An upstream reporting as unhealthy at the start of the write is
never requested (excluding probe requests).
These properties describe a mixture of invariants (don't replicate your
two copies of a write to the same ingester) and expected behaviour of
the replication logic (optimisations like "don't try writes when you
already know they'll fail").
This passes for the single-threaded replication logic used at the time
of this commit, and will be used to validate correctness of a concurrent
replication implementation - a concurrent approach should uphold these
properties the same way a single-threaded implementation does.
2023-05-12 16:59:48 +02:00
cf622c1b91
test: MockWriteClient tracks ACK response count
...
Changes the MockWriteClient to track how many success responses it has
returned in response to a write request.
2023-05-12 16:59:48 +02:00
ac656ab1f9
refactor: clearer NoHealthyUpstreams error name
...
Renames NoUpstreams -> NoHealthyUpstreams as it's confusing because we
also have "not enough replicas" which could be no upstreams? This has a
slightly clearer meaning.
2023-05-12 16:59:47 +02:00
8e74f0a568
test: proptest upstream snapshot cycles
...
Adds a proptest that ensures the set of upstream ingesters is cycled
over indefinitely, with each element yielded an equal number of times.
2023-05-12 16:59:44 +02:00
92e5036943
fix: Size of ColumnSet shouldn't be using ChunkId ( #7786 )
2023-05-12 14:58:03 +00:00
ac0ea99cde
chore(deps): Bump tracing-core from 0.1.30 to 0.1.31 ( #7781 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.30...tracing-core-0.1.31 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-12 13:40:04 +00:00
de62ee3130
Merge pull request #7782 from influxdata/dependabot/cargo/handlebars-4.3.7
...
chore(deps): Bump handlebars from 4.3.6 to 4.3.7
2023-05-12 14:34:26 +01:00
6c185bdec8
chore(deps): Bump handlebars from 4.3.6 to 4.3.7
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.3.6 to 4.3.7.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/v4.3.7/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.3.6...v4.3.7 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-12 13:11:56 +00:00
bd1d1f648c
chore(deps): Bump serde from 1.0.162 to 1.0.163 ( #7780 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.162 to 1.0.163.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.162...v1.0.163 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-12 13:07:56 +00:00
b7db7999e3
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-11 15:02:24 -05:00
36c1d23e19
chore(deps): Bump tokio from 1.28.0 to 1.28.1 ( #7776 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.0 to 1.28.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.0...tokio-1.28.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-11 19:46:58 +00:00
e5ee47c32e
chore: code cleanup
2023-05-11 13:21:15 -05:00
e73564ec0e
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-11 13:16:15 -05:00
81eb663122
refactor: accept impl Into<String> for schema methods
2023-05-11 01:44:14 +03:00
aac9911319
Merge pull request #7769 from influxdata/dom/rpc-client-error-split
...
refactor(router): separate RPC client error types
2023-05-10 10:45:59 +00:00
8cc94a710f
Merge branch 'main' into dom/rpc-client-error-split
2023-05-10 10:40:12 +00:00
ea086a913a
Merge pull request #7770 from influxdata/dependabot/cargo/console-subscriber-0.1.9
...
chore(deps): Bump console-subscriber from 0.1.8 to 0.1.9
2023-05-10 10:18:13 +01:00
8baf82071f
Merge branch 'main' into dependabot/cargo/console-subscriber-0.1.9
2023-05-10 10:12:24 +01:00
99f9069f81
Merge branch 'main' into dom/rpc-client-error-split
2023-05-10 10:09:08 +01:00
ab68998f9e
Merge pull request #7771 from influxdata/cn/rename-querier2
...
fix: Rename last remaining "2" suffixed things I could find
2023-05-10 09:56:29 +01:00
3cafa9ace3
fix: Rename query_tests2 to query_tests
2023-05-09 22:10:46 -04:00
35e8c52702
fix: Rename remaining test helper methods with a 2 suffix to not
2023-05-09 22:07:46 -04:00
e60f703e95
fix: Rename router2 to router
...
Including an alias and a test for continuing to support `influxdb_iox
run router2`.
2023-05-09 22:01:39 -04:00
6daf5660b4
fix: Rename querier2 to querier
...
Just some test helpers; there never was an `influxdb_iox run querier2`
command so no alias or test needed.
2023-05-09 22:01:06 -04:00
d99444fb2e
chore(deps): Bump console-subscriber from 0.1.8 to 0.1.9
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.8...console-subscriber-v0.1.9 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-10 01:59:08 +00:00
a9ee1bae6c
chore: Merge remote-tracking branch 'origin/main' into smith/remove-transactions-main
2023-05-09 13:26:02 -05:00
3f0e7745c2
Merge branch 'main' into dom/rpc-client-error-split
2023-05-09 14:41:12 +01:00
4af76cbe02
Merge pull request #7692 from influxdata/cn/custom-partitions-start
...
feat: If namespace or table partition templates are specified, use those
2023-05-09 14:03:44 +01:00
01205e9671
refactor: assert Column.table_id matches
...
Include an invariant assert when adding a Column to a TableSchema,
ensuring the table IDs match.
2023-05-09 14:55:04 +02:00
18c6d9e306
refactor: remove unnecessary "to_owned()" call
...
This method now takes an owned name, so no need to call to_owned()!
2023-05-09 14:55:03 +02:00
ab666ea5fa
refactor: owned ColumnsByName constructor only
...
Refactors the From<BtreeMap> impl that accepted a &str name for
ColumnsByName construction, instead allowing only the owned String, and
updating the test that makes use of it appropriately.
2023-05-09 14:55:03 +02:00
23c0110b32
feat: Create newtypes for different partition templates
...
So that the different kinds aren't mixed up. Also extracts the logic
having to do with which template takes precedence onto the
PartitionTemplate type itself.
2023-05-09 14:55:02 +02:00
ebceabb608
feat: Add an assertion for the new invariant that namespace partition templates never change
2023-05-09 14:55:02 +02:00
70dca8f60b
fix: Pass the NamespaceSchema through the dml write traits
2023-05-09 14:55:02 +02:00
c062d2d890
fix: Change NamespaceResolver to return the whole cached NamespaceSchema
...
Rather than picking out the ID and partition template to be passed
around separately
2023-05-09 14:55:01 +02:00
28d4f4743c
refactor: Move PartitionTemplate and friends to their own module
2023-05-09 14:55:01 +02:00
e8a480f5f6
fix: Give up ownership of Column when adding to a table
...
To enable reuse of existing allocations rather than borrowing, creating
new allocations, then dropping them.
2023-05-09 14:55:00 +02:00
e8655af52d
fix: Change ColumnsByName::new to enable taking ownership if caller wants to give it
2023-05-09 14:55:00 +02:00
wiedld
Carol (Nichols || Goulding)
Dom Dwyer
Carol (Nichols || Goulding)
dependabot[bot]
Jeffrey Smith II
kayagokalp
kodiakhq[bot]