* refactor(authz): move extract_header_token into authz
Move the extract_header_token method into the authz package so that
it can be shared by the query path. The method is renamed to reflect
the fact that it can now also extract a token from gRPC metadata.
The extract_token function is now a little more generic to allow
it to be used with HTTP header values and gRPC metadata values.
* feat(service_grpc_flight): JDBC compatible Handshake
While testing some JDBC based clients we found that some, Tableau
in this case, cannot be configured with authoriztion tokens. In
these cases we need to be able to support username/password. The
approach taken is to ignore the username and make the token the
password. This is the same approach being taken throughout the
product.
To facilitate this the Flight RPC Handshake command has been extended
to look for Basic authorization credentials and respond with the
appropriate Bearer authorization header.
While adding end-to-end tests the subprocess commands were causing
a deadlock. These have been changed to using the tonic::process
module.
There are also some small changes to the JDBC test application where
the hardcoded values were clashing with the authorization parameters.
* fix: lint
* chore: apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* chore: review suggestion
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This commit adds a randomised property test, that compares the results
of the new namespace cache schema merging (#7555) with a known-good
stdlib HashSet union (the cache implementation is effectively a more
specialised set union operation).
This property test also validates the "last writer wins" semantics for
other, non-schema data within the namespace.
Additionally the ChangeSet values returned over a pair of updates are
asserted to reflect the actual values added to the cache (but not each
call individually) to ensure accurate metrics are reported.
This re-introduces the potential racy conflicting schema updates, to
optimise for the expected read-heavy workload. This limits the point at
which write requests may race with schema updates to overlapping calls
to put, rather than the write call-path as a whole.
* feat(idpe-17265): authorization should occur as part of the single_tenant specific mod
* authz service is accessed only through the single_tenant mod handler
* authz service is wrapped in auth mod
* move auth integration test into auth mod
* push down the authorize() call into the query params parser call, in order to access query params in the extract_token
* provide configuration error when authz or single_tenant mode are not co-presented
* update authz e2e fixtures
* feat(idpe-17265): extract tokens based upon preferred ordering in spec, and write tests to verify behavior.
* chore(idpe-17265): update naming conventions for a unifying parser
* test: make MockAuthorizer have default, and add a test_delegate_to_authz for CST
* chore: record authz duration metric, and include in delegation test.
* chore: use authz terminology instead of auth_service
* chore: more explicit naming
* Revert "chore: record authz duration metric, and include in delegation test."
This reverts commit 05c36888ca7247b6953343d759a5185098fae679.
* refactor: extract_header_token versus the else condition
* refactor: make single_tenant mod and move auth within
* chore: make unreachable explicitly panic in the build
* test: make token values be const, to be consumed when MockAuthorizer is used
* test: use locking for calls_counter in test
* fix: add base64 encoding as expected for Basic header
* fix: merge conflict resolution. The AuthorizationHeaderExtension is now under the authz::http mod, which is a required feature for router package.
* chore: run rustfmt nightly with preferred import handling, on files with modified imports
* chore: code cleanup, to have minimal code needed
Provide a configuration item for the router (in RPC mode) that controls
the maximum outgoing RPC message size when communicating with an
Ingester.
Raises the maximum from the default 4MiB to 100MiB. This does not
increase exposure to memory-based DOS, as writes are size-limited by the
HTTP layer to 10MiB, preventing a user from submitting a write this
large (or larger!) across the RPC boundary.
This commit adds additive merge behaviour for tables missing
from the new NamespaceCache entry, as well as moving calculation
of change stat metrics down to the in-memory implementation.
The metrics no longer distinguish between insert and update
caches OPs as a result of the change to the `put_schema()` interface.
The "server_util" crate exists only to support HTTP authz operations, so
this commit moves it under the authz crate. This helper is gated by a
feature flag allowing callers to opt into this extra HTTP dependency
(disabled by default).
And return the "deletes unsupported" error sooner.
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
By moving the namespace schema into the Put cache method and returning
the new value wrapped in an Arc, it allows for the cache to merge the
new schema and the existing schema without calling clone() on either.
This has a side effect of allowing the metrics and stats capture
behaviour to be achieved without leaking into the traits definition.
The previous behaviour of the router's NamespaceCache was to provide
put semantics where the entire schema in the cache is replaced. With
the addition of the additive merging side-effect, the metrics decorator
could not compute the correct statistics. This calculates them during
the merge and surfaces the result to the caller.
Rather than unconditionally overwriting the whole namespace schema
in the namespace cache if an entry already exists the in-memory
cache will now merge any column schema missing from the new entry.
In order to calculate correct metrics for column count, the cache needs
to return extra data for an insert.
The NamespaceResolver was using its own very similar look-aside caching
to the DML handlers, this commit leverages the read-through cache
implementation to deduplicate more code and makes the read through
behavioural expectation explicit for namespace autocreation.
This removes the look-aside cache from the retention_validation
and schema_validation DML handlers, instead setting up the new
NamespaceCache decorator and using that to handle cache misses.
This commit refactors the NamespaceCache trait to return a result
instead of an option for calls to `get_schema()`, allowing callers and
decorators to differentiate between cache misses, namespaces not
existing and transient I/O errors. This allows implementations to
interact with backend catalog storage.
In order to implement a read-through NamespaceCache
decorator the `get_cache()` call will need to interact
with async catalog methods, so this allows implementations
to call await within the `get_cache()` body.
Part of the wider effort to consistently use tht term "database"
for the user-facing terminology, update the authorization system.
Whilst this system is technically user-facing, it is unlikely many
users will see it. It is however new enough that the change is
relatively little effort.
Dom Dwyer
Martin Hilton
Fraser Savage
dependabot[bot]
wiedld
wiedld
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Carol (Nichols || Goulding)