This was "internal". The mapping works like this: we take the
`DataFusionError` and call `find_root` which should traverse the
`External(...)` chain (even through Arrow) to find the last error that
is not within the Arrow/DataFusion land. This is then mapped by us.
`DataFusionError::External(...)` is no further inspected and mapped
straight to "internal". I think this if fine because in the end we're
mostly dealing w/ DataFusion stuff anyways.
I've slightly changed the error mapping in the planner to emit
`DataFusionError::Plan(...)` instead which we map to "invalid argument".
I think this is way better for the user.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds a single-line & batched write request benchmarks to ingester2.
When using the WAL, a floor of 10ms is added to all writes (the WAL
linger time, configurable). With the WAL disabled in code, benchmarked
writes complete in less than a millisecond:
single row/write
time: [1.9138 µs 1.9168 µs 1.9197 µs]
thrpt: [520.92 Kelem/s 521.70 Kelem/s 522.51 Kelem/s]
batched/write/1000
time: [129.15 µs 129.97 µs 131.28 µs]
thrpt: [7.6173 Melem/s 7.6941 Melem/s 7.7429 Melem/s]
Note these benchmarks exclude network I/O, and measure single-threaded,
synchronous write client performance.
Moves the ingester2 TestContext & builder to its own crate for reuse
between integration tests & benchmarks (and more?!)
This allows us to DRY & reuse the test code with it's nice API for
benchmarks too, without forcing it all to be in the "prod" build of
ingester2 (and the dependencies, and their dependencies, etc).
This also means this code can be built and cached instead of being
rebuilt all the time during normal development.
* chore: increasing concurrency a little more
This raises the threshold for single threading compactions to 100 column partitions. With the non-linear scaling, 70 column partitions would take 49% of the concurrency limit (allowing only 2 of such sized partitions to compact concurrently). Anything over 70 can only compact with something smaller than itself.
I'm gradually walking these up, partly to avoid causing OOMs in prod, and partly because I want to get a feel for how reactive the average concurrency is to these changes.
* chore: fix comment typo
The NamespaceResolver was using its own very similar look-aside caching
to the DML handlers, this commit leverages the read-through cache
implementation to deduplicate more code and makes the read through
behavioural expectation explicit for namespace autocreation.
This removes the look-aside cache from the retention_validation
and schema_validation DML handlers, instead setting up the new
NamespaceCache decorator and using that to handle cache misses.
This commit refactors the NamespaceCache trait to return a result
instead of an option for calls to `get_schema()`, allowing callers and
decorators to differentiate between cache misses, namespaces not
existing and transient I/O errors. This allows implementations to
interact with backend catalog storage.
In order to implement a read-through NamespaceCache
decorator the `get_cache()` call will need to interact
with async catalog methods, so this allows implementations
to call await within the `get_cache()` body.
Within our query tests and our CLI, we've used to print out empty
query responses as:
```text
++
++
```
This is pretty misleading. Why are there no columns?! The reason is that
while Flight provides us with schema information, we often have zero
record batches (because why would the querier send an empty batch). Now
lets fix this by creating an empty batch on the client side based on the
schema data we've received. This way, people know that there are columns
but no rows:
```text
+-------+--------+------+------+
| count | system | time | town |
+-------+--------+------+------+
+-------+--------+------+------+
```
An alternative fix would be to pass the schema in addition to
`Vec<RecordBatch>` to the formatting code, but that seemed to be more
effort.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: add CommandGetPrimaryKeys metadata endpoint and tests
* chore: update schema for the returned record batch
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: improve querier->ingester tracing
- add more hierarchy items on the querier side
- ensure that streaming is correctly traced by the querier
* refactor: improve span name
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* docs: `QueryDataTracer`
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Part of the wider effort to consistently use tht term "database"
for the user-facing terminology, update the authorization system.
Whilst this system is technically user-facing, it is unlikely many
users will see it. It is however new enough that the change is
relatively little effort.
* chore: end-to-end tests for authorization
Add tests to validate the behaviour of the authorization machinery
in the write and query paths.
In order to facilitate this an authorizer implentation has been
added to the the test helpers that runs an authorizer gRPC service
for the use of tests. The gRPC service is started in the process
that is running the test and listens on a OS-assigned port number.
The authorization service cannot be shared between tests so a
non-shared cluster must be used when the authorizer is configured.
The influxdb_iox_client has been enhanced so that the user can
configure additional headers in the flight client, which is used
for SQL and InfluxQL queries. This uses the same interface as the
Flight SQL client has for the same job.
* chore: fix lint errors
* chore: review suggestion
Consolate the authorization tests into fewer tests in order to avoid
repeating set-up and tear-down unnecessarily.
Not having frame pointers on x64 is a painful for profiling and debugging.
Sure we can use DWARF, but it is slow, memory-hungry (you easily need
gigabytes just for a simple benchmark), and often broken.
In constrat to 32bit x86, x64 has enough general purpose registers and
the processor likely has even more physical ones that it can access through
register renaming. One register more won't make the difference for
us.
Prior art:
- Go: https://github.com/golang/go/issues/15840
- RustC: https://github.com/rust-lang/rust/pull/107689 (not merged yet)
- Fedora tried: https://lwn.net/Articles/919940/
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Marco Neumann
Andrew Lamb
Joe-Blount
kodiakhq[bot]
Dom
Carol (Nichols || Goulding)
Fraser Savage
Stuart Carnie
Chunchun Ye
Martin Hilton
dependabot[bot]
wiedld