* feat(influxql): support TOP and BOTTOM functions
Add support for the TOP and BOTTOM functions which return the first
n rows in some ordered data set.
* fix: clippy
* refactor(influxql): use window aggregates for selectors
Change the implentation of ProjectionType::Selector to use a window
aggregate, rather than an aggregate with a custom selector function.
This is in preparation for implementing PERCENTILE.
* feat(influxql): PERCENTILE selector
Add a selector for the row containing the nth percentile of a
partition. This is the behaviour used when a single selector function
is used in an influxql query.
* feat(influxql): PERCENTILE aggregator
Add the PERCENTILE aggregation function for when the PERCENTILE
function is used in an aggregating projection. This implementation
buffers all non-null field values in memory in order to perform the
operation and therefore could be an expensive operation. This is
necessary for compatibility with earlier influxdb versions.
* refactor(influxql): move PERCENTILE implementation out of plan
The plan module is getting rather full of user-defined function
implementations. This breaks the new functions used to implement
percentile into some new top-level modules for aggregate and window
UDFs.
* fix: doc-lint
* chore: refactor `find_enumerated`
* chore: use `s` in format string
* chore: include the unexpected selector function in the error
* chore(influxql): review suggestions
Added some addition comments to help understanding.
Changed the handling os slector functions such that FIRST, LAST,
MAX & MIN behave the same as they did before PERCENTILE was added.
* chore(influxql): make percent_row_number a window UDF
Now that user-defined window functions are available make the
percent_row_number function be one of those. this allows the values
to be calculated for the entire window partition in one go.
For some reason the user-defined window function cannot return NULL
values. This function uses 0 where it would otherwise use NULL, as
row numbering starts at 1.
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: add `influxdb_iox debug build-catalog` command
* fix: tests
* fix: Use info! logs instead of println for status
* fix: Set partition_hash_id as well
* fix: remove leftover code
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor(iox_query_influxql): expand select projection
Change the SELECT projection in the planner to make it clearer how
each projection type works.
* feat(influxql): support TOP and BOTTOM functions
Add support for the TOP and BOTTOM functions which return the first
n rows in some ordered data set.
* fix: clippy
* chore: Use array / slice destructuring
* chore: review suggestion in iox_query_influxql/src/plan/planner.rs
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Do not (ab)use per-chunk delete predicates for the retention policy.
Instead use a per-table predicate.
This makes the code way cleaner, since the scoping is correct (i.e.
delete predicates are a table-wide attribute, not a chunk-based one) and
it is consistent time predicates that the user providers (e.g. via
`WHERE time > x`).
It also allows us to remove delete predicates (in their current,
non-scalable form) from the query path. A potential future version would
likely not use per chunk predicates (and "is processed" markers) but use
the timestamp / chunk order to determine to which data the predicate
should be applied.
Note that the lowering of the retention policy changed slightly from
```text
(time > (now() - retention)) AND (time < MAX)
```
to
```text
time > (now() - retention)
```
Since the `MAX` cut is just an artifact of the lowering and was unnecessary.
Closes#7409.
Closes#7410.
Add the DERIVATIVE and NON_NEGATIVE_DERIVATIVE functions to influxql.
These are used to calculate derivatives over arbitrary time units.
The implementation is modeled after the DIFFERENCE and
NON_NEGATIVE_DIFFERENCE functions, with a difference that the unit
parameters is a configuration of the user-defined aggregator function
and therefore there cannot be a single shared definition of the
function.
The NON_NEGATIVE_DIFFERENCE function implementation has been
refactored to be an arbitrary NON_NEGATIVE wrapper for any Accumulator
function.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* test: ensure that selectors check arg count
* feat: basic non-aggregates w/ InfluxQL selector functions
See #7533.
* refactor: clean up code
* feat: get more advanced cases to work
* docs: remove stale comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This commit adds an `inspect` command to read through the sequenced
operations in a WAL file and debug pretty print their contents to
stdout, optionally filtering by a sequence number range.
* feat: Allow passing service protection limits in create db gRPC call
* fix: Move the impl into the catalog namespace trait
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* chore: Update DataFusion pin
* chore: Update cargo
* fix: update for API changes
* fix: Update plans
* chore: Update for new api
* fix: Update plans
* chore: Update for API changes more
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This changes the e2e test to delete the WAL segment file, restart the
ingester and ensure the results returned by an ingester query after
feeding the regenerated line proto in are the same as those before.
This commit adds support for the CLI to query the namespace and schema
APIs to retrieve database and table names from the IDs found in WAL
entries being regenerated.
* test: integration test for tracing of queries to the ingester
* chore: add FlightFrameEncodeRecorder to record spans per each polling result
* refactor(trace): impl TraceCollector for Arc
Allow any Arc-wrapped TraceCollector implementation to be used as a
TraceCollector. This avoids needing to as_any() and downcast later.
* test: assert FlightFrameEncodeRecorder trace spans
This test exercises the FlightDataEncoder wrapped with the trace
decorator (FlightFrameEncodeRecorder) when executing against a data
source that yields data after varying numbers of Stream polls.
This test passing will validate the FlightFrameEncodeRecorder correctly
instruments the amount of time a client spends waiting on the
FlightDataEncoder to acquire or encode a protocol frame, but also
ensures the decorator correctly accounts for varying behaviours allowed
through the Stream abstraction. It does this by simulating a data source
that is not always immediately ready to provide data, such as a buffer
wrapped in a contended async mutex.
* refactor: move tracing decorator into separate mod
* fix: record spans
* refactor(test): update test
The frame encoder is not one-to-one - it emits two frames for the first
data payload, a schema and a payload. This commit updates the test to
account for it!
* refactor: remove unneeded mut ref, and use enum state method which panics when in a (should be unreachable) state
* chore: add more docs to FlightFrameEncodeRecorder and related
---------
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: consolidate pruning code
Let's have a single chunk pruning implementation in our code, not two.
Also removes a bit of crust from `QueryChunk` since it is technically no
longer responsible for pruning (this part has been pushed into the
querier for early pruning and bits for the `iox_query_influxrpc` for
some RPC shenanigans).
* test: regression test for incident
* fix: chunk pruning
* docs: add some test notes
Introduce a new header called `iox-debug` which when set enables certain
debug features. The first one will be the `system.queries` table which
is a process-local, namespace-scoped query log. In most prod setups this
is only useful for debugging and will confuse the user a lot because
when multiple queries are deployed then the K8s routing decides which
pod/process the users hits. This leads to an inconsistent view. However
the log is still useful for debugging.
This also wires the "debug header set" flag through the Flight ticket,
because JDBC proved (integration tests FTW!) that headers are only
passed to `GetFlightInfo` but not to `DoGet` and the ticket must encode
all the relevant information.
Closes#7119.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Martin Hilton
Fraser Savage
Andrew Lamb
Stuart Carnie
Marco Neumann
Carol (Nichols || Goulding)
Dom Dwyer
wiedld
Marko Mikulicic
Nga Tran
kodiakhq[bot]
Christopher M. Wolff