* feat: flags for pushing down new aggregates
* refactor: grouped aggregate rewrite rules
The storage operation ReadGroup aggregates per series on the storage
side. The planner will rewrite grouped aggregate queries to call
ReadGroup, which will perform a partial aggregation, followed by
another operation that will perform the rest of the aggregation on
the compute side.
* feat: storage capabilities for grouped aggregates
* fix: changes from review
* feat: group read operation name should include aggregate
This implements create empty for the window table reader and allows this
table read function to be used when it is specified. It will pass down
the create empty flag from the original window call into the storage
read function.
This also fixes the window table reader so it properly creates
individual tables for each window. Previously, it was constructing one
table for an entire series instead of one table per window.
Tests have been added to verify three edge case behaviors. The first is
the normal read operation where all values are present. The second is
when create empty is specified so null values may be created. The third
is with truncated boundaries to ensure that storage is read from and the
start and stop timestamps get correctly truncated.
Added a (disabled) planner rule that matches:
ReadGroupPhys -> { count }
It uses the same physical spec node for group to implement the aggregate. The
rule requires:
* the pushDownGroupAggregateCount feature flag enabled
* no existing aggregate present in the ReadGroup
* use of the "_value" column only
The column reader passed to `flux.Table.Do` is automatically released.
The function passed to the column reader should never release it
manually. This causes a double release which causes the table to be
erroneously freed when it might be referenced by another transformation.
In particular, this affected the following:
tables
|> yield()
|> to()
This is because this would produce a buffered table with two references
and pass it to both `yield()` and `to()` because `yield()` is a
pseudo-node that doesn't really exist. The real graph looks more like:
tables |> yield()
tables |> to()
The `yield()` would double release which would release the `to()`
transformation's copy of the column readers. The `to()` method would
then be invoked with an invalid column reader.
The e2e test driver in influxdb runs the tests twice to get past the fact that there
is no way to force order between the write to storage and the read back. When
the json.Marshal call became mandatory it was added to the first run, but not
the second.
Added a (disabled and feature-flagged) planner rule that matches:
ReadRange -> window -> { min, max, mean, count, sum }
The rule requires:
* the pushDownWindowAggregate{Count,Rest} feature flags enabled
* having WindowAggregateCapability
(which StorageReader does not currently have)
* use of "_value" columns only
* window.period == window.every
* window.every.months == 0
* window.every is positive
* window.offset == 0
* standard time columns
* createEmpty is false
This modifies the read window aggregate interfaces to future-proof it
if and when we add additional capabilities to the method. Previously,
the interface was all or nothing. If we modified the RPC call itself, we
would have to make a new interface to denote the change to the Go code.
This changes the interface so now a `WindowAggregateCapability` exists.
This way, we can modify the struct to include things like:
```
type WindowAggregateCapability struct {
WindowPeriodCapability bool
MeanAggregateCapability bool
}
```
This way we can learn if the RPC call itself supports some specific
option. If the first iteration doesn't support a mean aggregate or the
mean aggregate is only supported by single server implementations, the
window aggregate can tell the caller that it won't be able to compute
the mean aggregate.
Since it fills in a struct with these capabilities, the struct can
safely introduce new values. If a downstream consumer wants to take
advantage of that functionality, then all interfaces in the chain have
to be updated to consume the upstream capabilities.
The `ReadWindowAggregateSource` will invoke the `ReadWindowAggregate`
method on the `influxdb.Reader` and return the table. It is implemented
using the same common methods that are used for the other sources.
Added an interface for an additional storage capability. This interface
will allow for checking if the reader supports the window aggregate call
and another method for invoking the call if it does.
This is implemented using a single interface. If the reader implements
the interface, it indicates that the client is capable of reading the
response. The `HasXXX` method is intended to check if the store supports
the operation. This method also takes a context because it could require
a remote call or to wait for one.
This updates the semantic graph usage to accomodate the change to the
semantic graph that removed the ambiguity of the body so now it is
always a block instead of being a block or an expression.
The storage filters are modified to use the predicates directly so we do
not have to pass `semantic.FunctionExpression` around. Instead, since
simple expressions are all that are supported anyway, we transform
suitable function expressions into predicates as part of the push down
rule and this simplifies the influxdb reader code.
This also moves the storage predicate conversion code into the standard
library package as it is the only location that uses this code now that
the predicate conversion is done as part of the push down rule.
This refactor was prompted by another refactor of the
`semantic.FunctionExpression` that would cause it to always contain a
`semantic.Block`. Since the push down filter needs the expressions and
to combine them, this refactor allows us not do construct a combined
filter inside of blocks which allows us to have better type safety.
Faith Chikwekwe
Jonathan A. Sternberg
Yiqun (Ethan) Zhang
jlapacik
Christopher M. Wolff
Adrian Thurston
Johnny Steenbergen
Paul Hummer
Alirie Gray