Additional changes:
* fix(query/stdlib): update rewrite rules for schema mutation
The schema mutator was wrapped in a dual implementation spec so the
rewrite rules were type asserting on the wrong type.
This rule reorders group and window so it will switch from using
`ReadGroup` to using `ReadWindowAggregate` when the intent is to
aggregate a grouped window. It will then add a group node that groups by
the given columns and the start and stop columns and then reperform the
aggregate. This is more performant than performing the group first.
Annotate the context with feature flags when handling flux queries in influxdb.
Taking advantage of this in flux end-to-end tests. Using a custom flagger that
can set overrides based on the test case that is about to be run, allowing us
to enable features in the end-to-end tests.
This enables a new rule that will push down the full `aggregateWindow`
query including the `duplicate` and `window(every: inf)` that recombines
the tables. When the full rule is used, the table is not split into
tables for each window and instead retains itself as a single table. The
start or stop column is renamed to `_time` and `_start` and `_stop` will
be the boundaries of the query.
* feat: flags for pushing down new aggregates
* refactor: grouped aggregate rewrite rules
The storage operation ReadGroup aggregates per series on the storage
side. The planner will rewrite grouped aggregate queries to call
ReadGroup, which will perform a partial aggregation, followed by
another operation that will perform the rest of the aggregation on
the compute side.
* feat: storage capabilities for grouped aggregates
* fix: changes from review
* feat: group read operation name should include aggregate
This implements create empty for the window table reader and allows this
table read function to be used when it is specified. It will pass down
the create empty flag from the original window call into the storage
read function.
This also fixes the window table reader so it properly creates
individual tables for each window. Previously, it was constructing one
table for an entire series instead of one table per window.
Tests have been added to verify three edge case behaviors. The first is
the normal read operation where all values are present. The second is
when create empty is specified so null values may be created. The third
is with truncated boundaries to ensure that storage is read from and the
start and stop timestamps get correctly truncated.
Added a (disabled) planner rule that matches:
ReadGroupPhys -> { count }
It uses the same physical spec node for group to implement the aggregate. The
rule requires:
* the pushDownGroupAggregateCount feature flag enabled
* no existing aggregate present in the ReadGroup
* use of the "_value" column only
The e2e test driver in influxdb runs the tests twice to get past the fact that there
is no way to force order between the write to storage and the read back. When
the json.Marshal call became mandatory it was added to the first run, but not
the second.
Added a (disabled and feature-flagged) planner rule that matches:
ReadRange -> window -> { min, max, mean, count, sum }
The rule requires:
* the pushDownWindowAggregate{Count,Rest} feature flags enabled
* having WindowAggregateCapability
(which StorageReader does not currently have)
* use of "_value" columns only
* window.period == window.every
* window.every.months == 0
* window.every is positive
* window.offset == 0
* standard time columns
* createEmpty is false
This modifies the read window aggregate interfaces to future-proof it
if and when we add additional capabilities to the method. Previously,
the interface was all or nothing. If we modified the RPC call itself, we
would have to make a new interface to denote the change to the Go code.
This changes the interface so now a `WindowAggregateCapability` exists.
This way, we can modify the struct to include things like:
```
type WindowAggregateCapability struct {
WindowPeriodCapability bool
MeanAggregateCapability bool
}
```
This way we can learn if the RPC call itself supports some specific
option. If the first iteration doesn't support a mean aggregate or the
mean aggregate is only supported by single server implementations, the
window aggregate can tell the caller that it won't be able to compute
the mean aggregate.
Since it fills in a struct with these capabilities, the struct can
safely introduce new values. If a downstream consumer wants to take
advantage of that functionality, then all interfaces in the chain have
to be updated to consume the upstream capabilities.
The `ReadWindowAggregateSource` will invoke the `ReadWindowAggregate`
method on the `influxdb.Reader` and return the table. It is implemented
using the same common methods that are used for the other sources.
Added an interface for an additional storage capability. This interface
will allow for checking if the reader supports the window aggregate call
and another method for invoking the call if it does.
This is implemented using a single interface. If the reader implements
the interface, it indicates that the client is capable of reading the
response. The `HasXXX` method is intended to check if the store supports
the operation. This method also takes a context because it could require
a remote call or to wait for one.
This updates the semantic graph usage to accomodate the change to the
semantic graph that removed the ambiguity of the body so now it is
always a block instead of being a block or an expression.
The storage filters are modified to use the predicates directly so we do
not have to pass `semantic.FunctionExpression` around. Instead, since
simple expressions are all that are supported anyway, we transform
suitable function expressions into predicates as part of the push down
rule and this simplifies the influxdb reader code.
This also moves the storage predicate conversion code into the standard
library package as it is the only location that uses this code now that
the predicate conversion is done as part of the push down rule.
This refactor was prompted by another refactor of the
`semantic.FunctionExpression` that would cause it to always contain a
`semantic.Block`. Since the push down filter needs the expressions and
to combine them, this refactor allows us not do construct a combined
filter inside of blocks which allows us to have better type safety.
The `buckets()` and `v1.databases()` functions have been updated to
support their remote counterparts that were added to flux. These
functions now do the same thing as the `from()` call where they will
default to the current organization when run against the server and will
use the remote versions from the repl.
Algorithm W will return a semantic graph where every function block
always uses a block and a return statement. This is in contrast to the
Go code which would have the semantic graph be an expression or a block.
The push down code would not introspect blocks which meant that any
function expression produced by algorithm w would never be pushed down.
This fixes it so the code will now extract the semantic expression from
inside of a block if there is exactly one statement and the statement is
a return statement.
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
This removes the storage dependency on libflux by moving the interfaces
it implements to the `query` package so it can reference the definitions
rather than the package with the implementation and the registration
with the runtime. This breaks the dependency where a storage package
depends on a flux runtime package.
This updates the repl to support the new influxdb source and use it by
default in the repl. It will automatically set some default variables
for the influxdb source to make it easier to use the cli. In particular,
it will set the default organization, token, and the host. The
organization gets set to the one specified in the repl command and the
token gets filled in with the user installed one. The host defaults to
localhost but will change to whichever one was specified on the cli.
In addition, this will replace the http client with one that sets
insecure skip verify if the `--skip-verify` flag is used.
The storage engine isn't capable of sending back empty tables when a
series is empty. Because of this, we disable the push down and let flux
do the filtering in the case where there is a filter and it is specified
to keep the empty tables.
this is a step towards providing a shared http client that manages pooling connections,
timeouts, and reducing GC for by not creating/GCing a client each req. Bring on the red!
When `exists` was used in conjunction with any other pushed down
expression, the `exists` was not rewritten properly because the rewrite
did not descend into logical expressions.
This is now fixed so those expressions will be rewritten correctly. This
affected the following form:
filter(fn: (r) => r._measurement == "cpu" and exists r.host)
It did not affect the following:
filter(fn: (r) => r._measurement == "cpu")
|> filter(fn: (r) => exists r.host)
Writes directly to a PointsWriter require the tag key, value pairs
are sorted in lexicographically ascending order. This commit uses
new API from the `models` package to ensure this invariant is
maintained.
The `v1.databases()` call did not correctly filter buckets based on
auth. Fortunately, it did not cause any improper permissions such as
allowing a person to see buckets that they had no read access to.
The error instead was that if a user did not have read access to one of
the buckets that was returned, the entire command would fail rather than
filter out the bucket that didn't have permissions.
This changes it so that if the user doesn't gets an unauthorized error
when accessing a bucket, it will filter it from the list instead of
failing. It also changes it so the error message is marked as
`ENotFound` instead of as an internal error.
This change makes it so that if an org or orgID are missing on calls to the `to` function
that the orgID is retrieved from the request context.
This is consistent with how `from` works.
The `exists` operator now gets pushed down to storage correctly. If
`exists` is used on a tag, then it will be rewritten to `tag != ""`
which is how storage defines if a tag exists. If `not exists` is used,
then it will use `tag == ""` which is how you would query storage for
only if a tag doesn't exist.
The `tag == ""` and `tag != ""` are different. For `tag == ""`, the
predicate is impossible for the storage layer to return true with.
Ideally, we would just rewrite this to return nothing and we wouldn't
bother with even querying storage. Instead, we just do not rewrite this
predicate because it cannot be rewritten to make sense with storage. If
we see `tag != ""`, it is the only one that can be passed through as-is
because `tag != ""` returns the same values as `exists tag`. It will
return true for every non-null value.
The storage table reader will now work correctly when there are multiple
outputs. The table interface now implements the new table and column
reader interfaces and works properly with `execute.CopyTable`. The
source uses `execute.CopyTable` to buffer the table in memory when there
are multiple output transformations.
The RPC call should translate `_measurement` and `_field` to their
proper shortened byte strings when requesting the tag values.
This also fixes the planner rewrites to return the root node even when
no rewrite happened as this is required by the planner.
If a pattern is seen that matches the `v1.tagValues(...)` call, then it
will be replaced with a direct RPC call to read the tag values for the
selected tag key which should be better optimized than reading from the
storage engine tsm1 files.
If a pattern is seen that matches reading the tag keys, it will be
replaced with a direct RPC call to read the tag keys which should be
better optimized than reading from the storage engine tsm1 files.
jlapacik
Jonathan A. Sternberg
Yiqun (Ethan) Zhang
Adrian Thurston
Christopher M. Wolff
Johnny Steenbergen
Faith Chikwekwe
Paul Hummer
Alirie Gray
Lorenzo Affetti
Stuart Carnie
Nathaniel Cook
Adam
docmerlin (j. Emrys Landivar)
roshie548
Adam
Ben Johnson
Kelvin Wang
Jacob Marble