This updates the semantic graph usage to accomodate the change to the
semantic graph that removed the ambiguity of the body so now it is
always a block instead of being a block or an expression.
The storage filters are modified to use the predicates directly so we do
not have to pass `semantic.FunctionExpression` around. Instead, since
simple expressions are all that are supported anyway, we transform
suitable function expressions into predicates as part of the push down
rule and this simplifies the influxdb reader code.
This also moves the storage predicate conversion code into the standard
library package as it is the only location that uses this code now that
the predicate conversion is done as part of the push down rule.
This refactor was prompted by another refactor of the
`semantic.FunctionExpression` that would cause it to always contain a
`semantic.Block`. Since the push down filter needs the expressions and
to combine them, this refactor allows us not do construct a combined
filter inside of blocks which allows us to have better type safety.
The prometheus project doesn't adhere to the module suffix so any
attempt to use it as a library fails with go 1.13 or greater. The
workaround is to `go get` a specific commit revision that corresponds to
a tag and let go figure out a pseudo-version for it.
In this case, I updated the revision to the one pointed at by `v2.9.2`
since that is what the current `go.mod` file pointed at. I also updated
the go version to 1.13 inside of `go.mod` to be the same as influxdb.
See https://github.com/prometheus/prometheus/issues/6048 for details.
I think it is a nice utility function that I would like to use
elsewhere.
So I decided to move it out to the tracing package.
Signed-off-by: Gianluca Arbezzano <gianarb92@gmail.com>
The `buckets()` and `v1.databases()` functions have been updated to
support their remote counterparts that were added to flux. These
functions now do the same thing as the `from()` call where they will
default to the current organization when run against the server and will
use the remote versions from the repl.
Algorithm W will return a semantic graph where every function block
always uses a block and a return statement. This is in contrast to the
Go code which would have the semantic graph be an expression or a block.
The push down code would not introspect blocks which meant that any
function expression produced by algorithm w would never be pushed down.
This fixes it so the code will now extract the semantic expression from
inside of a block if there is exactly one statement and the statement is
a return statement.
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
This removes the spec and updates the lang package usage to make use of
passing in the runtime as a parameter.
It removes all direct dependendencies on the flux runtime from the http
package.
This removes the storage dependency on libflux by moving the interfaces
it implements to the `query` package so it can reference the definitions
rather than the package with the implementation and the registration
with the runtime. This breaks the dependency where a storage package
depends on a flux runtime package.
The tasks subsystem will now use the flux language service to parse and
evaluate flux instead of directly interacting with the parser or
runtime. This helps break the dependency on the libflux parser for the
base influxdb package.
This includes the task notification packages which were changed at the
same time.
This updates the repl to support the new influxdb source and use it by
default in the repl. It will automatically set some default variables
for the influxdb source to make it easier to use the cli. In particular,
it will set the default organization, token, and the host. The
organization gets set to the one specified in the repl command and the
token gets filled in with the user installed one. The host defaults to
localhost but will change to whichever one was specified on the cli.
In addition, this will replace the http client with one that sets
insecure skip verify if the `--skip-verify` flag is used.
This adds support for using pkg-config to build libflux inside of the
flux dependency. The build can occur by either installing `pkg-config`
into your path or the `env` script can be used to invoke it from the go
modules.
The repl no longer takes in a querier and it will run everything
locally. The spec interface will now not be used and will be removed
from the http endpoint at some point.
This change allows for the InfluxQL language type to be used with the
/v2/query API endpoint.
This change also introduces a way to give the transpiler an explicit
bucket name instead of using the DBRPMapping service.
Requests to the endpoint will know the bucket name directly but will
likely not have run the migration step to populate the DBRP mappings.
The storage engine isn't capable of sending back empty tables when a
series is empty. Because of this, we disable the push down and let flux
do the filtering in the case where there is a filter and it is specified
to keep the empty tables.
Update CONTRIBUTING.md
Added Security Vulnerability Reporting
Updated the text to include the simple changes. This branch still needs updating to reflect 2.0 API etc.
~
fix(contribution): updated the text for V2.
fixes#13370
Update the 'Getting the source' section'
Remove the 'Cloning a fork' seciton
* If they have forked the repo, it should be clear how to clone the fork.
refactor
last refactor
Use # for section headings
Minor grammar edit.
Update CONTRIBUTING.md
Fix tripple backticks
Backticks weren't being picked up by Github's md renderer properly.
Fixed formatting
Made tabs and spaces consistent (went for tabs, since that's what go uses). Made cli commands consistent by including $ at the start of the line. Fixed copy a little bit.
Softened the language
Fixes: https://github.com/influxdata/influxdb/pull/13370#discussion_r359393716
Softened the language a bit.
Update CONTRIBUTING.md
Co-Authored-By: Stuart Carnie <stuart.carnie@gmail.com>
chore: improve CONTRIBUTING.md
this is a step towards providing a shared http client that manages pooling connections,
timeouts, and reducing GC for by not creating/GCing a client each req. Bring on the red!
* chore: Remove several instances of WithLogger
* chore: unexport Logger fields
* chore: unexport some more Logger fields
* chore: go fmt
chore: fix test
chore: s/logger/log
chore: fix test
chore: revert http.Handler.Handler constructor initialization
* refactor: integrate review feedback, fix all test nop loggers
* refactor: capitalize all log messages
* refactor: rename two logger to log
Signed-off-by: Lorenzo Affetti <lorenzo.affetti@gmail.com>
Signed-off-by: Julius Volz <julius.volz@gmail.com>
move to internal
update flux to v0.50
Revert "move to internal"
This reverts commit bcd4caffbd44135f1dbeac4163cb2a22a751f45a.
promtests/internal --> internal/promtests
When `exists` was used in conjunction with any other pushed down
expression, the `exists` was not rewritten properly because the rewrite
did not descend into logical expressions.
This is now fixed so those expressions will be rewritten correctly. This
affected the following form:
filter(fn: (r) => r._measurement == "cpu" and exists r.host)
It did not affect the following:
filter(fn: (r) => r._measurement == "cpu")
|> filter(fn: (r) => exists r.host)
The controller now supports setting an initial memory limit and setting
a maximum amount of memory that the controller may use separately from
the memory quota per query and the concurrency quota.
This allows the controller to increase the concurrency quota to a larger
number while setting the maximum amount of memory to a lower amount than
would be required for all queries to use 100% of their allowable memory.
Functionally, this means that a query will have a soft limit for an
initial memory byte quota that a query is guaranteed to have, a shared
pool that it is allowed access to in the case it uses more, and a hard
limit that no query may exceed to prevent runaway queries from taking
over the entire pool.
This change is completely backwards compatible with older configurations
as the new options will default to values that mimic the old behavior
where a query is allocated the full amount of its memory quota and the
maximum amount of memory is based on the concurrency quota and this
maximum memory quota.
In addition to the above, this also fixes a bug in the controller that
allowed it to run more than its concurrency as executing queries. This
happened when the results had finished being sent by the executor, but
the query had not yet been read and/or serialized. The executor would be
freed up and would take the next query even though the previous query
hadn't yet been finalized with `Done()`.
The QueryServiceProxyBridge would not check for errors properly because
it would return any error encountered when running the query as a read
error on the `io.Reader`. This made it so that the csv decoder could not
identify if the error was related to the query or if it was related to
reading. The csv decoder needed to tell the difference because an error
with reading from the `io.Reader` needs to be returned as a decoder
error while an error from the query needs to be returned as-is.
Instead of adapting the csv decoder to do that, we instead lazily
initialize the result iterator when `More()` is called and call `Peek()`
on the reader. If no bytes can be read, we assume this was an error
while executing the query and return it as such. If we are able to read
at least one byte, we decode it through the csv decoder.
The http error schema has been changed to simplify the outward facing
API. The `op` and `error` attributes have been dropped because they
confused people. The `error` attribute will likely be readded in some
form in the future, but only as additional context and will not be
required or even suggested for the UI to use.
Errors are now output differently both when they are serialized to JSON
and when they are output as strings. The `op` is no longer used if it is
present. It will only appear as an optional attribute if at all. The
`message` attribute for an error is always output and it will be the
prefix for any nested error. When this is serialized to JSON, the
message is automatically flattened so a nested error such as:
influxdb.Error{
Msg: errors.New("something bad happened"),
Err: io.EOF,
}
This would be written to the message as:
something bad happened: EOF
This matches a developers expectations much more easily as most
programmers assume that wrapping an error will act as a prefix for the
inner error.
This is flattened when written out to HTTP in order to make this logic
immaterial to a frontend developer.
The code is still present and plays an important role in categorizing
the error type. On the other hand, the code will not be output as part
of the message as it commonly plays a redundant and confusing role when
humans read it. The human readable message usually gives more context
and a message like with the code acting as a prefix is generally not
desired. But, the code plays a very important role in helping to
identify categories of errors and so it is very important as part of the
return response.
We are planning to change the allocator interface within flux to use the
arrow allocator. To make the release easier, this updates the test in
advance to use the arrow allocator instead of the to be changed memory
allocator interface from flux.
Writes directly to a PointsWriter require the tag key, value pairs
are sorted in lexicographically ascending order. This commit uses
new API from the `models` package to ensure this invariant is
maintained.
The `v1.databases()` call did not correctly filter buckets based on
auth. Fortunately, it did not cause any improper permissions such as
allowing a person to see buckets that they had no read access to.
The error instead was that if a user did not have read access to one of
the buckets that was returned, the entire command would fail rather than
filter out the bucket that didn't have permissions.
This changes it so that if the user doesn't gets an unauthorized error
when accessing a bucket, it will filter it from the list instead of
failing. It also changes it so the error message is marked as
`ENotFound` instead of as an internal error.
The secret service is tested by creating a secret and then attempting to
use it in a flux query. There is one test where accessing the secret
should work and one where it should return that the action is forbidden.
This change makes it so that if an org or orgID are missing on calls to the `to` function
that the orgID is retrieved from the request context.
This is consistent with how `from` works.
The secret service is wrapped to be compatible with the flux interface
to the secret service.
The organization id is not part of the flux interface so this code
extracts the organization id from the query context so that it can lookup
a secret within the organization.
The `exists` operator now gets pushed down to storage correctly. If
`exists` is used on a tag, then it will be rewritten to `tag != ""`
which is how storage defines if a tag exists. If `not exists` is used,
then it will use `tag == ""` which is how you would query storage for
only if a tag doesn't exist.
The `tag == ""` and `tag != ""` are different. For `tag == ""`, the
predicate is impossible for the storage layer to return true with.
Ideally, we would just rewrite this to return nothing and we wouldn't
bother with even querying storage. Instead, we just do not rewrite this
predicate because it cannot be rewritten to make sense with storage. If
we see `tag != ""`, it is the only one that can be passed through as-is
because `tag != ""` returns the same values as `exists tag`. It will
return true for every non-null value.
If we handle the flux errors in the query controller, it makes it so we
are handling the errors in the location where the happen rather than at
a layer further up the stack.
This should simplify it so the errors are handled in this single
location instead.
In the QueueSize test, it was possible that after the `done` channel was
closed, one of the queries from the queue would begin executing. If all
three began executing before the shutdown was done, the third would
block on attempting to send a value to the `executing` channel and it
would never finish so the controller would report that shutdown failed.
This increases the queue size to a combination of the concurrency quota
and the queue size so all of the started queries will never block when
sending a signal to the executing channel.
The storage table reader will now work correctly when there are multiple
outputs. The table interface now implements the new table and column
reader interfaces and works properly with `execute.CopyTable`. The
source uses `execute.CopyTable` to buffer the table in memory when there
are multiple output transformations.
The controller implementation is primarily used by influxdb so it
shouldn't be part of the flux repository. This copies the code from flux
to influxdb so it can be removed from the next flux release.
The `databases()` function was moved into the `influxdata/influxdb/v1`
package and so it wouldn't work anymore. The `percentile()` call was
changed to `quantile()` and the argument was changed to `q`. This also
updates `median()` to use `median()` since we now produce AST's and not
the spec so we can use the `median()` definition instead.
The RPC call should translate `_measurement` and `_field` to their
proper shortened byte strings when requesting the tag values.
This also fixes the planner rewrites to return the root node even when
no rewrite happened as this is required by the planner.
If a pattern is seen that matches the `v1.tagValues(...)` call, then it
will be replaced with a direct RPC call to read the tag values for the
selected tag key which should be better optimized than reading from the
storage engine tsm1 files.
If a pattern is seen that matches reading the tag keys, it will be
replaced with a direct RPC call to read the tag keys which should be
better optimized than reading from the storage engine tsm1 files.
The ProxyQueryServiceAsyncBridge was not returning statistics when there
was an encoder error. Because the encoder was just writing to an
io.Writer, it was possible that a remote disconnect could happen and
statistics could not be reported.
BucketsAccessed doesn't work currently with a private flux.Spec.
See this issue: https://github.com/influxdata/influxdb/issues/13278
This set of changes just allows code to compile until #13278 is fixed.
Note that preauthorization is not working in the meantime.
Fixes#13275.
It is no longer necessary for the query logging to be a bridge as the
stats are available for consumption from the ProxyQueryService.
This change changes the logging bridge to directly implement the proxy
query service instead of implementing a bridge.
* deps: update flux-staging to master of both influxdb and flux (#12698)
* test(stdlib): skip failing window test due to new trigger optimization (#12708)
* chore: update Flux to v0.22.0
The asyncStatsResultIterator used inside QueryServiceProxyBridge assumed
that Release would only be called once. The godoc for ResultIterator
specifies that it is safe to call Release multiple times. Now, you can
do that without causing Release to block indefinitely.
The statistics are only finalized after release is called. Defer a call
to release to ensure they are released, but explicitly release on
success to ensure that the statistics are finalized from all sources
before returning them.
This refactors everything to generate and use a flux AST when
transpiling an influxql query. This also updates the spectests so they
use flux instead of writing out the AST and compare the resulting
AST's.
When trying to create a task that writes to an unknown bucket,
previously the error message would resemble:
could not find bucket {<nil> 0xc0000dfcc0 <nil> 0xc0000dfce0}: <not found> bucket not found
Now print something human-readable:
could not find write bucket with filter: [Bucket Name: b, Org Name: a]: <not found> bucket not found
Paul Hummer
jlapacik
Alirie Gray
Jonathan A. Sternberg
Yiqun (Ethan) Zhang
Gianluca Arbezzano
Lorenzo Affetti
Gianluca Arbezzano
Christopher M. Wolff
docmerlin (j. Emrys Landivar)
Nathaniel Cook
Faith Chikwekwe
Gavin Cabbage
Jacob Marble
timhallinflux
George
Johnny Steenbergen
Julius Volz
Lyon Hill
Stuart Carnie
Adam
roshie548
Adam
Ben Johnson
Kelvin Wang
Todd Persen
Mark Rushakoff
Jacob Marble
kelwang
zhulongcheng
Michael Desa