* deps: update flux-staging to master of both influxdb and flux (#12698)
* test(stdlib): skip failing window test due to new trigger optimization (#12708)
* chore: update Flux to v0.22.0
The asyncStatsResultIterator used inside QueryServiceProxyBridge assumed
that Release would only be called once. The godoc for ResultIterator
specifies that it is safe to call Release multiple times. Now, you can
do that without causing Release to block indefinitely.
The statistics are only finalized after release is called. Defer a call
to release to ensure they are released, but explicitly release on
success to ensure that the statistics are finalized from all sources
before returning them.
This refactors everything to generate and use a flux AST when
transpiling an influxql query. This also updates the spectests so they
use flux instead of writing out the AST and compare the resulting
AST's.
When trying to create a task that writes to an unknown bucket,
previously the error message would resemble:
could not find bucket {<nil> 0xc0000dfcc0 <nil> 0xc0000dfce0}: <not found> bucket not found
Now print something human-readable:
could not find write bucket with filter: [Bucket Name: b, Org Name: a]: <not found> bucket not found
filter out resources that have mission IDs
fix(influxdb): simplify auth check in PermissionAllowed
review(platform): update as noted in review
fix(influxdb): ensure permission has valid org id
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
all tests use a unique bucket based on the test file name. copied all tests over from flux repo
the tests are currently disabled due to engine consistency issues: https://github.com/influxdata/flux/issues/613
* refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused.
* chore(query/influxql): moved query_test.go and requisite files to the influxql dir to be closer to the code it tests. (#2013)
A standard Makefile is used now in all subdirs that run go generate.
Make will only generate the file if its source files changed.
The checkgenerate target runs clean to ensure all targets a generated
fresh.
The table interface was modified to expose the arrow buffers. The
storage table has now been converted to use this interface with the same
fixes so that it exposes arrow buffers.
The influxql package has also been updated to use the `DoArrow` method
from the `flux.Table` interface.
The flux query controller was updated to include a Shutdown method a
while ago. Explicitly handle query controller creation and shutdown
where applicable.
In influxd, this ensures that outstanding queries are handled before the
process dies. In tests, this ensures that query controller goroutines
aren't leaked, which drastically simplifies reading full stack traces.
This change also registers query controller metrics with the prometheus
registry in influxd.
feat(platform): add ToPermissions method to user resource mapping
The ToPermissions method returns a set of permissions that is granted
via a user resource mapping.
feat(bolt): resolve sessions permissions on lookup
feat(http): use authorizer instead of authorization service for write api
feat(bolt): create user resource mappings for org users in bucket create
feat(bolt): create user resource mapping for first org/user
fix(platform): use authorizer for query endpoint instead of authorization
test(http): use cmp instead of reflect for decode test
* Add default column to databases to support `show retention policies`
* add unit test for show retention policies
* update transpiler readme for show databases, show retention policies
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.
This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.
This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.
It is now possible to generate all files within this project by using:
go generate ./...
Or the Makefile can continue to be used.
This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
The pb package was only referenced in cmd/influx/query.go, but in
dead code, since it uses the same machinery as the repl, which goes
through the HTTP endpoints, rather than the gRPC endpoints.
We reorganized the functions in flux to have the structure:
/functions
/inputs
/transformations
/outputs
this PR catches up platform to work with the new package layout.
As a separate refactoring issue, we should discuss:
from(bucket: ) should migrate from flux --> platform
to_http and to_kafka should migrate from platform --> flux
It creates a simple client that follows the QueryService interface and
uses the `influxql.Compiler` type to determine where the query should be
routed to and to return the query as a `flux.ResultIterator`.
This will be useful for replaying transpiler queries against influxdb
1.x servers to verify correctness.
This iterface is supposed to be something that both sessions and
authorizations can share so that other components can authorize requests
as they see fit.
The `-i` flag causes a problem when a cross compilation happens because
it attempts to build files in GOROOT which is located in a directory
that is not writable by the build user. This same problem exists for
when the `-tags` are changed like adding `-tags assets` in the `all`
build that runs on the master branch.
This removes `-i` because, in the latest go release, the `-i` flag
doesn't actually do anything useful since there is another build cache
that the results are placed in regardless of whether `-i` is used.
This fixes the build on master so that `make all` and `make nightly`
will function properly.
This updates the dependency manager to use go modules instead of dep so
that we can remove dep as a dependency and we can begin using and giving
feedback on modules within this project. It should simplify dependency
management and make it faster to run the builds too.
The big change is updating the various Makefiles to stop relying on the
vendor directory because it no longer exists. This change creates a
`tools.go` file with the revisions of the tools we use that are currently
in the `required` section of `Gopkg.toml`. We are currently in other
discussions to modify how we handle tool dependency management, but this
change does not change that.
The transpiler will normalize the `_time` column by dropping any
existing time column and then duplicating `_start` when the query is an
aggregate type.
This works for the selectors because they did not normalize their
`_time` column at all and, while the aggregates did normalize their
`_time` column, we have made the decision to remove that functionality
and have aggregates not set a `_time` column at all.
Instead of generating multiple cursors, a pivot is used to join fields
within the same series.
This should be easier than generating a new cursor for everything.
WIP: saving state reword
feat(query/influxql): implement query.ResultIterator directly on response
review(influxql): add ResponseIterator that implement query.ResultIterator
review(influxql): update code is response to review
The previous default was just to have no limit at all. This adds a
configuration option to the planner so a static value can be set for the
memory limit on each individual query.
Using query request struct to query resources
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Use query.ProxyRequest instead query.Request
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Proxy request from idpd
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Comments about the desired results
Signed-off-by: Lorenzo Fontana <lo@linux.com>
V1 endpoints working with flux
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Influxql working for v1
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
V2 influxql query endpoint working
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
V2 Flux compiler support
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Improve comments in bolt sources and give error on self
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Review tests failing
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Avoid type casts for compiler types
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Using nil instead of dbrp mapping service for influxql v1
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Check if compiler types are valid for influxql
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Organization as query param in the flux external handler
Signed-off-by: Lorenzo Fontana <lo@linux.com>
feat(http): update swagger documentation for flux query endpoint
feat(http): document query endpoint design
The code documented does not currently work. It is indended that this
will be implemented in follow up PRs.
feat(platform): move source to platform package
The source Query endpoint implements what's in the query swagger docs
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
feat(platform): allow for encoding and decoding of csv dialects
feat(platform): specify dialect in flux page
Co-authored-by: Andrew Watkins <andrew.watkinz@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
The query statistics would only be read after the cancel call so we need
to wait for that before attempting to read the statistics rather than
attempting to read them immediately after the result is returned (before
it is read).
* Added default signature generators for Selector and Aggregate configs and update the functions that need them
* fix to percentile to collect the correct arguments for both aggregate and selector
The REPL's use of the interpreter was causing it to not get the builtins
defined in builtin scripts. For example the `top` function was missing.
This change fixes the issues by ensuring the builtins are only evaluated
once and that there is only one way to get the query Interpreter that is
guaranteed to have the proper builtins.
The logger is now threaded into the query controller, executor, and the
dispatcher so that we can log panics. They are logged at the info level
because the panics do not result in the system crashing and becoming
unusable.
Durations are changed to be a 3 vector to form a linear basis of
seconds, days, and months.
Interval comprehensions are introduced to be able to define complex
calendar intervals.
Specification is added around time zones.
The window function is update to default to not returning incomplete
windows.
kelwang
jlapacik
zhulongcheng
Jonathan A. Sternberg
Lorenzo Affetti
Christopher M. Wolff
Nathaniel Cook
Mark Rushakoff
Kelvin Wang
Jacob Marble
Jacob Marble
Ben Johnson
Michael Desa
Adam
Lyon Hill
Jorge Landivar
Mark Rushakoff
Adam
Andrew Watkins
Edd Robinson
Chris Goller
Stuart Carnie
Leonardo Di Donato
Jeff Wendling
Brandon Farmer
Adam Perlin