The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.
This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.
This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.
It is now possible to generate all files within this project by using:
go generate ./...
Or the Makefile can continue to be used.
This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
The pb package was only referenced in cmd/influx/query.go, but in
dead code, since it uses the same machinery as the repl, which goes
through the HTTP endpoints, rather than the gRPC endpoints.
We reorganized the functions in flux to have the structure:
/functions
/inputs
/transformations
/outputs
this PR catches up platform to work with the new package layout.
As a separate refactoring issue, we should discuss:
from(bucket: ) should migrate from flux --> platform
to_http and to_kafka should migrate from platform --> flux
It creates a simple client that follows the QueryService interface and
uses the `influxql.Compiler` type to determine where the query should be
routed to and to return the query as a `flux.ResultIterator`.
This will be useful for replaying transpiler queries against influxdb
1.x servers to verify correctness.
This iterface is supposed to be something that both sessions and
authorizations can share so that other components can authorize requests
as they see fit.
The `-i` flag causes a problem when a cross compilation happens because
it attempts to build files in GOROOT which is located in a directory
that is not writable by the build user. This same problem exists for
when the `-tags` are changed like adding `-tags assets` in the `all`
build that runs on the master branch.
This removes `-i` because, in the latest go release, the `-i` flag
doesn't actually do anything useful since there is another build cache
that the results are placed in regardless of whether `-i` is used.
This fixes the build on master so that `make all` and `make nightly`
will function properly.
This updates the dependency manager to use go modules instead of dep so
that we can remove dep as a dependency and we can begin using and giving
feedback on modules within this project. It should simplify dependency
management and make it faster to run the builds too.
The big change is updating the various Makefiles to stop relying on the
vendor directory because it no longer exists. This change creates a
`tools.go` file with the revisions of the tools we use that are currently
in the `required` section of `Gopkg.toml`. We are currently in other
discussions to modify how we handle tool dependency management, but this
change does not change that.
The transpiler will normalize the `_time` column by dropping any
existing time column and then duplicating `_start` when the query is an
aggregate type.
This works for the selectors because they did not normalize their
`_time` column at all and, while the aggregates did normalize their
`_time` column, we have made the decision to remove that functionality
and have aggregates not set a `_time` column at all.
Instead of generating multiple cursors, a pivot is used to join fields
within the same series.
This should be easier than generating a new cursor for everything.
WIP: saving state reword
feat(query/influxql): implement query.ResultIterator directly on response
review(influxql): add ResponseIterator that implement query.ResultIterator
review(influxql): update code is response to review
The previous default was just to have no limit at all. This adds a
configuration option to the planner so a static value can be set for the
memory limit on each individual query.
Using query request struct to query resources
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Use query.ProxyRequest instead query.Request
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Proxy request from idpd
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Comments about the desired results
Signed-off-by: Lorenzo Fontana <lo@linux.com>
V1 endpoints working with flux
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Influxql working for v1
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
V2 influxql query endpoint working
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
V2 Flux compiler support
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Improve comments in bolt sources and give error on self
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Review tests failing
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Avoid type casts for compiler types
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
Using nil instead of dbrp mapping service for influxql v1
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Check if compiler types are valid for influxql
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Organization as query param in the flux external handler
Signed-off-by: Lorenzo Fontana <lo@linux.com>
feat(http): update swagger documentation for flux query endpoint
feat(http): document query endpoint design
The code documented does not currently work. It is indended that this
will be implemented in follow up PRs.
feat(platform): move source to platform package
The source Query endpoint implements what's in the query swagger docs
Signed-off-by: Lorenzo Fontana <lo@linux.com>
Co-authored-by: Michael De Sa <mjdesa@gmail.com>
feat(platform): allow for encoding and decoding of csv dialects
feat(platform): specify dialect in flux page
Co-authored-by: Andrew Watkins <andrew.watkinz@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
The query statistics would only be read after the cancel call so we need
to wait for that before attempting to read the statistics rather than
attempting to read them immediately after the result is returned (before
it is read).
* Added default signature generators for Selector and Aggregate configs and update the functions that need them
* fix to percentile to collect the correct arguments for both aggregate and selector
The REPL's use of the interpreter was causing it to not get the builtins
defined in builtin scripts. For example the `top` function was missing.
This change fixes the issues by ensuring the builtins are only evaluated
once and that there is only one way to get the query Interpreter that is
guaranteed to have the proper builtins.
The logger is now threaded into the query controller, executor, and the
dispatcher so that we can log panics. They are logged at the info level
because the panics do not result in the system crashing and becoming
unusable.
Durations are changed to be a 3 vector to form a linear basis of
seconds, days, and months.
Interval comprehensions are introduced to be able to define complex
calendar intervals.
Specification is added around time zones.
The window function is update to default to not returning incomplete
windows.
The call to `setErr` would grab a lock that `Pop` used, but `setErr`
requires the controller run loop to be executing for it to work. If we
reverse the order of these calls, it should be fine.
When the controller moves to one of the finished states, it will finish
the parent span so that can be recorded. It presently will do this
multiple times when transitioning between different finished states.
This normally happens within the finishing states, but when compiling or
queueing fails it never enters those finished states and is instead
discarded. We need to signal that the query itself has finished in the
metrics.
This documents the responsibilities of what the Controller does and is
expected to do. It describes some behaviors that aren't implemented, but
acts as a guide for what the Controller should do as we continue
developing the query engine and improving the internal mechanics.
This commit provides a new join implementation that can join
tables with different group keys. It also implements a strategy
for evicting stale data from the join operation's cache when
it can be determined that a table will no longer be joined.
Currently early table eviction from the buffers of the join occurs
when the left-most group key column of both input streams are equal
and that column is part of the join key.
Moves idpe.QueryService into platform/query.ProxyQueryService
Splits the Request into ProxyRequest and Request.
Changes query.QueryService and query.AsyncQueryService to use a Request
type. This means that the Compiler interface is consumed by the service
to abstract out transpilation vs Flux compilation vs raw spec.
The transpiler handler is removed.
There are separate http handlers and service implementations for each of
the three query services.
Query logging types are moved into platform.
The ResultIterator now expects Cancel to always be called.
The fluxd binary exposes the query endpoint specified in the swagger
file.
For details, the user can always look at the spec, but a good getting
started guide that works is needed.
At the moment, this does not use the influx command. It does include
instructions for how to get started with docker since that is easiest to
setup and use.
* Add Flux Rename, Drop, and Keep functions.
* Add flux language tests for drop and rename; bugfixes around constructing drop/rename procedures
* Correct autcomplete error, remove debug statements
* Fix additional autocomplete error
* Add support for fn parameter to flux drop and rename functions
* Remove unneeded renameDropTransformation field
* WIP initial thoughts about merge/procedure rewrite logic
* Remove unnecessary external columnFn types
* Add additional query tests for drop/rename, add error when drop or rename refers to column that doesn't exist
* Change swapped test names
* Add keep function implementation
* Refactor rename/drop implementation to make use of helper methods
* Remove test skips forqueries that produce an expected error
* Make predicate return value logic clearer in drop implementation
* Add group key support to drop/rename/keep, update unit and query tests accordingly
* WIP refactor drop and rename to use new SchemaMutator interface
* Add unit tests for refactored drop, keep, and rename; bug fixes
* Update file structure for flux schema mutation functions
* Refactor mutation functions and interface
* Address final review drop/rename PR review comments
* Add builtin import to make sure builtins are initialized if test drop_rename_keep_test is run as standalone
* Small aesthetic change to schema_mutators.go
* Add initial rename specification to SPEC.md
* Tweak language in spec to be more explicit
* Update spec to be in accordance with final design decisions
This is so Chronograf doesn't have to import the builtin package, which
finalizes builtin registration.
Also clarify that the builtin package should only be imported from main
or test packages.
The now time is stamped by the influxql transpiler and used inside of
the actual query. It will result in more accuracy if we take the
timestamp we have created and send it as part of the spec to queryd
rather than force ourselves to ensure absolute times exist everywhere.
If a query is attempting to be enqueued and it gets canceled, it will
now stop attempting to add it to the new queries queue and return the
error reported by the context. This allows the http server to cancel a
running query when the client disconnects for whatever reason without
continuing to attempt to process the canceled query.
Introduces the Statisticser interface which ResultIterators may
implement.
The HTTP implementation uses HTTP trailers to preserve the statistics.
This way we do not need to have all encoders and decoders support
statistics.
These tests were part of a PR when the rename was made.
The changes were not rebased before merge so we did not discover the
failures till after the merge.
previous versions only supported the first parameter to time() that set the window size. This version supports the second parameter, which shifts the offset a fixed amount from the epoch
* feat(query/influxql): add regex support to transpiler
also added test case to querytest
* Add raw_with_condition test to transpiler unit tests
* add unit tests for regex conditions on raw query
The package contains all of the transpiler specs and allows them to be
put into different files instead of keeping all of the tests in the same
file. They are all Go code so they are type checked rather than being
loaded as JSON from disk.
Additionally, to make it easier for a developer, the tests will report
the exact file and line where the test was created. So rather than
hunting for which file a test is located in, you will get something nice
like the following:
--- FAIL: TestTranspiler/SELECT_count(value)_FROM_db0..cpu_WHERE_host_=_'server01' (0.00s)
testing.go:51: aggregates_with_condition.go:16: unexpected error: unimplemented function: "count"
As can be seen, the test that failed can be found in the
`aggregates_with_condition.go` file at line 16 which is where the test
was created by the `AggregateTest` function and the relevant spec can be
found in that same file.
The transpiler compilation tests will now not allow skip to be
specified. Instead, it must return an error message that starts with
`unimplemented` and then the reason will be used as the skip message.
This way, it will be easier to identify the failing tests in the
transpiler. In the previous method, it was possible for a test to be
marked as skip, but for the transpiler to return the wrong error message
because the test did not differentiate between an unimplemented error
message and an incorrect error message.
The transpiler now supports basic windowing. The window offsets are not
supported yet at all.
For windowing, we use the window function to split the points, perform
the aggregate/selector operation, and then we put them back into the
same window so they are within the same table as they originally were
located in. This is now reflected in the spec and the code.
The transpiler now supports basic windowing. The window offsets are not
supported yet at all.
For windowing, we use the window function to split the points, perform
the aggregate/selector operation, and then we put them back into the
same window so they are within the same table as they originally were
located in. This is now reflected in the spec and the code.
The compiler tests from the github.com/influxdata/influxdb/query have
been moved over to the influxql transpiler in platform. The framework
has updated to include a skip option so that all of the tests can be
there, but not all of them have to succeed at the moment. If a test
starts succeeding but is marked as one that should be skipped, it will
also cause an error to prevent us from doing work on the transpiler
without marking the test as something not to skip anymore (so progress
is always made).
This extra flexibility makes it easier for the transpiler to generate a
specification since the map step can be focused on only generating the
columns related to fields. In particular, it makes it easier to
implement wildcards for tags because the tags will get passed along with
the partition key.
The spec says to use the `_time` column for the time in the output, but
we were mapping `r._time` to `time` and using the `time` variable. This
modifies the encoder to use the `_time` column and rename it to `time`
for the column name.
This commit adds the option statement as a recognized node in
the semantic graph of a Flux query.
Flux interpreter must visit an option statement node
interpreter parses option statement
Semantically, variable declarations should be statements
Also update option statement test to validate proper scope
only test option statement
It was effectively a copied and pasted platform.ID, so change it to a
type alias. Once our known references to the query/id package are
updated to platform.ID, we'll delete the package.
The transpiler should use a bucket for the `from()` call instead of the
database parameter which will likely be deprecated. The bucket that it
will read data from is `db/rp` and, if the retention policy isn't
specified, `autogen` will be used as the default.
There are a few changes to how the transpiler works. The first is that
the streams are now abstracted behind a `cursor` interface. The
interface keeps track of which AST nodes (like variables or function
calls) are represented by the data inside of the stream and the method
of how to access the underlying data. This makes it easier to make a
generic interface for things like the join and map operations. This also
makes it easier to, in the future, use the same code from the map
operation for a filter so we can implement conditions.
This also follows the transpiler readme's methods and takes advantage of
the updates to the ifql language. This means it will group the relevant
cursors into a cursor group, perform any necessary joins, and allow us
to continue building on this as we flesh out more parts of the
transpiler and the language.
The cursor interface makes it so we no longer have to keep a symbol
table mapping the generated names to the locations because that is all
kept within the incoming cursor rather than as a separate data
structure.
It also splits the transpiler into more files so it is easier to find
the relevant code for each stage of the transpiler.
Update the transpiler docs to account for a change in the query
language. A slightly different method is now used with some
clarifications previously existing problems.
The readme is also indexed with a table of contents for easier reading.
* feat(query/querytest): first batch of tests ready for review
added helper executables for generating output, comparing specs. Also updated several tests and added code to skip red tests with a clear reason in the message
This fixes the encoder so that it will encode the response correctly to
a JSON blob using the outputs of the transpiler. The transpiler has also
been modified to pass through the correct values so the map function is
correctly constructed and the aggregate function is also correctly
constructed.
This removes the group function temporarily because it does not seem to
be working.
Jonathan A. Sternberg
Adam
jlapacik
Nathaniel Cook
Adam
Mark Rushakoff
Lyon Hill
Stuart Carnie
Chris Goller
Leonardo Di Donato
Jeff Wendling
Brandon Farmer
Michael Desa
Christopher M. Wolff
Adam Perlin
Jorge Landivar
Andrew Watkins