* refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused.
* chore(query/influxql): moved query_test.go and requisite files to the influxql dir to be closer to the code it tests. (#2013)
This commit moves the `Main.cancel()` execution to the `main_test.go`
file so it's only executed for tests. This was interfering with the
shutdown process on the regular `influxd` binary.
Previously, the WithTicker option would call TickScheduler.Tick every
time the underlying time.Ticker sent a time on its channel. This meant
we used a 1s period, which meant that in the worst case, we would see a
tick at about 999ms after the second rollover.
This change increases the underlying time.Ticker frequency, but only
calls TickScheduler.Tick after a second rolls over. Since we now use a
tick frequency of 100ms, during normal operation, TickScheduler.Tick
will be called within 0.1s after the second rolls over.
* feat(platform): add label struct
* feat(platform): add label service interface
* WIP label service
* chore(http): simplify URMapping validation
* WIP label service skeleton
* add new GET labels handler
* feat(http): add POST handler for label service
* feat(http): add label delete handler
* fix json tag
* add happy path testing
* feat(http): add label endpoints to dashboard service
* make fmt
* add label delete handler
* make fmt again :(
* feat(bolt): add label implementation
* wire boltdb implementation of labels
* feat(mock): add mock for label service
* fix tests, add delete test
* add remaining tests, flesh out GET params
* unused code
* remove comments
* add labels endpoints
* clean up labels after deleting a dashboard
* fix issues from code review
* WIP remove type from labels
* feat(http): add label cli functions
* fixes
* make fmt
* feat(http): add labels to tasks and buckets
* feat(bolt): remove labels if their resources are deleted
* remove bolt cleanup for now
* make fmt
* type conversion magic
* feat(inmem): implement label service
* prevent race conditions
* remove labels from inmem if their resources are deleted
* fix race condition
* fix formatting
The flux query controller was updated to include a Shutdown method a
while ago. Explicitly handle query controller creation and shutdown
where applicable.
In influxd, this ensures that outstanding queries are handled before the
process dies. In tests, this ensures that query controller goroutines
aren't leaked, which drastically simplifies reading full stack traces.
This change also registers query controller metrics with the prometheus
registry in influxd.
- Brought over enterprise's QueryLogReader, with small adjustments
- Time filters are for the run's ScheduledFor field, per spec
- Adjusted run log timestamps for consistent formatting:
- ScheduledFor is RFC3339 because it's a whole-second timestamp
- StartedAt, FinishedAt use RFC3339Nano for high precision
- Several test adjustments to use relative time, for easier integration
with storage retention
feat(internal/fs): move influxDir function to internal package in base of repo
feat(cmd/influx): add local flag that operates on local file system
feat(cmd/influx): add local support for org command
feat(cmd/influx): add local support for user command
eat(cmd/influx): add local support for bucket command
feat(cmd/influx): add error for local with query/task/setup cmds
fix(cmd/influx): unshadow error in delete bucket command
fix(cmd/influx): update copy for local flag commands
feat(platform): add dashboard operation log interface
feat(bolt): add dashboard operation log to bolt client
feat(platform): add links to dashboard load response
fix(http): update links in json response of dashboards
feat(bolt): add operation logs for each resource
feat(http): add routes for operation logs
feat(cmd/influxd): set operation logs on http api backend
This also adjusts the TaskService interface's RetryRun method to accept
a task ID rather than an org ID. Internally, we still look up runs by
organization, and maybe that will change later, but this is a more
natural way for clients to look it up.
These are the log messages that get printed immediately when starting
the application for the first time. This fixes the messages to conform
to the logging style guide.
Before, if you ran `influx query` and didn't have --org-id, you would
get an unhelpful runtime error like `id must have a length of 16 bytes`.
After this, you get a more helpful `required flag(s) "org-id" not set`
error.
Now at startup, there will be a log line like:
Opened bolt database {"log_id": "0B5Oks9W000", "path": "influxd.bolt"}
Which gives a somewhat better idea of what's going on.
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
The pb package was only referenced in cmd/influx/query.go, but in
dead code, since it uses the same machinery as the repl, which goes
through the HTTP endpoints, rather than the gRPC endpoints.
This pulls in the code that allows doing reads with flux into the
platform repo, and removes extra.go.
The reusable portion is under storage/reads, where the concrete
implementation for one of the platform's engines is in
storage/readservice.
In order to make this more reusable, the cursors had to move into
their own package, decoupling it from all of the other code in the
tsdb package. tsdb/cursors is this new package, and type/function
aliases have been added to the tsdb package to point at it.
The models package already is very light on transitive dependencies
and so it was allowed to be depended on in a concrete way in the
cursors package.
Finally, the protobuf definitions for issuing GRPC reads has been
moved into its own package for two reasons:
1. It's a clean separation, and helps keep it that way.
2. Many/most consumers will not be using GRPC. We just
use the datatypes to express the API which helps making
a GRPC server easier.
It is left up to future refactorings (specifically ones that involve
GPRC) to determine if these types should remain, or if there is a
cleaner way.
There's still some dependencies on both github.com/influxdata/influxql
and github.com/influxdata/influxdb/logger that we can hopefully remove
in future refactorings.
We reorganized the functions in flux to have the structure:
/functions
/inputs
/transformations
/outputs
this PR catches up platform to work with the new package layout.
As a separate refactoring issue, we should discuss:
from(bucket: ) should migrate from flux --> platform
to_http and to_kafka should migrate from platform --> flux
The race detector was picking up a data race because of the
unsynchronized reassignment of ctx:
```
$ go run -race ./cmd/influxd
...
^C
==================
WARNING: DATA RACE
Write at 0x00c00053d220 by main goroutine:
main.platformF()
/Users/mr/gomod/platform/cmd/influxd/main.go:381 +0x2cf7
github.com/spf13/cobra.(*Command).execute()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x8b2
github.com/spf13/cobra.(*Command).ExecuteC()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x432
github.com/spf13/cobra.(*Command).Execute()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800 +0x38
main.Execute()
/Users/mr/gomod/platform/cmd/influxd/main.go:388 +0x4e
main.main()
/Users/mr/gomod/platform/cmd/influxd/main.go:46 +0x2f
Previous read at 0x00c00053d220 by goroutine 147:
main.platformF.func1()
/Users/mr/gomod/platform/cmd/influxd/main.go:330 +0x3c
Goroutine 147 (running) created at:
main.platformF()
/Users/mr/gomod/platform/cmd/influxd/main.go:329 +0x2554
github.com/spf13/cobra.(*Command).execute()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x8b2
github.com/spf13/cobra.(*Command).ExecuteC()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x432
github.com/spf13/cobra.(*Command).Execute()
/Users/mr/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800 +0x38
main.Execute()
/Users/mr/gomod/platform/cmd/influxd/main.go:388 +0x4e
main.main()
/Users/mr/gomod/platform/cmd/influxd/main.go:46 +0x2f
==================
```
So just assign context.WithCancel to a new variable instead.
AJ Bond
Adam
Adam
Ben Johnson
Lyon Hill
Edd Robinson
Mark Rushakoff
Kelvin Wang
Jade McGough
Jorge Landivar
kelwang
Mark Rushakoff
zhulongcheng
Jeff Wendling
Michael Desa
Chris Goller
Jonathan A. Sternberg
Leonardo Di Donato