This command performs verification of TSM blocks
* expected and actual CRC-32 checksums match
* expected and actual min and max timestamps match decoded
data
Adds export tooling to `influxd inspect export-blocks` so that we
can dump out block data in SQL format for better analysis during
the debugging process.
The controller implementation is primarily used by influxdb so it
shouldn't be part of the flux repository. This copies the code from flux
to influxdb so it can be removed from the next flux release.
Co-authored-by: Jade McGough <jade@influxdata.com>
* Add session renew option to launcher and use in middlewhere
* pass session options to services
* Update SessionAutoRenew to SessionRenewDisabled
* Add test for service constructor defaults
* Update changelog
Until https://github.com/influxdata/flux/issues/1215 is fixed, the query
writer must make sure to always place a range directly after a from
operation. Otherwise the query will fail planning.
feat(http): add prometheus counters for tracking write/query usage
feat(http/metric): add metric recoder for recording http metrics
feat(prometheus): implement metric.Recorder for prometheus metrics
fix(prometheus): remove erroneous fmt.Printlns
feat(http): add prometheus registry to API backend
This was done as exposing prometheus metrics to a higher level was quite
difficult. It was much simple to simply pass the registry down to
anything that needs it.
feat(cmd/influxd/launcher): pass prom registry in on api backend
feat(http): collect metrics for write and query endpoints
This was much messier than I would have preferred. Future work is
outlined in TODOs.
review(influxdb): rename metric.Metric to metric.Event
This replaces usages of the spec compiler with the ast compiler and it
removes the error message referencing the spec compiler as an available
input.
It does not remove any of the code using the spec compiler that is
involved for proxying requests and it does not remove it from the API.
The synchronous executor was missing a call to ResultIterator.Release.
The asynchronous executor wasn't even calling Query.Statistics.
Also add a test that the scheduler records the statistics to the run
log, and that the statistics are visible from the launcher test. The
launcher test is the most likely place to catch if something goes wrong
in the full stack.
This commit consists of several improvements or changes:
* migrate the influxd binary to cobra.Command
* introduce a default run sub-command to start the server
* register the run sub-command flags with viper
to maintain compatibility with the existing behavior of automatic
binding of flags to environment variables.
Closes#12602
feat(influxdb): add generic store for documents
feat(influxdb): support authorizations in document store
feat(influxdb): support orgs in user resource mapping
feat(influxdb): add read-only included field on documents
feat(influxdb): add labels support to documents service
fix(influxdb): rename data field to content on documents
feat(influxdb): add with org id options for document store
feat(http): add templates swagger
feat(influxdb): add documentation to document options
doc(kv): add documentation for kv document store
test(kv): pull document tests in to the testing package
fix(http): fix swagger specification of templates endpoints
Previously the APIBackend understood only a ProxyQueryService,
but it needs to understand that there are two implementations of the
ProxyQueryService one for handling InfluxQL queries and one for handling
Flux queries. The names of the fields have been updated to make this
clear. As well as the FluxBackend is now initialized using the
FluxService explicitly.
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* fix(http): s/platform/influxdb/ for user service
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* feat(kv): implement labels generically on kv
* refactor(passwords): rename from BasicAuth to Passwords
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(http): initial support for flushing all key/values from kv store
* feat(kv): rename macro to variable
* feat(cmd/influxd/launcher): user kv services where appropriate
* refactor(passwords): rename from BasicAuth to Passwords
* feat(kv): implement macro service
* test(ui): introduce cypress
* test(ui): introduce first typescript test
* test(ui/e2e): add ci job
* chore: update gitignore to ignore test outputs
* feat(inmem): in memory influxdb
* test(e2e): adding pinger that checks if influxdb is alive
* hackathon
* hack
* hack
* hack
* hack
* Revert "feat(inmem): in memory influxdb"
This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295.
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* chore: lint ignore node_modules
* hack
* hack
* hack
* add user and flush
* hack
* remove unused vars
* hack
* hack
* ci(circle): prefix e2e artifacts
* change test to testid
* update cypress
* moar testid
* fix npm warnings
* remove absolte path
* chore(ci): remove /home/circleci proto mkdir hack
* wip: crud resources e2e
* fix(inmem): use inmem kv store services
* test(dashboard): add first dashboard crud tests
* hack
* undo hack
* fix: use response from setup for orgID
* chore: wip
* add convenience getByTitle function
* test(e2e): ui can create orgs
* test(e2e): add test for org deletion and update
* test(e2e): introduce task creation test
* test(e2e): create and update of buckets on org view
* chore: move types to declaration file
* chore: use route fixture in dashboard tests
* chore(ci): hack back
* test(ui): update snapshots
* chore: package-lock
* chore: remove macros
* fix: launcher rebase issues
* fix: compile errors
* fix: compile errors
* feat(cmd/influxdb): add explicit testing, asset-path, and store flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* fix(cmd/influxd): set default HTTP handler and flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* build(Makefile): add run-e2e and PHONY
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv): rename macro to variable
* refactor(kv): auth/bucket/org/user unique checks return errors now
* feat(inmem): add way to get all bucket names from store
* feat(inmem): Buckets to return slice of bytes rather than strings
* feat(inmem): add locks around Buckets to avoid races
* feat(cmd/influx): check for unauthorized error in wrapCheckSetup
* chore(e2e): add video and screenshot artifcats to gitignore
* docs(ci): add build instructions for e2e tests
* feat(kv): add id lookup for authorized resources
Task ID is now a required value on run and log filters. It was
effectively required by all implementations before anyway, so now those
types reflect that requirement.
Organization ID was removed from those same fields. The TaskService
looks up the organization ID via the task in cases where we need it at a
lower layer.
Immediately before the executor calls out to the query service, the
executor loads the authorizer associated with the task, and associates
that authorizer with the context used to execute the query.
Accept token when creating or updating a task, but only report back the
authorization ID.
This means the executor and the platform adapter are now both aware of
an Authorization Service.
With the ongoing authorization work, creation arguments will differ from
what's returned on reads. More specifically, creation will accept a
token, but reads will report back a token ID.
This refactor facilitates that authorization work, and also brings the
code closer to the swagger definition, for the TaskCreateRequest type in
particular.
Because the WAL relies on the tsm1.Value type, we move that into its own
tsm1/value package and set up some aliases forwarding them into tsm1. This
also required adding some methods and changing consumers to avoid the
unexported fields. I imagine this step will be useful one day when we make
the write path more efficient with respect to consuming points.
This commit additionally fixes some issues with generation. The iterator.tmpldata
and generation for array_cursor_* were removed accidentally when removing
iterators, making those generated files stale. Restore that and regenerate.
No change in functionality.
Previously, scrapers would scrape the target 10 times. This was
because each scraper subscriber was not put into a queue group.
I've added the queue group "metrics" and now we the subcribers
will only scrape the target once.
Additionally, I moved to using nats memory instead of nats file
store. We don't need durability for scraper runs across restarts.
fix(cmd/influx): format error for bare influx command
feat(cmd/influx): add org flag to query command
fix(cmd/influx): check correct error in wrapCheckSetup
fix(influx/cmd): use correct shorthand flag for org flag
feat(cmd/influx): wrapCheckSetup with wrapErrorFmt
feat(cmd/influx): cleanup language and errors during setup command
fix(http): use id query param in get buckets request
feat(cmd/influx): add helpful error messages for write command
feat(cmd/influx): add context to query command
We used go-bindata to embed all JSON files in proto to be shipped
with our released influxdb.
We are committing the bin_gen.go so that it simplifies the building
via make and go build.
If you would like to add another proto JSON file, run make
to generate a new bin_gen.go.
Co-Authored-By: Deniz Kusefoglu <deniz@influxdata.com>
feat(bolt): add function to find a resources organization id
rename platform to influxdb
Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
fix(bolt): rename FindResoureOrganization to FindResoureOrganizationID
feat(authorizer): add authorized user resource mapping service
Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
feat(influxdb): wire up authorized user resource mapping
Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
fix(authorizer): remove unused field from tests
Co-authored-by: Leonardo Di Donato <leodidonato@gmail.com>
Co-authored-by: Michael Desa <mjdesa@gmail.com>
test(http): get user off of session in create authz test
fix(http): allow user id to be specified explicitly on authorization
create authorization now allows specifying user id explicitly. If no
user id is specified then we use the user id from the authorizer.
fix(http): use influxdb import
fix(http): use platform error in http auth tests
feat(cmd/influx): allow create auth command to specify user explicitly
feat(http): add org id to permissions
filter out resources that have mission IDs
fix(influxdb): simplify auth check in PermissionAllowed
review(platform): update as noted in review
fix(influxdb): ensure permission has valid org id
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
feat(http): add http handler for proto service
feat(mock): add mock proto service
test(http): add proto handler tests
fix(platform): add view as option when adding a cell
feat(platform): add dashboard to proto struct
feat(fs): add filesystem implementation of proto
feat(http): add protos endpoints to api handler
feat(cmd/influxd/launcher): add protos path to server
doc(http): add protos to swagger
test(cmd/influxd/launcher): add --protos-path to launcher tests
fix(fs): remove unused args from test
fix(http): use platform.Error where appropriate
feat(platform): add functional options for platform errors
fix(testing): set dashboard ids properly in dashboard tests
feat(bolt): add dashboard specific views
fix(bolt): delete view when cell is removed or dashboard is deleted
all tests use a unique bucket based on the test file name. copied all tests over from flux repo
the tests are currently disabled due to engine consistency issues: https://github.com/influxdata/flux/issues/613
fix(http): add members/secrets/labels links on org response
fix(http:cmd/influxd): use secret service in api backend
fix(bolt): return empty list if there are no secrets for an org
chore(vault): add description of vault usage
* refactor(cmd/influxd): move driver code for influxd main package to sub-package so it can be reused.
* chore(query/influxql): moved query_test.go and requisite files to the influxql dir to be closer to the code it tests. (#2013)
This commit moves the `Main.cancel()` execution to the `main_test.go`
file so it's only executed for tests. This was interfering with the
shutdown process on the regular `influxd` binary.
Previously, the WithTicker option would call TickScheduler.Tick every
time the underlying time.Ticker sent a time on its channel. This meant
we used a 1s period, which meant that in the worst case, we would see a
tick at about 999ms after the second rollover.
This change increases the underlying time.Ticker frequency, but only
calls TickScheduler.Tick after a second rolls over. Since we now use a
tick frequency of 100ms, during normal operation, TickScheduler.Tick
will be called within 0.1s after the second rolls over.
* feat(platform): add label struct
* feat(platform): add label service interface
* WIP label service
* chore(http): simplify URMapping validation
* WIP label service skeleton
* add new GET labels handler
* feat(http): add POST handler for label service
* feat(http): add label delete handler
* fix json tag
* add happy path testing
* feat(http): add label endpoints to dashboard service
* make fmt
* add label delete handler
* make fmt again :(
* feat(bolt): add label implementation
* wire boltdb implementation of labels
* feat(mock): add mock for label service
* fix tests, add delete test
* add remaining tests, flesh out GET params
* unused code
* remove comments
* add labels endpoints
* clean up labels after deleting a dashboard
* fix issues from code review
* WIP remove type from labels
* feat(http): add label cli functions
* fixes
* make fmt
* feat(http): add labels to tasks and buckets
* feat(bolt): remove labels if their resources are deleted
* remove bolt cleanup for now
* make fmt
* type conversion magic
* feat(inmem): implement label service
* prevent race conditions
* remove labels from inmem if their resources are deleted
* fix race condition
* fix formatting
The flux query controller was updated to include a Shutdown method a
while ago. Explicitly handle query controller creation and shutdown
where applicable.
In influxd, this ensures that outstanding queries are handled before the
process dies. In tests, this ensures that query controller goroutines
aren't leaked, which drastically simplifies reading full stack traces.
This change also registers query controller metrics with the prometheus
registry in influxd.
- Brought over enterprise's QueryLogReader, with small adjustments
- Time filters are for the run's ScheduledFor field, per spec
- Adjusted run log timestamps for consistent formatting:
- ScheduledFor is RFC3339 because it's a whole-second timestamp
- StartedAt, FinishedAt use RFC3339Nano for high precision
- Several test adjustments to use relative time, for easier integration
with storage retention
feat(internal/fs): move influxDir function to internal package in base of repo
feat(cmd/influx): add local flag that operates on local file system
feat(cmd/influx): add local support for org command
feat(cmd/influx): add local support for user command
eat(cmd/influx): add local support for bucket command
feat(cmd/influx): add error for local with query/task/setup cmds
fix(cmd/influx): unshadow error in delete bucket command
fix(cmd/influx): update copy for local flag commands
feat(platform): add dashboard operation log interface
feat(bolt): add dashboard operation log to bolt client
feat(platform): add links to dashboard load response
fix(http): update links in json response of dashboards
feat(bolt): add operation logs for each resource
feat(http): add routes for operation logs
feat(cmd/influxd): set operation logs on http api backend
This also adjusts the TaskService interface's RetryRun method to accept
a task ID rather than an org ID. Internally, we still look up runs by
organization, and maybe that will change later, but this is a more
natural way for clients to look it up.
These are the log messages that get printed immediately when starting
the application for the first time. This fixes the messages to conform
to the logging style guide.
Before, if you ran `influx query` and didn't have --org-id, you would
get an unhelpful runtime error like `id must have a length of 16 bytes`.
After this, you get a more helpful `required flag(s) "org-id" not set`
error.
Now at startup, there will be a log line like:
Opened bolt database {"log_id": "0B5Oks9W000", "path": "influxd.bolt"}
Which gives a somewhat better idea of what's going on.
Max U
Kelvin Wang
maxunt
Edd Robinson
Adam Perlin
Christopher Wolff
Stuart Carnie
Ben Johnson
Lyon Hill
Alirie Gray
Jonathan A. Sternberg
kelwang
Iris Scholten
Palak Bhojani
jlapacik
Jade McGough
Nathaniel Cook
Todd Persen
Michael Desa
zhulongcheng
Jacob Marble
Lorenzo Affetti
Mark Rushakoff
Jacob Marble
Chris Goller
Jeff Wendling
Julius Volz
bpopovschi
docmerlin (j. Emrys Landivar)
Leonardo Di Donato
Mark Rushakoff
Jeff Wendling
Jeff Wendling
Adam
Andrew Watkins
AJ Bond
Adam
Mark Rushakoff