The synchronous executor was missing a call to ResultIterator.Release.
The asynchronous executor wasn't even calling Query.Statistics.
Also add a test that the scheduler records the statistics to the run
log, and that the statistics are visible from the launcher test. The
launcher test is the most likely place to catch if something goes wrong
in the full stack.
The late measurement filter, after a pivot, had the potential to result
in empty groups without a runID, which would cause a runtime error,
which would cause the whole query to fail.
Experimentation has shown that those empty tables will no longer arrive
by filtering early on measurement.
This should considerably simplify debugging when things go wrong with
the tasks, as this error can be displayed from the UI or CLI. Prior to
this change, you would have to view the console output from influxd.
Fixes#12548.
In the platform adapter, we ask the URM for a list of tasks the user
owns, and then we look up each task individually.
The task service tests uncovered a legitimate bug where FindTasks would
return a "task not found" error, originating from looking up a task that
was present when we interrogated the URM but was deleted before we could
find it in the task store.
This change also removes duplicated URM logic from the HTTP handler
which has since been pushed down into the platform adapter.
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* fix(http): s/platform/influxdb/ for user service
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* feat(kv): implement labels generically on kv
* refactor(passwords): rename from BasicAuth to Passwords
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(http): initial support for flushing all key/values from kv store
* feat(kv): rename macro to variable
* feat(cmd/influxd/launcher): user kv services where appropriate
* refactor(passwords): rename from BasicAuth to Passwords
* feat(kv): implement macro service
* test(ui): introduce cypress
* test(ui): introduce first typescript test
* test(ui/e2e): add ci job
* chore: update gitignore to ignore test outputs
* feat(inmem): in memory influxdb
* test(e2e): adding pinger that checks if influxdb is alive
* hackathon
* hack
* hack
* hack
* hack
* Revert "feat(inmem): in memory influxdb"
This reverts commit 30ddf032003e704643b07ce80df61c3299ea7295.
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* hack
* chore: lint ignore node_modules
* hack
* hack
* hack
* add user and flush
* hack
* remove unused vars
* hack
* hack
* ci(circle): prefix e2e artifacts
* change test to testid
* update cypress
* moar testid
* fix npm warnings
* remove absolte path
* chore(ci): remove /home/circleci proto mkdir hack
* wip: crud resources e2e
* fix(inmem): use inmem kv store services
* test(dashboard): add first dashboard crud tests
* hack
* undo hack
* fix: use response from setup for orgID
* chore: wip
* add convenience getByTitle function
* test(e2e): ui can create orgs
* test(e2e): add test for org deletion and update
* test(e2e): introduce task creation test
* test(e2e): create and update of buckets on org view
* chore: move types to declaration file
* chore: use route fixture in dashboard tests
* chore(ci): hack back
* test(ui): update snapshots
* chore: package-lock
* chore: remove macros
* fix: launcher rebase issues
* fix: compile errors
* fix: compile errors
* feat(cmd/influxdb): add explicit testing, asset-path, and store flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* fix(cmd/influxd): set default HTTP handler and flags
Co-authored-by: Andrew Watkins <watts@influxdb.com>
* build(Makefile): add run-e2e and PHONY
* feat(kv:inmem:bolt): implement user service in a kv
* refactor(kv): use consistent func receiver name
* feat(kv): add initial basic auth service
* refactor(passwords): move auth interface into own file
* refactor(passwords): rename basic auth files to passwords
* refactor(passwords): rename from BasicAuth to Passwords
* refactor(kv): copy bolt user test into kv
Co-authored-by: Michael Desa <mjdesa@gmail.com>
* feat(kv): add inmem testing to kv store
* fix(kv): remove extra user index initialization
* feat(kv): attempt at making errors nice
* fix(http): return not found error if filter is invalid
* fix(http): s/platform/influxdb/ for user service
* fix(http): s/platform/influxdb/ for user service
* feat(kv): initial port of telegraf configs to kv
* feat(kv): initial port of scrapers in bolt to kv
* feat(kv): first pass at migrating bolt org service to kv
* feat(kv): first pass at bucket service
* feat(kv): first pass at migrating kvlog to kv package
* feat(kv): add resource op logs
* feat(kv): first pass at user resource mapping migration
* feat(kv): add urm usage to bucket and org services
* feat(kv): first pass at kv authz service
* feat(kv): add cascading auth delete for users
* feat(kv): first pass d authorizer.OrganizationService in kv
* feat(cmd/influxd/launcher): user kv services where appropriate
* fix(kv): initialize authorizations
* fix(influxdb): use same buckets while slowly migrating stuff
* fix(kv): make staticcheck pass
* feat(kv): add dashboards to kv
review: make suggestions from pr review
fix: use common bucket names for bolt/kv stores
* test(kv): add complete password test coverage
* chore(kv): fixes for staticcheck
* feat(kv): implement labels generically on kv
* feat(kv): implement macro service
* feat(kv): add source service
* feat(kv): add session service
* feat(kv): add kv secret service
* refactor(kv): update telegraf and urm with error messages
* feat(kv): add lookup service
* feat(kv): add kv onboarding service
* refactor(kv): update telegraf to avoid repetition
* feat(cmd/influxd): use kv lookup service
* feat(kv): add telegraf to lookup service
* feat(cmd/influxd): use kv telegraf service
* feat(kv): update scraper error messaging
* feat(cmd/influxd): add kv scraper
* feat(kv): add inmem backend tests
* refactor(kv): copy paste errors
* refactor(kv): add code to password errors
* fix(testing): update error messages for incorrect passwords
* feat(kv): rename macro to variable
* refactor(kv): auth/bucket/org/user unique checks return errors now
* feat(inmem): add way to get all bucket names from store
* feat(inmem): Buckets to return slice of bytes rather than strings
* feat(inmem): add locks around Buckets to avoid races
* feat(cmd/influx): check for unauthorized error in wrapCheckSetup
* chore(e2e): add video and screenshot artifcats to gitignore
* docs(ci): add build instructions for e2e tests
* feat(kv): add id lookup for authorized resources
Task ID is now a required value on run and log filters. It was
effectively required by all implementations before anyway, so now those
types reflect that requirement.
Organization ID was removed from those same fields. The TaskService
looks up the organization ID via the task in cases where we need it at a
lower layer.
This was a missed case from #11817.
This case currently occurs when creating a task through the UI, using a
session rather than a full-fledged authorization. It doesn't fix that
case yet, but at least it will log an informative message.
Immediately before the executor calls out to the query service, the
executor loads the authorizer associated with the task, and associates
that authorizer with the context used to execute the query.
Accept token when creating or updating a task, but only report back the
authorization ID.
This means the executor and the platform adapter are now both aware of
an Authorization Service.
With the ongoing authorization work, creation arguments will differ from
what's returned on reads. More specifically, creation will accept a
token, but reads will report back a token ID.
This refactor facilitates that authorization work, and also brings the
code closer to the swagger definition, for the TaskCreateRequest type in
particular.
The earliest time needed to be captured before creating the task, not
after; otherwise a test running close to a second boundary would fail.
Also make the failure message slightly more readable.
This switches run status from a tag to a field. This is likely a
breaking change to existing task logs.
Using a one-off local query, for 250 records, the previous approach took
around 10 seconds and the new approach is about 30 milliseconds. At 1000
records, the previous approach was roughly 110 seconds and the new
approach is around 70 milliseconds.
filter out resources that have mission IDs
fix(influxdb): simplify auth check in PermissionAllowed
review(platform): update as noted in review
fix(influxdb): ensure permission has valid org id
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
CreateTasks now check that the user has the write permission to the
tasks resource belonging to an organization. This change comes after
https://github.com/influxdata/platform/pull/2157 modified the structure
of authorization.
Also rename RetryAlreadyQueuedError by running:
gorename -from '"github.com/influxdata/platform/task/backend".RetryAlreadyQueuedError' -to RequestStillQueuedError
and some further manual cleanup for comments.
A standard Makefile is used now in all subdirs that run go generate.
Make will only generate the file if its source files changed.
The checkgenerate target runs clean to ensure all targets a generated
fresh.
Previously, the WithTicker option would call TickScheduler.Tick every
time the underlying time.Ticker sent a time on its channel. This meant
we used a 1s period, which meant that in the worst case, we would see a
tick at about 999ms after the second rollover.
This change increases the underlying time.Ticker frequency, but only
calls TickScheduler.Tick after a second rolls over. Since we now use a
tick frequency of 100ms, during normal operation, TickScheduler.Tick
will be called within 0.1s after the second rolls over.
* feat(task): Allow the most recent run time to be shown in the api.
When showing tasks in the api latest_completed will now show in the api if it has been run.
fixes#1705
Previously, using every=1m would run every minute from when the task was
created. This change restores the original intent, which is that "every
1m" happens on the minute, "every 1h" on the hour, etc.
The keys used to be strings, back when platform.ID was a byte slice, not
a uint64.
And rename the field from runners to meta, which is much more accurate.
Mark Rushakoff
Jorge Landivar
Lyon Hill
kelwang
Kelvin Wang
Chris Goller
Jacob Marble
Jacob Marble
Jonathan A. Sternberg
Michael Desa
Brandon Farmer
Lorenzo Affetti
Mark Rushakoff
Nathaniel Cook
jlapacik
Andrew Watkins
Christopher M. Wolff
Jakub Bednar
Mark Rushakoff
Edd Robinson
zhulongcheng
Ben Johnson