* chore: update task tests to use the tenant service
After the introduction of the tenant system we need to switch the testing frameworks
to use it instead of the old kv system
* chore: update onboarding to allow injected middleware
This commit adds `mincore.Limiter` which throttles page faults caused
by mmap() data. It works by periodically calling `mincore()` to determine
which pages are not resident in memory and using `rate.Limiter` to
throttle accessing using a token bucket algorithm.
* feat(task): Add new permission lookup pattern for executor
We can now use the user service to populate task owners permissions.
This should improve the task lookup time and decouple the task system
from the URM system. In the future we will have the ability to better isolate
tenant pieces from the rest of the service.
* feat: add feature flagging
We can now use the user service to populate task owners permissions.
This should improve the task lookup time and decouple the task system
from the URM system. In the future we will have the ability to better isolate
tenant pieces from the rest of the service.
* refactor: migrator and introduce Store.(Create|Delete)Bucket
feat: kvmigration internal utility to create / managing kv store migrations
fix: ensure migrations applied in all test cases
* chore: update kv and migration documentation
note: going to make this evolution in two steps so that we have a simple
rollback to get back to a working state. We'll be maintaining both packages
and the new templates and stacks endpoints for a while as users start to
move onto a newer CLI version. Sunsetting by end of July.
references: #18580
Annotate the context with feature flags when handling flux queries in influxdb.
Taking advantage of this in flux end-to-end tests. Using a custom flagger that
can set overrides based on the test case that is about to be run, allowing us
to enable features in the end-to-end tests.
* feat: start using the new org handler from the tenant service.
The rest of the tenant system is in place except the org http api handler and the
user api handler.
* fix: update the label service in org handler and add links
This adds a launcher test for the read window aggregate push down to
verify that it is done when a query is sent with the appropriate
pattern, the output is correct, and that the metric is incremented that
signals the push down happened.
Switch to use the new user handler. We have been using the tenant backend for some
time now and just need to switch over to using tenant front to back.
We have reached the stage wehre the new tenant service is being used and
is stable but we want to get it in more hands and used as the default service.
* fix: allow authorized label service to be called indirectly
17071 exists because pkger loads all service resources as authorized on
start, resulting in them all being authorized when referenced indirectly
(not hit directly via api by consumer). Rather than restructure pkger to
only authorize direct services, this allows proper indirect auth to
labels (the cause of 17071).
* Add orgService to tests
* Add resource types to find orgID from
The tasks subsystem will now use the flux language service to parse and
evaluate flux instead of directly interacting with the parser or
runtime. This helps break the dependency on the libflux parser for the
base influxdb package.
This includes the task notification packages which were changed at the
same time.
Prior to this change influxql requests were sent to the same back end as Flux queries.
This MAY not always be the case. Now InfluxQL queries are specifically routed to the InfluxQLService.
In the case of this OSS build the FluxService and InfluxQLService are the same.
this is the last step for pkger to follow the service definition pattern
that is in the works. Some bits from http were moved into kit/transport/http
for reusability. End result is to hopefully axe http pkg for resuable types in
kit. Long ways off still...
* feat(backup): `influx backup` creates data backup
* feat(backup): initial restore work
* feat(restore): initial restore impl
Adds a restore tool which does offline restore of data and metadata.
* fix(restore): pr cleanup
* fix(restore): fix data dir creation
* fix(restore): pr cleanup
* chore: amend CHANGELOG
* fix: restore to empty dir fails differently
* feat(backup): backup and restore credentials
Saves the credentials file to backups and restores it from backups.
Additionally adds some logging for errors when fetching backup files.
* fix(restore): add missed commit
* fix(restore): pr cleanup
* fix(restore): fix default credentials restore path
* fix(backup): actually copy the credentials file for the backup
* fix: dirs get 0777, files get 0666
* fix: small review feedback
Co-authored-by: tmgordeeva <tanya@influxdata.com>
this is a blocker for anyone who hits the endpoint services internally. They
had to know that they need to also know of hte secret service then do all that
put/delete alongside the operation. This makes that unified inside the store tx.
one other thing this does is make obvious the dependencies that
notification services has. In this case it is the secrets service it
depends on.
* chore: Remove several instances of WithLogger
* chore: unexport Logger fields
* chore: unexport some more Logger fields
* chore: go fmt
chore: fix test
chore: s/logger/log
chore: fix test
chore: revert http.Handler.Handler constructor initialization
* refactor: integrate review feedback, fix all test nop loggers
* refactor: capitalize all log messages
* refactor: rename two logger to log
the flags have to match the flags with the exception of beign lower case and
all `_` be changed to `-` in the flag. This is a result of using the cobra
flag to env var mapping with the `-` replaced to `_`.
The secret service is tested by creating a secret and then attempting to
use it in a flux query. There is one test where accessing the secret
should work and one where it should return that the action is forbidden.
To have checks and notifications happen transactionally we need to be
able to alert the task system when a new task was created using the checks and notifications systems.
These two new middlewares allow us to inform the task system of a update
to a task that was created through the check or notification systems.
The controller implementation is primarily used by influxdb so it
shouldn't be part of the flux repository. This copies the code from flux
to influxdb so it can be removed from the next flux release.
Co-authored-by: Jade McGough <jade@influxdata.com>
* Add session renew option to launcher and use in middlewhere
* pass session options to services
* Update SessionAutoRenew to SessionRenewDisabled
* Add test for service constructor defaults
* Update changelog
feat(http): add prometheus counters for tracking write/query usage
feat(http/metric): add metric recoder for recording http metrics
feat(prometheus): implement metric.Recorder for prometheus metrics
fix(prometheus): remove erroneous fmt.Printlns
feat(http): add prometheus registry to API backend
This was done as exposing prometheus metrics to a higher level was quite
difficult. It was much simple to simply pass the registry down to
anything that needs it.
feat(cmd/influxd/launcher): pass prom registry in on api backend
feat(http): collect metrics for write and query endpoints
This was much messier than I would have preferred. Future work is
outlined in TODOs.
review(influxdb): rename metric.Metric to metric.Event
Stuart Carnie
George
Edd Robinson
Lyon Hill
Alirie Gray
Johnny Steenbergen
Ben Johnson
David McKay
Brett Buddin
Adrian Thurston
Jonathan A. Sternberg
jlapacik
Gianluca Arbezzano
Gavin Cabbage
Lorenzo Affetti
Greg
Sebastian Borza
Nathaniel Cook
Jacob Marble
Chris Goller
docmerlin (j. Emrys Landivar)
Johnny Steenbergen
Brandon Farmer
Kelvin Wang
David McKay
Adam
Max U
Christopher Wolff
kelwang
Iris Scholten
Todd Persen
Michael Desa
Jacob Marble