_http.NewRequestWithContext_ (available since golang 1.13) ensures that the supplied context also controls the entire lifetime of a request and its response.
This commit adds `user_id` as a tag for traces. It helps to lookup and
filter traces we need by userID.
OrgID is harder to get right, so I will open an issue, but it will be
nice to have it in as well.
Signed-off-by: Gianluca Arbezzano <gianarb92@gmail.com>
Co-authored-by: George MacRorie <gmacrorie@influxdata.com>
Renaming Generate in anticipation of a new method that will onboard
users other than the initial user. The intent is to simplify multi-user
setups.
Co-authored-by: Chris Goller <goller@gmail.com>
* fix: allow authorized label service to be called indirectly
17071 exists because pkger loads all service resources as authorized on
start, resulting in them all being authorized when referenced indirectly
(not hit directly via api by consumer). Rather than restructure pkger to
only authorize direct services, this allows proper indirect auth to
labels (the cause of 17071).
* Add orgService to tests
* Add resource types to find orgID from
This removes the spec and updates the lang package usage to make use of
passing in the runtime as a parameter.
It removes all direct dependendencies on the flux runtime from the http
package.
This moves a few types and constants to the global package so it can be
used without importing the `task/backend` package. These constants are
referenced in non tasks-specific code.
This is needed to break a dependency chain where the task backend will
call into the flux runtime to perform parsing or evaluation of a script
and to prevent the http package from inheriting that dependency.
The tasks subsystem will now use the flux language service to parse and
evaluate flux instead of directly interacting with the parser or
runtime. This helps break the dependency on the libflux parser for the
base influxdb package.
This includes the task notification packages which were changed at the
same time.
* fix(backup): handle backup with no credentials file
Backups and restores should work whether or not the original installation uses
a credentials file and whether or not the backup contains a credentials file.
* Revert "fix(kv): Don't stop when key not found from index."
This reverts commit bd9167d383.
* Revert "fix(kv): push down org ID to skip in delete URM (#16841)"
This reverts commit a5f508de77.
* Revert "fix(kv): delete authorization from correct index bucket (#16835)"
This reverts commit 7349216e94.
* Revert "feat(kv): Index Authorizations by User ID (#16818)"
This reverts commit df36fe957b.
* Revert "feat: add indexes to urm for user lookups (#16789)"
This reverts commit 9561d0a4f4.
Prior to this change influxql requests were sent to the same back end as Flux queries.
This MAY not always be the case. Now InfluxQL queries are specifically routed to the InfluxQLService.
In the case of this OSS build the FluxService and InfluxQLService are the same.
2 issues from investigating this error. First is the status check func
did not identify it was a media unsupported issue adn tries to unmarshal
the empty response body. The 2nd, was the double content type headers were
causing an error. Locally this error does not surface, cannot repoduce on
macos, but in cloud it is persistent.
closes: #16819
* feat(kv): add user id index on authorizations
* chore(auths): test FindAuthorizations both with and without a populated index
* chore(kv): cleanup index skipping flag in auths service
* fix(kv): bad flag around auth by user index population
* fix(kv): auth by user index lookup use correct buckets
* chore(kv): ensure indexer is called as expected when auth user index missing
* chore(kv): add benchmarks around authorization lookup
This change allows for the InfluxQL language type to be used with the
/v2/query API endpoint.
This change also introduces a way to give the transpiler an explicit
bucket name instead of using the DBRPMapping service.
Requests to the endpoint will know the bucket name directly but will
likely not have run the migration step to populate the DBRP mappings.
this is the last step for pkger to follow the service definition pattern
that is in the works. Some bits from http were moved into kit/transport/http
for reusability. End result is to hopefully axe http pkg for resuable types in
kit. Long ways off still...
* refactor: move views logic to separate directory
* refactor: normalize views
* fix: spinners
* fix: dont render views until status is done
* fix(http/dashboards): view shape not returning from getDashboard
* test: delete irrelevant and redundant test
* fix: go tidy
* test: skipping monaco test
* chore: sort type exports
* chore: cleanup
* feat(backup): `influx backup` creates data backup
* feat(backup): initial restore work
* feat(restore): initial restore impl
Adds a restore tool which does offline restore of data and metadata.
* fix(restore): pr cleanup
* fix(restore): fix data dir creation
* fix(restore): pr cleanup
* chore: amend CHANGELOG
* fix: restore to empty dir fails differently
* feat(backup): backup and restore credentials
Saves the credentials file to backups and restores it from backups.
Additionally adds some logging for errors when fetching backup files.
* fix(restore): add missed commit
* fix(restore): pr cleanup
* fix(restore): fix default credentials restore path
* fix(backup): actually copy the credentials file for the backup
* fix: dirs get 0777, files get 0666
* fix: small review feedback
Co-authored-by: tmgordeeva <tanya@influxdata.com>
also makes the yaml decoder the default. To foten we end up in application/octet-stream
which is the default for many different mime types. This provides a mechanism
around that so that when the automagical detection fails it can allow the user
to provide it via the CLI.
* feat(checks): Add custom check type
* feat(checks): Remove alert builder from custom check
* feat(checks): Add AlertBuilderAction to list of possible actions
* feat(checks): Query visualization does not make sense for custom check
* feat(check): check editor should only reexecute queries if view query changes
* Update ui/src/timeMachine/components/TimeMachineFluxEditor.tsx
Co-Authored-By: Bucky Schwarz <hoorayimhelping@users.noreply.github.com>
* Address PR review
Co-authored-by: Bucky Schwarz <hoorayimhelping@users.noreply.github.com>
feat(ui): added last run status checks for notification rules and check rules, readded updateCheck to fix linter and functionality issues with program and added tests to ensure check creation and update stability
this work is to support pkger, but was able to add back in the
skipped tests. seeing failures upstream, and didn't catch it in
influxdb b/c the tests were being skipped.
closes: #14799
this is a blocker for anyone who hits the endpoint services internally. They
had to know that they need to also know of hte secret service then do all that
put/delete alongside the operation. This makes that unified inside the store tx.
one other thing this does is make obvious the dependencies that
notification services has. In this case it is the secrets service it
depends on.
noticed that I had not used the http server as the entry point for server tests.
This was work to make that happen. Along the way, found a bunch of issues I hadn't
seen before 🤦. There are a number of changes tucked away inside the
other types, that make it possible to encode/decode a type with zero value for
influxdb.ID.
* added date-time format for start and stop DeletePredicateRequest
* fixed malformed reference to ViewProperties in PkgChart model
* define separate model for RetentionRule as is in Organizations, Buckets, Labels
* "labels" property from Check and PostCheck should be part of CheckBase (it is ancestor for all Check types)
* "labels" property from NotificationRule and PostNotificationRule should be part of NotificationRuleBase (it is ancestor for all NotificationRule and types)
* "labels" property from NotificationEndpoint and PostNotificationEndpoint should be part of NotificationEndpointBase (it is ancestor for all NotificationEndpoint and types)
* The url property of HTTPNotificationRuleBase should not be required
* Added query link for CheckBase and NotificationRuleBase
note: tests are seriously borked here. Cannot reuse any existing testing
as the setup is very particular and the http layer doesn't suppport everyting.
that being said, there are goign to be implicit testing in the
`launcher/pkger_test.go` file. This feels broken, and probably needs to be
readdressed before we GA a 2.0 influxdb....
this is a step towards providing a shared http client that manages pooling connections,
timeouts, and reducing GC for by not creating/GCing a client each req. Bring on the red!
Now, traces will have all headers except authorization and user-agent.
Additionally, I've removed the referer and remoteaddr as they did not
contribute much.
Eventually, this may change to just a small set of headers to record.
This removes the bucket name validation from the KV BucketService,
and moves it to the http implementation of the service.
The effect is that API user requests still get validated but direct KV access does not
* chore: Remove several instances of WithLogger
* chore: unexport Logger fields
* chore: unexport some more Logger fields
* chore: go fmt
chore: fix test
chore: s/logger/log
chore: fix test
chore: revert http.Handler.Handler constructor initialization
* refactor: integrate review feedback, fix all test nop loggers
* refactor: capitalize all log messages
* refactor: rename two logger to log
* fix(http): Return an empty array if organization has no secret keys
* fix(http): Return an empty array if organization has no secret keys instead nil
* fix(endpoint): when looking up a endpoint we should allow org only lookup
In the current system the api always adds "UserID" to the filter. This only
allows the system to look up endpoints that user created. The behavior should be
that we filter based on user input and use authorizor to hide things they shouldn't see.
* feat(tracing): dont trace spans with full URL path names in ExtractFromHTTPRequest
* chore(multiple): replace all occurrences of julienschmidt/httprouter with influxdata/httprouter
the errors changed during the pkger http server error improvements and this
fixes it to be similar to what it once was, only now its in a flatter fashion
very similar to the pkger http server apply response.
one thing to note here is that new endpoint was created. there was no
endpoint for setting an initial password that worked. The existin endpoint
was a bit messy and coupled across multiple routes. Having multiple auth
schemes proved incredibly taxing to write against.
also adds some extra user friendliness. it sorts the pkg created via an
export by resource kinds. It also titles the kinds to make them match the
documentation even though the kind is case insensitive. Easier to read this
way.
no associations included at this time. Also fixes http response to be just
the pkg without the envelope. Having that envelope makes the API icky to
work with from any shell script or just saving it to file. This feels more
organic to just drop that envelope.
* fix(flakey-test): refactored getSortedBucketNames for more consistency and predictability. Finished DWP API functionality
* fix(FilterRow): removed unnecessary FeatureFlag from component
* chore: updated yml and tests to reflect API changes
* feat(auth): add createdAt and updatedAt to authorization
Co-Authored-By: Ariel <ariel.salem1989@gmail.com>
* feat(auth): passing createAuth tests
* test: ensured that createdAt and updatedAt are valid on authorizations
Previously we overwrote the tasks existing latestCompleted to be used for latestCompleted as well as latestScheduled.
For obvious reasons this is confusing and missleading. I believe by seperating the two fields we can have a clear seperation
of concerns.
* feat(kv): unique variable names
- adds system bucket for creating an index of unique variable names
- adds tests
- deleted unit tests for dead code
- removed a test runner for the variable service from http
* feat(task): Allow tasks to run more isolated from other task systems
To allow the task internal system to be used for user created tasks as well
as checks, notification and other future additions we needed to take 2 actions:
1 - We need to use type as a first class citizen, meaning that task's have a type
and each system that will be creating tasks will set the task type through the api.
This is a change to the previous assumption that any user could set task types. This change
will allow us to have other service's white label the task service for their own purposes and not
have to worry about colissions between the types.
2 - We needed to allow other systems to add data specific to the problem they are trying to solve.
For this purpose adding a `metadata` field to the internal task system which should allow other systems to
use the task service.
These changes will allow us in the future to allow for the current check's and notifications implementations
to create a task with meta data instead of creating a check object and a task object in the database.
By allowing this new behavior checks, notifications, and user task's can all follow the same pattern:
Field an api request in a system specific http endpoint, use a small translation to the `TaskService` function call,
translate the results to what the api expects for this system, and return results.
* fix(task): undo additional check for ownerID because check is not ready
* NotificationRuleBase: the endpointID should be required and editable, there is missing a links property
* NotificationRule: add discriminator
* add a missing TraceSpan parameter into a GetNotificationRules and GetChecks
* Add TraceSpan parameter to GetNotificationEndpoints operation
* Fixed handler path for a list of all labels for a notification endpoint
* Fixed filter NotificationEndpoints by limit and offset
The http error schema has been changed to simplify the outward facing
API. The `op` and `error` attributes have been dropped because they
confused people. The `error` attribute will likely be readded in some
form in the future, but only as additional context and will not be
required or even suggested for the UI to use.
Errors are now output differently both when they are serialized to JSON
and when they are output as strings. The `op` is no longer used if it is
present. It will only appear as an optional attribute if at all. The
`message` attribute for an error is always output and it will be the
prefix for any nested error. When this is serialized to JSON, the
message is automatically flattened so a nested error such as:
influxdb.Error{
Msg: errors.New("something bad happened"),
Err: io.EOF,
}
This would be written to the message as:
something bad happened: EOF
This matches a developers expectations much more easily as most
programmers assume that wrapping an error will act as a prefix for the
inner error.
This is flattened when written out to HTTP in order to make this logic
immaterial to a frontend developer.
The code is still present and plays an important role in categorizing
the error type. On the other hand, the code will not be output as part
of the message as it commonly plays a redundant and confusing role when
humans read it. The human readable message usually gives more context
and a message like with the code acting as a prefix is generally not
desired. But, the code plays a very important role in helping to
identify categories of errors and so it is very important as part of the
return response.
The `/query` swagger endpoint now specifies that error messages are
returned as the standard JSON schema. The standard JSON schema has also
been changed slightly so that only `code` and `message` are documented
and the intention is that we will flatten the message from an
`influxdb.Error` before we encode the JSON.
We have been tracking down odd error messages when writing data and
found the problem to be internal server errors when writing empty
bodies.
I added fairly comprehensive test coverage for /api/v2/write as well
as simplify and clarify the error messages.
It was possible to create checks with invalid data causing
odd internal errors to return from the layers far down below.
Likely, we need to add more checks to the validation, but, I have
not thoroughly checked.
Closes https://github.com/influxdata/influxdb/issues/15042
Previously, there was an optional URL provided for the pagerduty
endpoint. However, the pagerduty API url does not change and as a result
it should not have been a parameter. The Pagerduty API does require a
`clientURL` that is presented in the pagerduty UI when an alert is
triggered. Currently that value will default to the alerts history page
for the organization.
At times snowflake id generation would create org and bucket IDs with
characters that had special meaning for the storage engine.
The storage engine concats the org and bucket bytes together into a
single 128 bit value. That value is used in the old measurement
section. Measurement was transformed into the tag, _measurement.
However, certain properties of the older measurement data location
are still required for the org/bucket bytes. We cannot have
commas, spaces, nor backslashes.
This PR puts a specific ID generator in place during the creation of
orgs and buckets. The IDs are just random numbers but with each
of the restricted chars incremented by one. While this changes the
entropy distribution somewhat, it does not matter too much for our
purposes.
... because now org and bucket ids are checked for previous existence
transactionally in the key-value stores. If the ID does already exist
then we try to generate a new key up to 100 times.
* WIP
* Fix UI linter errors from swagger changes to Level Rule
* Prevent same level selection on changes from
* Remove unused get
* Fix prettier error
* chore(notification/rule): change level rule to check level for rules
* Make endpoint description field normal sized
* Add password type to secret inputs
* de-snarkify and remove defaults from new-endpoint-draft
* Make new tag rules unique
* Remove hardcoded fields in initRuleDraft
* Add tag rule make statusRule is equal to level
* Status rule can be "is equal to"
* Remove cron option from rule
* Add the text lastupdated to cards
* Fix offset and every copy
* Improve Tags in checks
* Move threshold deadman selector to conditions card
* Filter empty tag rules on notification rules
* Make dismiss button gray
* Fix review comments
* remove irrelevant tests
* Fix missing url field from http rule
fix(notification/check): include tags in check object in generated flux
Closes https://github.com/influxdata/influxdb/issues/14769
fix(notification/check): use selected field in threshold functions
Closes https://github.com/influxdata/influxdb/issues/14776
fix(testing): add selected field for check tests
fix(check): use real flux for threshold check
feat(notification/check): generate flux for deadman checks
chore(endpoint): rename webhook endpoint to http endpoint
fix(notification/rule): fetch url for flux script off of endpoint
fix(notification/rule): clean up slack and http rules
fix(notification/rule): change MessageTemp to MessageTemplate
fix(rules): pass endpoint in to rule during create
fix(ui): rename webhook to http
feat(notification/check): namespace deadman under alerts
fix(notification/check): nest tags under tags key in data object in flux
wip
feat(kv): log error if urm cannot be deleted for notification rule
fix(notification/rule): remove name from notify call in slack rule
chore(ui/cypress/e2e): skip rule create test
* added missing Permission types: notificationRules, notificationEndpoints, checks
* Authorization links use Link as a type
* Run's logs are readonly
* TaskCreateRequest and TaskUpdateRequest has description property to update Task description
Create view with checkID in NewVEO
fix ruleToDraftRule bug
Fix no thresholds error
Get correct view status for rendering cell
Fix editVEO flow
Add notification endpoint ID to initRule
Remove task type from check and notification rule
mock generate AST
use check query in EditCheckEO
wip
Create default http notification endpoint
Co-authored-by: Michael Desa <mjdesa@gmail.com>
Dont save view if no dashboardID in saveCheck
Execute queries in editCheckEO
Fix check remove bug
Disable Advanced view and multiple queries in check builder
Disable alerting button if draftQuery is not alertable
chore(kv/check): remove createCheckTask from KV as the result of #14728
* feat(task): Remove token's from task structures
We had previously removed token's from the task api but left the token in place in several locations in the stack.
Now we can cleanly remove the extra tokens.
Johnny Steenbergen
jlapacik
Lyon Hill
Alirie Gray
Pavel Závora
Brett Buddin
Gianluca Arbezzano
Ariel Salem
Jonathan A. Sternberg
Lorenzo Affetti
Christopher M. Wolff
Marc Delagrammatikas
Gavin Cabbage
Kelvin Wang
Greg
Jorge Landivar
pierwill
tmgordeeva
George
Nathaniel Cook
kelwang
Michael Desa
Jade McGough
kun
Andrew Watkins
Jacob Marble
Zoe Steinkamp
Stuart Carnie
Gavin Cabbage
Deniz Kusefoglu
Yiqun Zhang
Jakub Bednář
Deary Hudson III
Chris Goller
Edd Robinson
nwneisen
Timmy Luong
Bucky Schwarz
Scott Anderson
Deary Hudson
Brandon Farmer
Johnny Steenbergen
Russ Savage