The HTTP handler logs URLs, but not body values for POST requests.
This means that queries sent by GET are logged, because the query
is in the URL, but queries sent by POST have no query text in the
log. This feature prints all the key-value pairs in the post body,
which includes the query text, except passwords, which are redacted.
Closes https://github.com/influxdata/influxdb/issues/20653
(cherry picked from commit 70755bf42c)
Closes https://github.com/influxdata/influxdb/issues/21000
* fix: infinite recursion bug (#20862)
* Revert "fix(error): unsupported value: +Inf" error not handled gracefully (#20250)"
This reverts commit 6ac0bb3fe3.
* fix: No infinite recursion on write error
If there is some error writing to the response writer, we
would previous have infinite recursion.
Re-closes https://github.com/influxdata/influxdb/issues/20249
(cherry picked from commit d6f7716924)
* chore: update CHANGELOG.md
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.
Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries
https://github.com/influxdata/influxdb/issues/20736
(cherry picked from commit 092c7a9976)
* chore: move context.Context to first argument in methods per convention
(cherry picked from commit a8b2129df5)
* fix(error): unsupported value: +Inf" error not handled gracefully
JSON marshalling errors should be returned properly formatted in JSON
like other errors. This fix formats marshalling errors the same way
influxdb formats other query errors.
Fixes https://github.com/influxdata/influxdb/issues/20249
(cherry picked from commit 2407077576)
* chore(changelog): update CHANGELOG.md for PR #20276
Clustering requires taking the hash of synthetic points, so
allow this function to work on anything with a HashID.
(cherry picked from commit d96c8fb125)
This commit updates the dependencies for influxdb to require v4.0.0-preview1 of
the jwt-go package. This required updating the go.mod and go.sum files as well
as any source file that directly imported that package.
Prior to this commit, the TestHandler_Query_Auth() tests would fail as it
checked for specific error strigns returned by the jwt-go package.
Version 4.0.0-preview1 of the package changed the verbiage of those errors a
bit. This patch updates the test to detect the new error string.
* feat: generate modern profiles
Prior to this commit, influxd was writing legacy profiling data which
often (always?) required an accompanying executable to use.
This commit instructs influxd to write profiles in the new format which
can be examined without a binary.
While we're at it, this commit also adds the allocs and threadcreate
profiles.
Finally, this patch also changes the format of the downloaded tar in the
following ways:
* The profiles are added to the profile/ directory -- so instead of
extracting the profiles into your current directory, they're placed in
a "profiles" directory.
* This commit adds the .pb.gz extension to each of the files since
they're gzipped protobuf files and not .txt.
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.
This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.
This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.
Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.
This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
Backport of #19566.
* feat(engine/tsm1): Add WritePointsWithContext()
Add WritePontsWithContext() and make WritePoints() a thin wrapper for
it.
The purpose is to add statistics context values that we'll use to
propagate the number of fields and points written to calls up the call
chain.
* feat(tsdb): Add WriteToShardWithContext()
When applied, this patch adds WriteToShardWithContext() and wraps it
with WriteToShard() to preserve the API.
The the purpose of this addition is to propagate a context.Context value
to Shard.WritePointsWithContext().
* feat(tsdb/shard): Add WritePointsWithContext()
The purpose of adding WritePointsWithContext() is to propage context
values down to engine code and propage statistics via the context.Value
up to callers.
This patch also adds values written statistics to the shard.
* feat(http): Gather values written stats
WritePointsWithContext() was added to propagate context values down to
the engine and communicate stats to the caller.
* feat(http): Gather values written stats
WritePointsWithContext() was added to propagate context values down to
the engine and communicate stats to the caller.
* refactor: Change MetricKey to ContextKey
This patch gives the type we're useing for context keys a better name.
This patch adds the [http.headers] subsection to the configuration file
that allows users to supply headers that will be returned in all HTTP
responses.
Applying this patch will:
* Add code to implement new configuration items.
* Add test to ensure configuration is properly parsed.
* Add test to ensure http response headers are set
* Update sample configuration file
Have AuthorizerIsOpen() assert if a given authizer has an
AuthorizeUnrestricted() method and if so, call that to provide the
result of AuthorizerIsOpen().
Otherwise we check if the supplied Authorizer is nil.
This preserves the fast-path for checking tag-level (and other) tsdb
operations.
This simplifies how we handle such authorizers by handling this case in
only one place.
The storage read service would return a nil cursor if there was no data
and no shards to read from. This modifies it to return an empty cursor
instead.
This commit adds a /api/v2/write endpoint that maps the supplied bucket
and org to a v1 database and retention policy.
* Add AllowedOrgs to httpd Config type.
* Add /api/v2/write handler
If a series was split by the encoder because of chunking and it was
reconstructed by the http handler, it would not reset the partial
indicator for the series to indicate if the series was still partial or
not. That meant that a result that returned more than the 10,000 values
in a single series with chunking disabled would say that the series was
partial, but it was not.
This fixes it so the handler now correctly sets the partial attribute of
the series to indicate if the series is still partial or not. This was
done when merging results, but was not done with series.
The flux in influxdb has been upgraded to use v0.33.2. A lot of
interfaces for the storage engine were changed during this so code had
to change to accomodate the new interfaces and remove the old ones.
Included in this commit is a patch file for the changes that were made.
A patch was generated for the following packages:
* `flux/stdlib/influxdata/influxdb`
* `storage/reads`
* `tsdb/cursors`
These are the three packages that are in common with version 2 of the
database and the first of these packages contains the specific
implementations that are used for version 1.
It is very possible that the next time we upgrade this, the patch will
not apply cleanly just like it wouldn't have applied cleanly to this
update. The patch is mostly meant to document exactly what changed
during the copy over to help ensure we don't forget things when adapting
the interfaces.
Add a patch file to hopefully make this easier in the future
When a Flux predicate is transformed to a Store.Read / GroupRead request,
the `_measurement` and `_field` keys are remapped to match 2.x internal
tag keys.
This change does not modify the 2.x behavior, but rather updates the
1.x mapping, so merging future updates from the 2.x storage/reads
package should have fewer conflicts.
This integrates the influxdb 1.x series to the latest version of Flux
and updates the code to use it. It also removes the dependency on
platform and copies the necessary code from storage into the 1.x series
so the dependency is unneeded.
The flux functions specific to 1.x have been moved to the same structure
that flux changed to with having a `stdlib` directory instead of a
`functions` directory. It also adds a `databases()` function that
returns the databases from the meta client.
This commit extends the Prometheus remote write endpoint to drop
unsupported Prometheus values, rather than reject the entire batch.
InfluxDB does not support NaN, -Inf or +Inf, but Prometheus does. The
remote write endpoint will now drop these and write valid values in the
provided batch.
If the user enabled write trace logging (`[http] write-tracing = true`)
then summaries of any dropped values within a batch will be logged.
If a batch of values contains any values that are subsequently dropped,
the returned status code will be `204`.
* Add AuthorizeDatabase API to QueryAuthorizer to verify a user has
appropriate access to the specified database
* Update serverFluxQuery handler to require a meta.User when auth is
enabled
* update Flux createFromSource and createBucketsSource dependencies to
require Authorizer when auth is enabled in configuration
* update createFromSource to verify read permissions for each bucket
specified in a Flux query
* update BucketsDecoder, which implements the buckets() Flux function,
to return buckets that the user has read or write permissions to
* add unit tests to verify authentication is required for Flux HTTP
requests when auth is enabled in configuration
Sam Arnold
davidby-influx
Yun Zhao
Ayan George
Christopher M. Wolff
Jonathan A. Sternberg
David Norton
Jakub Bednář
jsign
ricky
Gunnar Aasen
Adam
Stuart Carnie
Edd Robinson
Ahmed Ashour
Dilan Bellinghoven
Ben Wells