Closes#22033
* fix: tsi index should compact old log files that are too large
* chore: run automated formatter
* chore: update changelog
* fix: review comments
(cherry picked from commit b64c2c3dcf)
The dgrijalva/jwt-go project is no longer maintained[1] and they have
transferred ownership to golang-jwt/jwt[2][3][4]. We should move to the
supported golang-jwt/jwt.
The following was performed:
1. update services/httpd/handler*.go to import golang-jwt/jwt
2. revert testcase string comparison changes from 225bcecd (back to v3)
2. go mod edit -require github.com/golang-jwt/jwt@v3.2.1+incompatible
3. go mod edit -droprequire github.com/dgrijalva/jwt-go
4. go mod tidy # see note
5. go clean ./... && go build ./...
6. go test ./...
Note: 'go mod tidy' had unrelated changes (perhaps it wasn't run in
recent commits) so I removed the unrelated delta to keep this PR focused
on the dgrijalva/jwt-go to golang-jwt/jwt changes.
References:
[1] dgrijalva/jwt-go#462
[2] dgrijalva/jwt-go#463
[3] https://github.com/dgrijalva/jwt-go/blob/master/README.md
[4] https://github.com/golang-jwt/jwt
[5] https://github.com/influxdata/influxdb/issues/21927
* chore: harmonize OSS and Enterprise subscriber initialization
* refactor: chanwriter close blocks until chanwriter is flushed
* refactor: remove redundant state
* test: add test for blocked subscriber during subscription update
* fix: blocked subscriber does not fail subscription update
* fix: only iterate matching subscriptions
* fix: use serialized points in subscriber queues
* feat: add total-buffer-bytes config parameter to subscriptions
* fix: put subscription serialization on the write path
* fix: subscription service only needs regular mutex, not RWMutex
* fix: review comments
* chore: update changelog
(cherry picked from commit 85abb3af36)
* build(flux): update flux to v0.113.0
* feat(query): enable min/max pushdown
* fix(query): fix the group last pushdown to use descending cursors
* test(storage): add read group test with no agg
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
Co-authored-by: Christopher M. Wolff <chris.wolff@influxdata.com>
Co-authored-by: Faith Chikwekwe <faithchikwekwe01@gmail.com>
* fix(storage): Detect need for descending cursor in WindowAggregate
* test: add tests for bare aggregate pushdowns
* test: add test cases for window aggregate pushdowns
* test: add tests for aggregate-by-time (aggregateWindow pushdown)
Co-authored-by: Sean Brickley <sean@wabr.io>
* feat: add cursors and readers for window aggregates
* fix: backport fix + tests for race condition in flux tag cache
* test: port 2.x test for array_cursor
* test: add script to run flux tests
* feat(flux): enable test capabilities in Flux controller
* feat(flux): add MergeFiltersRule
* build: bump existing Dockerfiles to go 1.15
* build: add flux tests to CI
* refactor: allow for overriding tcp.Mux logger
* build: upgrade to Flux v0.111.0
The HTTP handler logs URLs, but not body values for POST requests.
This means that queries sent by GET are logged, because the query
is in the URL, but queries sent by POST have no query text in the
log. This feature prints all the key-value pairs in the post body,
which includes the query text, except passwords, which are redacted.
Closes https://github.com/influxdata/influxdb/issues/20653
The storage read service would return a nil cursor if there was no data
and no shards to read from. This modifies it to return an empty cursor
instead.
(cherry picked from commit 2852bf0399)
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.
This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.
This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.
Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.
This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
Backport of #19566.
(cherry picked from commit ceead88bd5)
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
* Revert "fix(error): unsupported value: +Inf" error not handled gracefully (#20250)"
This reverts commit 6ac0bb3fe3.
* fix: No infinite recursion on write error
If there is some error writing to the response writer, we
would previous have infinite recursion.
Re-closes https://github.com/influxdata/influxdb/issues/20249
* chore: Update flux to 0.67
* chore: Builds against 0.68 flux
* chore: Builds against 0.80.0
* chore: Builds against 0.90.0
* chore: Everything builds on latest flux
* chore: goimports fixed
* chore: fix tests locally
* chore: fix CI dockerfiles
* chore: clean up some unused code
* chore: remove flux repl and Spec in flux query json
* chore: port flux end to end tests from 2.x
* chore: fix up goimports
* chore: remove 32 bit build support
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.
Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries
https://github.com/influxdata/influxdb/issues/20736
Extending the context instead of fixing the API breaks type safety.
For tracking the number of points / values written, it is much clearer
to pass an explicit tracker.
JSON marshalling errors should be returned properly formatted in JSON
like other errors. This fix formats marshalling errors the same way
influxdb formats other query errors.
Fixes https://github.com/influxdata/influxdb/issues/20249
* fix: Upgrade version of jwt-go package to v4.0.0
This commit updates the dependencies for influxdb to require v4.0.0-preview1 of
the jwt-go package. This required updating the go.mod and go.sum files as well
as any source file that directly imported that package.
Prior to this commit, the TestHandler_Query_Auth() tests would fail as it
checked for specific error strigns returned by the jwt-go package.
Version 4.0.0-preview1 of the package changed the verbiage of those errors a
bit. This patch updates the test to detect the new error string.
Prior to this commit, we had our own recursive file walker which
required a condition based on if s.Config.TypesDB pointed to a directory
or a regular file.
This commit replaces our own readdir() with filepath.Walk() and treats
recursing directories and loading one file as a single case. This
simplifies the code quite a bit.
* feat: generate modern profiles
Prior to this commit, influxd was writing legacy profiling data which
often (always?) required an accompanying executable to use.
This commit instructs influxd to write profiles in the new format which
can be examined without a binary.
While we're at it, this commit also adds the allocs and threadcreate
profiles.
Finally, this patch also changes the format of the downloaded tar in the
following ways:
* The profiles are added to the profile/ directory -- so instead of
extracting the profiles into your current directory, they're placed in
a "profiles" directory.
* This commit adds the .pb.gz extension to each of the files since
they're gzipped protobuf files and not .txt.
* feat(engine/tsm1): Add WritePointsWithContext()
Add WritePontsWithContext() and make WritePoints() a thin wrapper for
it.
The purpose is to add statistics context values that we'll use to
propagate the number of fields and points written to calls up the call
chain.
* feat(tsdb): Add WriteToShardWithContext()
When applied, this patch adds WriteToShardWithContext() and wraps it
with WriteToShard() to preserve the API.
The the purpose of this addition is to propagate a context.Context value
to Shard.WritePointsWithContext().
* feat(tsdb/shard): Add WritePointsWithContext()
The purpose of adding WritePointsWithContext() is to propage context
values down to engine code and propage statistics via the context.Value
up to callers.
This patch also adds values written statistics to the shard.
* feat(http): Gather values written stats
WritePointsWithContext() was added to propagate context values down to
the engine and communicate stats to the caller.
* feat(http): Gather values written stats
WritePointsWithContext() was added to propagate context values down to
the engine and communicate stats to the caller.
* refactor: Change MetricKey to ContextKey
This patch gives the type we're useing for context keys a better name.
This patch adds the [http.headers] subsection to the configuration file
that allows users to supply headers that will be returned in all HTTP
responses.
Applying this patch will:
* Add code to implement new configuration items.
* Add test to ensure configuration is properly parsed.
* Add test to ensure http response headers are set
* Update sample configuration file
Sam Arnold
Dane Strandboge
Jamie Strandboge
davidby-influx
Daniel Moran
davidby-influx
Ayan George
Pavel Zavora
David Norton
Tristan Su