* feat: update protos
This PR updates the proto files to use protoc-gen-go v1.34.1
and protoc to use v5.29.2
* feat: update makefile target to use v1.34.1 of protoc-gen-go
When querying data before 1970-01-01 (UNIX time 0)
validateArgs would set start to -in64 max and end to int64 max.
closes https://github.com/influxdata/influxdb/issues/24669
Co-authored-by: Paul Hegenberg <paul.hegenberg@gmail.com>
* fix: prevent retention service from hanging (#25055)
Fix issue that can cause the retention service to hang waiting on a
`Shard.Close` call. When this occurs, no other shards will be deleted
by the retention service. This is usually noticed as an increase in
disk usage because old shards are not cleaned up.
The fix adds to new methods to `Store`, `SetShardNewReadersBlocked`
and `InUse`. `InUse` can be used to poll if a shard has active readers,
which the retention service uses to skip over in-use shards to prevent
the service from hanging. `SetShardNewReadersBlocked` determines if
new read access may be granted to a shard. This is required to prevent
race conditions around the use of `InUse` and the deletion of shards.
If the retention service skips over a shard because it is in-use, the
shard will be checked again the next time the retention service is run.
It can be deleted on subsequent checks if it is no longer in-use. If
the shards is stuck in-use, the retention service will not be able to
delete the shards, which can be observed in the logs for manual
intervention. Other shards can still be deleted by the retention service
even if a shard is stuck with readers.
This is a port of ad68ec8 from master-1.x to main-2.x.
closes: #25076
(cherry picked from commit b4bd607eef)
This adds locking to the load method and renames it to Reload(). This
method replaces the cached data from the underlying kv.Store and needs a
write lock. The restore api uses it and may have been an issue with
concurrent writes into the cached data during a restore.
* fixes#24895
Under certain circumstances, the retention service can fail to delete shards from
the store in a timely manner. When the shard groups are pruned based on age, this
leaves orphaned shard files on the disk. The retention service will then not attempt
to remove the obsolete shard files because the meta store does not know about them.
This can cause excessive disk space usage for some users.
This corrects that by requiring shards files be deleted before they can be removed
from the meta store.
fixes: #24529
(cherry picked from commit 7bd3f89d18)
closes https://github.com/influxdata/influxdb/issues/24545
Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com>
* fix: remove nats for scraper processing
Scrapers now use go channels instead of NATS and interprocess communication.
This should fix#23085 .
Additionally, found and fixed#23106 .
* chore: fix formatting
* chore: fix static check and go.mod
* test: fix some flaky tests
* fix: mark NATS arguments as deprecated
* feat: works with custom iterator
* feat: works with existing iterators
* chore: cleanup
* test: consistent assertions for tests
* fix: better log message if trying to filter on the value of a field key
* fix: comment for handling boolean literal; handle false boolean as well
* fix: make time range checking inclusive
* fix: field metaqueries take fast path if predicate is only on `_measurement`
* chore: update CHANGELOG
* test: add test for fields with measurement predicate
* Regenerate protos using gogo 1.3.2
* Add protos to generate, add checkgenerate to CI
* Address proto warning
* Add generator tooling to Makefile
* Delete recursive Makefiles, simplify generation run by goreleaser
* Use env bash for fetch-ui-assets
* Add static-data to clean target
* feat: new metadata backup endpoint
* feat: added restore/sql API endpoint
* fix: content-type is multipart/mixed, part names are kv and sql
* fix: changed multipart manifest to buckets and made it .json
* feat: added lock for backing up sqlite and bolt dbs
* fix: use read lock instead of write lock on kv during backup
* fix: use filepath.Join for temp dirs
* feat(query): enable min/max pushdown
* fix(query): fix the group last pushdown to use descending cursors
* test(storage): add read group test with no agg
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
* fix(storage): Detect need for descending cursor in WindowAggregate
* chore: Format + comments
* chore: PR cosmetic feedback (#21141)
Co-authored-by: Phil Bracikowski <pbracikowski+git@influxdata.com>
* chore: rename testcase and fix comments
Co-authored-by: Phil Bracikowski <pbracikowski+git@influxdata.com>
* refactor: allow newIndexSeriesCursor() to accept an influxql.Expr
In order to let TagKeys and TagValues get the right answer,
sometimes they will need to examine their predicate and
see if using the index is possible, or if a block scan is needed.
For those cases we want to examine the predicate before creating
the index series cursor. This change allows us to create an
index series cursor with an already-deserialized influxql.Expr.
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.
This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.
This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.
Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.
This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
devanbenz
davidby-influx
Geoffrey Wossum
Phil Bracikowski
Jeffrey Smith II
Dane Strandboge
Jonathan A. Sternberg
Sam Arnold
Dane Strandboge
William Baker
Daniel Moran
Faith Chikwekwe
Sean Brickley
Ben Johnson
Christopher M. Wolff
Faith Chikwekwe
vlastahajek
Stuart Carnie