Commit Graph

49 Commits (c889d0f22bf3d3c4a207d4800bc9f8fb1ccf6a08)

Author SHA1 Message Date
Sam Arnold b970e359dc
feat: remaining storage metrics from OSS engine (#22938)
* fix: simplify disk size tracking

* refactor: EngineTags in tsdb package

* fix: fewer compaction buckets and dead code removal

* feat: shard metrics

* chore: formatting

* feat: tsdb store metrics

* feat: retention check metrics

* chore: fix go vet

* fix: review comments
2021-12-02 09:01:46 -05:00
Dane Strandboge f4e9ae94ca
build: upgrade protobuf library (#22654) 2021-11-02 16:00:54 -05:00
Sam Arnold 0a6f562a44
fix: do not allow shard creation to create overlapping shards (#22604) 2021-10-01 08:07:34 -04:00
William Baker ec7841b355
feat: support for flux cardinality query (#22441)
* feat: works with custom iterator

* feat: works with existing iterators

* chore: cleanup

* test: consistent assertions for tests

* fix: better log message if trying to filter on the value of a field key

* fix: comment for handling boolean literal; handle false boolean as well

* fix: make time range checking inclusive
2021-09-13 13:20:56 -06:00
Sam Arnold 5015297d40
fix: more expressive errors (#22448)
* fix: more expressive errors

Closes #22446

* fix: server only logging for untyped errors

* chore: fix formatting
2021-09-13 15:12:35 -04:00
William Baker 3e275a123d
feat: multi-measurement query optimization (#22301)
* feat: multi-measurement query optimization
2021-09-13 13:00:08 -06:00
William Baker 1482bdfa38
feat: partial support for series cardinality flux query (#22378) 2021-09-09 12:37:11 -06:00
Daniel Moran dc3b501298
refactor: rename Lock/Unlock on KV stores to RLock/RUnlock (#22357) 2021-08-31 17:03:54 -04:00
Daniel Moran 22c6b78c62
build: fix generation when vendoring dependencies (#22089) 2021-08-06 15:06:36 -04:00
William Baker 8e80798c33
fix: field metaqueries take fast path if predicate is only on `_measurement` (#21962)
* fix: field metaqueries take fast path if predicate is only on `_measurement`

* chore: update CHANGELOG

* test: add test for fields with measurement predicate
2021-07-29 13:57:06 -06:00
Daniel Moran 858be90139
fix: regenerate protos using gogo 1.3.2 (#21880)
* Regenerate protos using gogo 1.3.2
* Add protos to generate, add checkgenerate to CI
* Address proto warning
* Add generator tooling to Makefile
* Delete recursive Makefiles, simplify generation run by goreleaser
* Use env bash for fetch-ui-assets
* Add static-data to clean target
2021-07-20 15:11:46 -04:00
William Baker b0ea3b6675
feat: backup and restore sqlite database (#21584)
* feat: new metadata backup endpoint

* feat: added restore/sql API endpoint

* fix: content-type is multipart/mixed, part names are kv and sql

* fix: changed multipart manifest to buckets and made it .json

* feat: added lock for backing up sqlite and bolt dbs

* fix: use read lock instead of write lock on kv during backup

* fix: use filepath.Join for temp dirs
2021-06-02 19:07:53 -04:00
Daniel Moran 00420fb54c
fix(influxql): make meta queries respect query timeout (#21545)
Co-authored-by: davidby-influx <dbyrne@influxdata.com>
2021-05-24 21:10:53 -04:00
Sam Arnold d89b9cbaca
refactor: last cursor optimization function name (#21353)
Closes: #21330
2021-05-13 14:44:00 -04:00
Faith Chikwekwe 7bde3413b3
feat(query): enable min/max pushdown (#20994)
* feat(query): enable min/max pushdown

* fix(query): fix the group last pushdown to use descending cursors

* test(storage): add read group test with no agg

Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
2021-04-20 12:56:43 -07:00
Sean Brickley f862f8cb1f
fix(storage): Detect need for descending cursor in WindowAggregate (#21140)
* fix(storage): Detect need for descending cursor in WindowAggregate

* chore: Format + comments

* chore: PR cosmetic feedback (#21141)

Co-authored-by: Phil Bracikowski <pbracikowski+git@influxdata.com>

* chore: rename testcase and fix comments

Co-authored-by: Phil Bracikowski <pbracikowski+git@influxdata.com>
2021-04-05 18:33:07 -04:00
Daniel Moran f6b073348b
build: delete vendored copy of gogo protos (#21098) 2021-03-30 14:26:05 -04:00
Daniel Moran 00afd95cb7
refactor: automated move of errors and id from root to kit (#21101)
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-03-30 14:10:02 -04:00
Daniel Moran b1283bc998
build: fix 'go generate' and regenerate the universe (#21089) 2021-03-30 10:32:14 -04:00
Daniel Moran ba4099ddb9
feat(cmd/influx): allow setting shard-group durations for buckets via API and CLI (#20911)
Co-authored-by: Alexander Savinykh <asavinykh@nbrx.de>
2021-03-11 14:51:03 -05:00
Daniel Moran 5fc9240dc5
refactor: delete unused v1 monitoring service (#20818) 2021-03-02 12:39:38 -05:00
Daniel Moran 727a7b58c1
test: replace influxlogger with zaptest logger (#20589) 2021-02-11 10:12:39 -05:00
Daniel Moran 5b29ef7060
fix(v1): optimize shard lookup in groups containing only one shard (#20201)
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
2020-11-30 15:16:03 -05:00
Daniel Moran 15b9531273
fix: correct various typos (#19987)
Co-authored-by: kumakichi <xyesan@gmail.com>
2020-11-11 13:54:21 -05:00
Ben Johnson 419b0cf76b feat: Implement full restore 2020-11-05 10:05:01 -07:00
Ben Johnson 23679c2375 feat: Implement backup/restore CLI subcommands. 2020-11-05 10:05:01 -07:00
Christopher M. Wolff 14443a5c39
fix: return empty iterator instead of null in tagValues (#19894)
Fixes influxdata/flux#3300.
2020-11-04 08:09:27 -08:00
Christopher M. Wolff 5a60488729
fix: make tagKeys and tagValues work for edge cases involving _field (#19856) 2020-10-30 10:37:08 -07:00
Faith Chikwekwe 5e1cdda597 feat(storage): enable window agg mean pushdown 2020-10-27 12:53:54 -07:00
Christopher M. Wolff 790165b05a
refactor: allow newIndexSeriesCursor() to accept an influxql.Expr (#19833)
* refactor: allow newIndexSeriesCursor() to accept an influxql.Expr

In order to let TagKeys and TagValues get the right answer,
sometimes they will need to examine their predicate and
see if using the index is possible, or if a block scan is needed.
For those cases we want to examine the predicate before creating
the index series cursor. This change allows us to create an
index series cursor with an already-deserialized influxql.Expr.
2020-10-27 09:31:46 -07:00
vlastahajek 8763bb1af9 feat(upgrade): upgrade databases 2020-10-08 09:39:30 -07:00
Jonathan A. Sternberg 025319c387
fix(services/storage): multi measurement queries return all applicable series (#19566)
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.

This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.

This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.

Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.

This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
2020-09-17 14:28:24 -05:00
Stuart Carnie 7e16da8f74 chore: Consolidate `v1/internal` to `internal`
Take the opportunity to combines duplicated types in both packages
under the root `internal` package
2020-09-11 11:37:43 -07:00
Stuart Carnie 6f0cf049ca feat: Port shard precreation service from InfluxDB 1.x
Provides new configuration parameters:

```
--storage-shard-precreator-advance-period
--storage-shard-precreator-check-interval
```

Closes #19520
2020-09-09 13:34:22 -07:00
Stuart Carnie e265f60b55 fix: Improvements in response to PR feedback
* Pass context.Context to Service.Open
* Remove redundant comments
* Bind to retention.Config configuration to be consistent with 1.x
2020-09-09 10:32:06 -07:00
Stuart Carnie 7c2be6e780 feat: Port 1.x retention policy enforcement service
Configuration of the check interval is available via

```
--storage-retention-check-interval
```

Closes #19309
2020-09-09 10:32:06 -07:00
Stuart Carnie 7f94b0d5db
fix: Rewrite predicate to remove _field from schema physical operations
Fixes #19488
2020-09-02 11:10:40 -07:00
Stuart Carnie b3138d2ead
fix: Ensure TemporaryEngine returns new inner TSDB store 2020-09-01 10:50:44 -07:00
Brett Buddin b917d8d9b0
chore(influxdb): Placate the linter. 2020-08-27 15:46:32 -04:00
Stuart Carnie dee8977d2c
chore: move v2/v1/tsdb → v2/tsdb 2020-08-26 10:46:47 -07:00
Stuart Carnie afd5120221
fix(storage): Ensure _field tag key has special handling
Fixes #19120
2020-08-21 13:06:00 -07:00
Stuart Carnie d67b034868
fix(storage): Fix panic when no shards exists 2020-08-03 10:26:32 -07:00
Stuart Carnie 21c3733464
fix(storage): meta.Client persists to kv and updates via BucketService
Includes initial migration to add metadata bucket
2020-08-03 09:21:02 -07:00
Stuart Carnie e772333b6b
feat(meta): Use kv.Store abstraction to read and write to files or kv 2020-08-03 09:20:51 -07:00
Stuart Carnie 89532f0c29
refactor: move v2/v1/models to v2/models 2020-08-03 09:20:51 -07:00
Stuart Carnie 7c225cb3ac
chore: More cleanup 2020-08-03 09:20:50 -07:00
Stuart Carnie 26c18ac0a5
chore: Converging on 1.x tsdb 2020-08-03 09:20:48 -07:00
Stuart Carnie a97ec1f34e
feat: Initial commit with working reads 2020-08-03 09:18:34 -07:00
Stuart Carnie 92efddbfbe
chore(tsdb): Initial commit of tsdb package
* pulls in 1.x tsdb, compiles and passes test
2020-08-03 09:17:23 -07:00