Commit Graph

371 Commits (60c4984f518bf39080ca5bd7c15b0772623be89b)

Author SHA1 Message Date
Stuart Carnie 8576d1df99 feat: Implement DELETE FROM and DROP MEASUREMENT
Closes #19416
2020-09-11 11:37:43 -07:00
Stuart Carnie 6f0cf049ca feat: Port shard precreation service from InfluxDB 1.x
Provides new configuration parameters:

```
--storage-shard-precreator-advance-period
--storage-shard-precreator-check-interval
```

Closes #19520
2020-09-09 13:34:22 -07:00
Stuart Carnie e265f60b55 fix: Improvements in response to PR feedback
* Pass context.Context to Service.Open
* Remove redundant comments
* Bind to retention.Config configuration to be consistent with 1.x
2020-09-09 10:32:06 -07:00
Stuart Carnie 7c2be6e780 feat: Port 1.x retention policy enforcement service
Configuration of the check interval is available via

```
--storage-retention-check-interval
```

Closes #19309
2020-09-09 10:32:06 -07:00
Stuart Carnie b3138d2ead
fix: Ensure TemporaryEngine returns new inner TSDB store 2020-09-01 10:50:44 -07:00
Stuart Carnie a607ac30b9
chore: Re-enable `TemporaryEngine` via `--e2e-testing` 2020-08-31 13:50:20 -07:00
Brett Buddin b917d8d9b0
chore(influxdb): Placate the linter. 2020-08-27 15:46:32 -04:00
Stuart Carnie dee8977d2c
chore: move v2/v1/tsdb → v2/tsdb 2020-08-26 10:46:47 -07:00
Stuart Carnie 06fac044e5
fix(storage): Implement SeriesCardinality, which is per-bucket
Fixed unit tests
2020-08-25 14:52:00 -07:00
Stuart Carnie 9bebacf92f
fix(reads): Fix query cursor read that caused cursor truncation.
Port of #17610

Closes #19310
2020-08-25 09:01:47 -07:00
Stuart Carnie 047ed7836a
fix(Makefile): Remove redundant make targets
Fixes #19407
2020-08-21 13:24:03 -07:00
Stuart Carnie c01a62d5d4
chore(task): Remove unused code 2020-08-20 16:59:26 -07:00
Stuart Carnie 0a644dceed
fix(storage): BufferedPointsWriter is for a single org / bucket only
Fixes experimental.to tests
2020-08-03 15:15:00 -07:00
Stuart Carnie d7434216e2
fix(storage): Use correct engine paths
Remove redundant repl.go after rebasing on 8dd8d1f79e
2020-08-03 09:21:04 -07:00
Stuart Carnie 21c3733464
fix(storage): meta.Client persists to kv and updates via BucketService
Includes initial migration to add metadata bucket
2020-08-03 09:21:02 -07:00
Stuart Carnie 89532f0c29
refactor: move v2/v1/models to v2/models 2020-08-03 09:20:51 -07:00
Stuart Carnie a71b43ff17
refactor: remove redundant files
* retention will be enforced by the InfluxDB 1.x engine
* storage will be a shim for `tsdb`, so don't need tests here
2020-08-03 09:20:50 -07:00
Stuart Carnie d32709ed16
chore: More cleanup 2020-08-03 09:20:50 -07:00
Stuart Carnie 7c225cb3ac
chore: More cleanup 2020-08-03 09:20:50 -07:00
Stuart Carnie 26c18ac0a5
chore: Converging on 1.x tsdb 2020-08-03 09:20:48 -07:00
Edd Robinson 2b175291be
refactor: WIP removing tsbd 2020-08-03 09:18:34 -07:00
Stuart Carnie a97ec1f34e
feat: Initial commit with working reads 2020-08-03 09:18:34 -07:00
Edd Robinson 89efcb3d74
refactor: writes work 2020-08-03 09:18:34 -07:00
Stuart Carnie e3060c291c
refactor: tsdb store builds an runs 2020-08-03 09:18:32 -07:00
Edd Robinson 5f426195bf
refactor: start inserting tsdb store into engine 2020-08-03 09:17:24 -07:00
Edd Robinson 8c4bb1d65f
refactor: strip contents of storage engine 2020-08-03 09:17:24 -07:00
Yiqun (Ethan) Zhang baa8323431
fix: group resultset produces 1 table per group (#18968)
This helps resolve #18098
2020-07-29 15:31:29 -05:00
jlapacik 17391d6924
fix: call Next on read cursor at end of loop (#19012) 2020-07-23 15:06:17 -07:00
Jonathan A. Sternberg 8678a22ee3
feat(storage/flux): add tests for min/max pushdowns (#19028) 2020-07-23 16:07:30 -05:00
Ben Johnson 1247e7449d
Merge pull request #19018 from benbjohnson/tsi-series-file-limiter
feat(tsdb): Add optional mincore limiter to TSI & Series File
2020-07-23 12:34:23 -06:00
Christopher M. Wolff 8b106bcee1
feat(storage): add mean aggregate array cursor and tests (#19019) 2020-07-23 09:49:53 -07:00
Ben Johnson 3cc2638bbf feat(tsi1): Add optional mincore limiter to TSI 2020-07-22 10:17:42 -06:00
Jonathan A. Sternberg f854762cac
fix(storage/flux): do not wait for empty tables in the windower (#19008)
When splitting windows into their own individual tables, do not wait for
empty tables to be read. Empty tables may never be read so waiting for
them can result in a deadlock.

This was found while writing a test case for in progress work and so a
test case for this condition will follow in a future commit.
2020-07-21 17:38:33 -05:00
jlapacik b7e2330fa3 test: offset tests for min and max read cursors 2020-07-20 12:23:37 -07:00
jlapacik 36987bec0f feat: add offset parameter to window cursors 2020-07-20 12:23:37 -07:00
Christopher M. Wolff c695be2f56
feat(storage/reads): add min/max aggregate array cursors (#18957) 2020-07-20 10:17:04 -07:00
Ben Johnson c28eb70856 feat(mincore): Add page fault limiter
This commit adds `mincore.Limiter` which throttles page faults caused
by mmap() data. It works by periodically calling `mincore()` to determine
which pages are not resident in memory and using `rate.Limiter` to
throttle accessing using a token bucket algorithm.
2020-07-17 09:37:31 -06:00
Brett Buddin 51406f4f62
feat(tsdb): SHOW TAG KEYS (no time) query using only TSI data. (#18905)
* feat(tsdb): SHOW TAG KEYS (no time) query using only TSI data.

* fix(tsdb): Allow for earlier return when scanning during show tag keys.

* fix(tsdb): Speed things up by using the key merger to reduce allocs.

* chore(tsm1): Fix golint.

* fix(tsdb): Remove sorting, because these keys should already be sorted.

* fix(tsdb): Remove dead code to placate the linter.
2020-07-09 18:01:42 -04:00
Stuart Carnie d2dd19b70e
feat(storage): InfluxQL schema APIs without time range
These changes introduce optimized schema APIs for InfluxQL that
utilize the time series index (TSI) exclusively for significant
performance gains.
2020-07-09 10:09:19 -07:00
Ben Johnson 6b0b4d7c4e fix(storage): Refactor LoggingPointsWriter to use BucketFinder
This commit changes the writer to use `storage.BucketFinder` which
is a subset of the `influxdb.BucketService` interface.
2020-07-02 12:46:43 -06:00
jlapacik 3b50d08cd9 fix: window tables are empty if cursor is empty
Previously windowed aggregate tables would construct the _start and
_stop arrays before checking if the underlying array cursor was empty.
The table would therefore report itself as non-empty even though
there were no points within the table's time range. Now we check if
the underlying array cursor is empty before we construct any output
arrays. If the cursor is empty, meaning the series is empty, the
table will be dropped.

Fixes https://github.com/influxdata/influxdb/issues/18704.
2020-07-01 16:58:50 -07:00
Ben Johnson 482442ecc1 feat(storage): Add logging points writer
This commit adds a LoggingPointsWriter which wraps a PointsWriter
and provides logging when a write fails.
2020-07-01 11:41:29 -06:00
George 96d84b9126
refactor: migrator and introduce Store.(Create|Delete)Bucket (#18570)
* refactor: migrator and introduce Store.(Create|Delete)Bucket

feat: kvmigration internal utility to create / managing kv store migrations

fix: ensure migrations applied in all test cases

* chore: update kv and migration documentation
2020-07-01 12:08:20 +01:00
jlapacik c86ce5b325 feat(storage): read cursor for window last 2020-06-29 13:12:47 -07:00
jlapacik 81681fc7dc feat(storage): turn on capability for last selection 2020-06-29 13:12:47 -07:00
Brett Buddin 0c268e205b
fix(storage): Push-down a predicate to match tags for SHOW MEASUREMENT calls (#18740)
* fix(storage): Push-down a predicate to match tags for SHOW MEASUREMENTS calls.

* chore: Address feedback.

* fix(tsm1): Split behavior based on existence of predicate for show measurements.

* fix(tsm1): Allow parenthesis expression on the LHS of a predicate.

* fix(tsm1): Create a separate tag predicate verifier that rejects negative comparisons.

* fix(tsm1): Additional test cases for show measurements with predicate.
2020-06-29 14:31:54 -04:00
Jonathan A. Sternberg ec6e610cd5
fix(storage/flux): use the correct constant for CountKind in the reader (#18755) 2020-06-26 16:02:53 -05:00
Jonathan A. Sternberg 58bd5b9854
fix(storage/flux): the count aggregate will fill in zero values instead of null (#18735) 2020-06-26 15:43:03 -05:00
jlapacik f1c557ccd8 fix(storage): remove libflux dependency from storage
Storage should not have a dependency on libflux. This commit removes
some constants that were used from the flux universe package and
replaces them with local copies so that storage no longer has a
dependecy on the universe package.
2020-06-26 09:50:39 -07:00
jlapacik 5e4dd3f77a feat: flux table iterators for windowed selectors 2020-06-25 14:37:07 -07:00