Commit Graph

12784 Commits (80cd5e63af7c39ff111c91c7fdbcf0eb8bf49655)

Author SHA1 Message Date
Jason Wilder 80cd5e63af Optimize DeleteSeriesRange
This removes more allocations and speeds up some critical sections.
2017-11-13 09:02:10 -07:00
Jason Wilder aee395d3bd Make DeleteSeriesRange take SeriesIterator 2017-11-13 09:02:10 -07:00
Jason Wilder f893beb6d8 Use MeasurementSeriesKeysByExprIterator for deletes 2017-11-13 09:02:10 -07:00
Jason Wilder 16d1f4309b Extract MeasurementSeriesKeysByExprIterator 2017-11-13 09:02:10 -07:00
Jason Wilder 000768371f Optimized deletes in TSM index
This optimizes how deletes are processed to reduce memory usage
and improve efficiency.
2017-11-13 09:02:08 -07:00
Jason Wilder eebd88f825 Don't write tombstones for keys that do not exist
This filters out keys that do not exist in a TSM file to avoid
writing entries that would end up being ignored when applied.
2017-11-13 08:50:07 -07:00
Jason Wilder ca4998f29e Fix pid test to ignore local data
If you have lots of data stored locally, this test takes a while to
complete since it loads it all up from the users home dir.
2017-11-13 08:50:07 -07:00
Jason Wilder 88c48ec78b Rework Engine.DeleteSeriesRange to avoid allocations
This removes the containsSeries func which ends up creating a map
sized to the slice of keys passed in.  This doesn't scale well to
high cardinalities and creates a lot of garbage.
2017-11-13 08:50:07 -07:00
Jason Wilder cb658774bb Reduce allocations when reading tombstone v4 2017-11-13 08:50:07 -07:00
Jason Wilder 1c65bb3bb1 Fix leaked goroutine in FileStore.WalkKeys
If fn returned and error, the goroutines sending keys from TSM files
would get blocked indefinitely and leak.
2017-11-13 08:50:07 -07:00
Jason Wilder b0c7a44eaa Adjust min/max time to work in the engine
The query language min and max times are slighly different than the
values used in the engine.  This allows faster codes to be used when
the whole time range is deleted.
2017-11-13 08:50:07 -07:00
Jason Wilder 2959b8d2eb Make BatchDeleters concurrent 2017-11-13 08:50:07 -07:00
Jason Wilder 5a775c50d9 Add DeleteRangeWith
This is a version of DeleteRange that take a func predicate to determine
whether a series key should be deleted or not.  This avoids the large
slice allocations with higher cardinalities.
2017-11-13 08:50:07 -07:00
Jason Wilder de5592cda9 Extract ParseKeyBytes from ParseKey
Allows callers to use []byte and avoid a string allocation
2017-11-13 08:48:03 -07:00
Jason Wilder 6b19d2b673 Add BatchDeleters type 2017-11-13 08:48:03 -07:00
Jason Wilder 9ac83601cf Use BatchDeleter in FileStore 2017-11-13 08:48:03 -07:00
Jason Wilder 4ed19348fd Add a BatchDelete capability to TSMReader 2017-11-13 08:48:03 -07:00
Jason Wilder 44e782f173 Store temporary tombstones on disk
This removes the in-memory tombstone buffer when writing tombstones
which eliminates one source of large memory spikes during deletes.
2017-11-13 08:48:03 -07:00
Jason Wilder bd15d37c70 Extract commit func 2017-11-13 08:48:03 -07:00
Jason Wilder 1e56894097 Extract writeTombstone func 2017-11-13 08:48:03 -07:00
Jason Wilder b958c68ce5 Avoid re-reading tombstones when writing new ones
This adds a new v4 tombstone format that extends the v3 format
by allowing multiple batches of tombstones to be written without
having to re-read all the existing tombstones.  This uses gzip
multi stream to append multiple v3 files together to create a v4
format.
2017-11-13 08:48:03 -07:00
Jason Wilder 17bae05370 Allow buffering tombstones before writing to disk 2017-11-13 08:48:03 -07:00
Jonathan A. Sternberg 7cae889b13
Merge pull request #9055 from influxdata/js-update-zap-logger
Update the zap logger dependency
2017-11-10 22:01:00 -06:00
Jonathan A. Sternberg 0b7c56bcd8 Update the zap logger dependency
The previous sha was taken from a revision on a devel branch that I
thought would continue staying in the tree after it was merged. That
revision was rebased away and the API was changed for the logger.

This updates the usage of the logger and adds a simple package for
constructing the base logger.

The 1.0 version of zap changed the format of the default console logger
so this change moves over to this new logger instead of attempting to
retain backwards compatibility with the old format.
2017-11-10 16:27:16 -06:00
Stuart Carnie ef42c367f0
Merge pull request #9091 from influxdata/sgc-ifql
don't panic here; nil cursor is handled for now. fixes #9090
2017-11-10 11:25:45 -07:00
Stuart Carnie 57677be010 don't panic here; nil cursor is handled for now. fixes #9090 2017-11-10 11:21:31 -07:00
Jason Wilder fe35aee4c4
Merge pull request #9086 from influxdata/jw-points-panic
Fix panic: runtime error: slice bounds out of range
2017-11-09 08:50:55 -07:00
Jason Wilder ed246db55a Fix panic: runtime error: slice bounds out of range
Fixes #8538
2017-11-08 17:00:25 -07:00
Edd Robinson 0614ebb1d1
Merge pull request #9085 from influxdata/er-iterator-pb
Pass through SystemIterator in PB
2017-11-08 20:28:45 +00:00
Edd Robinson 6443355467 Pass through SystemIterator in PB 2017-11-08 19:57:16 +00:00
Edd Robinson 25d1976e95
Merge pull request #9083 from influxdata/er-show-tag-key-vals
SHOW TAG {KEYS, VALUES} across RPs
2017-11-08 16:49:57 +00:00
Edd Robinson d581aee285 Ensure all retention policies queried 2017-11-08 16:27:57 +00:00
Edd Robinson 59c4e4b1bc Skip shards we don't have 2017-11-08 13:33:52 +00:00
Jonathan A. Sternberg 8e6c208b55
Merge pull request #9080 from influxdata/js-influxql-data-type
Update influxql so DataType constants are the type DataType
2017-11-07 20:05:23 -06:00
Jonathan A. Sternberg b09396f977 Update influxql so DataType constants are the type DataType 2017-11-07 18:34:53 -06:00
Edd Robinson 081d0649a6
Merge pull request #9078 from influxdata/er-race-tests
Skip cardinality tests in race mode
2017-11-07 19:52:30 +00:00
Ben Johnson af9931198c
Merge pull request #9073 from benbjohnson/show-query-perf-2
Improve SHOW TAG KEYS performance.
2017-11-07 12:35:38 -07:00
Edd Robinson d03f1a68d6
Merge pull request #9077 from influxdata/er-go19-appveyor
Update AppVeyor to use Go 1.9
2017-11-07 19:17:30 +00:00
Edd Robinson 5a85f0d239 Skip cardinality tests in race mode 2017-11-07 18:55:43 +00:00
Ben Johnson 156f25ac23
Improve SHOW TAG KEYS performance. 2017-11-07 10:59:19 -07:00
Stuart Carnie f43d42e961
Merge pull request #9076 from influxdata/sgc-yamux
update yamux
2017-11-07 10:50:56 -07:00
Edd Robinson b212d927c0 Update AppVeyor to use Go 1.9 2017-11-07 17:42:30 +00:00
Stuart Carnie b653b5ffcf update yamux 2017-11-07 10:35:59 -07:00
Jonathan A. Sternberg 9e7427f319
Merge pull request #9075 from influxdata/js-pid-test-env-vars
Use the isolated temp dir for the test pid server
2017-11-07 11:27:12 -06:00
Jonathan A. Sternberg e13ad18160 Use the isolated temp dir for the test pid server 2017-11-07 10:40:46 -06:00
Jonathan A. Sternberg 0b5178ca73
Merge pull request #9074 from influxdata/js-fix-cmd-integration-test
Use the appropriate environment variable for the wal directory
2017-11-07 10:30:02 -06:00
Jonathan A. Sternberg 0a73dcec96 Use the appropriate environment variable for the wal directory
The integration test was intended to use the temporary directory for the
files that were created, but `INFLUXDB_WAL_DIR` is supposed to be
`INFLUXDB_DATA_WAL_DIR`.
2017-11-07 10:07:33 -06:00
Edd Robinson 126db1b5f9
Merge pull request #9068 from influxdata/er-show-query-perf
Add time support to some meta queries
2017-11-07 15:48:58 +00:00
Edd Robinson e762da9aca Fix race on store close
There was a very small window where it was possible to deadlock during
the close of the Store. When closing, the Store waited on its Waitgroup
under a `Lock`. Naturally, all other goroutines must have been in a
position to call `Done` on the `Waitgroup` before the `Wait` call in
`Close` would return.

For the goroutine running the `monitorShards` method it was possible
that it would be unable to do this. Specifically, if the `monitorShards`
goroutine was jumping into the `t.C` case as the `Close()` goroutine was
acquiring the `Lock` then then `monitorShards` goroutine would be unable
to acquire the `RLock`. Since it would also be unable to progress around
its loop to jump into the `s.closing` case, it would be unable to call
`Done` on the `WaitGroup` and we would have a deadlock.

This was identified during an AppVeyor CI run, though I was unable to
reproduce this locally.
2017-11-07 15:26:46 +00:00
Edd Robinson e69217440b Remove noisy log line 2017-11-07 10:50:49 +00:00