1443b22379
auth: add series auth to 'show tag values'
2017-09-27 20:01:18 -04:00
ea104596f0
Implement TSI index versioning
...
This commit adds a basic TSI versioning scheme, by adding a Version field
to an index's MANIFEST file.
Existing TSI indexes will not have this field present in their MANIFEST
files, and thus will be deemed incomatible with the current version.
Users with existing TSI indexes will be able to remove them, and convert the
resulting inmem indexes to the current version of a TSI index using the
influx_inspect tooling.
2017-09-22 17:59:39 +01:00
44691847e9
Merge branch 'master' into er-8678-tsi1-where
2017-09-22 16:54:49 +01:00
d5d9f9acfe
Remove debug line
2017-09-11 15:31:28 -06:00
ee4d3c7b3d
Invalidate all bloom filters.
2017-09-11 15:29:26 -06:00
3c2487b97a
Clean up tsi bloom filter invalidation.
2017-09-11 15:29:26 -06:00
6af936ee61
Fix bloom filter invalidation.
2017-09-11 15:29:26 -06:00
a40b2bb210
Simplify bloom filter invalidation.
2017-09-11 15:29:26 -06:00
408a78d904
Increase size of SeriesBlock partition
2017-09-11 15:29:26 -06:00
0ec2736f23
Incrementally rebuild tsi bloom filters.
2017-09-11 15:29:25 -06:00
a5a2957567
Reduce allocation in log_file
2017-09-11 15:29:25 -06:00
a9e89ede75
Reduce lock contenton on Index
...
Stat and Size are read-only and can take an RLock.
2017-09-11 15:26:25 -06:00
94e229ff59
Merge branch 'master' into jw-drop-series
2017-09-08 15:34:32 -06:00
4fb35b373b
auth: apply series auth to TSI
2017-09-08 09:09:53 -04:00
a8d9eeef36
Reduce lock contention when deleting high cardinality series
...
Deleting high cardinality series could take a very long time, cause
write timeouts as well as dead lock the process. This fixes these
issue to by changing the approach for cleaning up the indexes and
reducing lock contention.
The prior approach delete each series and updated every index (inmem)
during the delete. This was very slow and cause the index to be locked
while it items in a slice were removed one by one. This has been changed
to mark series as deleted and then rebuild the index asynchronously which
speeds up the process.
There was also a dead lock that could occur when deleing the field set.
Deleting the field set held a write lock and the function it invoked under
the lock could try to take a read lock on the field set. This would then
deadlock. This approach was also very slow and caused time out for writes.
It now uses faster approach that checks for the existing of the measurment
in the cache and filestore which does not take write locks.
2017-09-07 11:36:02 -06:00
732a0c2eaa
Merge pull request #8769 from influxdata/jl-map-cleanup
...
cleanup: remove poor usage of ',ok' with maps
2017-08-31 09:18:42 -04:00
a95647b720
cleanup: remove poor usage of ',ok' with maps
...
There are several places in the code where comma-ok map retrieval was
being used poorly. Some were benign, like checking existence before
issuing an unconditional delete with no cleanup. Others were potentially
far more serious: assuming that if 'ok' was true, then the resulting
pointer retrieved from the map would be non-nil. `nil` is a perfectly
valid value to store in a map of pointers, and the comma-ok syntax is
meant for when membership is distinct from having a non-zero value.
There was only one or two cases that I saw that being used correctly for
maps of pointers.
2017-08-30 09:49:31 -04:00
51eb85193c
release lock to avoid dead lock when calling WalkWhereForSeriesIDs
...
* WalkWhereForSeriesIDs may call SeriesIDs, which may attempt to
upgrade from a `RLock` to a `Lock`, causing the dead lock
2017-08-29 16:12:51 -07:00
0ced270197
fix race condition reading map
2017-08-28 13:36:49 -07:00
697759613c
Remove time comparisons from the inner sections of the storage engine
2017-08-16 16:51:13 -05:00
9a2357c2c0
Separate the query engine into a separate package
...
This change provides a clear separation between the query engine
mechanics and the query language so that the language can be parsed and
dealt with separate from the query engine itself.
2017-08-16 13:38:43 -05:00
06bc3b6fbf
TSI Index Migration
2017-08-15 11:40:24 -06:00
45969ef3c6
Allow tag filtering when using DELETE with tsi1
2017-08-14 19:09:36 +01:00
1121b69a9e
auth: apply FGA to SHOW SERIES
2017-08-09 14:56:53 -04:00
0f648e5170
Remove unsafe shenanigans
2017-08-03 16:38:05 +01:00
2d57b599e9
Remove debugging statement
2017-08-02 17:24:00 +01:00
da676a79ae
Implement TSI iterator
2017-08-02 16:29:14 +01:00
aa7095be5a
Use a merge-based approach for TagValues
2017-08-02 14:10:52 +01:00
94a48774b7
Pull in new index filter
2017-08-02 14:10:52 +01:00
c25f7b8b3f
Fix duplicate points returned after delete
...
The sortedSeriesIds slice was not getting reset to 0 which caused
the same series ids to exist in the slice more than once. Since
the size of the slice never matched the size of the seriesID map,
it kept appendending to the slice and sorting it which cause multiple
cursor to get created for the same series.
Fixes #8531
2017-07-10 10:37:01 -06:00
2ccdda72a1
Free RLock prior to returning
2017-07-08 07:14:50 -07:00
9e64813db8
Defer unlock all write locks in inmem index.
...
Currently two write locks in `inmem` are obtained and then
manually unlocked at function exit points. However, we have
reports that the `inmem` index is hanging on a write lock and
cannot track the issue down to anything else besides a lock
that could have been left unlocked because of a panic.
This commit changes the two locks to always defer their unlocks
to prevent these hangs.
2017-06-29 10:23:13 -06:00
f9dc61928a
Fix TSI issue with spaces in tag values.
2017-06-28 11:39:48 -06:00
9bd703d597
Fix possible deadlocks in inmem index
2017-06-21 12:07:40 -06:00
b51f604030
Fix TSI non-contiguous compaction panic.
...
This fixes the case where log files are compacted out of order
and cause non-contiguous sets of index files to be compacted.
Previously, the compaction planner would fetch a list of index files
for each level and compact them in order starting with the oldest
ones. This can be a problem for level 1 because level 0 (log files)
are compacted individually and in some cases a log file can finish
compacting before older log files are finished compacting. This
causes there to be a gap in the list of level 1 files that is
ignored when fetching a list of index files.
Now, the planner reads the list of index files starting from the
oldest but stops once it hits a log file. This prevents that gap
from being ignored.
2017-06-13 10:53:26 -06:00
d17c205b54
fix typo
2017-06-12 11:20:08 +08:00
6e6f92c99a
fix index file fd leak
2017-06-12 10:58:05 +08:00
bcc6ef769b
Check file count before attempting a TSI level compaction.
...
This check was previously in a different section of code which
was lost during a refactor to the new compaction strategy. The
compaction planning now makes a check to ensure at least two
files are available for compaction in a level.
2017-06-06 11:08:59 -06:00
47f97ea134
use parsed measurement and models.Tags
2017-05-26 13:21:59 -07:00
3ec9b401f7
fix benchmark test
2017-05-26 13:21:59 -07:00
46796d932f
add database to index, engine and shard; call AuthorizeSeriesRead
2017-05-26 13:21:50 -07:00
815f740f4c
initial fga work
...
wip
wip
fix tests / build
2017-05-26 13:16:27 -07:00
c30c33dbcb
Merge remote-tracking branch 'origin/master' into sgc-tagsets
2017-05-26 09:10:18 -07:00
c89d98dc02
gofmt
2017-05-25 16:00:23 -07:00
386720b2e7
improvements to inmem/Measurement.TagSets API
...
```
benchmark old ns/op new ns/op delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8 234054 117315 -49.88%
BenchmarkMeasurement_TagSetsDimensions_1000-8 996838 313313 -68.57%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8 58940464 39452117 -33.06%
BenchmarkMeasurement_TagSetsDimensions_100000-8 175612060 70195562 -60.03%
benchmark old allocs new allocs delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8 1026 26 -97.47%
BenchmarkMeasurement_TagSetsDimensions_1000-8 8026 2029 -74.72%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8 100064 64 -99.94%
BenchmarkMeasurement_TagSetsDimensions_100000-8 800064 200067 -74.99%
benchmark old bytes new bytes delta
BenchmarkMeasurement_TagSetsNoDimensions_1000-8 117080 69080 -41.00%
BenchmarkMeasurement_TagSetsDimensions_1000-8 549081 117176 -78.66%
BenchmarkMeasurement_TagSetsNoDimensions_100000-8 23298264 18498265 -20.60%
BenchmarkMeasurement_TagSetsDimensions_100000-8 66498276 23298360 -64.96%
```
2017-05-25 15:52:27 -07:00
14b54e08cb
Fix compile error
2017-05-25 15:18:35 -06:00
24446a0297
Implement zap logging in TSI.
2017-05-25 08:57:50 -06:00
547db32d01
Fix tsi go vet issues.
2017-05-23 13:42:38 -06:00
48456d80ad
Remove tsi commented code.
2017-05-23 10:24:37 -06:00
2524df3405
Convert tsi1 series keys to uint32.
2017-05-23 09:48:13 -06:00
c744e2f562
TSI pull request fixes.
2017-05-23 09:01:05 -06:00
57eeae03fc
Add note about SeriesIDs() limitation.
2017-05-23 08:42:25 -06:00
e7f39c06ab
Refactor TSI1 compaction.
2017-05-23 08:42:25 -06:00
1975940f76
intermediate compaction commit
2017-05-23 08:42:25 -06:00
79edc0979c
Add temporary debugging stats for offset lookups.
2017-05-23 08:41:31 -06:00
48a06432df
Add tsi1 bloom filter.
2017-05-23 08:41:31 -06:00
f3e08c5871
Delta encode tag and measurement block series data.
2017-05-23 08:41:31 -06:00
6f58149052
Increase tsi compaction factor.
2017-05-23 08:40:26 -06:00
5c5bea2baa
move Measurement and Series to inmem package
2017-05-19 08:17:09 -07:00
2cac46ebbc
Convert usage of strings to []byte
...
Measurement name and field were converted between []byte and string
repetively causing lots of garbage. This switches the code to use
[]byte in the write path.
2017-05-12 14:05:19 -06:00
489c89bea4
Add tsi support tooling.
2017-05-08 11:00:15 -06:00
73ddd4787b
Fix race in SeriesN and CreateSeriesIfNotExists
2017-05-04 14:40:50 -06:00
7371f1067b
Fix deadlock in Index.ForEachMeasurementTagKey
...
Index.ForEachMeasurementTagKey held an RLock while call the fn,
if the fn made another call into the index which acquired an RLock
and after another goroutine tried to acquire a Lock, it would deadlock.
2017-05-03 22:48:10 -06:00
ba7108f94e
Add TSI documentation.
2017-04-21 14:45:03 -06:00
02b663b651
Fix lock contention in Index.CreateSeriesListIfNotExists
...
There was contention on the write lock which only needs to be acquired
when checking to see if the log file should be rolled over.
2017-04-20 12:28:42 -06:00
40ec85aacd
Fix lock contention in LogFile.SeriesWithBuffer
...
Under high write load, the check for each series was done sequentially
which caused a lot of CPU time to acquire/release the RLock on LogFile.
This switches the code to check multiple series at once under an RLock
similar to the chang for inmem.
2017-04-20 12:28:42 -06:00
3c2825a851
Reduce lock thrashing when checking series
...
The inmem index would call CreateSeriesIfNotExist for each series
which takes and releases and RLock to see if a series exists. Under
high write load, the lock shows up in profiles quite a bit. This
adds a filtering step that obtains a single RLock and checks all the
series and returns the non-existent series to contine though the slow
path.
2017-04-20 12:28:41 -06:00
a19ce9c10f
Reduce index lock contention
...
Series and Measurment have their own locks and we do not need to
hold locks on the index while using those types.
2017-04-18 16:32:33 -06:00
9c97cd8601
Merge remote-tracking branch 'upstream/master' into tsi
2017-04-04 12:46:09 -06:00
0d74497abe
Reset rhh map elements to reuse allocations.
2017-04-04 11:57:37 -06:00
6ff27c95e5
Fix tsi assertions.
2017-04-04 11:29:21 -06:00
95d4016ff2
Merge branch 'tsi' of https://github.com/influxdata/influxdb into tsi-series-block-partitioning
2017-04-04 10:14:03 -06:00
bf49b176f5
Partition tsi1 series index.
2017-04-04 09:46:04 -06:00
fddaff2cc8
Merge master in
2017-03-29 18:00:28 +01:00
116230b427
Use varint for tag count
2017-03-29 16:31:13 +01:00
2edfb1c92d
Ignore series limit on database load.
2017-03-24 16:27:16 -06:00
ee03fbb164
Fix series tombstone sketch not updated when dropping measurment
2017-03-24 15:49:00 -06:00
9fb8f1ec1d
Fix database and tag limits.
2017-03-24 09:48:10 -06:00
85c1ae4dd4
Remove sort.Slice
2017-03-21 12:33:05 -06:00
1e9fa7bc2c
Fix 32-bit rhh implementation.
2017-03-21 11:44:13 -06:00
58c8736ebc
Merge pull request #8172 from influxdata/jw-dropped-points
...
Fix series not getting created
2017-03-21 09:44:31 -06:00
92c722913d
Unlock index if dropping non-existent series
2017-03-21 09:19:44 -06:00
5cf41adcb8
Optimize tsi1 write path.
...
Removes sorted series list in log, uses a buffer for HasSeries(),
and prepends a length for series key encoding.
2017-03-21 08:44:35 -06:00
7f78c6ad8e
Fix series not getting created
2017-03-20 17:19:22 -06:00
5d67c424bf
Refactor tsi1 write locking.
2017-03-17 11:20:50 -06:00
70efc70abe
Reduce lock contention, fix rhh lookup.
2017-03-17 09:44:11 -06:00
1807772388
Fix tsi tests.
2017-03-15 11:23:58 -06:00
ee2e046853
Merge remote-tracking branch 'upstream/tsi-log-compact' into tsi
2017-03-15 10:22:32 -06:00
cf7ba96377
Merge branch 'tsi-log-compact' into tsi
2017-03-15 10:18:40 -06:00
358b1e0b05
Merge remote-tracking branch 'upstream/master' into tsi
2017-03-15 10:13:32 -06:00
ddcea1c322
WHY YOU SMITE ME BEN. B. JOHNSON?
2017-03-15 12:50:03 +00:00
d23f2971c3
Refactor TagBlockEncoder.
2017-03-10 10:08:16 -07:00
7ee0b9bb28
Rewrite SeriesBlockEncoder.
2017-02-16 09:39:51 -07:00
7e6f33ef48
TSI1 Index file compaction.
2017-02-15 08:44:22 -07:00
71d13fed6d
Immediate, parallel tsl compactions.
2017-02-15 08:44:21 -07:00
7749bc1c67
Store sorted series in logFile
2017-02-13 17:03:42 +00:00
a6a2f9d5f0
Don't load meta data for tsi
2017-02-09 17:59:14 +00:00
2d59fb788c
Modify arguments to reduce allocations
2017-02-09 15:01:32 +00:00
c2bbc18e4b
Reduce repeated string allocations
2017-02-09 11:23:23 +00:00
21e821e5f9
Reduce memory footprint by pre-allocating
2017-02-08 18:49:03 +00:00
a9d58cc879
Add log file benchmarks
2017-02-08 16:50:46 +00:00
d6fedcacd5
Refactor tsi.LogFile compaction.
2017-02-08 09:00:08 -07:00
c4e9430de3
Recover after partial tsi log file write.
2017-02-07 10:02:31 -07:00
d91e6eabac
Add max-values-per-tag to inmem index.
2017-02-06 11:14:13 -07:00
908959a78a
Fix vet
2017-02-03 11:57:35 +00:00
c246f3d9b0
Use inmem index on existing shards.
2017-02-02 10:04:25 -07:00
5a37d51f02
Refactor DropSeries
2017-02-01 22:07:44 +00:00
57f44d5f0c
Include index in snapshot.
2017-02-01 14:19:42 -07:00
0a9fc910f6
Fix potential race on inmem
2017-02-01 16:29:52 +00:00
ffced9b538
Move sketch merging into File
2017-02-01 15:33:30 +00:00
ab238251f4
Refactor measurement sketches
2017-02-01 14:56:35 +00:00
d8a73a1954
Go vet
2017-02-01 14:51:15 +00:00
05bbda2c0a
Fix measurement block sketches.
2017-02-01 07:40:00 -07:00
86e908c384
Remove debug println().
2017-01-31 09:24:58 -07:00
f4a4d1e196
Fix series sketches.
2017-01-31 08:47:18 -07:00
ab94c1b743
Fixes #7882
2017-01-30 19:12:24 +00:00
42959ead1f
Fix CreateSeriesListIfNotExists() data race.
2017-01-30 10:07:07 -07:00
2ae9406c1d
Reincorporate memory leak fix
...
7832 got lost during a rebase/merge. This reincorporate the fix
for the in-memory index as well as tsi.
2017-01-24 20:13:48 -07:00
173003bc38
Fix go vet
2017-01-24 20:13:37 -07:00
047c21f4d9
Merge remote-tracking branch 'upstream/master' into tsi
2017-01-24 09:28:58 -07:00
c459d24a60
Test coverage.
2017-01-23 09:38:27 -07:00
f43b0f7ec9
Fix series & measurement deletion.
2017-01-12 09:29:40 -07:00
352817e8c4
Convert 32-bit offsets to 64-bit.
2017-01-11 08:58:10 -07:00
64c7715243
Rebase fixes.
2017-01-09 10:10:12 -07:00
d1f1e19591
Fixing rebase.
2017-01-06 09:31:25 -07:00
1003db0067
Add active log file tracking, time-based compaction.
2017-01-05 10:17:12 -07:00
c1c98223ec
Fix and optimize tsi1 FileSet.
2017-01-05 10:17:12 -07:00
31e74d809b
Add tsi FileSet.
2017-01-05 10:17:11 -07:00
dcd2a771b0
Optimizing tsi compaction.
2017-01-05 10:17:11 -07:00
1ce99e797f
Use series map in tsi1.LogFile.
2017-01-05 10:17:11 -07:00
9b1e8215e0
Remove dictionary encoding, add bulk series insertion.
2017-01-05 10:17:11 -07:00
5f7654173e
Add locking to sketch merge.
2017-01-05 10:17:11 -07:00
9bd19cdc69
Fix inmem DELETE SERIES.
2017-01-05 10:17:11 -07:00
f9efcb3365
Re-add shared in-memory index.
2017-01-05 10:17:09 -07:00
0f9b2bfe6a
Fix tests
2017-01-05 10:16:15 -07:00
4ccb8dbab1
Move series count check to shard
2017-01-05 10:16:13 -07:00
0cb74eedbf
Add log file (WAL) sketches
2017-01-05 10:15:38 -07:00
190c78c644
Add series sketches
2017-01-05 10:15:37 -07:00
695adafc00
Add measurement sketches
2017-01-05 10:15:37 -07:00
745b1973a8
tsi compaction
2017-01-05 10:15:37 -07:00
83e80f6d0b
Fix in-mem index integration tests.
2017-01-05 10:15:37 -07:00
183418dcbd
Fix tsi TAG KEYS iterator.
2017-01-05 10:15:36 -07:00
759ff4ab80
Add tsi1 term hash index.
2017-01-05 10:15:35 -07:00
75cfe244c4
Add series hash index.
2017-01-05 10:15:35 -07:00
9f8b206b51
Fix measurement system queries.
2017-01-05 10:15:34 -07:00
4aa78383d1
Fix tsi1 series deletion.
2017-01-05 10:14:48 -07:00
5965610de6
Refactoring tsi tombstoning.
2017-01-05 10:14:02 -07:00
e7940cc556
Add tsi1 series system iterator.
2017-01-05 10:14:00 -07:00
87f4e0ec0a
Add regex support in tsi1.
2017-01-05 10:12:29 -07:00
d13afa8f47
Iterator refactoring
2017-01-05 10:11:49 -07:00
f0427d180e
Fix tsi index panics
...
Hardcoded panics cause the server to crash in 10s due to stats collection.
2017-01-05 10:11:12 -07:00
4bf7b2bb19
Allow tsi to be enabled via config option
2017-01-05 10:11:12 -07:00
2b96c5d4d0
Set Tags on entry
...
These were lost when reloading from the index. Fixes queries not
returning any data.
2017-01-05 10:11:12 -07:00
a6490920fd
Fix reslicing indices
...
The slicing was backwards causing the buffer to grow indefinitely
and filling the disks on writes.
2017-01-05 10:11:12 -07:00
59864226b7
Add RWMutex to LogFile
...
Fixes concurrent map access panic
2017-01-05 10:11:12 -07:00
fbe7f464ee
Improve insert performance.
2017-01-05 10:11:12 -07:00
33412782ed
Fix go vet issue.
2017-01-05 10:11:10 -07:00
2b864c72c5
Refactor MeasurementBlockTrailer read/write.
2017-01-05 10:11:10 -07:00
cb93f10120
Remove per-shard in-memory index.
2017-01-05 10:11:09 -07:00
409b0165f5
shared in-memory index
2017-01-05 10:09:57 -07:00
a812502ea3
reintegrating in-memory index
2017-01-05 10:07:35 -07:00
Joe LeGasse
Edd Robinson
Jason Wilder
Ben Johnson
Jason Wilder
Stuart Carnie
Jonathan A. Sternberg
Summer