f712828016
tsm1: refactor and rename some methods
2019-01-08 14:52:30 -07:00
8744a82665
tsm1: add DeletePrefix to the reader
2019-01-07 21:11:49 -07:00
f65b0933f6
tsm1: move code around into smaller files and add tests
2019-01-07 21:11:49 -07:00
fed3154506
tsm1: DeletePrefix on the indirectIndex
2019-01-07 21:08:32 -07:00
ad5352926f
tsm1: log when error reading entries for tsm key
2019-01-07 11:00:35 -07:00
9cdefa8e4f
tsm1: fix staticcheck and refactor closure out
2019-01-07 11:00:35 -07:00
1ffcd77342
tsm1: fix remaining issues and add small benchmarks
...
- notice when keys are deleted during iteration and return an error
- make sure all the consumers check the error
- add some benchmarks for small indexes to compare
- allow concurrent readers to flag deletes
benchmarks against base:
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 70.0ms ±17% 71.0ms ±12% ~ (p=1.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 1.48µs ± 1% 0.28µs ± 5% -81.29% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull/Large-8 786ms ± 1% 363ms ± 3% -53.89% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull/Small-8 2.37ms ± 0% 1.14ms ± 3% -52.02% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 384ms ± 2% 188ms ± 3% -51.04% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 470µs ± 1% 190µs ± 1% -59.71% (p=0.000 n=8+7)
IndirectIndex_Delete/Large-8 74.0ms ± 1% 128.7ms ± 1% +73.80% (p=0.001 n=7+7)
IndirectIndex_Delete/Small-8 142µs ± 1% 130µs ± 1% -8.24% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeLast-8 3.26kB ± 0% 0.00kB ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 233MB ± 0% 161MB ± 0% -30.75% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 2.13MB ± 0% 1.40MB ± 0% -34.53% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 12.4MB ± 0% 0.4MB ± 0% -96.82% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 120kB ± 0% 0kB ± 0% -99.89% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 4.54kB ± 0% 0.21kB ± 0% -95.26% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 80.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 3.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 1.53M ± 0% 0.52M ± 0% -65.98% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 15.2k ± 0% 5.2k ± 0% -65.97% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 620 ± 0% 124 ± 0% -80.00% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 10.0 ± 0% 2.0 ± 0% -80.00% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 246 ± 0% 1 ± 0% -99.59% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 4.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
14cf01911e
tsm1: change TSMFile to use an iterator style api
2019-01-07 11:00:35 -07:00
917584b054
tsm1: use readerOffsetsIterator for deletes
...
This reduces the amount of disk hits at some costs in cpu on some benchmarks. Notably, the
DeleteRangeFull_Covered and Delete benchmarks both went to approximately zero page faults
meaning they read from the index file linearly.
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 68.8ms ±10% 63.1ms ±16% -8.28% (p=0.021 n=8+8)
IndirectIndex_Entries-8 9.09µs ± 3% 9.62µs ± 1% +5.84% (p=0.000 n=8+7)
IndirectIndex_ReadEntries-8 5.86µs ± 1% 6.15µs ± 3% +5.03% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 562ns ± 6% 308ns ± 2% -45.25% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 363ms ±10% 376ms ± 5% ~ (p=0.054 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 574ms ± 2% 746ms ± 0% +30.01% (p=0.000 n=8+7)
IndirectIndex_Delete-8 51.2ms ± 0% 88.2ms ± 0% +72.38% (p=0.000 n=8+7)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.7MB ± 0% 11.7MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.798 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.857 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.04kB ± 0% +0.90% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 42.0 ± 0% 42.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.743 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.856 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 18.1MB/s ± 9% 17.5MB/s ± 7% ~ (p=0.105 n=8+8)
IndirectIndex_Delete-8 116MB/s ± 0% 0MB/s ± 0% -99.96% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
6f5c94f3f7
tsm1: introduce readerOffsets to manage the offsets slice
...
It exposes an API that will clean up the bodies of many methods and
provide a safe abstraction around iteration that will be able to
handle reads with concurrent deletes.
Benchmarks are flat.
2019-01-07 11:00:35 -07:00
f860305124
tsm1: keep first 8 bytes of each key in memory
...
Since most keys will share the first 8 bytes, we collapse them into
a slice containing partial sums of the counts. We can then binary search
into that slice to find the associated prefix for a given offset index.
Compressing in this way causes the overhead to be negligable and reduces
disk misses by about 30% in these benchmarks (500k series across 100 orgs).
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.5ms ± 1% 64.6ms ± 1% -4.33% (p=0.000 n=8+7)
IndirectIndex_Entries-8 9.41µs ± 2% 9.39µs ± 1% ~ (p=0.959 n=8+8)
IndirectIndex_ReadEntries-8 5.99µs ± 1% 6.07µs ± 1% +1.29% (p=0.001 n=8+8)
IndirectIndex_DeleteRangeLast-8 369ns ± 2% 566ns ± 1% +53.37% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 368ms ± 9% 369ms ± 2% ~ (p=0.232 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 600ms ± 1% 618ms ± 0% +3.03% (p=0.000 n=8+7)
IndirectIndex_Delete-8 50.0ms ± 1% 47.6ms ± 9% ~ (p=0.463 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.776 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.01kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.457 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 24.7MB/s ±10% 17.8MB/s ± 2% -28.18% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 14.2MB/s ± 1% 9.6MB/s ± 0% -32.30% (p=0.000 n=8+7)
IndirectIndex_Delete-8 171MB/s ± 1% 126MB/s ±10% -26.35% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeLast went from 17 page faults, or ~180GB/sec at 369ns/op
to zero page faults. So even though it got 50% slower, it was actually I/O bound
and no longer is.
2019-01-07 11:00:35 -07:00
0becfc6239
tsm1: add helper to track page faults in index
...
Since the methods inline and dead code is eliminated, it has no runtime
overhead in the benchmarks when disabled.
benchmark recorded faults
BenchmarkIndirectIndex_Entries-8 11
BenchmarkIndirectIndex_ReadEntries-8 11
BenchmarkIndirectIndex_DeleteRangeLast-8 17
BenchmarkIndirectIndex_DeleteRangeFull-8 2218
BenchmarkIndirectIndex_Delete-8 2084
2019-01-07 11:00:35 -07:00
91e820a9d8
tsm1: fix multiple issues with DeleteRange
...
1. Correctly acquires locks
2. Seeks for discontiguous key ranges (like delete ["aaa", "zzz"])
3. Is precise about deleting a key when it contains no data
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.3ms ± 1% 63.2ms ±15% ~ (p=0.463 n=7+8)
IndirectIndex_Entries-8 9.14µs ± 1% 9.01µs ± 0% -1.40% (p=0.004 n=8+7)
IndirectIndex_ReadEntries-8 5.83µs ± 1% 5.68µs ± 2% -2.62% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 283ns ± 2% 191ns ± 1% -32.37% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull-8 612ms ± 1% 361ms ± 1% -41.02% (p=0.000 n=8+8)
IndirectIndex_Delete-8 49.0ms ± 1% 49.8ms ± 1% +1.80% (p=0.001 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.6MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 64.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 168MB ± 0% 162MB ± 0% -3.71% (p=0.000 n=8+8)
IndirectIndex_Delete-8 3.94kB ± 0% 3.94kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 35.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 2.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 1.04M ± 0% 0.52M ± 0% -49.77% (p=0.000 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
2019-01-07 11:00:35 -07:00
aed17cfedd
tsm1: speed up IndirectIndex benchmarks
...
rather than create the indirectIndex every Benchmark iteration
reuse a global one. reduces a couple benchmarks where Start/Stop timer
weren't in palce.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 67710057 69216355 +2.22%
BenchmarkIndirectIndex_Entries-8 9239 9762 +5.66%
BenchmarkIndirectIndex_ReadEntries-8 5964 5886 -1.31%
BenchmarkIndirectIndex_DeleteRangeLast-8 317 284 -10.41%
BenchmarkIndirectIndex_DeleteRangeFull-8 615346992 598392398 -2.76%
BenchmarkIndirectIndex_Delete-8 52906315 44400269 -16.08%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1038932 1038722 -0.02%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 0 -100.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 168112352 168061952 -0.03%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
d40c3e662f
tsm1: use uint32 key for tombstones
...
rough, noisy benchmarks.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 62462250 67710057 +8.40%
BenchmarkIndirectIndex_Entries-8 9601 9239 -3.77%
BenchmarkIndirectIndex_ReadEntries-8 5984 5964 -0.33%
BenchmarkIndirectIndex_DeleteRangeLast-8 314 317 +0.96%
BenchmarkIndirectIndex_DeleteRangeFull-8 813838165 615346992 -24.39%
BenchmarkIndirectIndex_Delete-8 52079181 52906315 +1.59%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532670 1038932 -32.21%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 232738960 168112352 -27.77%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
ffd35ce1aa
tsm1: use a uint32 for offsets globally
...
benchmarks are flat.
2019-01-07 11:00:35 -07:00
7a7a4b6d58
tsm1: remove offsets from mmap
...
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 74525387 66439305 -10.85%
BenchmarkIndirectIndex_Entries-8 8892 9200 +3.46%
BenchmarkIndirectIndex_ReadEntries-8 5816 5691 -2.15%
BenchmarkIndirectIndex_DeleteRangeLast-8 1550 311 -79.94%
BenchmarkIndirectIndex_DeleteRangeFull-8 773649708 767030277 -0.86%
BenchmarkIndirectIndex_Delete-8 79755991 52015903 -34.78%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3 2 -33.33%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532589 1532344 -0.02%
BenchmarkIndirectIndex_Delete-8 246 123 -50.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3264 64 -98.04%
BenchmarkIndirectIndex_DeleteRangeFull-8 232710448 232624208 -0.04%
BenchmarkIndirectIndex_Delete-8 4432 3936 -11.19%
2019-01-07 11:00:35 -07:00
9d8114ef88
Fix tombstone error logic
2018-12-18 13:00:13 +00:00
419db63f7a
Address PR feedback
2018-12-18 12:33:28 +00:00
262772544a
Add prefix key to tombstoner
...
This commit adds support for "prefix keys". Prefix keys differ from
regular tombstone key entries in that the key of the entry should act as
a prefix that matches all series with the same prefix key for the given
time range.
This means only one entry is needed to delete many series.
The tombstone entries now have a maximum length of 16777215 (24 bits),
with the remaining 8 high bits available for setting further options /
meta information about the tombstone entry.
In this case, the top bit is used to indicate that the tombstone entry
is intended to be a prefix. This leaves 7 spare bits for future use.
2018-12-18 12:33:28 +00:00
1b5ad5e129
Ensure v4 tombstone file read correctly
2018-12-18 12:33:28 +00:00
90f3583fe0
Remove old tombstone version support
2018-12-18 12:33:28 +00:00
04605eb266
tsm1: speed up deleterange for large keys
...
rather than starting at the first key, do a binary search to the
first key. changes O(N) when deleting the largest key to O(log N).
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_DeleteRangeFull-8 17884166763 738717473 -95.87%
2018-12-14 10:06:24 -07:00
687a390aaf
tsm1: add benchmarks for deletes
2018-12-14 10:06:24 -07:00
f383e8337a
test(tsdb/tsi1): test series id cache delete concurrently
...
Report the total number of gets, puts, and deletes at the end of the
test. I've found this kind of output to be a useful sanity check in
similar tests that exercise concurrency involving tasks.
Use a local random source in each goroutine. I unscientifically
eyeballed that to increase total operations by 5-10%.
Also call t.Parallel in a few more tests that involve disk access. This
shaves 1-2 seconds off the full tsi1 test suite on my machine.
2018-12-12 08:21:17 -08:00
eba485f2be
Fix nil tracker for full compactions
2018-12-11 18:30:59 +00:00
c4b42c72be
Add option to disable TSI metrics
2018-12-10 15:02:26 +00:00
6b63a3def7
Add option to disable sfile metrics
2018-12-10 14:36:28 +00:00
3ff39cd9dc
Merge pull request #1789 from influxdata/er-cache-size
...
Allow TSI cache to be dynamically altered
2018-12-10 11:23:15 +00:00
df0b084543
Merge pull request #1785 from influxdata/bj-tss-file-observer
...
Allow stats files to be observed for finishing/unlinking.
2018-12-07 18:46:35 -07:00
e0cddadffd
Allow TSI cache to be dynamically altered
2018-12-07 18:35:25 +00:00
73d8c85aa2
Allow stats files to be observed for finishing/unlinking.
...
This commit adds the `.tss` files generated for TSM statistics to
the `FileObserver` so that package users can be notified when new
stats files are created and removed.
2018-12-07 10:20:32 -07:00
e13309ebbe
Fix metric names
2018-12-07 16:37:17 +00:00
b015757c06
Ensure all tsi1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
bff655786f
Ensure tsdb metrics properly registered
2018-12-07 14:32:34 +00:00
aa936df138
Ensure all tsm1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
d94f898c8b
WIP
2018-12-07 14:32:34 +00:00
79b108d174
Fix bug with slice reuse
2018-12-07 14:32:34 +00:00
de491968ba
Fix rebase
2018-12-07 14:32:34 +00:00
93892c20ab
Fix test
2018-12-07 14:32:34 +00:00
a8b6827c9e
megacheck
2018-12-07 14:32:34 +00:00
2bb558a9d1
Ensure fileset protected by lock
2018-12-07 14:32:34 +00:00
a1804d27be
Fix race
2018-12-07 14:32:34 +00:00
f9a2f7a017
go fmt
2018-12-07 14:32:34 +00:00
e0c10227d0
Fix metric issue in series file
2018-12-07 14:32:34 +00:00
7960ccc320
Add TSI index metrics
2018-12-07 14:32:34 +00:00
55caa0fe54
Add RHH metrics
2018-12-07 14:32:34 +00:00
d1fe2bc188
Add series file metrics
2018-12-07 14:32:34 +00:00
8ca637bd80
Refactor default labels and retention metrics
2018-12-07 14:32:34 +00:00
6c5dec8f88
Refactor tracker names
2018-12-07 14:32:34 +00:00
44e5fbae0a
Convert WAL stats
2018-12-07 14:32:34 +00:00
3b980ed7e3
Convert Cache statistics
2018-12-07 14:32:34 +00:00
d61b9f1645
Convert Filestore stats
2018-12-07 14:32:34 +00:00
f56bc0853f
Convert TSM compaction stats to Prom metrics
...
This commits converts all the 1.x TSM compaction statistics, which
previously were written to an _internal db, to Prometheus metrics.
2018-12-07 14:32:34 +00:00
186e0392ed
Address PR feedback
2018-11-30 10:54:24 +00:00
e11789f46a
Omit unused receiver name: ST1006
2018-11-30 10:54:24 +00:00
eaa4a4f49a
Removes unused methods: U1000
2018-11-30 10:54:24 +00:00
9403c1ec8e
Ensure error strings not capitalised ST1005
2018-11-30 10:54:24 +00:00
308a5148cf
Remove iterators
2018-11-30 10:54:24 +00:00
98d24f7e3c
Merge pull request #1625 from influxdata/remove-influxdb-dependency
...
Remove influxdb dependency.
2018-11-29 14:23:00 -07:00
0084d4d824
Remove influxdb dependency.
2018-11-29 11:44:22 -07:00
7ccb201b80
Merge pull request #1332 from zhulongcheng/rm-create-series
...
Remove Index.CreateSeriesIfNotExists
2018-11-29 18:36:48 +00:00
1862b4421d
Integrate scanned values statistics tracking.
2018-11-28 15:32:06 -07:00
e22aff46cb
Force create TSS files.
...
This commit replaces an `os.OpenFile()` call with an `os.Create()`
call which drops `O_EXCL` for `O_TRUNC` since `.tss` files are only
created after `.tsm` files so lingering temporary files are safe to
overwrite.
2018-11-27 08:07:21 -07:00
ed799a3d6c
remove Index.CreateSeriesIfNotExists
2018-11-21 20:16:45 +08:00
dbfa140cc4
remove Engine.CreateSeriesIfNotExists
2018-11-21 20:16:45 +08:00
085ce852b7
remove CreateSeriesIfNotExists from engine tests
2018-11-21 20:16:45 +08:00
8ab01c99c0
test(tsdb/tsm1): skip long tests in short mode
...
The tsdb/tsm1 package was one of the test suites that took the longest
to run in platform with go test -short. The rule of thumb on the Go
project is that short mode should skip any individual test that takes
longer than one second. This change skips two such tests, and it
eliminates a string concatenation loop in two other tests, so that they
report completion in "0.00s" rather than about 0.94s, on my machine.
These cumulative changes take `go test -short ./tsdb/tsm1` from about 14
seconds to about 7 seconds on my machine.
2018-11-16 08:06:23 -08:00
bbd460e7d9
Add method QueryRawJSON to influxql.service (for querytest tool) ( #1402 )
2018-11-15 10:45:38 -08:00
e7bc29a590
reduce parsing and copying of tags
2018-11-15 20:45:16 +08:00
305ebb8729
fix: Allow compactor to make progress if v.MaxTime() != entry.MaxTime
2018-11-14 12:14:45 +00:00
b35533e7f7
chore: Compactor test which replicates issue #10465
...
Due to an encoding bug with simple8b, it is possible that the
MaxTime for a TSM index entry does not match the last encoded timestamp.
2018-11-14 12:14:43 +00:00
f731ed595d
Merge pull request #1358 from influxdata/jmw-test-explode-points
...
test(tsdb): add test for explode points
2018-11-13 14:41:29 -07:00
1ab9c80ae8
fix(tsdb): eliminate data race from *SeriesIDSet.Clone
...
And add a test to cover that.
The data race would look roughly like:
```
WARNING: DATA RACE
Write at 0x00c000024e18 by goroutine 8:
github.com/RoaringBitmap/roaring.(*roaringArray).markAllAsNeedingCopyOnWrite()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:881 +0x6b
github.com/RoaringBitmap/roaring.(*roaringArray).clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:266 +0x808
github.com/RoaringBitmap/roaring.(*Bitmap).Clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaring.go:385 +0x58
github.com/influxdata/platform/tsdb.(*SeriesIDSet).CloneNoLock()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:229 +0x73
github.com/influxdata/platform/tsdb.(*SeriesIDSet).Clone()
Previous write at 0x00c000024e18 by goroutine 7:
github.com/RoaringBitmap/roaring.(*roaringArray).markAllAsNeedingCopyOnWrite()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:881 +0x6b
github.com/RoaringBitmap/roaring.(*roaringArray).clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:266 +0x808
github.com/RoaringBitmap/roaring.(*Bitmap).Clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaring.go:385 +0x58
github.com/influxdata/platform/tsdb.(*SeriesIDSet).CloneNoLock()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:229 +0x73
github.com/influxdata/platform/tsdb.(*SeriesIDSet).Clone()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:223 +0x7b
```
2018-11-13 08:12:38 -08:00
704941d624
test(tsdb): add test for explode points
2018-11-12 17:36:33 -07:00
39f4908946
fix(storage): allow disabling the WAL
...
We were passing a non-nil tsm1.Log containing a nil *tsm1.WAL which
would cause a panic when it was attempted to be used. Instead, always
pass a non-nil WAL.
We change the storage engine code to not pass in a nil WAL, and
additionally add a defensive check to change any nil WALs into a
NopWAL.
2018-11-09 10:45:24 -07:00
25532778df
fix(tsm1): fix max concurrent compaction logic
2018-11-09 10:14:32 -07:00
4b504b84df
respond to review feedback
...
- Add some documentation.
- Move compaction planner to an option instead of config.
The latter fits with the general theme of having config be things
that can be specified in a toml, and everything else being an
option.
2018-11-08 11:39:36 -07:00
a1b5b322bb
some more refactoring
...
- add helpers to get directories out
- change FileStoreObserver to be an option rather than config.
2018-11-08 11:39:36 -07:00
22e23d6e31
final touches
...
- move default directories to the storage package
- make the directory layout match before
- clean up some dead missed functions
2018-11-08 11:39:36 -07:00
2cbc2ee896
refactor wal out, paths, and options
2018-11-08 11:39:36 -07:00
932b0bf01a
compat: Package to convert old to new config
2018-11-08 11:39:36 -07:00
0d411023f2
config: clean up
...
- Breaks the weird cycle that existed with the EngineOptions
- Removes a bunch of useless parameters
- Moves around a bunch of defaults
2018-11-08 11:39:36 -07:00
594664a876
remove RegisteredIndexes tests
2018-11-05 00:00:02 +08:00
aeefeb2eed
remove RegisteredIndexes method
2018-11-04 21:56:57 +08:00
1857bf1084
Fix TSM index maxtime modification
2018-11-02 18:39:30 -06:00
985c260af7
chore(storage,tsdb): fix megacheck errors
2018-11-01 12:54:46 -07:00
a0300064df
feat(tsm1): Improve performance of Gorilla float block decoding
...
```
name old time/op new time/op delta
FloatArrayDecodeAll/1-8 45.9ns ± 1% 13.8ns ± 1% -70.00% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 686ns ± 0% 232ns ± 1% -66.10% (p=0.000 n=9+8)
FloatArrayDecodeAll/550-8 5.78µs ± 0% 2.22µs ± 1% -61.61% (p=0.000 n=9+9)
FloatArrayDecodeAll/1000-8 10.2µs ± 2% 4.0µs ± 5% -60.47% (p=0.000 n=10+10)
name old speed new speed delta
FloatArrayDecodeAll/1-8 414MB/s ± 1% 1383MB/s ± 1% +233.76% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 144MB/s ± 0% 424MB/s ± 1% +194.19% (p=0.000 n=9+9)
FloatArrayDecodeAll/550-8 133MB/s ± 0% 346MB/s ± 1% +160.09% (p=0.000 n=9+10)
FloatArrayDecodeAll/1000-8 135MB/s ± 2% 340MB/s ± 5% +153.03% (p=0.000 n=10+10)
```
2018-11-01 18:59:20 +00:00
353df7edca
Fix imports
2018-11-01 18:59:20 +00:00
e282d012c8
Address PR feedback
2018-11-01 18:59:20 +00:00
c21336af0a
fix(encoding): Improve array string encoding perf a little more
...
Encode the compressed data at the start internal buffer. This ensures
the returned slice maintains the entire capacity and is available for
subsequent use.
When we pool / reuse string buffers, this will help considerably.
Improvements over previous commit:
```
name old time/op new time/op delta
EncodeStrings/10/batch-8 542ns ± 1% 355ns ± 2% -34.53% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 5.29µs ± 1% 3.58µs ± 2% -32.20% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 48.6µs ± 0% 36.2µs ± 2% -25.40% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
EncodeStrings/10/batch-8 704B ± 0% 0B -100.00% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 9.47kB ± 0% 0.00kB -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 90.1kB ± 0% 0.0kB -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
EncodeStrings/10/batch-8 0.00 0.00 ~ (all equal)
EncodeStrings/100/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
296d39059a
fix(encoding): Improve simple8b another 6%; fix inconsequential bug
...
simple8b encodes deltas[1:], thus deltas[0] >= simple8b.MaxValue is
invalid.
Also changed loop calculating deltas, RLE and max to be similar to
batch timestamp, for greater consistency.
Improvements over previous commit:
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 1.50µs ± 1% 1.48µs ± 1% -1.40% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 6.10µs ± 0% 5.69µs ± 2% -6.58% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 1.50µs ± 1% 1.49µs ± 0% -1.21% (p=0.008 n=5+5)
```
Improvements overall:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.48µs ± 1% -27.25% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 5.69µs ± 2% -35.29% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.49µs ± 0% -26.93% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
9fa01f7115
feat(encoding): Improve timestamp encoding
...
Timestamp improvements prior to any improvements to simple8b
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 1.36µs ± 1% -48.25% (p=0.008 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 32.2µs ± 1% -49.64% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 1.30µs ± 1% -86.06% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
a339f8f620
feat(encoding): Improve integer and simple8b encoding performance
...
simple8b EncodeAll improvements should
```
name old time/op new time/op delta
EncodeAll/1_bit-8 28.5µs ± 1% 28.6µs ± 1% ~ (p=0.133 n=9+10)
EncodeAll/2_bits-8 28.9µs ± 2% 28.7µs ± 0% ~ (p=0.068 n=10+8)
EncodeAll/3_bits-8 29.3µs ± 1% 28.8µs ± 0% -1.70% (p=0.000 n=10+10)
EncodeAll/4_bits-8 29.6µs ± 1% 29.1µs ± 1% -1.85% (p=0.000 n=10+10)
EncodeAll/5_bits-8 30.6µs ± 1% 29.8µs ± 2% -2.70% (p=0.000 n=10+10)
EncodeAll/6_bits-8 31.3µs ± 1% 30.0µs ± 1% -4.08% (p=0.000 n=9+9)
EncodeAll/7_bits-8 32.6µs ± 1% 30.8µs ± 0% -5.49% (p=0.000 n=9+9)
EncodeAll/8_bits-8 33.6µs ± 2% 31.0µs ± 1% -7.77% (p=0.000 n=10+9)
EncodeAll/10_bits-8 34.9µs ± 0% 31.9µs ± 2% -8.55% (p=0.000 n=9+10)
EncodeAll/12_bits-8 36.8µs ± 1% 32.6µs ± 1% -11.35% (p=0.000 n=9+10)
EncodeAll/15_bits-8 39.8µs ± 1% 34.1µs ± 2% -14.40% (p=0.000 n=10+10)
EncodeAll/20_bits-8 45.2µs ± 3% 36.2µs ± 1% -19.97% (p=0.000 n=10+9)
EncodeAll/30_bits-8 55.0µs ± 0% 40.9µs ± 1% -25.62% (p=0.000 n=9+9)
EncodeAll/60_bits-8 86.2µs ± 1% 55.2µs ± 1% -35.92% (p=0.000 n=10+10)
EncodeAll/combination-8 582µs ± 2% 502µs ± 1% -13.80% (p=0.000 n=9+9)
```
EncodeIntegers:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.50µs ± 1% -26.22% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 6.10µs ± 0% -30.73% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.50µs ± 1% -26.04% (p=0.008 n=5+5)
```
EncodeTimestamps (ran is improved due to simple8b improvements)
```
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 2.65µs ± 2% ~ (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 33.8µs ± 1% -47.23% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 9.28µs ± 1% ~ (p=0.087 n=5+5)
```
2018-11-01 18:59:20 +00:00
5e7b2cb273
Fix index bug in float encoder
2018-11-01 18:59:20 +00:00
80c953b774
Add TSM batch key iterator
...
The batch focussed TSM key iterator iterates TSM blocks, decoding and
merging blocks where appropriate using the the batch focussed
approaches.
2018-11-01 18:59:20 +00:00
5074b834cd
Add batch block encoders
2018-11-01 18:59:19 +00:00
ab68204683
Batch oriented unsigned encoder
2018-11-01 18:59:19 +00:00
aeeef803c0
Batch oriented boolean encoders
...
This commit adds a tsm1 function for encoding a batch of booleans into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated sets of booleans.
2018-11-01 18:59:19 +00:00
5376530392
Improvements to batch float encoder
...
- Inlined the closure to avoid a function call.
- Changed append(b, make([]byte, 8)...) to inline the make call.
- Check for NaN once at the end assuming NaN is infrequent.
New performance delta comparing the current iterators to the new batch
function:
name old time/op new time/op delta
EncodeFloats/10_seq 1.32µs ± 2% 0.17µs ± 2% -87.39% (p=0.000 n=10+10)
EncodeFloats/10_ran 2.09µs ± 1% 0.15µs ± 0% -92.97% (p=0.000 n=10+9)
EncodeFloats/100_seq 8.37µs ± 2% 1.28µs ± 2% -84.74% (p=0.000 n=10+10)
EncodeFloats/100_ran 19.1µs ± 1% 1.3µs ± 1% -93.08% (p=0.000 n=9+9)
EncodeFloats/1000_seq 60.4µs ± 1% 12.6µs ± 0% -79.13% (p=0.000 n=9+7)
EncodeFloats/1000_ran 212µs ± 1% 12µs ± 1% -94.53% (p=0.000 n=9+8)
name old alloc/op new alloc/op delta
EncodeFloats/10_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/10_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/100_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/100_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_ran 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeFloats/10_seq 0.00 0.00 ~ (all equal)
EncodeFloats/10_ran 0.00 0.00 ~ (all equal)
EncodeFloats/100_seq 0.00 0.00 ~ (all equal)
EncodeFloats/100_ran 0.00 0.00 ~ (all equal)
EncodeFloats/1000_seq 0.00 0.00 ~ (all equal)
EncodeFloats/1000_ran 0.00 0.00 ~ (all equal)
2018-11-01 18:59:19 +00:00
Jeff Wendling
Edd Robinson
Mark Rushakoff
Ben Johnson
zhulongcheng
Christopher M. Wolff
Stuart Carnie
Jeff Wendling
Jessica Obermark