e503ef40d1
tsm1: add comments responding to review feedback
2019-01-09 11:35:06 -07:00
73c0ea410e
tsm1: add test for engine DeletePrefix
2019-01-09 10:56:10 -07:00
0a85e3b0dd
tsm1: add initial index cleanup to DeletePrefix
2019-01-08 16:32:43 -07:00
0fe2f02812
tsm1: initial DeletePrefix impl
2019-01-08 16:03:34 -07:00
f712828016
tsm1: refactor and rename some methods
2019-01-08 14:52:30 -07:00
8744a82665
tsm1: add DeletePrefix to the reader
2019-01-07 21:11:49 -07:00
f65b0933f6
tsm1: move code around into smaller files and add tests
2019-01-07 21:11:49 -07:00
fed3154506
tsm1: DeletePrefix on the indirectIndex
2019-01-07 21:08:32 -07:00
ad5352926f
tsm1: log when error reading entries for tsm key
2019-01-07 11:00:35 -07:00
9cdefa8e4f
tsm1: fix staticcheck and refactor closure out
2019-01-07 11:00:35 -07:00
1ffcd77342
tsm1: fix remaining issues and add small benchmarks
...
- notice when keys are deleted during iteration and return an error
- make sure all the consumers check the error
- add some benchmarks for small indexes to compare
- allow concurrent readers to flag deletes
benchmarks against base:
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 70.0ms ±17% 71.0ms ±12% ~ (p=1.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 1.48µs ± 1% 0.28µs ± 5% -81.29% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull/Large-8 786ms ± 1% 363ms ± 3% -53.89% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull/Small-8 2.37ms ± 0% 1.14ms ± 3% -52.02% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 384ms ± 2% 188ms ± 3% -51.04% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 470µs ± 1% 190µs ± 1% -59.71% (p=0.000 n=8+7)
IndirectIndex_Delete/Large-8 74.0ms ± 1% 128.7ms ± 1% +73.80% (p=0.001 n=7+7)
IndirectIndex_Delete/Small-8 142µs ± 1% 130µs ± 1% -8.24% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeLast-8 3.26kB ± 0% 0.00kB ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 233MB ± 0% 161MB ± 0% -30.75% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 2.13MB ± 0% 1.40MB ± 0% -34.53% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 12.4MB ± 0% 0.4MB ± 0% -96.82% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 120kB ± 0% 0kB ± 0% -99.89% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 4.54kB ± 0% 0.21kB ± 0% -95.26% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 80.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 3.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 1.53M ± 0% 0.52M ± 0% -65.98% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 15.2k ± 0% 5.2k ± 0% -65.97% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 620 ± 0% 124 ± 0% -80.00% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 10.0 ± 0% 2.0 ± 0% -80.00% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 246 ± 0% 1 ± 0% -99.59% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 4.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
14cf01911e
tsm1: change TSMFile to use an iterator style api
2019-01-07 11:00:35 -07:00
917584b054
tsm1: use readerOffsetsIterator for deletes
...
This reduces the amount of disk hits at some costs in cpu on some benchmarks. Notably, the
DeleteRangeFull_Covered and Delete benchmarks both went to approximately zero page faults
meaning they read from the index file linearly.
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 68.8ms ±10% 63.1ms ±16% -8.28% (p=0.021 n=8+8)
IndirectIndex_Entries-8 9.09µs ± 3% 9.62µs ± 1% +5.84% (p=0.000 n=8+7)
IndirectIndex_ReadEntries-8 5.86µs ± 1% 6.15µs ± 3% +5.03% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 562ns ± 6% 308ns ± 2% -45.25% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 363ms ±10% 376ms ± 5% ~ (p=0.054 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 574ms ± 2% 746ms ± 0% +30.01% (p=0.000 n=8+7)
IndirectIndex_Delete-8 51.2ms ± 0% 88.2ms ± 0% +72.38% (p=0.000 n=8+7)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.7MB ± 0% 11.7MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.798 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.857 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.04kB ± 0% +0.90% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 42.0 ± 0% 42.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.743 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.856 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 18.1MB/s ± 9% 17.5MB/s ± 7% ~ (p=0.105 n=8+8)
IndirectIndex_Delete-8 116MB/s ± 0% 0MB/s ± 0% -99.96% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
6f5c94f3f7
tsm1: introduce readerOffsets to manage the offsets slice
...
It exposes an API that will clean up the bodies of many methods and
provide a safe abstraction around iteration that will be able to
handle reads with concurrent deletes.
Benchmarks are flat.
2019-01-07 11:00:35 -07:00
f860305124
tsm1: keep first 8 bytes of each key in memory
...
Since most keys will share the first 8 bytes, we collapse them into
a slice containing partial sums of the counts. We can then binary search
into that slice to find the associated prefix for a given offset index.
Compressing in this way causes the overhead to be negligable and reduces
disk misses by about 30% in these benchmarks (500k series across 100 orgs).
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.5ms ± 1% 64.6ms ± 1% -4.33% (p=0.000 n=8+7)
IndirectIndex_Entries-8 9.41µs ± 2% 9.39µs ± 1% ~ (p=0.959 n=8+8)
IndirectIndex_ReadEntries-8 5.99µs ± 1% 6.07µs ± 1% +1.29% (p=0.001 n=8+8)
IndirectIndex_DeleteRangeLast-8 369ns ± 2% 566ns ± 1% +53.37% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 368ms ± 9% 369ms ± 2% ~ (p=0.232 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 600ms ± 1% 618ms ± 0% +3.03% (p=0.000 n=8+7)
IndirectIndex_Delete-8 50.0ms ± 1% 47.6ms ± 9% ~ (p=0.463 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.776 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.01kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.457 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 24.7MB/s ±10% 17.8MB/s ± 2% -28.18% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 14.2MB/s ± 1% 9.6MB/s ± 0% -32.30% (p=0.000 n=8+7)
IndirectIndex_Delete-8 171MB/s ± 1% 126MB/s ±10% -26.35% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeLast went from 17 page faults, or ~180GB/sec at 369ns/op
to zero page faults. So even though it got 50% slower, it was actually I/O bound
and no longer is.
2019-01-07 11:00:35 -07:00
0becfc6239
tsm1: add helper to track page faults in index
...
Since the methods inline and dead code is eliminated, it has no runtime
overhead in the benchmarks when disabled.
benchmark recorded faults
BenchmarkIndirectIndex_Entries-8 11
BenchmarkIndirectIndex_ReadEntries-8 11
BenchmarkIndirectIndex_DeleteRangeLast-8 17
BenchmarkIndirectIndex_DeleteRangeFull-8 2218
BenchmarkIndirectIndex_Delete-8 2084
2019-01-07 11:00:35 -07:00
91e820a9d8
tsm1: fix multiple issues with DeleteRange
...
1. Correctly acquires locks
2. Seeks for discontiguous key ranges (like delete ["aaa", "zzz"])
3. Is precise about deleting a key when it contains no data
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.3ms ± 1% 63.2ms ±15% ~ (p=0.463 n=7+8)
IndirectIndex_Entries-8 9.14µs ± 1% 9.01µs ± 0% -1.40% (p=0.004 n=8+7)
IndirectIndex_ReadEntries-8 5.83µs ± 1% 5.68µs ± 2% -2.62% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 283ns ± 2% 191ns ± 1% -32.37% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull-8 612ms ± 1% 361ms ± 1% -41.02% (p=0.000 n=8+8)
IndirectIndex_Delete-8 49.0ms ± 1% 49.8ms ± 1% +1.80% (p=0.001 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.6MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 64.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 168MB ± 0% 162MB ± 0% -3.71% (p=0.000 n=8+8)
IndirectIndex_Delete-8 3.94kB ± 0% 3.94kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 35.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 2.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 1.04M ± 0% 0.52M ± 0% -49.77% (p=0.000 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
2019-01-07 11:00:35 -07:00
aed17cfedd
tsm1: speed up IndirectIndex benchmarks
...
rather than create the indirectIndex every Benchmark iteration
reuse a global one. reduces a couple benchmarks where Start/Stop timer
weren't in palce.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 67710057 69216355 +2.22%
BenchmarkIndirectIndex_Entries-8 9239 9762 +5.66%
BenchmarkIndirectIndex_ReadEntries-8 5964 5886 -1.31%
BenchmarkIndirectIndex_DeleteRangeLast-8 317 284 -10.41%
BenchmarkIndirectIndex_DeleteRangeFull-8 615346992 598392398 -2.76%
BenchmarkIndirectIndex_Delete-8 52906315 44400269 -16.08%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1038932 1038722 -0.02%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 0 -100.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 168112352 168061952 -0.03%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
d40c3e662f
tsm1: use uint32 key for tombstones
...
rough, noisy benchmarks.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 62462250 67710057 +8.40%
BenchmarkIndirectIndex_Entries-8 9601 9239 -3.77%
BenchmarkIndirectIndex_ReadEntries-8 5984 5964 -0.33%
BenchmarkIndirectIndex_DeleteRangeLast-8 314 317 +0.96%
BenchmarkIndirectIndex_DeleteRangeFull-8 813838165 615346992 -24.39%
BenchmarkIndirectIndex_Delete-8 52079181 52906315 +1.59%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532670 1038932 -32.21%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 232738960 168112352 -27.77%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
ffd35ce1aa
tsm1: use a uint32 for offsets globally
...
benchmarks are flat.
2019-01-07 11:00:35 -07:00
7a7a4b6d58
tsm1: remove offsets from mmap
...
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 74525387 66439305 -10.85%
BenchmarkIndirectIndex_Entries-8 8892 9200 +3.46%
BenchmarkIndirectIndex_ReadEntries-8 5816 5691 -2.15%
BenchmarkIndirectIndex_DeleteRangeLast-8 1550 311 -79.94%
BenchmarkIndirectIndex_DeleteRangeFull-8 773649708 767030277 -0.86%
BenchmarkIndirectIndex_Delete-8 79755991 52015903 -34.78%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3 2 -33.33%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532589 1532344 -0.02%
BenchmarkIndirectIndex_Delete-8 246 123 -50.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3264 64 -98.04%
BenchmarkIndirectIndex_DeleteRangeFull-8 232710448 232624208 -0.04%
BenchmarkIndirectIndex_Delete-8 4432 3936 -11.19%
2019-01-07 11:00:35 -07:00
9d8114ef88
Fix tombstone error logic
2018-12-18 13:00:13 +00:00
419db63f7a
Address PR feedback
2018-12-18 12:33:28 +00:00
262772544a
Add prefix key to tombstoner
...
This commit adds support for "prefix keys". Prefix keys differ from
regular tombstone key entries in that the key of the entry should act as
a prefix that matches all series with the same prefix key for the given
time range.
This means only one entry is needed to delete many series.
The tombstone entries now have a maximum length of 16777215 (24 bits),
with the remaining 8 high bits available for setting further options /
meta information about the tombstone entry.
In this case, the top bit is used to indicate that the tombstone entry
is intended to be a prefix. This leaves 7 spare bits for future use.
2018-12-18 12:33:28 +00:00
1b5ad5e129
Ensure v4 tombstone file read correctly
2018-12-18 12:33:28 +00:00
90f3583fe0
Remove old tombstone version support
2018-12-18 12:33:28 +00:00
04605eb266
tsm1: speed up deleterange for large keys
...
rather than starting at the first key, do a binary search to the
first key. changes O(N) when deleting the largest key to O(log N).
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_DeleteRangeFull-8 17884166763 738717473 -95.87%
2018-12-14 10:06:24 -07:00
687a390aaf
tsm1: add benchmarks for deletes
2018-12-14 10:06:24 -07:00
f383e8337a
test(tsdb/tsi1): test series id cache delete concurrently
...
Report the total number of gets, puts, and deletes at the end of the
test. I've found this kind of output to be a useful sanity check in
similar tests that exercise concurrency involving tasks.
Use a local random source in each goroutine. I unscientifically
eyeballed that to increase total operations by 5-10%.
Also call t.Parallel in a few more tests that involve disk access. This
shaves 1-2 seconds off the full tsi1 test suite on my machine.
2018-12-12 08:21:17 -08:00
eba485f2be
Fix nil tracker for full compactions
2018-12-11 18:30:59 +00:00
c4b42c72be
Add option to disable TSI metrics
2018-12-10 15:02:26 +00:00
6b63a3def7
Add option to disable sfile metrics
2018-12-10 14:36:28 +00:00
3ff39cd9dc
Merge pull request #1789 from influxdata/er-cache-size
...
Allow TSI cache to be dynamically altered
2018-12-10 11:23:15 +00:00
df0b084543
Merge pull request #1785 from influxdata/bj-tss-file-observer
...
Allow stats files to be observed for finishing/unlinking.
2018-12-07 18:46:35 -07:00
e0cddadffd
Allow TSI cache to be dynamically altered
2018-12-07 18:35:25 +00:00
73d8c85aa2
Allow stats files to be observed for finishing/unlinking.
...
This commit adds the `.tss` files generated for TSM statistics to
the `FileObserver` so that package users can be notified when new
stats files are created and removed.
2018-12-07 10:20:32 -07:00
e13309ebbe
Fix metric names
2018-12-07 16:37:17 +00:00
b015757c06
Ensure all tsi1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
bff655786f
Ensure tsdb metrics properly registered
2018-12-07 14:32:34 +00:00
aa936df138
Ensure all tsm1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
d94f898c8b
WIP
2018-12-07 14:32:34 +00:00
79b108d174
Fix bug with slice reuse
2018-12-07 14:32:34 +00:00
de491968ba
Fix rebase
2018-12-07 14:32:34 +00:00
93892c20ab
Fix test
2018-12-07 14:32:34 +00:00
a8b6827c9e
megacheck
2018-12-07 14:32:34 +00:00
2bb558a9d1
Ensure fileset protected by lock
2018-12-07 14:32:34 +00:00
a1804d27be
Fix race
2018-12-07 14:32:34 +00:00
f9a2f7a017
go fmt
2018-12-07 14:32:34 +00:00
e0c10227d0
Fix metric issue in series file
2018-12-07 14:32:34 +00:00
7960ccc320
Add TSI index metrics
2018-12-07 14:32:34 +00:00
55caa0fe54
Add RHH metrics
2018-12-07 14:32:34 +00:00
d1fe2bc188
Add series file metrics
2018-12-07 14:32:34 +00:00
8ca637bd80
Refactor default labels and retention metrics
2018-12-07 14:32:34 +00:00
6c5dec8f88
Refactor tracker names
2018-12-07 14:32:34 +00:00
44e5fbae0a
Convert WAL stats
2018-12-07 14:32:34 +00:00
3b980ed7e3
Convert Cache statistics
2018-12-07 14:32:34 +00:00
d61b9f1645
Convert Filestore stats
2018-12-07 14:32:34 +00:00
f56bc0853f
Convert TSM compaction stats to Prom metrics
...
This commits converts all the 1.x TSM compaction statistics, which
previously were written to an _internal db, to Prometheus metrics.
2018-12-07 14:32:34 +00:00
186e0392ed
Address PR feedback
2018-11-30 10:54:24 +00:00
e11789f46a
Omit unused receiver name: ST1006
2018-11-30 10:54:24 +00:00
eaa4a4f49a
Removes unused methods: U1000
2018-11-30 10:54:24 +00:00
9403c1ec8e
Ensure error strings not capitalised ST1005
2018-11-30 10:54:24 +00:00
308a5148cf
Remove iterators
2018-11-30 10:54:24 +00:00
98d24f7e3c
Merge pull request #1625 from influxdata/remove-influxdb-dependency
...
Remove influxdb dependency.
2018-11-29 14:23:00 -07:00
0084d4d824
Remove influxdb dependency.
2018-11-29 11:44:22 -07:00
7ccb201b80
Merge pull request #1332 from zhulongcheng/rm-create-series
...
Remove Index.CreateSeriesIfNotExists
2018-11-29 18:36:48 +00:00
1862b4421d
Integrate scanned values statistics tracking.
2018-11-28 15:32:06 -07:00
e22aff46cb
Force create TSS files.
...
This commit replaces an `os.OpenFile()` call with an `os.Create()`
call which drops `O_EXCL` for `O_TRUNC` since `.tss` files are only
created after `.tsm` files so lingering temporary files are safe to
overwrite.
2018-11-27 08:07:21 -07:00
ed799a3d6c
remove Index.CreateSeriesIfNotExists
2018-11-21 20:16:45 +08:00
dbfa140cc4
remove Engine.CreateSeriesIfNotExists
2018-11-21 20:16:45 +08:00
085ce852b7
remove CreateSeriesIfNotExists from engine tests
2018-11-21 20:16:45 +08:00
8ab01c99c0
test(tsdb/tsm1): skip long tests in short mode
...
The tsdb/tsm1 package was one of the test suites that took the longest
to run in platform with go test -short. The rule of thumb on the Go
project is that short mode should skip any individual test that takes
longer than one second. This change skips two such tests, and it
eliminates a string concatenation loop in two other tests, so that they
report completion in "0.00s" rather than about 0.94s, on my machine.
These cumulative changes take `go test -short ./tsdb/tsm1` from about 14
seconds to about 7 seconds on my machine.
2018-11-16 08:06:23 -08:00
bbd460e7d9
Add method QueryRawJSON to influxql.service (for querytest tool) ( #1402 )
2018-11-15 10:45:38 -08:00
e7bc29a590
reduce parsing and copying of tags
2018-11-15 20:45:16 +08:00
305ebb8729
fix: Allow compactor to make progress if v.MaxTime() != entry.MaxTime
2018-11-14 12:14:45 +00:00
b35533e7f7
chore: Compactor test which replicates issue #10465
...
Due to an encoding bug with simple8b, it is possible that the
MaxTime for a TSM index entry does not match the last encoded timestamp.
2018-11-14 12:14:43 +00:00
f731ed595d
Merge pull request #1358 from influxdata/jmw-test-explode-points
...
test(tsdb): add test for explode points
2018-11-13 14:41:29 -07:00
1ab9c80ae8
fix(tsdb): eliminate data race from *SeriesIDSet.Clone
...
And add a test to cover that.
The data race would look roughly like:
```
WARNING: DATA RACE
Write at 0x00c000024e18 by goroutine 8:
github.com/RoaringBitmap/roaring.(*roaringArray).markAllAsNeedingCopyOnWrite()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:881 +0x6b
github.com/RoaringBitmap/roaring.(*roaringArray).clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:266 +0x808
github.com/RoaringBitmap/roaring.(*Bitmap).Clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaring.go:385 +0x58
github.com/influxdata/platform/tsdb.(*SeriesIDSet).CloneNoLock()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:229 +0x73
github.com/influxdata/platform/tsdb.(*SeriesIDSet).Clone()
Previous write at 0x00c000024e18 by goroutine 7:
github.com/RoaringBitmap/roaring.(*roaringArray).markAllAsNeedingCopyOnWrite()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:881 +0x6b
github.com/RoaringBitmap/roaring.(*roaringArray).clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaringarray.go:266 +0x808
github.com/RoaringBitmap/roaring.(*Bitmap).Clone()
/Users/mr/go/pkg/mod/github.com/!roaring!bitmap/roaring@v0.4.16/roaring.go:385 +0x58
github.com/influxdata/platform/tsdb.(*SeriesIDSet).CloneNoLock()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:229 +0x73
github.com/influxdata/platform/tsdb.(*SeriesIDSet).Clone()
/Users/mr/go/src/github.com/influxdata/platform/tsdb/series_set.go:223 +0x7b
```
2018-11-13 08:12:38 -08:00
704941d624
test(tsdb): add test for explode points
2018-11-12 17:36:33 -07:00
39f4908946
fix(storage): allow disabling the WAL
...
We were passing a non-nil tsm1.Log containing a nil *tsm1.WAL which
would cause a panic when it was attempted to be used. Instead, always
pass a non-nil WAL.
We change the storage engine code to not pass in a nil WAL, and
additionally add a defensive check to change any nil WALs into a
NopWAL.
2018-11-09 10:45:24 -07:00
25532778df
fix(tsm1): fix max concurrent compaction logic
2018-11-09 10:14:32 -07:00
4b504b84df
respond to review feedback
...
- Add some documentation.
- Move compaction planner to an option instead of config.
The latter fits with the general theme of having config be things
that can be specified in a toml, and everything else being an
option.
2018-11-08 11:39:36 -07:00
a1b5b322bb
some more refactoring
...
- add helpers to get directories out
- change FileStoreObserver to be an option rather than config.
2018-11-08 11:39:36 -07:00
22e23d6e31
final touches
...
- move default directories to the storage package
- make the directory layout match before
- clean up some dead missed functions
2018-11-08 11:39:36 -07:00
2cbc2ee896
refactor wal out, paths, and options
2018-11-08 11:39:36 -07:00
932b0bf01a
compat: Package to convert old to new config
2018-11-08 11:39:36 -07:00
0d411023f2
config: clean up
...
- Breaks the weird cycle that existed with the EngineOptions
- Removes a bunch of useless parameters
- Moves around a bunch of defaults
2018-11-08 11:39:36 -07:00
594664a876
remove RegisteredIndexes tests
2018-11-05 00:00:02 +08:00
aeefeb2eed
remove RegisteredIndexes method
2018-11-04 21:56:57 +08:00
1857bf1084
Fix TSM index maxtime modification
2018-11-02 18:39:30 -06:00
985c260af7
chore(storage,tsdb): fix megacheck errors
2018-11-01 12:54:46 -07:00
a0300064df
feat(tsm1): Improve performance of Gorilla float block decoding
...
```
name old time/op new time/op delta
FloatArrayDecodeAll/1-8 45.9ns ± 1% 13.8ns ± 1% -70.00% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 686ns ± 0% 232ns ± 1% -66.10% (p=0.000 n=9+8)
FloatArrayDecodeAll/550-8 5.78µs ± 0% 2.22µs ± 1% -61.61% (p=0.000 n=9+9)
FloatArrayDecodeAll/1000-8 10.2µs ± 2% 4.0µs ± 5% -60.47% (p=0.000 n=10+10)
name old speed new speed delta
FloatArrayDecodeAll/1-8 414MB/s ± 1% 1383MB/s ± 1% +233.76% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 144MB/s ± 0% 424MB/s ± 1% +194.19% (p=0.000 n=9+9)
FloatArrayDecodeAll/550-8 133MB/s ± 0% 346MB/s ± 1% +160.09% (p=0.000 n=9+10)
FloatArrayDecodeAll/1000-8 135MB/s ± 2% 340MB/s ± 5% +153.03% (p=0.000 n=10+10)
```
2018-11-01 18:59:20 +00:00
353df7edca
Fix imports
2018-11-01 18:59:20 +00:00
e282d012c8
Address PR feedback
2018-11-01 18:59:20 +00:00
c21336af0a
fix(encoding): Improve array string encoding perf a little more
...
Encode the compressed data at the start internal buffer. This ensures
the returned slice maintains the entire capacity and is available for
subsequent use.
When we pool / reuse string buffers, this will help considerably.
Improvements over previous commit:
```
name old time/op new time/op delta
EncodeStrings/10/batch-8 542ns ± 1% 355ns ± 2% -34.53% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 5.29µs ± 1% 3.58µs ± 2% -32.20% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 48.6µs ± 0% 36.2µs ± 2% -25.40% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
EncodeStrings/10/batch-8 704B ± 0% 0B -100.00% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 9.47kB ± 0% 0.00kB -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 90.1kB ± 0% 0.0kB -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
EncodeStrings/10/batch-8 0.00 0.00 ~ (all equal)
EncodeStrings/100/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
296d39059a
fix(encoding): Improve simple8b another 6%; fix inconsequential bug
...
simple8b encodes deltas[1:], thus deltas[0] >= simple8b.MaxValue is
invalid.
Also changed loop calculating deltas, RLE and max to be similar to
batch timestamp, for greater consistency.
Improvements over previous commit:
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 1.50µs ± 1% 1.48µs ± 1% -1.40% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 6.10µs ± 0% 5.69µs ± 2% -6.58% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 1.50µs ± 1% 1.49µs ± 0% -1.21% (p=0.008 n=5+5)
```
Improvements overall:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.48µs ± 1% -27.25% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 5.69µs ± 2% -35.29% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.49µs ± 0% -26.93% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
9fa01f7115
feat(encoding): Improve timestamp encoding
...
Timestamp improvements prior to any improvements to simple8b
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 1.36µs ± 1% -48.25% (p=0.008 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 32.2µs ± 1% -49.64% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 1.30µs ± 1% -86.06% (p=0.008 n=5+5)
```
2018-11-01 18:59:20 +00:00
a339f8f620
feat(encoding): Improve integer and simple8b encoding performance
...
simple8b EncodeAll improvements should
```
name old time/op new time/op delta
EncodeAll/1_bit-8 28.5µs ± 1% 28.6µs ± 1% ~ (p=0.133 n=9+10)
EncodeAll/2_bits-8 28.9µs ± 2% 28.7µs ± 0% ~ (p=0.068 n=10+8)
EncodeAll/3_bits-8 29.3µs ± 1% 28.8µs ± 0% -1.70% (p=0.000 n=10+10)
EncodeAll/4_bits-8 29.6µs ± 1% 29.1µs ± 1% -1.85% (p=0.000 n=10+10)
EncodeAll/5_bits-8 30.6µs ± 1% 29.8µs ± 2% -2.70% (p=0.000 n=10+10)
EncodeAll/6_bits-8 31.3µs ± 1% 30.0µs ± 1% -4.08% (p=0.000 n=9+9)
EncodeAll/7_bits-8 32.6µs ± 1% 30.8µs ± 0% -5.49% (p=0.000 n=9+9)
EncodeAll/8_bits-8 33.6µs ± 2% 31.0µs ± 1% -7.77% (p=0.000 n=10+9)
EncodeAll/10_bits-8 34.9µs ± 0% 31.9µs ± 2% -8.55% (p=0.000 n=9+10)
EncodeAll/12_bits-8 36.8µs ± 1% 32.6µs ± 1% -11.35% (p=0.000 n=9+10)
EncodeAll/15_bits-8 39.8µs ± 1% 34.1µs ± 2% -14.40% (p=0.000 n=10+10)
EncodeAll/20_bits-8 45.2µs ± 3% 36.2µs ± 1% -19.97% (p=0.000 n=10+9)
EncodeAll/30_bits-8 55.0µs ± 0% 40.9µs ± 1% -25.62% (p=0.000 n=9+9)
EncodeAll/60_bits-8 86.2µs ± 1% 55.2µs ± 1% -35.92% (p=0.000 n=10+10)
EncodeAll/combination-8 582µs ± 2% 502µs ± 1% -13.80% (p=0.000 n=9+9)
```
EncodeIntegers:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.50µs ± 1% -26.22% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 6.10µs ± 0% -30.73% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.50µs ± 1% -26.04% (p=0.008 n=5+5)
```
EncodeTimestamps (ran is improved due to simple8b improvements)
```
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 2.65µs ± 2% ~ (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 33.8µs ± 1% -47.23% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 9.28µs ± 1% ~ (p=0.087 n=5+5)
```
2018-11-01 18:59:20 +00:00
5e7b2cb273
Fix index bug in float encoder
2018-11-01 18:59:20 +00:00
80c953b774
Add TSM batch key iterator
...
The batch focussed TSM key iterator iterates TSM blocks, decoding and
merging blocks where appropriate using the the batch focussed
approaches.
2018-11-01 18:59:20 +00:00
5074b834cd
Add batch block encoders
2018-11-01 18:59:19 +00:00
ab68204683
Batch oriented unsigned encoder
2018-11-01 18:59:19 +00:00
aeeef803c0
Batch oriented boolean encoders
...
This commit adds a tsm1 function for encoding a batch of booleans into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated sets of booleans.
2018-11-01 18:59:19 +00:00
5376530392
Improvements to batch float encoder
...
- Inlined the closure to avoid a function call.
- Changed append(b, make([]byte, 8)...) to inline the make call.
- Check for NaN once at the end assuming NaN is infrequent.
New performance delta comparing the current iterators to the new batch
function:
name old time/op new time/op delta
EncodeFloats/10_seq 1.32µs ± 2% 0.17µs ± 2% -87.39% (p=0.000 n=10+10)
EncodeFloats/10_ran 2.09µs ± 1% 0.15µs ± 0% -92.97% (p=0.000 n=10+9)
EncodeFloats/100_seq 8.37µs ± 2% 1.28µs ± 2% -84.74% (p=0.000 n=10+10)
EncodeFloats/100_ran 19.1µs ± 1% 1.3µs ± 1% -93.08% (p=0.000 n=9+9)
EncodeFloats/1000_seq 60.4µs ± 1% 12.6µs ± 0% -79.13% (p=0.000 n=9+7)
EncodeFloats/1000_ran 212µs ± 1% 12µs ± 1% -94.53% (p=0.000 n=9+8)
name old alloc/op new alloc/op delta
EncodeFloats/10_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/10_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/100_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/100_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_ran 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeFloats/10_seq 0.00 0.00 ~ (all equal)
EncodeFloats/10_ran 0.00 0.00 ~ (all equal)
EncodeFloats/100_seq 0.00 0.00 ~ (all equal)
EncodeFloats/100_ran 0.00 0.00 ~ (all equal)
EncodeFloats/1000_seq 0.00 0.00 ~ (all equal)
EncodeFloats/1000_ran 0.00 0.00 ~ (all equal)
2018-11-01 18:59:19 +00:00
d8b5f9d432
Batch oriented string encoders
...
This commit adds a tsm1 function for encoding a batch of strings into a
provided buffer. The new function also shares the buffer between the
input data and the snappy encoded output, reducing allocations.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated strings.
name old time/op new time/op delta
EncodeStrings/10 2.14µs ± 4% 1.42µs ± 4% -33.56% (p=0.000 n=10+10)
EncodeStrings/100 12.7µs ± 3% 10.9µs ± 2% -14.46% (p=0.000 n=10+10)
EncodeStrings/1000 132µs ± 2% 114µs ± 2% -13.88% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
EncodeStrings/10 657B ± 0% 704B ± 0% +7.15% (p=0.000 n=10+10)
EncodeStrings/100 6.14kB ± 0% 9.47kB ± 0% +54.14% (p=0.000 n=10+10)
EncodeStrings/1000 61.4kB ± 0% 90.1kB ± 0% +46.66% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
EncodeStrings/10 3.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeStrings/100 3.00 ± 0% 1.00 ± 0% -66.67% (p=0.000 n=10+10)
EncodeStrings/1000 3.00 ± 0% 1.00 ± 0% -66.67% (p=0.000 n=10+10)
2018-11-01 18:59:19 +00:00
7032aed1c3
Batch oriented timestamp encoders
...
This commit adds a tsm1 function for encoding a batch of timestamps into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders. They look
at a sequential input slice, a randomly generated input slice and a
duplicate slice. All slices are sorted.
name old time/op new time/op delta
EncodeTimestamps/10_seq 153ns ± 2% 104ns ± 2% -31.62% (p=0.000 n=9+10)
EncodeTimestamps/10_ran 191ns ± 2% 142ns ± 0% -25.73% (p=0.000 n=10+9)
EncodeTimestamps/10_dup 114ns ± 1% 68ns ± 4% -39.77% (p=0.000 n=8+10)
EncodeTimestamps/100_seq 704ns ± 2% 321ns ± 2% -54.44% (p=0.000 n=9+9)
EncodeTimestamps/100_ran 7.27µs ± 4% 7.01µs ± 2% -3.59% (p=0.000 n=10+10)
EncodeTimestamps/100_dup 756ns ± 3% 396ns ± 2% -47.57% (p=0.000 n=10+10)
EncodeTimestamps/1000_seq 6.32µs ± 1% 2.46µs ± 2% -61.01% (p=0.000 n=8+10)
EncodeTimestamps/1000_ran 108µs ± 0% 68µs ± 3% -37.57% (p=0.000 n=8+10)
EncodeTimestamps/1000_dup 7.26µs ± 1% 3.64µs ± 1% -49.80% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
EncodeTimestamps/10_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/10_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/10_dup 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_dup 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_dup 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeTimestamps/10_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/10_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/10_dup 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_dup 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_dup 0.00 0.00 ~ (all equal)
2018-11-01 18:59:19 +00:00
b463f97b15
Batch oriented int encoders
...
This commit adds a tsm1 function for encoding a batch of ints into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders. They look
at a sequential input slice, a randomly generated input slice and a
duplicate slice:
name old time/op new time/op delta
EncodeIntegers/10_seq 144ns ± 2% 41ns ± 1% -71.46% (p=0.000 n=10+10)
EncodeIntegers/10_ran 304ns ± 7% 140ns ± 2% -53.99% (p=0.000 n=10+10)
EncodeIntegers/10_dup 147ns ± 4% 41ns ± 2% -72.14% (p=0.000 n=10+9)
EncodeIntegers/100_seq 483ns ± 7% 208ns ± 1% -56.98% (p=0.000 n=10+9)
EncodeIntegers/100_ran 1.64µs ± 7% 1.01µs ± 1% -38.42% (p=0.000 n=9+9)
EncodeIntegers/100_dup 484ns ±14% 210ns ± 2% -56.63% (p=0.000 n=10+10)
EncodeIntegers/1000_seq 3.11µs ± 2% 1.81µs ± 2% -41.68% (p=0.000 n=10+10)
EncodeIntegers/1000_ran 16.9µs ±10% 11.0µs ± 2% -34.58% (p=0.000 n=10+10)
EncodeIntegers/1000_dup 3.05µs ± 3% 1.81µs ± 2% -40.71% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
EncodeIntegers/10_seq 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/10_ran 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/10_dup 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_seq 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_ran 128B ± 0% 0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_dup 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_seq 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_ran 1.15kB ± 0% 0.00kB -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_dup 32.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
EncodeIntegers/10_seq 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/10_ran 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/10_dup 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_seq 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_ran 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/100_dup 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_seq 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_ran 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeIntegers/1000_dup 1.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
2018-11-01 18:59:19 +00:00
8190edbf14
Batch oriented float encoders
...
This commit adds a tsm1 function for encoding a batch of floats into a
buffer. Further, it replaces the `bitstream` library used in the
existing encoders (and all the current decoders) with inlined bit
expressions within the encoder, significantly reducing the function call
overhead for larger batches.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders. They look
at a sequential input slice and a randomly generated input slice.
name old time/op new time/op delta
EncodeFloats/10_seq 1.14µs ± 3% 0.24µs ± 3% -78.94% (p=0.000 n=10+10)
EncodeFloats/10_ran 1.69µs ± 2% 0.21µs ± 3% -87.43% (p=0.000 n=10+10)
EncodeFloats/100_seq 7.07µs ± 1% 1.72µs ± 1% -75.62% (p=0.000 n=7+9)
EncodeFloats/100_ran 15.8µs ± 4% 1.8µs ± 1% -88.60% (p=0.000 n=10+9)
EncodeFloats/1000_seq 50.2µs ± 3% 16.2µs ± 2% -67.66% (p=0.000 n=10+10)
EncodeFloats/1000_ran 174µs ± 2% 16µs ± 2% -90.77% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
EncodeFloats/10_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/10_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/100_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/100_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_ran 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeFloats/10_seq 0.00 0.00 ~ (all equal)
EncodeFloats/10_ran 0.00 0.00 ~ (all equal)
EncodeFloats/100_seq 0.00 0.00 ~ (all equal)
EncodeFloats/100_ran 0.00 0.00 ~ (all equal)
EncodeFloats/1000_seq 0.00 0.00 ~ (all equal)
EncodeFloats/1000_ran 0.00 0.00 ~ (all equal)
2018-11-01 18:59:19 +00:00
29114ec5f2
Rename time batch decoders
2018-11-01 18:59:19 +00:00
095ed44f48
Rename unsigned batch decoders
2018-11-01 18:59:19 +00:00
d7a4b814d4
Rename string batch decoders
2018-11-01 18:59:19 +00:00
db84dfae92
Rename boolean batch decoders
2018-11-01 18:59:19 +00:00
bcb7b5d44a
Rename integer batch decoders
2018-11-01 18:59:19 +00:00
2e00954703
Rename float batch decoders
2018-11-01 18:59:19 +00:00
6830329ef4
review feedback
2018-10-31 15:41:39 -06:00
a7657ac409
tsdb: remove hll sketches
...
This keeps file compatability by just writing out zeros for the
sizes and offsets. Perhaps it's ok to just nuke everything and
remove the data.
It also keeps the hll package because it seems generally useful
even if it's not currently being used.
2018-10-31 15:41:39 -06:00
381d449b82
tsm1: remove digests and backup/restore
2018-10-31 15:41:07 -06:00
d8548d41e1
chore(fmt): update formating with make fmt
2018-10-30 07:40:28 -05:00
46a7b8155a
Merge pull request #1170 from zhulongcheng/rm-index
...
refactor(tsdb): remove tsdb.Index and tsdb.IndexSet
2018-10-30 11:10:54 +00:00
67dc4d8cdd
fix: conform to logging style guide for initial log messages
...
These are the log messages that get printed immediately when starting
the application for the first time. This fixes the messages to conform
to the logging style guide.
2018-10-29 16:42:55 -05:00
1dd0d33b1e
fix type assertion err
2018-10-27 02:08:31 +08:00
268832ee64
remove unused seriesPointIterator
2018-10-27 02:08:31 +08:00
f6104a7e78
remove unused Shard
2018-10-27 02:08:31 +08:00
9d29874e20
move SeriesFileDirectory constant to defaults package
2018-10-27 02:08:31 +08:00
5d66bbed48
remove functions for registering engine
...
This fix is to resolve import cycle
2018-10-27 02:08:31 +08:00
0e9185f764
remove tsdb.Index interface
...
This fix is to resolve #886 .
2018-10-27 02:08:31 +08:00
c89c79dc02
replace tsdb.Index interface with tsi1.Index instance
...
This fix is to remove tsdb.Index interface to resolve #886 .
2018-10-27 02:08:31 +08:00
c1e732782e
remove tsdb.IndexSet
...
This fix is to resolve #886 .
2018-10-27 02:08:31 +08:00
427d719af8
remove tsdb.IndexSet tests
...
This fix is to remove tsdb.IndexSet to resolve #886 .
2018-10-27 02:08:31 +08:00
28fecc1f6f
replace tsdb.IndexSet with tsi1.Index
...
This fix is to remove tsdb.IndexSet to resolve #886 .
2018-10-27 02:08:31 +08:00
e739afb77d
add tsi1.Index methods
...
These methods are copied from tsdb.IndexSet and modified slightly.
This fix is to remove tsdb.IndexSet to resolve #886 .
2018-10-27 02:08:31 +08:00
39b9c26db4
chore: add String method to FieldType
...
Type conflicts should be rare, but when they do happen, printing out the
string name should save developers a couple minutes of digging compared
to looking up which numeric value means which type.
2018-10-26 08:24:18 -07:00
b6a1c0e9c7
storage: MeasurementStats.ReadFrom requires ByteReader
2018-10-19 14:16:20 -07:00
d856116b00
Add tsi1 measurement cardinality stats.
2018-10-17 08:38:41 -06:00
5c8318d181
Merge pull request #1080 from influxdata/js-go-generate-with-go-run
...
refactor: modify tooling to take advantage of go run
2018-10-15 11:27:13 -05:00
ce3e1fa90d
Remove the fields index
...
This commit removes the remaining bits of the fields index. In doing
so, the buildCursor method on the engine would need to be updated.
It turns out, that code was statically dead, so delete it and anything
that depended on it. Additionally, delete anything as reported by
the unused tool in the tsdb package.
2018-10-15 10:22:07 -06:00
e9600b1f0b
refactor: modify tooling to take advantage of go run
...
The generate commands have been modified to take advantage of the new
functionality in Go 1.11 that allows `go run` to execute a package
instead of individual files.
This functionality combined with Go modules allows us to execute a
package directly out of our pinned dependencies rather than accidentally
picking up another binary outside of the build environment.
This also simplifies the Makefile because they no longer have to be
responsible for installing the correct tooling since the Go command
takes care of that logic. It also makes it so that the Makefiles with
file generation can now be invoked from their appropriate subdirectories
so they are contained within the directory itself rather than relying on
values in the top level Makefile.
It is now possible to generate all files within this project by using:
go generate ./...
Or the Makefile can continue to be used.
This commit also copies over the special copy of `tmpl` that the storage
engine uses within the influxdb repository. It was never copied over so
using `go generate` on these packages did not work.
2018-10-15 11:01:27 -05:00
eeb9580bff
fix(engine): remove SeriesIDSets check in deletion
...
We leave in the options and stuff for now as it's tied in with
a lot of the tests/multiple shard stuff that will be removed
eventually, anyway.
2018-10-12 15:25:10 -06:00
8a67febf35
Merge pull request #1044 from influxdata/js-move-toml-package
...
refactor(toml): copy the toml utility package from influxdb to platform
2018-10-11 13:03:15 -05:00
8dee884038
refactor(toml): copy the toml utility package from influxdb to platform
2018-10-11 12:08:43 -05:00
590967c092
Add skeleton buildtsi
2018-10-11 17:45:47 +01:00
21475c22cd
feature(tsdb): decode name back into IDs
...
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
2018-10-11 09:54:19 -05:00
919aae41c8
refactor(id): update storage, read, and flux from to use IDs
2018-10-11 09:54:19 -05:00
639a245fa9
fix(tsdb): encode org/bucket bytes
2018-10-11 09:54:18 -05:00
e7e17d6972
refactor: move the logger package from influxdb to platform
...
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
2018-10-10 15:49:07 -05:00
0af8de9a3c
Merge pull request #1001 from influxdata/jmw-refactor-storage-reads
...
Refactor storage reads
2018-10-09 10:54:12 -06:00
4d58c7df8c
Merge pull request #1000 from influxdata/bj-fix-series-index-tombstone
...
fix(tsdb): Fix series file tombstoning.
2018-10-09 10:41:18 -06:00
810833f33f
chore: refactor reads service and make it consumable externally
...
This pulls in the code that allows doing reads with flux into the
platform repo, and removes extra.go.
The reusable portion is under storage/reads, where the concrete
implementation for one of the platform's engines is in
storage/readservice.
In order to make this more reusable, the cursors had to move into
their own package, decoupling it from all of the other code in the
tsdb package. tsdb/cursors is this new package, and type/function
aliases have been added to the tsdb package to point at it.
The models package already is very light on transitive dependencies
and so it was allowed to be depended on in a concrete way in the
cursors package.
Finally, the protobuf definitions for issuing GRPC reads has been
moved into its own package for two reasons:
1. It's a clean separation, and helps keep it that way.
2. Many/most consumers will not be using GRPC. We just
use the datatypes to express the API which helps making
a GRPC server easier.
It is left up to future refactorings (specifically ones that involve
GPRC) to determine if these types should remain, or if there is a
cleaner way.
There's still some dependencies on both github.com/influxdata/influxql
and github.com/influxdata/influxdb/logger that we can hopefully remove
in future refactorings.
2018-10-09 09:51:13 -06:00
555c454cd8
Remove the radix tree series type conflict stuff
2018-10-09 09:05:20 -06:00
90494a5309
chore(fmt): fix and simplify go formatting
2018-10-08 17:23:54 -05:00
3ce51e7f02
fix(tsdb): Fix series file tombstoning.
...
This commit fixes an issue with the series file compaction
process where tombstones are lost after compaction and series
existence checks are not correct. This commit also fixes some
smaller flushing issues within the series file that mainly
related to testing.
2018-10-08 14:16:11 -06:00
cf3c70a1a0
Merge pull request #927 from influxdata/bj-tsm1-stats
...
feat(tsdb/tsm1): Add TSM1 measurement stats.
2018-10-08 13:44:02 -06:00
0360fc418c
Remove empty index package for tsi1
2018-10-08 13:18:30 -06:00
260ed3eb13
chore: don't depend on details of what platform.ID is
...
It may become a uint64 in the future, for example. This does mean
that we have to call Decode on some data that we just Encoded, but
we can fix that later.
2018-10-08 12:31:39 -06:00
68450681ef
Add TSM1 measurement stats.
...
This commit generates an additional `.tss` stats file alongside each
TSM file when it is written that contains size stats for all measurements
within the TSM file. These files can be combined to generate stats for
all measurements across all TSM files.
2018-10-08 10:43:53 -06:00
6408640836
fix: dont conflict on metrics namespace and mod tidy
2018-10-05 16:25:59 -06:00
512a015add
Address PR feedback
2018-10-05 18:01:59 +01:00
6b23165146
Add Engine tests
2018-10-05 15:04:09 +01:00
81e0fbabeb
Integrate WAL into engine
2018-10-05 12:44:27 +01:00
e85999ed45
Enable the WAL
2018-10-05 12:44:26 +01:00
69015d55ef
Address PR feedback
2018-10-04 18:18:55 +01:00
41aba8b539
Tidy up tag keys
2018-10-04 10:52:36 +01:00
cfb3df203a
Working storage engine
2018-10-04 10:24:43 +01:00
981b2cdbea
Skeleton storage engine
2018-10-04 10:24:43 +01:00
d18814267e
Add very basic write support
2018-10-04 10:21:22 +01:00
6a0fc9237c
Read/write points with the appropriate measurement
2018-10-03 17:39:30 -06:00
c659c24f9a
Add very basic write support
2018-10-03 17:39:30 -06:00
ea02772fc5
Pull in more pkg packages
2018-10-01 12:08:37 +01:00
ed81da6600
Update roaring dependency
2018-10-01 12:08:37 +01:00
6e57211d36
Fix tests
2018-10-01 12:08:37 +01:00
7de29bfe01
Update tsi1 package from OSS
2018-10-01 12:08:37 +01:00
3385f389f7
Update tsdb package from OSS
2018-10-01 12:08:37 +01:00
074f263e08
Initial import of tsm1.Engine
2018-10-01 12:08:37 +01:00
1280f4ac5e
Initial import of tsm1 package
2018-10-01 12:08:37 +01:00
aa433a723a
Removed some inmem references
2018-10-01 12:08:37 +01:00
e4cca868f4
Get TSI tests passing
2018-10-01 12:08:37 +01:00
6d3c74ecd9
Update imports
2018-10-01 12:08:37 +01:00
a7e23519b9
Initial import of tsi1 package
2018-10-01 12:08:33 +01:00
53f38073cc
Change import
2018-10-01 12:03:20 +01:00
fb0db04bc1
Initial import pkg package
2018-10-01 12:03:20 +01:00
f0ba72f227
Remove FGA auth hooks
2018-10-01 12:03:19 +01:00
04818c7859
Initial import of models package
2018-10-01 12:03:19 +01:00
d44b583c4d
remove code as reported by the unused tool
2018-10-01 12:03:19 +01:00
b0a317a34c
remove and document some things
2018-10-01 12:03:19 +01:00
8a21a3568a
prune Shard and reorganize
2018-10-01 12:03:19 +01:00
f2381de7bf
regenerate internal protobuf with non-conflicting package
2018-10-01 12:03:19 +01:00
992884ab6c
initial import of tsdb package
2018-10-01 12:03:19 +01:00
Jeff Wendling
Edd Robinson
Mark Rushakoff
Ben Johnson
zhulongcheng
Christopher M. Wolff
Stuart Carnie
Jeff Wendling
Jessica Obermark
Chris Goller
Jonathan A. Sternberg
Jacob Marble
Leonardo Di Donato