2e5ebbe251
perf(storage): reduce allocations when deleting from cache
...
When deleting from the cache, each cache key must be checked to
determine if it matches the prefix we're deleting. Since the keys are
stored as strings in the cache (map keys) there were a lot of allocations
happening because `applySerial` expects `[]byte` keys.
It's beneficial to reduce allocations by refacting `applySerial` to work
on strings. Whilst some allocations now have to happen the other way
(string -> []byte), they only happen if we actually need to delete the
key from the cache. Most of the keys don't get deleted so it's better
doing it this way.
Performance on the benchmark from the previous commit improved by ~40-50%.
name old time/op new time/op delta
Engine_DeletePrefixRange_Cache/exists-24 102ms ±11% 59ms ± 3% -41.95% (p=0.000 n=10+8)
Engine_DeletePrefixRange_Cache/not_exists-24 97.1ms ± 4% 45.0ms ± 1% -53.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
Engine_DeletePrefixRange_Cache/exists-24 25.5MB ± 1% 3.1MB ± 2% -87.83% (p=0.000 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 23.9MB ± 1% 0.1MB ±86% -99.65% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Engine_DeletePrefixRange_Cache/exists-24 305k ± 1% 28k ± 1% -90.77% (p=0.000 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 299k ± 1% 1k ±63% -99.74% (p=0.000 n=9+10)
Raw benchmarks on a 24T/32GB/NVME machine are as follows:
goos: linux
goarch: amd64
pkg: github.com/influxdata/influxdb/tsdb/tsm1
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 50379720 ns/op 3054106 B/op 27859 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 57326032 ns/op 3124764 B/op 28217 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 58943855 ns/op 3162146 B/op 28527 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 60565115 ns/op 3138811 B/op 28176 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 200 59775969 ns/op 3087910 B/op 27921 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59530451 ns/op 3120986 B/op 28207 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59185532 ns/op 3113066 B/op 28302 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59295867 ns/op 3100832 B/op 28108 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59599776 ns/op 3100686 B/op 28113 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 200 62065907 ns/op 3048527 B/op 27879 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44979062 ns/op 123026 B/op 1244 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44733344 ns/op 52650 B/op 479 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44534180 ns/op 35119 B/op 398 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45179881 ns/op 105256 B/op 706 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44918964 ns/op 47426 B/op 621 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45000465 ns/op 63164 B/op 564 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45332999 ns/op 117008 B/op 1146 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45652342 ns/op 66221 B/op 616 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45083957 ns/op 154354 B/op 1143 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44560228 ns/op 65024 B/op 724 allocs/op
PASS
ok github.com/influxdata/influxdb/tsdb/tsm1 1690.583s
2019-08-30 20:35:05 +01:00
945f16ff4b
docs(tsm1): TagKeys and TagValues return partial results with non-nil errors
2019-08-01 13:47:11 -07:00
e7a0068e46
fix(tsm1): TagKeys/TagValues returns when context canceled
2019-08-01 13:47:11 -07:00
bf774b66ce
fix(storage): Ensure Tag(Keys|Values) APIs never return (nil, nil)
...
Formalized this post condition in the documentation and added additional
unit tests.
Added a nil guard and unit test to WriteStringIterator.
2019-05-02 09:45:38 -07:00
7b97a41dcb
feat(storage): Teach TagKeys, TagValues how to accumulate statistics
...
This commit teaches the storage schema APIs how to track statistics
and make them available via the returned `cursors.StringIterator`.
Statistics are only tracked when decoding TSM blocks or when scanning
the in-memory cache.
Closes #13541
2019-04-24 11:14:22 -07:00
ed344d25f8
feat(storage): Teach storage how to find a distinct set of tag keys
...
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagKeys expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement. This is tracked by issue #13497 .
2019-04-24 11:14:22 -07:00
972cda1775
feedback: Changes in response to PR feedback
2019-04-18 16:19:18 -07:00
d3790aa072
feat: Teach storage engine how to find tag values for a given key
...
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagValues expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement.
2019-04-18 16:19:18 -07:00
Edd Robinson
jlapacik
Stuart Carnie