2e5ebbe251
perf(storage): reduce allocations when deleting from cache
...
When deleting from the cache, each cache key must be checked to
determine if it matches the prefix we're deleting. Since the keys are
stored as strings in the cache (map keys) there were a lot of allocations
happening because `applySerial` expects `[]byte` keys.
It's beneficial to reduce allocations by refacting `applySerial` to work
on strings. Whilst some allocations now have to happen the other way
(string -> []byte), they only happen if we actually need to delete the
key from the cache. Most of the keys don't get deleted so it's better
doing it this way.
Performance on the benchmark from the previous commit improved by ~40-50%.
name old time/op new time/op delta
Engine_DeletePrefixRange_Cache/exists-24 102ms ±11% 59ms ± 3% -41.95% (p=0.000 n=10+8)
Engine_DeletePrefixRange_Cache/not_exists-24 97.1ms ± 4% 45.0ms ± 1% -53.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
Engine_DeletePrefixRange_Cache/exists-24 25.5MB ± 1% 3.1MB ± 2% -87.83% (p=0.000 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 23.9MB ± 1% 0.1MB ±86% -99.65% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Engine_DeletePrefixRange_Cache/exists-24 305k ± 1% 28k ± 1% -90.77% (p=0.000 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 299k ± 1% 1k ±63% -99.74% (p=0.000 n=9+10)
Raw benchmarks on a 24T/32GB/NVME machine are as follows:
goos: linux
goarch: amd64
pkg: github.com/influxdata/influxdb/tsdb/tsm1
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 50379720 ns/op 3054106 B/op 27859 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 57326032 ns/op 3124764 B/op 28217 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 58943855 ns/op 3162146 B/op 28527 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 60565115 ns/op 3138811 B/op 28176 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 200 59775969 ns/op 3087910 B/op 27921 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59530451 ns/op 3120986 B/op 28207 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59185532 ns/op 3113066 B/op 28302 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59295867 ns/op 3100832 B/op 28108 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 300 59599776 ns/op 3100686 B/op 28113 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 200 62065907 ns/op 3048527 B/op 27879 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44979062 ns/op 123026 B/op 1244 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44733344 ns/op 52650 B/op 479 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44534180 ns/op 35119 B/op 398 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45179881 ns/op 105256 B/op 706 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44918964 ns/op 47426 B/op 621 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45000465 ns/op 63164 B/op 564 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45332999 ns/op 117008 B/op 1146 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45652342 ns/op 66221 B/op 616 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 45083957 ns/op 154354 B/op 1143 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 300 44560228 ns/op 65024 B/op 724 allocs/op
PASS
ok github.com/influxdata/influxdb/tsdb/tsm1 1690.583s
2019-08-30 20:35:05 +01:00
d160585a34
refactor(storage): add deeper tracing around deletes
2019-08-22 11:08:33 +01:00
740d669514
tsm1: teach the cache about predicates
2019-05-01 13:40:40 -06:00
fad1e07d1d
tsm1: clean up some dead/useless code in the cache
...
The storer interface isn't necessary if the init/Free logic is
removed, which is unnecessary in a world with only one shard.
Additionally, there were some cases where an init/Free call could
race and cause data loss in the cache. Not doing it at all fixes
all of those races.
2019-03-30 10:24:15 -06:00
a424bf3e4c
tsm1: implement DeleteBucketRange for the Cache
2019-02-04 10:32:52 -07:00
376b347d56
wal: change deletes to be based on DeleteBucket
2019-02-04 10:32:52 -07:00
c9bb55b889
refactor: move the tsm1/wal into the storage/wal package
...
Because the WAL relies on the tsm1.Value type, we move that into its own
tsm1/value package and set up some aliases forwarding them into tsm1. This
also required adding some methods and changing consumers to avoid the
unexported fields. I imagine this step will be useful one day when we make
the write path more efficient with respect to consuming points.
This commit additionally fixes some issues with generation. The iterator.tmpldata
and generation for array_cursor_* were removed accidentally when removing
iterators, making those generated files stale. Restore that and regenerate.
No change in functionality.
2019-02-04 10:32:52 -07:00
6c5dec8f88
Refactor tracker names
2018-12-07 14:32:34 +00:00
3b980ed7e3
Convert Cache statistics
2018-12-07 14:32:34 +00:00
074f263e08
Initial import of tsm1.Engine
2018-10-01 12:08:37 +01:00
Edd Robinson
Jeff Wendling