Commit Graph

23 Commits (3f3b7b5160a882ba513943fa0a72908a21ea6c50)

Author SHA1 Message Date
Edd Robinson efdddbb31a Allow TSI bitset cache size to be configured
This commit adds a config option to the tsdb Config allowing the size of
the bitset cached in the TSI index to be specified.

Setting the cache size to 0 will disable the cache.
2019-01-24 17:41:45 +00:00
Edd Robinson 3a055a6107 Fix cardinality estimation error
This commit fixes an error in the TSI index with estimating the
cardinality of series recently added and then removed.
2019-01-10 17:46:30 +00:00
Jonathan A. Sternberg af8bf99256
Do not panic when a series id iterator is nil 2018-10-11 15:16:59 -05:00
Edd Robinson d8af622333 Add benchmark for TagSets across indexes 2018-09-18 15:58:38 -07:00
Ben Johnson 88d006a18c
Remove TSI1 HLL sketches from heap.
This commit removes the HLL sketches on each `tsi1.LogFile` and
`tsi1.IndexFile` and instead caches the data at the `tsi1.Index`
level. This reduces the heap size significantly for servers with
many TSI-enabled shards.
2018-09-12 08:48:40 -06:00
Edd Robinson dece5b847f Refactor index names 2018-08-21 14:32:30 +01:00
Edd Robinson 6b3860e9a1 Reduce allocations in TSI TagSets implementation
Since all tag sets are materialised to strings before this method
returns, a large number of allocations can be avoided by carefully
resuing buffers and containers.

This commit reduces allocations by about 75%, which can be very
significant for high cardinality workloads.

The benchmark results shown below are for a benchmark that asks for all
series keys matching `tag5=value0'.

name                                               old time/op    new time/op    delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8     5.66s ± 4%     5.70s ± 5%     ~     (p=0.739 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8      26.5s ± 8%     26.8s ±12%     ~     (p=0.579 n=10+10)
IndexSet_TagSets/1M_series/inmem-8                   11.9ms ±18%    10.4ms ± 2%  -12.81%  (p=0.000 n=10+10)
IndexSet_TagSets/1M_series/tsi1-8                    23.4ms ± 5%    18.9ms ± 1%  -19.07%  (p=0.000 n=10+9)

name                                               old alloc/op   new alloc/op   delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8    2.50GB ± 0%    2.50GB ± 0%     ~     (p=0.315 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8     32.6GB ± 0%    32.6GB ± 0%     ~     (p=0.247 n=10+10)
IndexSet_TagSets/1M_series/inmem-8                   3.56MB ± 0%    3.56MB ± 0%     ~     (all equal)
IndexSet_TagSets/1M_series/tsi1-8                    12.7MB ± 0%     5.2MB ± 0%  -59.02%  (p=0.000 n=10+10)

name                                               old allocs/op  new allocs/op  delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8     24.0M ± 0%     24.0M ± 0%     ~     (p=0.353 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8      96.6M ± 0%     96.7M ± 0%     ~     (p=0.579 n=10+10)
IndexSet_TagSets/1M_series/inmem-8                     51.0 ± 0%      51.0 ± 0%     ~     (all equal)
IndexSet_TagSets/1M_series/tsi1-8                     80.4k ± 0%     20.4k ± 0%  -74.65%  (p=0.000 n=10+10)
2018-08-10 16:01:49 +01:00
Edd Robinson 9f883c8dee Add benchmark for TagSets across indexes 2018-08-10 16:01:49 +01:00
Jacob Marble 2dde4e6174
Fix broken unit tests after function signature change (#9848) 2018-05-15 16:22:24 -07:00
Jacob Marble 7f8b7af61e
Cleanup index memory footprint counting code (#9828)
* Fix IndexSet.DedupeInmemIndexes

* Cleanup index memory footprint code
2018-05-15 11:25:19 -07:00
Ben Johnson ed9c0576d4
Add series sketches, fix tombstones in index files. 2018-02-07 14:52:13 -07:00
Edd Robinson 0d164f3164
WIP - tsi integration sketches 2018-02-07 14:52:13 -07:00
Edd Robinson 64171229e8
Add test for index sketches 2018-02-07 14:52:13 -07:00
Edd Robinson e610e7c21d Track undeleted series IDs per-shard with inmem
This commit adds a bitset into each shard's in-memory index, to be used to
track undeleted series ids. Currently tsi1 support is not implemented.

When new series are added to the shard, the series id is added
to the bitset. When series are deleted from the shard, the series
ids are removed from the bitset.

Becasue each shard shares the same inmem index reference, the bitset
is stored in the `ShardIndex`, which is local to each shard, and then
different references are passed into the shared `Index` object, depending
on which shard is writing the series.
2018-01-11 01:01:54 +00:00
Edd Robinson f9ea54198f rename series directory 2018-01-03 15:44:58 +00:00
Ben Johnson 56980b0d24
Segment series file 2017-12-29 11:57:45 -07:00
Edd Robinson bde66f19bc Adjust series file size and partitions 2017-12-18 13:17:42 +00:00
Edd Robinson 3bfe525705 Add 32-bit support to series file
This commit ensures that the series file should work appropriately on
32-bit architecturs. It does this by reducing the maximum size of a
series file to 512MB on 32-bit systems, which should be fully
addressable.

It further updates tests so that the series file size can be reduced
further when running many tests in parallel on 32-bit architectures.
2017-12-15 15:47:26 +00:00
Edd Robinson f1bcc97e89 Fix auth tests 2017-12-12 21:25:35 +00:00
Edd Robinson f6835632e7 Merge master into branch 2017-12-08 17:11:07 +00:00
Ben Johnson e0df47d54f
Fixing up tests. 2017-12-02 16:52:34 -07:00
Edd Robinson a5af19fc06 Address PR feedback 2017-11-17 12:43:48 +00:00
Edd Robinson 3967e78885 Consolidate tests to tsdb package 2017-11-17 11:06:43 +00:00