4022db03c2
Provide explicit cache snapshot reasons
2019-03-25 11:44:01 +00:00
c4cc3ca7bc
Fix
2019-03-19 15:12:35 +00:00
f383ec9225
Add ability to use report-tsm programmatically
2019-03-19 14:29:25 +00:00
3b39832ba5
Reduce garbage
2019-03-19 14:28:51 +00:00
a6447b6ca5
Refactor tsm report for 2.0
2019-03-19 14:25:53 +00:00
fdae1ae5ea
Expose field key sep
2019-03-19 14:25:53 +00:00
a33c325890
storage: pr review changes
2019-03-12 22:15:28 +08:00
2554f1c5dd
storage: add SeriesOffsetSize constant
2019-03-12 10:51:22 +08:00
603a1f26e0
use tracing.StartSpanFromContext
2019-03-07 12:12:31 -07:00
582ed6834c
ddress PR feedback
2019-03-07 09:56:07 +00:00
1cb20b654d
ExplodePoints now complies with new keys
2019-03-07 09:56:07 +00:00
f029f1645d
Change location and value for internal tag keys
2019-03-07 09:56:07 +00:00
f53f9cd949
storage: detect conflicting types in a single batch of points
...
When the WAL was moved up, the validation that happened at the cache
was skipped. This moves the field type validation for a batch of
points up ahead of the WAL again.
2019-03-06 10:30:52 -07:00
b9c7ec439e
feat(influxd): Tracing refactor ( #12318 )
...
* feat(launcher): Tracing to log disabled by default
* remove traceLogger and use opentracing directly
* add Jaeger tracing
* go vet && go fmt
2019-03-04 11:48:11 -08:00
052421d5d6
Merge pull request #12207 from influxdata/jmw-fix-resource-ownership
...
storage: fix problems with keeping resources alive
2019-03-04 10:58:46 -07:00
12d35f1a50
Revert "Merge point parse & explode."
...
This reverts commit 1004abc3e1
2019-03-02 06:23:04 -07:00
1004abc3e1
Merge point parse & explode.
2019-03-01 15:55:37 -07:00
ef425b7bf9
tsdb: fix disabling metrics in the series index
...
During Recover, we forgot to propagate the disabled flag to the
keyIDMap options like we do during Open. Since we still do propagate
the singleton `ims` which is initialized lazily, if the first
initialization has a different set of labels, it will cause an
inconsistent usage even if the metrics are disabled.
2019-03-01 12:11:16 -07:00
0fae44e219
storage: fix problems with keeping resources alive
...
This commit adds the pkg/lifecycle.Resource to help manage opening,
closing, and leasing out references to some resource. A resource
cannot be closed until all acquired references have been released.
If the debug_ref tag is enabled, all resource acquisitions keep
track of the stack trace that created them and have a finalizer
associated with them to print on stderr if they are leaked. It also
registers a handler on SIGUSR2 to dump all of the currently live
resources.
Having resources tracked in a uniform way with a data type allows us
to do more sophisticated tracking with the debug_ref tag, as well.
For example, we could panic the process if a resource cannot be
closed within a certain time frame, or attempt to figure out the
DAG of resource ownership dynamically.
This commit also fixes many issues around resources, correctness
during error scenarios, reporting of errors, idempotency of
close, tracking of memory for some data structures, resource leaks
in tests, and out of order dependency closes in tests.
2019-02-28 10:22:01 -07:00
4e5253d581
Feat/add zeros to tsm filename ( #12174 )
...
* unit tests to confirm The Old Way®
* feat: Increase TSM generation max value to 1 trillion
2019-02-27 14:59:38 -08:00
4f9f85de84
tsdb: cleanup shard errors
2019-02-18 21:25:30 +08:00
26ca30e97a
Ensure that cached series id sets are Go heap backed
2019-02-12 16:33:35 -07:00
cf29b6bca4
Convert TagValueSeriesIDCache to use string fields
2019-02-12 14:45:38 -07:00
3bb765279b
storage: respond to review comments
2019-02-04 12:26:26 -07:00
b4823d11bf
storage: double check the cache to avoid deleting keys that still exist
2019-02-04 10:58:17 -07:00
3014733b20
chore: fix staticcheck issues
2019-02-04 10:32:52 -07:00
a424bf3e4c
tsm1: implement DeleteBucketRange for the Cache
2019-02-04 10:32:52 -07:00
376b347d56
wal: change deletes to be based on DeleteBucket
2019-02-04 10:32:52 -07:00
7f54e816e3
refactor: have retention use DeleteBucketRange
2019-02-04 10:32:52 -07:00
aa12144fc7
storage: replay the WAL through the whole engine
2019-02-04 10:32:52 -07:00
6deced1215
refactor: make the WAL part of snapshots again
2019-02-04 10:32:52 -07:00
2989936d5a
refactor: write to the WAL again
2019-02-04 10:32:52 -07:00
a3e66755ca
refactor: move value aliases into its own file
2019-02-04 10:32:52 -07:00
2f46937527
refactor: move value package up to tsdb
2019-02-04 10:32:52 -07:00
d2ddd48eea
refactor: hook up metrics and wal to storage engine
...
It turns out that LastModified and DiskSize are unused, and so it
was easy to change to not care about the WAL.
This hooks up metrics and starts the WAL again.
2019-02-04 10:32:52 -07:00
95de3d52b2
refactor: use concrete WAL in tsm1
...
At the cost of some nil checks, we don't have to have an interface, defend against
subtle bugs with nils in non-nil interfaces, an empty implementation, etc.
Also, the tsm1 engine is losing the WAL anyway.
2019-02-04 10:32:52 -07:00
c9bb55b889
refactor: move the tsm1/wal into the storage/wal package
...
Because the WAL relies on the tsm1.Value type, we move that into its own
tsm1/value package and set up some aliases forwarding them into tsm1. This
also required adding some methods and changing consumers to avoid the
unexported fields. I imagine this step will be useful one day when we make
the write path more efficient with respect to consuming points.
This commit additionally fixes some issues with generation. The iterator.tmpldata
and generation for array_cursor_* were removed accidentally when removing
iterators, making those generated files stale. Restore that and regenerate.
No change in functionality.
2019-02-04 10:32:52 -07:00
e5d773cee3
Merge pull request #11492 from asashour/typos
...
Fix typo
2019-02-04 08:33:41 -06:00
1188d75a99
Merge pull request #11202 from influxdata/er-tsi-times
...
Add skeleton TSI design doc
2019-01-28 11:54:59 -08:00
19a36e0dc7
Remove copy-on-write when caching bitmaps
...
In the case of caching TSI bitmaps belonging to immutable .tsi files,
the underlying bitset data can be mmapped. It is possible, though rare,
for this data to be unmapped (e.g., via a TSI compaction) but for the
cached bitmap to be subsequently read. This leads to a segfault.
This only happens when copy-on-write is set to true on the roaring
bitmap, because in that case only the internal pointers are cloned.
This change will reduce the TSI cache performance by around 10%, which I
have deemed to account for only a few microseconds typically.
2019-01-25 13:38:22 +00:00
0d1d2c841e
Fix typo
2019-01-23 13:33:09 +01:00
07b8eacf34
Fix bucket delete for all buckets
...
If a bucket had bytes in it that would be escaped by the models
parser/package, then the index would not be correctly purged of those
series data when the bucket was dropped.
2019-01-18 17:28:58 +00:00
045bb64c5e
Add skeleton TSI design doc
2019-01-17 12:22:08 +00:00
810b5d9281
Bulk log file delete
...
This commit adds a method to delete many series ids from the LogFile in
bulk, reducing the number of fsyncs required.
2019-01-15 11:45:12 +00:00
9ff65f6016
Track deleted series ids to remove from series file
...
Previously series that were being removed were tracked at the key level.
This means that when removing them from the series file, the series id
first had to be looked up. This can cause lock thrashing when there are
many series ids to look up (such as with a bulk delete), because there
are no bulk methods to do this.
This commit changes how the series file delete is done by extracting
the series ids from the index before we remove the index entries. It's
then possible to delete all those series ids from the series file
without having to lookup the ids.
2019-01-15 11:45:10 +00:00
b025d9afa9
Improve efficiency of TSI index series drop
...
This commit improves the performance of a mass delete on the TSI index
by deleting at the measurement level instead of deleting each series
individually.
2019-01-14 12:46:55 +00:00
7ee4f499e1
Clarify best method of set difference
2019-01-14 12:46:53 +00:00
c7d26d8950
Rename delete method
2019-01-14 11:23:13 +00:00
20a8528337
Ensure TSI bitset cache cleaned up on m drop
2019-01-14 11:23:13 +00:00
d73d73c0d4
chore: rename imports from platform to influxdb
...
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
703c3c15ca
Hook up DeleteBucket to the tsm1 engine
2019-01-09 15:24:26 -07:00
b5bfb836c0
tsm1: remove unsafe in prefixTree
2019-01-09 12:43:01 -07:00
e503ef40d1
tsm1: add comments responding to review feedback
2019-01-09 11:35:06 -07:00
73c0ea410e
tsm1: add test for engine DeletePrefix
2019-01-09 10:56:10 -07:00
0a85e3b0dd
tsm1: add initial index cleanup to DeletePrefix
2019-01-08 16:32:43 -07:00
0fe2f02812
tsm1: initial DeletePrefix impl
2019-01-08 16:03:34 -07:00
f712828016
tsm1: refactor and rename some methods
2019-01-08 14:52:30 -07:00
8744a82665
tsm1: add DeletePrefix to the reader
2019-01-07 21:11:49 -07:00
f65b0933f6
tsm1: move code around into smaller files and add tests
2019-01-07 21:11:49 -07:00
fed3154506
tsm1: DeletePrefix on the indirectIndex
2019-01-07 21:08:32 -07:00
ad5352926f
tsm1: log when error reading entries for tsm key
2019-01-07 11:00:35 -07:00
9cdefa8e4f
tsm1: fix staticcheck and refactor closure out
2019-01-07 11:00:35 -07:00
1ffcd77342
tsm1: fix remaining issues and add small benchmarks
...
- notice when keys are deleted during iteration and return an error
- make sure all the consumers check the error
- add some benchmarks for small indexes to compare
- allow concurrent readers to flag deletes
benchmarks against base:
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 70.0ms ±17% 71.0ms ±12% ~ (p=1.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 1.48µs ± 1% 0.28µs ± 5% -81.29% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull/Large-8 786ms ± 1% 363ms ± 3% -53.89% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull/Small-8 2.37ms ± 0% 1.14ms ± 3% -52.02% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 384ms ± 2% 188ms ± 3% -51.04% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 470µs ± 1% 190µs ± 1% -59.71% (p=0.000 n=8+7)
IndirectIndex_Delete/Large-8 74.0ms ± 1% 128.7ms ± 1% +73.80% (p=0.001 n=7+7)
IndirectIndex_Delete/Small-8 142µs ± 1% 130µs ± 1% -8.24% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeLast-8 3.26kB ± 0% 0.00kB ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 233MB ± 0% 161MB ± 0% -30.75% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 2.13MB ± 0% 1.40MB ± 0% -34.53% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 12.4MB ± 0% 0.4MB ± 0% -96.82% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 120kB ± 0% 0kB ± 0% -99.89% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 4.54kB ± 0% 0.21kB ± 0% -95.26% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 80.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 3.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 1.53M ± 0% 0.52M ± 0% -65.98% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 15.2k ± 0% 5.2k ± 0% -65.97% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 620 ± 0% 124 ± 0% -80.00% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 10.0 ± 0% 2.0 ± 0% -80.00% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 246 ± 0% 1 ± 0% -99.59% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 4.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
14cf01911e
tsm1: change TSMFile to use an iterator style api
2019-01-07 11:00:35 -07:00
917584b054
tsm1: use readerOffsetsIterator for deletes
...
This reduces the amount of disk hits at some costs in cpu on some benchmarks. Notably, the
DeleteRangeFull_Covered and Delete benchmarks both went to approximately zero page faults
meaning they read from the index file linearly.
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 68.8ms ±10% 63.1ms ±16% -8.28% (p=0.021 n=8+8)
IndirectIndex_Entries-8 9.09µs ± 3% 9.62µs ± 1% +5.84% (p=0.000 n=8+7)
IndirectIndex_ReadEntries-8 5.86µs ± 1% 6.15µs ± 3% +5.03% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 562ns ± 6% 308ns ± 2% -45.25% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 363ms ±10% 376ms ± 5% ~ (p=0.054 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 574ms ± 2% 746ms ± 0% +30.01% (p=0.000 n=8+7)
IndirectIndex_Delete-8 51.2ms ± 0% 88.2ms ± 0% +72.38% (p=0.000 n=8+7)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.7MB ± 0% 11.7MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.798 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.857 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.04kB ± 0% +0.90% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 42.0 ± 0% 42.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.743 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.856 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 18.1MB/s ± 9% 17.5MB/s ± 7% ~ (p=0.105 n=8+8)
IndirectIndex_Delete-8 116MB/s ± 0% 0MB/s ± 0% -99.96% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
6f5c94f3f7
tsm1: introduce readerOffsets to manage the offsets slice
...
It exposes an API that will clean up the bodies of many methods and
provide a safe abstraction around iteration that will be able to
handle reads with concurrent deletes.
Benchmarks are flat.
2019-01-07 11:00:35 -07:00
f860305124
tsm1: keep first 8 bytes of each key in memory
...
Since most keys will share the first 8 bytes, we collapse them into
a slice containing partial sums of the counts. We can then binary search
into that slice to find the associated prefix for a given offset index.
Compressing in this way causes the overhead to be negligable and reduces
disk misses by about 30% in these benchmarks (500k series across 100 orgs).
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.5ms ± 1% 64.6ms ± 1% -4.33% (p=0.000 n=8+7)
IndirectIndex_Entries-8 9.41µs ± 2% 9.39µs ± 1% ~ (p=0.959 n=8+8)
IndirectIndex_ReadEntries-8 5.99µs ± 1% 6.07µs ± 1% +1.29% (p=0.001 n=8+8)
IndirectIndex_DeleteRangeLast-8 369ns ± 2% 566ns ± 1% +53.37% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 368ms ± 9% 369ms ± 2% ~ (p=0.232 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 600ms ± 1% 618ms ± 0% +3.03% (p=0.000 n=8+7)
IndirectIndex_Delete-8 50.0ms ± 1% 47.6ms ± 9% ~ (p=0.463 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 162MB ± 0% 162MB ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 82.4MB ± 0% 82.4MB ± 0% ~ (p=0.776 n=8+8)
IndirectIndex_Delete-8 4.01kB ± 0% 4.01kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeFull-8 522k ± 0% 522k ± 0% ~ (p=0.382 n=8+8)
IndirectIndex_DeleteRangeFull_Covered-8 3.31k ± 0% 3.31k ± 0% ~ (p=0.457 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
name old speed new speed delta
IndirectIndex_DeleteRangeFull-8 24.7MB/s ±10% 17.8MB/s ± 2% -28.18% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull_Covered-8 14.2MB/s ± 1% 9.6MB/s ± 0% -32.30% (p=0.000 n=8+7)
IndirectIndex_Delete-8 171MB/s ± 1% 126MB/s ±10% -26.35% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeLast went from 17 page faults, or ~180GB/sec at 369ns/op
to zero page faults. So even though it got 50% slower, it was actually I/O bound
and no longer is.
2019-01-07 11:00:35 -07:00
0becfc6239
tsm1: add helper to track page faults in index
...
Since the methods inline and dead code is eliminated, it has no runtime
overhead in the benchmarks when disabled.
benchmark recorded faults
BenchmarkIndirectIndex_Entries-8 11
BenchmarkIndirectIndex_ReadEntries-8 11
BenchmarkIndirectIndex_DeleteRangeLast-8 17
BenchmarkIndirectIndex_DeleteRangeFull-8 2218
BenchmarkIndirectIndex_Delete-8 2084
2019-01-07 11:00:35 -07:00
91e820a9d8
tsm1: fix multiple issues with DeleteRange
...
1. Correctly acquires locks
2. Seeks for discontiguous key ranges (like delete ["aaa", "zzz"])
3. Is precise about deleting a key when it contains no data
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 67.3ms ± 1% 63.2ms ±15% ~ (p=0.463 n=7+8)
IndirectIndex_Entries-8 9.14µs ± 1% 9.01µs ± 0% -1.40% (p=0.004 n=8+7)
IndirectIndex_ReadEntries-8 5.83µs ± 1% 5.68µs ± 2% -2.62% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 283ns ± 2% 191ns ± 1% -32.37% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull-8 612ms ± 1% 361ms ± 1% -41.02% (p=0.000 n=8+8)
IndirectIndex_Delete-8 49.0ms ± 1% 49.8ms ± 1% +1.80% (p=0.001 n=7+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.6MB ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 32.8kB ± 0% 32.8kB ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 64.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 168MB ± 0% 162MB ± 0% -3.71% (p=0.000 n=8+8)
IndirectIndex_Delete-8 3.94kB ± 0% 3.94kB ± 0% ~ (all samples are equal)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 35.0 ± 0% ~ (all samples are equal)
IndirectIndex_Entries-8 1.00 ± 0% 1.00 ± 0% ~ (all samples are equal)
IndirectIndex_ReadEntries-8 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
IndirectIndex_DeleteRangeLast-8 2.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull-8 1.04M ± 0% 0.52M ± 0% -49.77% (p=0.000 n=8+8)
IndirectIndex_Delete-8 123 ± 0% 123 ± 0% ~ (all samples are equal)
2019-01-07 11:00:35 -07:00
aed17cfedd
tsm1: speed up IndirectIndex benchmarks
...
rather than create the indirectIndex every Benchmark iteration
reuse a global one. reduces a couple benchmarks where Start/Stop timer
weren't in palce.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 67710057 69216355 +2.22%
BenchmarkIndirectIndex_Entries-8 9239 9762 +5.66%
BenchmarkIndirectIndex_ReadEntries-8 5964 5886 -1.31%
BenchmarkIndirectIndex_DeleteRangeLast-8 317 284 -10.41%
BenchmarkIndirectIndex_DeleteRangeFull-8 615346992 598392398 -2.76%
BenchmarkIndirectIndex_Delete-8 52906315 44400269 -16.08%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1038932 1038722 -0.02%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 0 -100.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 168112352 168061952 -0.03%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
d40c3e662f
tsm1: use uint32 key for tombstones
...
rough, noisy benchmarks.
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 62462250 67710057 +8.40%
BenchmarkIndirectIndex_Entries-8 9601 9239 -3.77%
BenchmarkIndirectIndex_ReadEntries-8 5984 5964 -0.33%
BenchmarkIndirectIndex_DeleteRangeLast-8 314 317 +0.96%
BenchmarkIndirectIndex_DeleteRangeFull-8 813838165 615346992 -24.39%
BenchmarkIndirectIndex_Delete-8 52079181 52906315 +1.59%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 2 2 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532670 1038932 -32.21%
BenchmarkIndirectIndex_Delete-8 123 123 +0.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 64 64 +0.00%
BenchmarkIndirectIndex_DeleteRangeFull-8 232738960 168112352 -27.77%
BenchmarkIndirectIndex_Delete-8 3936 3936 +0.00%
2019-01-07 11:00:35 -07:00
ffd35ce1aa
tsm1: use a uint32 for offsets globally
...
benchmarks are flat.
2019-01-07 11:00:35 -07:00
7a7a4b6d58
tsm1: remove offsets from mmap
...
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_UnmarshalBinary-8 74525387 66439305 -10.85%
BenchmarkIndirectIndex_Entries-8 8892 9200 +3.46%
BenchmarkIndirectIndex_ReadEntries-8 5816 5691 -2.15%
BenchmarkIndirectIndex_DeleteRangeLast-8 1550 311 -79.94%
BenchmarkIndirectIndex_DeleteRangeFull-8 773649708 767030277 -0.86%
BenchmarkIndirectIndex_Delete-8 79755991 52015903 -34.78%
benchmark old allocs new allocs delta
BenchmarkIndirectIndex_UnmarshalBinary-8 35 35 +0.00%
BenchmarkIndirectIndex_Entries-8 1 1 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 0 0 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3 2 -33.33%
BenchmarkIndirectIndex_DeleteRangeFull-8 1532589 1532344 -0.02%
BenchmarkIndirectIndex_Delete-8 246 123 -50.00%
benchmark old bytes new bytes delta
BenchmarkIndirectIndex_UnmarshalBinary-8 11648760 11648760 +0.00%
BenchmarkIndirectIndex_Entries-8 32768 32768 +0.00%
BenchmarkIndirectIndex_ReadEntries-8 1 1 +0.00%
BenchmarkIndirectIndex_DeleteRangeLast-8 3264 64 -98.04%
BenchmarkIndirectIndex_DeleteRangeFull-8 232710448 232624208 -0.04%
BenchmarkIndirectIndex_Delete-8 4432 3936 -11.19%
2019-01-07 11:00:35 -07:00
9d8114ef88
Fix tombstone error logic
2018-12-18 13:00:13 +00:00
419db63f7a
Address PR feedback
2018-12-18 12:33:28 +00:00
262772544a
Add prefix key to tombstoner
...
This commit adds support for "prefix keys". Prefix keys differ from
regular tombstone key entries in that the key of the entry should act as
a prefix that matches all series with the same prefix key for the given
time range.
This means only one entry is needed to delete many series.
The tombstone entries now have a maximum length of 16777215 (24 bits),
with the remaining 8 high bits available for setting further options /
meta information about the tombstone entry.
In this case, the top bit is used to indicate that the tombstone entry
is intended to be a prefix. This leaves 7 spare bits for future use.
2018-12-18 12:33:28 +00:00
1b5ad5e129
Ensure v4 tombstone file read correctly
2018-12-18 12:33:28 +00:00
90f3583fe0
Remove old tombstone version support
2018-12-18 12:33:28 +00:00
04605eb266
tsm1: speed up deleterange for large keys
...
rather than starting at the first key, do a binary search to the
first key. changes O(N) when deleting the largest key to O(log N).
benchmark old ns/op new ns/op delta
BenchmarkIndirectIndex_DeleteRangeFull-8 17884166763 738717473 -95.87%
2018-12-14 10:06:24 -07:00
687a390aaf
tsm1: add benchmarks for deletes
2018-12-14 10:06:24 -07:00
f383e8337a
test(tsdb/tsi1): test series id cache delete concurrently
...
Report the total number of gets, puts, and deletes at the end of the
test. I've found this kind of output to be a useful sanity check in
similar tests that exercise concurrency involving tasks.
Use a local random source in each goroutine. I unscientifically
eyeballed that to increase total operations by 5-10%.
Also call t.Parallel in a few more tests that involve disk access. This
shaves 1-2 seconds off the full tsi1 test suite on my machine.
2018-12-12 08:21:17 -08:00
eba485f2be
Fix nil tracker for full compactions
2018-12-11 18:30:59 +00:00
c4b42c72be
Add option to disable TSI metrics
2018-12-10 15:02:26 +00:00
6b63a3def7
Add option to disable sfile metrics
2018-12-10 14:36:28 +00:00
3ff39cd9dc
Merge pull request #1789 from influxdata/er-cache-size
...
Allow TSI cache to be dynamically altered
2018-12-10 11:23:15 +00:00
df0b084543
Merge pull request #1785 from influxdata/bj-tss-file-observer
...
Allow stats files to be observed for finishing/unlinking.
2018-12-07 18:46:35 -07:00
e0cddadffd
Allow TSI cache to be dynamically altered
2018-12-07 18:35:25 +00:00
73d8c85aa2
Allow stats files to be observed for finishing/unlinking.
...
This commit adds the `.tss` files generated for TSM statistics to
the `FileObserver` so that package users can be notified when new
stats files are created and removed.
2018-12-07 10:20:32 -07:00
e13309ebbe
Fix metric names
2018-12-07 16:37:17 +00:00
b015757c06
Ensure all tsi1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
bff655786f
Ensure tsdb metrics properly registered
2018-12-07 14:32:34 +00:00
aa936df138
Ensure all tsm1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
d94f898c8b
WIP
2018-12-07 14:32:34 +00:00
79b108d174
Fix bug with slice reuse
2018-12-07 14:32:34 +00:00
de491968ba
Fix rebase
2018-12-07 14:32:34 +00:00
93892c20ab
Fix test
2018-12-07 14:32:34 +00:00
a8b6827c9e
megacheck
2018-12-07 14:32:34 +00:00
2bb558a9d1
Ensure fileset protected by lock
2018-12-07 14:32:34 +00:00
a1804d27be
Fix race
2018-12-07 14:32:34 +00:00
f9a2f7a017
go fmt
2018-12-07 14:32:34 +00:00
Edd Robinson
zhulongcheng
Jacob Marble
Jeff Wendling
Jacob Marble
Jeff Wendling
Ben Johnson
Chris Goller
Ahmed Ashour
Mark Rushakoff
Jeff Wendling
Mark Rushakoff