550da3c5a5
fix(fs): merge branch 'master' into HEAD
2019-07-25 10:23:48 -07:00
ce3f58d9f0
Merge pull request #13787 from zhulongcheng/KeyCursor-overlapping
...
fix(tsm1): check if blocks are overlapping in KeyCursor.Next
2019-07-24 15:36:39 +01:00
2202d727da
fixes merge conflicts
2019-07-08 14:07:04 -04:00
ca5a599261
Merge branch 'master' into er-rename
2019-07-08 13:42:24 -04:00
fe748128e3
replaces os.Rename calls w api calls to fs.RenameFile. tests now are failing
2019-07-03 13:14:43 -04:00
fe4333e8e0
fix(storage): fix tracking disk bytes in memory
2019-06-27 16:36:00 -07:00
3ff15a8b41
fix(storage): fix counts for level 4+ files
...
The counts wreen't adding all the level 4+ files, so the last one to be counted
would override the rest.
2019-06-27 16:36:00 -07:00
12549c859e
feat(tsdb): Add basic tsdb read metrics
...
Adds a total cursor counter and seek location counter to a new
`readMetrics` that is added to each `Engine`. Default labels group
by `engine_id` and `node_id`.
2019-06-26 16:16:24 -06:00
fb69c5d06c
Merge pull request #13698 from influxdata/tg-fix-metrics
...
fix(storage): reduce tsm level metrics cardinality
2019-06-20 17:57:37 -07:00
6428cdbce6
fix(storage): initialize tsm file metrics, update after compaction
...
These metrics weren't being properly intialized on opening the file store, and
weren't being properly updated on compaction.
2019-06-20 14:37:53 -07:00
85dc52a93b
fix(storage): reduce tsm level metrics cardinality
...
This should have cut off TSM file levels at 4+.
2019-06-20 14:37:33 -07:00
14980d55b8
fix(storage): Add WithCurrentGenerationFunc() for generation injection.
...
Adds the ability to set the current generation to use when compacting
the cache only. Previously, we used the current generation for all
files but this causes issues and we should only use the current
generation for level 1 compaction.
2019-06-20 08:54:38 -06:00
fbd6e9f5c4
fix(tsm1): check if blocks are overlapping in KeyCursor
2019-05-04 22:25:09 +08:00
dcf797f111
tsm1: basic predicate implementation at index layer
...
Only wires it up. No tests, no tombstone tracking, nothing.
2019-05-01 13:40:40 -06:00
97572ee878
feat(storage): add tsm level metrics
...
Adds prometheus metrics recording compaction levels for TSM files.
2019-04-19 13:33:52 -07:00
d3790aa072
feat: Teach storage engine how to find tag values for a given key
...
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagValues expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement.
2019-04-18 16:19:18 -07:00
35e0094a28
feat: TimeRangeIterator for checking if keys have data in a TSM file
...
The TimeRangeIterator permits linear or random index scans and
can answer whether the current key has data for the specified time
interval, considering any tombstones.
When there are no tombstones there are some opportunities for
optimization to skip decoding blocks. Specifically, if the
queried time interval overlaps any boundaries of the TSM index entries.
2019-04-18 16:19:18 -07:00
138c17f22c
Fix typos in tsdb package
2019-04-17 12:55:38 -07:00
fad1e07d1d
tsm1: clean up some dead/useless code in the cache
...
The storer interface isn't necessary if the init/Free logic is
removed, which is unnecessary in a world with only one shard.
Additionally, there were some cases where an init/Free call could
race and cause data loss in the cache. Not doing it at all fixes
all of those races.
2019-03-30 10:24:15 -06:00
603a1f26e0
use tracing.StartSpanFromContext
2019-03-07 12:12:31 -07:00
b9c7ec439e
feat(influxd): Tracing refactor ( #12318 )
...
* feat(launcher): Tracing to log disabled by default
* remove traceLogger and use opentracing directly
* add Jaeger tracing
* go vet && go fmt
2019-03-04 11:48:11 -08:00
4e5253d581
Feat/add zeros to tsm filename ( #12174 )
...
* unit tests to confirm The Old Way®
* feat: Increase TSM generation max value to 1 trillion
2019-02-27 14:59:38 -08:00
bd8a167a3e
Rename file package to fs
2019-02-12 11:24:11 +00:00
d73d73c0d4
chore: rename imports from platform to influxdb
...
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
2019-01-09 20:51:47 -08:00
0a85e3b0dd
tsm1: add initial index cleanup to DeletePrefix
2019-01-08 16:32:43 -07:00
f712828016
tsm1: refactor and rename some methods
2019-01-08 14:52:30 -07:00
8744a82665
tsm1: add DeletePrefix to the reader
2019-01-07 21:11:49 -07:00
fed3154506
tsm1: DeletePrefix on the indirectIndex
2019-01-07 21:08:32 -07:00
ad5352926f
tsm1: log when error reading entries for tsm key
2019-01-07 11:00:35 -07:00
1ffcd77342
tsm1: fix remaining issues and add small benchmarks
...
- notice when keys are deleted during iteration and return an error
- make sure all the consumers check the error
- add some benchmarks for small indexes to compare
- allow concurrent readers to flag deletes
benchmarks against base:
name old time/op new time/op delta
IndirectIndex_UnmarshalBinary-8 70.0ms ±17% 71.0ms ±12% ~ (p=1.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 1.48µs ± 1% 0.28µs ± 5% -81.29% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeFull/Large-8 786ms ± 1% 363ms ± 3% -53.89% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull/Small-8 2.37ms ± 0% 1.14ms ± 3% -52.02% (p=0.000 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 384ms ± 2% 188ms ± 3% -51.04% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 470µs ± 1% 190µs ± 1% -59.71% (p=0.000 n=8+7)
IndirectIndex_Delete/Large-8 74.0ms ± 1% 128.7ms ± 1% +73.80% (p=0.001 n=7+7)
IndirectIndex_Delete/Small-8 142µs ± 1% 130µs ± 1% -8.24% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
IndirectIndex_UnmarshalBinary-8 11.6MB ± 0% 11.7MB ± 0% +0.02% (p=0.000 n=8+7)
IndirectIndex_DeleteRangeLast-8 3.26kB ± 0% 0.00kB ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 233MB ± 0% 161MB ± 0% -30.75% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 2.13MB ± 0% 1.40MB ± 0% -34.53% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 12.4MB ± 0% 0.4MB ± 0% -96.82% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 120kB ± 0% 0kB ± 0% -99.89% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 4.54kB ± 0% 0.21kB ± 0% -95.26% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 80.0B ± 0% 0.0B ±NaN% -100.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
IndirectIndex_UnmarshalBinary-8 35.0 ± 0% 42.0 ± 0% +20.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeLast-8 3.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Large-8 1.53M ± 0% 0.52M ± 0% -65.98% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull/Small-8 15.2k ± 0% 5.2k ± 0% -65.97% (p=0.000 n=8+8)
IndirectIndex_DeleteRangeFull_Covered/Large-8 620 ± 0% 124 ± 0% -80.00% (p=0.002 n=7+8)
IndirectIndex_DeleteRangeFull_Covered/Small-8 10.0 ± 0% 2.0 ± 0% -80.00% (p=0.000 n=8+8)
IndirectIndex_Delete/Large-8 246 ± 0% 1 ± 0% -99.59% (p=0.000 n=8+8)
IndirectIndex_Delete/Small-8 4.00 ± 0% 0.00 ±NaN% -100.00% (p=0.000 n=8+8)
2019-01-07 11:00:35 -07:00
14cf01911e
tsm1: change TSMFile to use an iterator style api
2019-01-07 11:00:35 -07:00
df0b084543
Merge pull request #1785 from influxdata/bj-tss-file-observer
...
Allow stats files to be observed for finishing/unlinking.
2018-12-07 18:46:35 -07:00
73d8c85aa2
Allow stats files to be observed for finishing/unlinking.
...
This commit adds the `.tss` files generated for TSM statistics to
the `FileObserver` so that package users can be notified when new
stats files are created and removed.
2018-12-07 10:20:32 -07:00
aa936df138
Ensure all tsm1 metrics support multiple instances
2018-12-07 14:32:34 +00:00
6c5dec8f88
Refactor tracker names
2018-12-07 14:32:34 +00:00
3b980ed7e3
Convert Cache statistics
2018-12-07 14:32:34 +00:00
d61b9f1645
Convert Filestore stats
2018-12-07 14:32:34 +00:00
0084d4d824
Remove influxdb dependency.
2018-11-29 11:44:22 -07:00
0d411023f2
config: clean up
...
- Breaks the weird cycle that existed with the EngineOptions
- Removes a bunch of useless parameters
- Moves around a bunch of defaults
2018-11-08 11:39:36 -07:00
68450681ef
Add TSM1 measurement stats.
...
This commit generates an additional `.tss` stats file alongside each
TSM file when it is written that contains size stats for all measurements
within the TSM file. These files can be combined to generate stats for
all measurements across all TSM files.
2018-10-08 10:43:53 -06:00
074f263e08
Initial import of tsm1.Engine
2018-10-01 12:08:37 +01:00
Max U
Edd Robinson
maxunt
Tanya Gordeeva
Ben Johnson
zhulongcheng
Jeff Wendling
Stuart Carnie
Todd Persen
Jacob Marble
Jacob Marble
Mark Rushakoff
Jeff Wendling