da2fb27cb9
perf(storage): reduce amount of tracing
...
In a previous PR I added some tracing to help investigate delete
performance within the cache. Ironically this makes performance
significantly worse when you have a very high cardinality cache.
This keeps the main benefits of the tracing, but reduces the number of
spans created. The remaining spans are smarter with context, and include
useful information about the size of the operation being traced.
Performance on a benchmark shows a significant improvement:
name old time/op new time/op delta
Engine_DeletePrefixRange_Cache/exists-24 262ms ± 6% 113ms ± 8% -57.06% (p=0.000 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 266ms ± 4% 96ms ± 2% -64.09% (p=0.000 n=8+10)
name old alloc/op new alloc/op delta
Engine_DeletePrefixRange_Cache/exists-24 62.7MB ± 0% 29.6MB ± 1% -52.82% (p=0.000 n=9+10)
Engine_DeletePrefixRange_Cache/not_exists-24 59.2MB ± 0% 24.3MB ± 2% -59.03% (p=0.000 n=8+10)
name old allocs/op new allocs/op delta
Engine_DeletePrefixRange_Cache/exists-24 711k ± 0% 334k ± 0% -53.07% (p=0.000 n=9+8)
Engine_DeletePrefixRange_Cache/not_exists-24 700k ± 0% 302k ± 1% -56.79% (p=0.000 n=8+10)
Raw benchmarks on a 24T/32GB/Nvme machine:
goos: linux
goarch: amd64
pkg: github.com/influxdata/influxdb/tsdb/tsm1
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 104851012 ns/op 29442514 B/op 333599 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 107838824 ns/op 29485649 B/op 334369 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 108020671 ns/op 29443324 B/op 333610 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 106507506 ns/op 29977931 B/op 338597 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 116393032 ns/op 29443516 B/op 333614 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 112581877 ns/op 29691455 B/op 334699 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 50 119833106 ns/op 29444712 B/op 333625 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 100 113851895 ns/op 29921119 B/op 337419 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 50 121735395 ns/op 29445551 B/op 333634 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 50 115387319 ns/op 29444513 B/op 333627 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 94474658 ns/op 24696698 B/op 306702 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 94767020 ns/op 24004763 B/op 300066 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 97869523 ns/op 24556560 B/op 305827 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 93916119 ns/op 24172163 B/op 301244 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 96591891 ns/op 24006021 B/op 300081 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 93521244 ns/op 24266467 B/op 303190 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 95419569 ns/op 24006501 B/op 300087 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 96694570 ns/op 24521126 B/op 306041 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 95075965 ns/op 24299409 B/op 301649 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 100 97182864 ns/op 24007644 B/op 300101 allocs/op
PASS
ok github.com/influxdata/influxdb/tsdb/tsm1 490.287s
2019-08-30 20:35:05 +01:00
15ade8c162
perf(storage): remeove erroneous variable
...
This commit removes an unused slice that was being built up. Comparting
to the base-line performance yields a slight improvement when deleting
from the cache.
name old time/op new time/op delta
Engine_DeletePrefixRange_Cache/exists-24 268ms ± 5% 262ms ± 6% ~ (p=0.218 n=10+10)
Engine_DeletePrefixRange_Cache/not_exists-24 265ms ± 5% 266ms ± 4% ~ (p=0.965 n=10+8)
name old alloc/op new alloc/op delta
Engine_DeletePrefixRange_Cache/exists-24 64.1MB ± 0% 62.7MB ± 0% -2.16% (p=0.000 n=9+9)
Engine_DeletePrefixRange_Cache/not_exists-24 59.2MB ± 0% 59.2MB ± 0% ~ (p=0.505 n=8+8)
name old allocs/op new allocs/op delta
Engine_DeletePrefixRange_Cache/exists-24 711k ± 0% 711k ± 0% -0.00% (p=0.000 n=9+9)
Engine_DeletePrefixRange_Cache/not_exists-24 700k ± 0% 700k ± 0% ~ (p=0.687 n=8+8)
Raw benchmarks using a 24T / 32GB / NVME machine:
goos: linux
goarch: amd64
pkg: github.com/influxdata/influxdb/tsdb/tsm1
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 267664312 ns/op 62689106 B/op 711400 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 255017152 ns/op 62688809 B/op 711398 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 258136039 ns/op 62689626 B/op 711404 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 273982453 ns/op 62688325 B/op 711395 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 252670795 ns/op 62688704 B/op 711397 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 277700985 ns/op 61801204 B/op 702520 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 272353886 ns/op 62688767 B/op 711403 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 258717468 ns/op 62689461 B/op 711408 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 252909070 ns/op 62688949 B/op 711404 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 255436837 ns/op 62689712 B/op 711409 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 241173429 ns/op 59202122 B/op 700036 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 247961098 ns/op 60507541 B/op 714102 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 20 263380230 ns/op 59202750 B/op 700044 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 266035285 ns/op 59202758 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 266512878 ns/op 59202759 B/op 700044 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 262065769 ns/op 59202726 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 270485538 ns/op 59202733 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 263355678 ns/op 62562757 B/op 727794 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 260440337 ns/op 59203324 B/op 700050 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 276122362 ns/op 59203316 B/op 700050 allocs/op
PASS
ok github.com/influxdata/influxdb/tsdb/tsm1 259.435s
2019-08-30 20:35:05 +01:00
f2d6c93e65
test: add benchmark to track cache deletion perf
...
Benchmarks using a 24T / 32GB / NVME disk machine:
goos: linux
goarch: amd64
pkg: github.com/influxdata/influxdb/tsdb/tsm1
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 20 280039668 ns/op 64073374 B/op 711421 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 271810284 ns/op 64073207 B/op 711420 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 263464797 ns/op 64072589 B/op 711415 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 269460489 ns/op 64073344 B/op 711420 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 268319443 ns/op 64073947 B/op 711425 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 254945449 ns/op 64073463 B/op 711421 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 270202990 ns/op 65616337 B/op 724440 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 274113444 ns/op 64074764 B/op 711435 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 264234897 ns/op 64073748 B/op 711428 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/exists-24 30 264406196 ns/op 64073797 B/op 711429 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 250130623 ns/op 59202124 B/op 700036 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 255092042 ns/op 59552365 B/op 706287 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 274121068 ns/op 59202753 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 273088065 ns/op 59202702 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 264184087 ns/op 59202724 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 268075364 ns/op 59202718 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 265067057 ns/op 59202709 B/op 700043 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 254749976 ns/op 60118957 B/op 701435 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 266953837 ns/op 59203376 B/op 700051 allocs/op
BenchmarkEngine_DeletePrefixRange_Cache/not_exists-24 30 275083559 ns/op 59203329 B/op 700050 allocs/op
PASS
ok github.com/influxdata/influxdb/tsdb/tsm1 261.273s
2019-08-30 20:35:05 +01:00
36d3a6ea82
refactor(tsi1): address comments to clean up tool
2019-08-23 14:08:00 -07:00
9fc99c2724
feat(tsi1): port the dump-tsi tool to 2.x
2019-08-23 14:07:30 -07:00
6f5c81f0dc
feat(inspect): add verify-seriesfile to 2.x
2019-08-22 11:01:02 -07:00
d160585a34
refactor(storage): add deeper tracing around deletes
2019-08-22 11:08:33 +01:00
9964312bad
feat(cursors): Implement Size method so it doesn't panic
...
Returns the size (in bytes) of the associated array. Used by
the reads.ResponseWriter to estimate the size of buffered data.
2019-08-16 10:34:43 -07:00
9f3cbdc80e
test(storage): add benchmark for series creation
...
This benchmark exercises creating (or checking if series need creating)
in the TSI index and the Series File.
2019-08-12 13:50:02 +01:00
396c6b0364
refactor(storage): ensure tests clean up
2019-08-12 13:49:52 +01:00
62d1474442
Merge pull request #14246 from influxdata/mu-tsm-clean-14058
...
Clears tsm files when replace fails
2019-08-07 11:34:17 -07:00
757fb4f80c
Merge pull request #14280 from influxdata/er-rename
...
feat(fs): API for replacing os calls
2019-08-07 11:33:57 -07:00
a2fc43ff0c
Merge pull request #14470 from influxdata/mu-reportTSI-2.x
...
feat(tsi1): add report-tsi tool to 2.x
2019-08-07 19:30:58 +01:00
ad188d6465
refactor(tsi1): remove extraneous logging
2019-08-05 13:21:13 -07:00
64747e9781
refactor(tsi1): address config changes to report-tsi tool
2019-08-05 10:03:32 -07:00
945f16ff4b
docs(tsm1): TagKeys and TagValues return partial results with non-nil errors
2019-08-01 13:47:11 -07:00
e7a0068e46
fix(tsm1): TagKeys/TagValues returns when context canceled
2019-08-01 13:47:11 -07:00
66662dca7f
test(tsm1): TagKeys/TagValues stops scanning when context is canceled
2019-08-01 13:47:11 -07:00
4fef1683a0
refactor(tsi1): address review comments for report-tsi tool
2019-07-26 16:21:11 -07:00
a0f4d714ea
chore(tsi1): rename tsi1_report.go -> report.go
2019-07-26 11:17:02 -07:00
d47a578258
fix(tsi1): map org to bucket in report-tsi tool so output is more useful
2019-07-26 11:17:02 -07:00
7ce1b8109f
chore(tsi1): Clean up flags and naming in report-tsi tool; add comments
2019-07-26 11:16:59 -07:00
9bd6200f15
fix(tsi1): make mergeable
2019-07-26 11:16:12 -07:00
2c1f3e2987
fix(tsi1): remove obnoxious log messages
2019-07-26 11:16:12 -07:00
17b1dd8562
feat(tsi1): add shorthand for --top flag as -t, plus cleaning
2019-07-26 11:12:15 -07:00
aa2f7a8ff7
feat(tsi1): add a --top flag for limiting output, output now sorted
2019-07-26 11:12:15 -07:00
32b283d25a
feat(tsi1/report): Add ability to filter by measurement; add additional maps for efficient retrieval of total org/bucket cardinalities
2019-07-26 11:12:15 -07:00
5e5fa96c5b
feat(tsi1): add flags for --org-id and --bucket-id
2019-07-26 11:12:15 -07:00
bfd38d93d8
feat(tsi1): provide API tooling for use in testing
2019-07-26 11:12:15 -07:00
8f99d20deb
feat(tsi1): port report-tsi tool to influxdb 2.x
2019-07-26 11:12:15 -07:00
eb6d0f4478
feat(tsi): report cardinality for all indexes, still needs to be cleaned
...
Fix iteration logic and clean up
2019-07-26 11:12:00 -07:00
36e578122e
feat(tsi): placeholder
2019-07-26 11:11:22 -07:00
41cc23cc35
fix(tsi1): clean up some error checking
2019-07-26 11:10:47 -07:00
b9ede87508
fix(tsi): error trace for engine failure, not working
2019-07-26 11:09:40 -07:00
fc1dc37886
fix(tsdb): add missing err in SeriesPartition.Open
...
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-07-26 10:48:32 +08:00
550da3c5a5
fix(fs): merge branch 'master' into HEAD
2019-07-25 10:23:48 -07:00
ce3f58d9f0
Merge pull request #13787 from zhulongcheng/KeyCursor-overlapping
...
fix(tsm1): check if blocks are overlapping in KeyCursor.Next
2019-07-24 15:36:39 +01:00
48ee7ada04
fix(storage): move retention snapshot out of per bucket calls ( #14420 )
...
* fix(storage): move retention snapshot out of per bucket calls
Also adds tracking for snapshots from retention and full compactions.
2019-07-23 11:40:05 -07:00
979c075931
Merge branch 'master' into HEAD
2019-07-22 10:21:59 -07:00
1d56544587
Merge pull request #14263 from zhulongcheng/keyRange
...
fix(storage): add key field to keyRange
2019-07-15 22:46:39 -04:00
00561d5a1b
feedback: Move verify routines to `tsm1` package for consistency
...
Should have left it there to begin with 🤣
2019-07-09 09:00:41 +10:00
46952afe37
feat(influxd): New influxd verify tsm-blocks command
...
This command performs verification of TSM blocks
* expected and actual CRC-32 checksums match
* expected and actual min and max timestamps match decoded
data
2019-07-09 09:00:41 +10:00
fe2aceb5e4
Merge branch 'master' into mu-tsm-clean-14058
2019-07-08 15:19:41 -04:00
6f14314e97
change log level from info to error
2019-07-08 14:49:14 -04:00
2202d727da
fixes merge conflicts
2019-07-08 14:07:04 -04:00
ca5a599261
Merge branch 'master' into er-rename
2019-07-08 13:42:24 -04:00
39f51969e9
replaced os.Create calls w API calls to fs.CreateFile, includes unit test
2019-07-08 13:01:42 -04:00
12f2eeda1f
fix(storage): ensure query range is (start, stop]
2019-07-05 17:10:56 +01:00
94d4c846a8
fix(storage): add key field to keyRange
...
Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>
2019-07-05 00:14:19 +08:00
150165e92f
fix(tsdb): Fix tsm1 block merge. ( #14254 )
...
fix(tsdb): Fix tsm1 block merge.
2019-07-04 08:35:43 -06:00
948f04ef8d
allow for compaction to same file with fs api
2019-07-03 14:28:27 -04:00
c669a32ff3
change fs.RenameFile to fs.RenameFileWithReplacement when compacting partition stats files as they must go to the same place
2019-07-03 14:23:21 -04:00
fe748128e3
replaces os.Rename calls w api calls to fs.RenameFile. tests now are failing
2019-07-03 13:14:43 -04:00
10a2063dcc
fix(tsdb): Fix tsm1 block merge.
...
Fixes the `tsm1.BlockIterator` so that it returns the current
key if there are still additional entries remaining. This previously
caused multiple entries not to be merged together during compaction
because the iterator would check if the next key matched the current
key but the key for the next set of entries was returned.
2019-07-03 10:08:51 -06:00
a42ff1628d
fix(influxd): --pattern flag matches specified substring
...
Previously the logic was inverted so `--pattern` matched
everything but the specified value.
2019-07-03 12:02:19 +10:00
af257e93ff
initial commit for clearing tsm files when replace fails
2019-07-02 13:48:31 -04:00
08e24faf4c
feat(tsdb): Add block exporter.
...
Adds export tooling to `influxd inspect export-blocks` so that we
can dump out block data in SQL format for better analysis during
the debugging process.
2019-07-01 10:10:52 -06:00
fe4333e8e0
fix(storage): fix tracking disk bytes in memory
2019-06-27 16:36:00 -07:00
3ff15a8b41
fix(storage): fix counts for level 4+ files
...
The counts wreen't adding all the level 4+ files, so the last one to be counted
would override the rest.
2019-06-27 16:36:00 -07:00
b3d7986d4b
chore(tsdb): Fix read metrics declaration.
2019-06-27 09:25:27 -06:00
12549c859e
feat(tsdb): Add basic tsdb read metrics
...
Adds a total cursor counter and seek location counter to a new
`readMetrics` that is added to each `Engine`. Default labels group
by `engine_id` and `node_id`.
2019-06-26 16:16:24 -06:00
fb69c5d06c
Merge pull request #13698 from influxdata/tg-fix-metrics
...
fix(storage): reduce tsm level metrics cardinality
2019-06-20 17:57:37 -07:00
6428cdbce6
fix(storage): initialize tsm file metrics, update after compaction
...
These metrics weren't being properly intialized on opening the file store, and
weren't being properly updated on compaction.
2019-06-20 14:37:53 -07:00
85dc52a93b
fix(storage): reduce tsm level metrics cardinality
...
This should have cut off TSM file levels at 4+.
2019-06-20 14:37:33 -07:00
14980d55b8
fix(storage): Add WithCurrentGenerationFunc() for generation injection.
...
Adds the ability to set the current generation to use when compacting
the cache only. Previously, we used the current generation for all
files but this causes issues and we should only use the current
generation for level 1 compaction.
2019-06-20 08:54:38 -06:00
a181e60d70
fix(tsdb): Fix series file count ( #13770 )
...
fix(tsdb): Fix series file count
2019-06-11 10:07:12 -06:00
a82e2cb180
chore(tsdb): skip flaky test
2019-05-30 16:29:31 -07:00
576da8f9d2
fix(swagger): add log property to task runs endpoint docs
2019-05-17 14:08:10 -07:00
faa5fddf7b
Merge branch 'master' into flux-staging
2019-05-15 10:12:14 -06:00
52a98aae2b
chore(tsdb): skip flaky test
...
https://github.com/influxdata/influxdb/issues/13755
2019-05-14 12:52:37 -07:00
95f28cb571
fix(series file): Sync series segment after truncate ( #13836 ) ( #13859 )
2019-05-10 11:25:43 -07:00
aa5c77409d
backport: Fix open/close race in SeriesFile ( #13837 )
2019-05-08 11:39:24 -07:00
fbd6e9f5c4
fix(tsm1): check if blocks are overlapping in KeyCursor
2019-05-04 22:25:09 +08:00
3588c0505e
fix(storage): don't remap renamed TSM file
...
There exists a possibility for an in-flight read on a TSMReader to read
a stale reference to an mmapped TSM file index, which has become
unmapped.
This commit resolves that issue by simply renaming the file, leaving the
original file handler open and the data mapped. The path is updated so
that if any callers need to refer to the name of the TSM file after it's
renamed, the new name will be reflected.
The orphaned file handler will be closed when the TSM file is closed.
2019-05-03 22:36:35 +01:00
a5ccf5ce9a
fix(tsdb): Fix series file count
...
Previously the series file did not include tombstones in the total
count. This commit now includes tombstones in the count as well as
fixes an issue where replayed tombstone records could exist but
their underlying ID did not exist. This caused the count to become
negative and with the count being `uint64` it caused the count to
rollover to `math.Uint64Max`.
2019-05-03 09:58:13 -06:00
ef0768db31
tsm1: predicate deletes ( #13371 )
...
tsm1: predicate deletes
2019-05-03 14:27:25 +00:00
bf774b66ce
fix(storage): Ensure Tag(Keys|Values) APIs never return (nil, nil)
...
Formalized this post condition in the documentation and added additional
unit tests.
Added a nil guard and unit test to WriteStringIterator.
2019-05-02 09:45:38 -07:00
16e9eb4cb9
tsdb: respond to feedback and improve test coverage
...
predicate.go:
UnmarshalPredicate 100.0%
NewProtobufPredicate 100.0%
Matches 100.0%
Marshal 100.0%
walkPredicateNodes 100.0%
buildPredicateNode 100.0%
newPredicateState 100.0%
Reset 100.0%
Set 100.0%
newPredicateCache 100.0%
Cached 100.0%
Store 100.0%
Update 100.0%
Update 92.9%
Update 94.1%
predicateEval 90.9%
predicatePopTag 100.0%
predicatePopTagEscape 100.0%
2019-05-01 13:40:40 -06:00
4b4a814d7d
storage: fix predicate matching on field tags
2019-05-01 13:40:40 -06:00
740d669514
tsm1: teach the cache about predicates
2019-05-01 13:40:40 -06:00
4fb7bf1730
tsm1: implement predicate matcher from protobufs
2019-05-01 13:40:40 -06:00
4096f93891
tsm1: implement reading and writing predicates in tombstone files
2019-05-01 13:40:40 -06:00
dcf797f111
tsm1: basic predicate implementation at index layer
...
Only wires it up. No tests, no tombstone tracking, nothing.
2019-05-01 13:40:40 -06:00
7403fd8aa9
tsm1: rename engine method to DeletePrefixRange
...
The storage/engine knows about buckets, but the tsm1/engine doesn't, so
name the tsm1/engine method Prefix and keep the storage/engine named
Bucket.
2019-05-01 13:40:40 -06:00
8c269e0153
chore(log): Put trace_id back in logs ( #13712 )
...
* chore(log): Put trace_id back in logs
* fix tests
2019-04-30 18:51:22 -07:00
65e4e3c5de
Merge pull request #13701 from influxdata/sgc/bp/2.x/13687
...
Don't panic when encoding string blocks and length of source slice is too large
2019-04-30 10:02:40 -07:00
369a4610e6
fix(storage): Don't panic when length of source slice is too large
...
StringArrayEncodeAll will panic if the total length of strings
contained in the src slice is > 0xffffffff. This change adds a unit
test to replicate the issue and an associated fix to return an error.
This also raises an issue that compactions will be unable to make
progress under the following condition:
* multiple string blocks are to be merged to a single block and
* the total length of all strings exceeds the maximum block size that
snappy will encode (0xffffffff)
The observable effect of this is errors in the logs indicating a
compaction failure.
Fixes #13687
2019-04-29 13:29:41 -07:00
9cd7c0f7e3
tsi1: don't do verbose debug logging unless test fails
2019-04-29 14:01:45 -06:00
7b97a41dcb
feat(storage): Teach TagKeys, TagValues how to accumulate statistics
...
This commit teaches the storage schema APIs how to track statistics
and make them available via the returned `cursors.StringIterator`.
Statistics are only tracked when decoding TSM blocks or when scanning
the in-memory cache.
Closes #13541
2019-04-24 11:14:22 -07:00
ed344d25f8
feat(storage): Teach storage how to find a distinct set of tag keys
...
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagKeys expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement. This is tracked by issue #13497 .
2019-04-24 11:14:22 -07:00
272f340c30
Merge point parse & explode.
2019-04-24 10:12:15 -06:00
59279837e5
tsi1: partition close deadlock
...
When a tsi1 partition closes, it waits on the wait group for compactions
and then acquires the lock. Unfortunately, a compaction may start in the
mean time, holding on to some resources. Then, close will attempt to
close those resources while holding the lock. That will block until
the compaction has finished, but it also needs to acquire the lock
in order to finish, leading to deadlock.
One cannot just move the wait group wait into the lock because, once
again, the compaction must acquire the lock before finishing. Compaction
can't finish before acquiring the lock because then it might be operating
on an invalid resource.
This change splits the locks into two: one to protect just against
concurrent Open and Close calls, and one to protect all of the other
state. We then just close the partition, acquire the lock, then free
the resources. Starting a compaction requires acquiring a resource
to the partition itself, so that it can't start one after it has
started closing.
This change also introduces a cancellation channel into a reference
to a resource that is closed when the resource is being closed, allowing
processes that have acquired a reference to clean up quicker if someone
is trying to close the resource.
2019-04-22 09:06:32 -06:00
97572ee878
feat(storage): add tsm level metrics
...
Adds prometheus metrics recording compaction levels for TSM files.
2019-04-19 13:33:52 -07:00
d5341a1a4a
feedback: Fix comments in template
2019-04-18 16:19:19 -07:00
972cda1775
feedback: Changes in response to PR feedback
2019-04-18 16:19:18 -07:00
904c91aecc
chore: Fix staticcheck complaints
2019-04-18 16:19:18 -07:00
d3790aa072
feat: Teach storage engine how to find tag values for a given key
...
The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagValues expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement.
2019-04-18 16:19:18 -07:00
35e0094a28
feat: TimeRangeIterator for checking if keys have data in a TSM file
...
The TimeRangeIterator permits linear or random index scans and
can answer whether the current key has data for the specified time
interval, considering any tombstones.
When there are no tombstones there are some opportunities for
optimization to skip decoding blocks. Specifically, if the
queried time interval overlaps any boundaries of the TSM index entries.
2019-04-18 16:19:18 -07:00
7544ea0a5b
feat: Teach Values how to determine it contains data for a time interval
...
Add a Contains API which is a peer to the TimestampArray.Contains
function. This is used by the schema APIs to determine if data exists
in the cache for a given key and time interval.
2019-04-18 16:19:18 -07:00
1ddd0445d8
feat(tsm1): Add Seek API to TSMIndexIterator
...
Permits random access of the iterator, correctly maintaining state,
so that Next may be called to iterator from a given key.
This API will be used by the schema APIs when a predicate is specified,
typically requiring random access.
2019-04-18 16:19:18 -07:00
Edd Robinson
Max U
Nathaniel Cook
Stuart Carnie
maxunt
jlapacik
Adam Perlin
zhulongcheng
tmgordeeva
kelwang
Ben Johnson
Christopher Wolff
Alirie Gray
Jacob Marble
Jeff Wendling
Jeff Wendling