fc3beb7d0a
fix: last-modified of empty shard directory shouldn't be Unix epoch. ( #21481 )
...
Co-authored-by: davidby-influx <72418212+davidby-influx@users.noreply.github.com>
2021-05-17 13:36:36 -04:00
531c984d37
chore: fix typo ( #21284 )
2021-05-01 08:55:47 -04:00
7766672797
fix(storage): cursor requests are [start, stop] instead of [start, stop) ( #21318 )
...
The cursors were previously [start, stop) to be consistent with how flux
requests data, but the underlying storage file store was [start, stop]
because that's how influxql read data. This reverts back the cursor
behavior so that it is now [start, stop] everywhere and the conversion
from [start, stop) to [start, stop] is performed when doing the cursor
request to get the next cursor.
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-04-30 12:15:06 -04:00
7169df3b51
refactor(tsm1): delete unused Write method on cache ( #20890 )
2021-03-09 09:09:20 -05:00
265c1f311e
fix(tsm1): fix wal's totalOldDiskSize statistics ( #20811 )
2021-03-03 15:20:24 -05:00
1068d1de6f
refactor: Remove unused function add and unused variable keysHint ( #20803 )
2021-02-25 08:31:00 -05:00
3abd65a1a1
fix(tsm1): fix data race and validation in cache ring ( #20797 )
...
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
2021-02-24 16:28:30 -05:00
f7516e31fe
fix(tsm1): mark tombstone stats as loaded to enable caching ( #20782 )
2021-02-19 11:11:51 -05:00
efd766d60f
fix(tsm1): fix data race when accessing tombstone stats ( #20773 )
2021-02-18 20:23:57 -05:00
727a7b58c1
test: replace influxlogger with zaptest logger ( #20589 )
2021-02-11 10:12:39 -05:00
1a00f2f123
fix(tsm): should not check write-ahead-log size against default size ( #20585 )
...
it should check against the local saved SegmentSize instead of the
default const DefaultSegmentSize.
2021-02-10 10:32:53 -05:00
781fa0e846
chore: add goimports
2021-01-29 14:06:52 -05:00
743aef4a98
fix(tsdb): allow backups during snapshotting, and don't leak tmp files ( #20527 )
...
Co-authored-by: davidby-influx <dbyrne@influxdata.com>
2021-01-18 19:02:26 -08:00
7f3f562b67
test: bump timeouts and retry counts to try fixing flaky test failures. ( #20412 )
2020-12-23 11:12:11 -08:00
9aefa6f868
fix(tsdb): never use an inmem index ( #20313 )
...
And fix the logging setup for the TSDB storage engine
2020-12-23 07:46:57 -08:00
15b9531273
fix: correct various typos ( #19987 )
...
Co-authored-by: kumakichi <xyesan@gmail.com>
2020-11-11 13:54:21 -05:00
23679c2375
feat: Implement backup/restore CLI subcommands.
2020-11-05 10:05:01 -07:00
fbe85ef548
feat: enable window pushdowns
2020-10-30 18:09:38 -07:00
7dcaf5c639
fix: typos ( #19734 )
2020-10-13 09:50:32 -07:00
bde1209349
fix(tsdb): exclude the stop time from the array cursor ( #19622 )
...
This is a backport of #14262 to the 1.x storage engine. The 1.x storage
engine is now the primary engine for open source so when we switched we
regressed to the old behavior.
This also fixes `go generate` for the tsm1 package by running `tmpl`
with `go run` instead of assuming the correct one is installed in the
path.
2020-09-28 12:04:30 -05:00
a24edb2b1c
chore: Skip tests on circleci
...
This is derived from 2fd82644f850b5
2020-08-31 12:14:27 -07:00
b917d8d9b0
chore(influxdb): Placate the linter.
2020-08-27 15:46:32 -04:00
dee8977d2c
chore: move v2/v1/tsdb → v2/tsdb
2020-08-26 10:46:47 -07:00
f2898d1992
Wipe out workspace in preparation for v2 merge
...
"Knock knock."
"Who's there?"
"InfluxDB Veet."
...
2019-01-11 10:38:50 -08:00
348dac1672
Add repro test case for UTF-8 issue
2018-12-19 14:38:31 +00:00
40db64d0b9
Limit force-full and cold compaction size.
...
This commit limits the number of files that can be compacted in
a single group when forcing a full compaction or when a shard
becomes cold. This is to prevent too many files being compacted
at the same time.
2018-12-05 10:18:56 -07:00
39a3d2335e
chore(flux): Update to Flux 0.7.1
...
Resolve breaking API changes
2018-11-30 10:38:56 -07:00
0a2f6191a6
tsdb: clean up fields index for every kind of delete
...
Before this, if you deleted everything with `delete where true`
for example, then you would be left with all of your measurements
in the fields index. That would cause ghost fields to reappear
if someone reinserted to the measurement.
This fixes that by making it so the deepest most delete code
checks if the measurement was removed from the index, and if so
cleaning it up out of the fields index.
Additionally, it fixes bugs in that cleanup code where if you had
a measurement like "m1" and "m10", when iterating over the cache
or file store, "m1" would match "m10" due to it only checking the
prefix. This also has it check the character right after the
measurement to be either a comma because tags started, or the first
character of the field separator.
2018-11-27 16:12:06 -07:00
c3d7f3de2b
fix: Allow compactor to make progress if v.MaxTime() != entry.MaxTime
2018-11-14 09:13:13 -07:00
5d083887a5
chore: Compactor test which replicates issue #10465
...
Due to an encoding bug with simple8b, it is possible that the
MaxTime for a TSM index entry does not match the last encoded timestamp.
2018-11-14 09:13:13 -07:00
a16096cbc4
Merge pull request #9943 from michaelyou/hotfix-typo
...
Some typo and Wrong position of comment
2018-11-05 12:36:05 -06:00
be662a5853
Fix TSM index maxtime modification
2018-10-29 15:44:31 +00:00
3ad44c0ff4
error if manifest is read/written more than once
...
This change makes the shard digest writer and reader return an error if
the manifest is written or read more than once.
2018-10-22 14:42:05 -04:00
3d01051dfc
make digest reader skip manifest if needed
...
This change makes the digest reader read and discard the manifest if
needed. Not all readers of a digest are interested in the manifest.
This change also makes it a requirement for the writer to write a
manifest because it is a non-optional part of a digest file.
2018-10-22 13:14:35 -04:00
0734f6fe21
feat(tsm1): Improve performance of Gorilla float block decoding
...
```
name old time/op new time/op delta
FloatArrayDecodeAll/1-8 45.9ns ± 1% 13.8ns ± 1% -70.00% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 686ns ± 0% 232ns ± 1% -66.10% (p=0.000 n=9+8)
FloatArrayDecodeAll/550-8 5.78µs ± 0% 2.22µs ± 1% -61.61% (p=0.000 n=9+9)
FloatArrayDecodeAll/1000-8 10.2µs ± 2% 4.0µs ± 5% -60.47% (p=0.000 n=10+10)
name old speed new speed delta
FloatArrayDecodeAll/1-8 414MB/s ± 1% 1383MB/s ± 1% +233.76% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 144MB/s ± 0% 424MB/s ± 1% +194.19% (p=0.000 n=9+9)
FloatArrayDecodeAll/550-8 133MB/s ± 0% 346MB/s ± 1% +160.09% (p=0.000 n=9+10)
FloatArrayDecodeAll/1000-8 135MB/s ± 2% 340MB/s ± 5% +153.03% (p=0.000 n=10+10)
```
2018-10-16 17:28:36 -07:00
4dccba29c3
chore(tsm1): go fmt file
2018-10-16 17:07:19 -07:00
a989b01356
Merge pull request #10249 from hpbieker/hpb-delete-from-prevent-rebuild-series
...
Prevent DELETE FROM to rebuild series files for shards where nothing is deleted
2018-10-16 14:53:09 -06:00
5054d6fae4
Address PR feedback
2018-10-16 13:37:49 +01:00
a792fbbdfa
fix(encoding): Improve array string encoding perf a little more
...
Encode the compressed data at the start internal buffer. This ensures
the returned slice maintains the entire capacity and is available for
subsequent use.
When we pool / reuse string buffers, this will help considerably.
Improvements over previous commit:
```
name old time/op new time/op delta
EncodeStrings/10/batch-8 542ns ± 1% 355ns ± 2% -34.53% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 5.29µs ± 1% 3.58µs ± 2% -32.20% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 48.6µs ± 0% 36.2µs ± 2% -25.40% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
EncodeStrings/10/batch-8 704B ± 0% 0B -100.00% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 9.47kB ± 0% 0.00kB -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 90.1kB ± 0% 0.0kB -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
EncodeStrings/10/batch-8 0.00 0.00 ~ (all equal)
EncodeStrings/100/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
964bc3c19e
fix(encoding): Improve simple8b another 6%; fix inconsequential bug
...
simple8b encodes deltas[1:], thus deltas[0] >= simple8b.MaxValue is
invalid.
Also changed loop calculating deltas, RLE and max to be similar to
batch timestamp, for greater consistency.
Improvements over previous commit:
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 1.50µs ± 1% 1.48µs ± 1% -1.40% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 6.10µs ± 0% 5.69µs ± 2% -6.58% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 1.50µs ± 1% 1.49µs ± 0% -1.21% (p=0.008 n=5+5)
```
Improvements overall:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.48µs ± 1% -27.25% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 5.69µs ± 2% -35.29% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.49µs ± 0% -26.93% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
43f96a6ddf
feat(encoding): Improve timestamp encoding
...
Timestamp improvements prior to any improvements to simple8b
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 1.36µs ± 1% -48.25% (p=0.008 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 32.2µs ± 1% -49.64% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 1.30µs ± 1% -86.06% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
e9531b7830
feat(encoding): Improve integer and simple8b encoding performance
...
simple8b EncodeAll improvements should
```
name old time/op new time/op delta
EncodeAll/1_bit-8 28.5µs ± 1% 28.6µs ± 1% ~ (p=0.133 n=9+10)
EncodeAll/2_bits-8 28.9µs ± 2% 28.7µs ± 0% ~ (p=0.068 n=10+8)
EncodeAll/3_bits-8 29.3µs ± 1% 28.8µs ± 0% -1.70% (p=0.000 n=10+10)
EncodeAll/4_bits-8 29.6µs ± 1% 29.1µs ± 1% -1.85% (p=0.000 n=10+10)
EncodeAll/5_bits-8 30.6µs ± 1% 29.8µs ± 2% -2.70% (p=0.000 n=10+10)
EncodeAll/6_bits-8 31.3µs ± 1% 30.0µs ± 1% -4.08% (p=0.000 n=9+9)
EncodeAll/7_bits-8 32.6µs ± 1% 30.8µs ± 0% -5.49% (p=0.000 n=9+9)
EncodeAll/8_bits-8 33.6µs ± 2% 31.0µs ± 1% -7.77% (p=0.000 n=10+9)
EncodeAll/10_bits-8 34.9µs ± 0% 31.9µs ± 2% -8.55% (p=0.000 n=9+10)
EncodeAll/12_bits-8 36.8µs ± 1% 32.6µs ± 1% -11.35% (p=0.000 n=9+10)
EncodeAll/15_bits-8 39.8µs ± 1% 34.1µs ± 2% -14.40% (p=0.000 n=10+10)
EncodeAll/20_bits-8 45.2µs ± 3% 36.2µs ± 1% -19.97% (p=0.000 n=10+9)
EncodeAll/30_bits-8 55.0µs ± 0% 40.9µs ± 1% -25.62% (p=0.000 n=9+9)
EncodeAll/60_bits-8 86.2µs ± 1% 55.2µs ± 1% -35.92% (p=0.000 n=10+10)
EncodeAll/combination-8 582µs ± 2% 502µs ± 1% -13.80% (p=0.000 n=9+9)
```
EncodeIntegers:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.50µs ± 1% -26.22% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 6.10µs ± 0% -30.73% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.50µs ± 1% -26.04% (p=0.008 n=5+5)
```
EncodeTimestamps (ran is improved due to simple8b improvements)
```
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 2.65µs ± 2% ~ (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 33.8µs ± 1% -47.23% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 9.28µs ± 1% ~ (p=0.087 n=5+5)
```
2018-10-16 12:08:12 +01:00
91d0a8c3d2
Fix index bug in float encoder
2018-10-16 12:08:12 +01:00
09da18c08e
Add TSM batch key iterator
...
The batch focussed TSM key iterator iterates TSM blocks, decoding and
merging blocks where appropriate using the the batch focussed
approaches.
2018-10-16 12:08:12 +01:00
51233b71a5
Add batch block encoders
2018-10-16 12:05:52 +01:00
592127e411
Batch oriented unsigned encoder
2018-10-16 12:05:52 +01:00
a7a70a920e
Batch oriented boolean encoders
...
This commit adds a tsm1 function for encoding a batch of booleans into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated sets of booleans.
2018-10-16 12:05:52 +01:00
a4d4ef6999
Improvements to batch float encoder
...
- Inlined the closure to avoid a function call.
- Changed append(b, make([]byte, 8)...) to inline the make call.
- Check for NaN once at the end assuming NaN is infrequent.
New performance delta comparing the current iterators to the new batch
function:
name old time/op new time/op delta
EncodeFloats/10_seq 1.32µs ± 2% 0.17µs ± 2% -87.39% (p=0.000 n=10+10)
EncodeFloats/10_ran 2.09µs ± 1% 0.15µs ± 0% -92.97% (p=0.000 n=10+9)
EncodeFloats/100_seq 8.37µs ± 2% 1.28µs ± 2% -84.74% (p=0.000 n=10+10)
EncodeFloats/100_ran 19.1µs ± 1% 1.3µs ± 1% -93.08% (p=0.000 n=9+9)
EncodeFloats/1000_seq 60.4µs ± 1% 12.6µs ± 0% -79.13% (p=0.000 n=9+7)
EncodeFloats/1000_ran 212µs ± 1% 12µs ± 1% -94.53% (p=0.000 n=9+8)
name old alloc/op new alloc/op delta
EncodeFloats/10_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/10_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/100_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/100_ran 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_seq 0.00B 0.00B ~ (all equal)
EncodeFloats/1000_ran 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeFloats/10_seq 0.00 0.00 ~ (all equal)
EncodeFloats/10_ran 0.00 0.00 ~ (all equal)
EncodeFloats/100_seq 0.00 0.00 ~ (all equal)
EncodeFloats/100_ran 0.00 0.00 ~ (all equal)
EncodeFloats/1000_seq 0.00 0.00 ~ (all equal)
EncodeFloats/1000_ran 0.00 0.00 ~ (all equal)
2018-10-16 12:05:52 +01:00
ee607f9288
Batch oriented string encoders
...
This commit adds a tsm1 function for encoding a batch of strings into a
provided buffer. The new function also shares the buffer between the
input data and the snappy encoded output, reducing allocations.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders using
randomly generated strings.
name old time/op new time/op delta
EncodeStrings/10 2.14µs ± 4% 1.42µs ± 4% -33.56% (p=0.000 n=10+10)
EncodeStrings/100 12.7µs ± 3% 10.9µs ± 2% -14.46% (p=0.000 n=10+10)
EncodeStrings/1000 132µs ± 2% 114µs ± 2% -13.88% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
EncodeStrings/10 657B ± 0% 704B ± 0% +7.15% (p=0.000 n=10+10)
EncodeStrings/100 6.14kB ± 0% 9.47kB ± 0% +54.14% (p=0.000 n=10+10)
EncodeStrings/1000 61.4kB ± 0% 90.1kB ± 0% +46.66% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
EncodeStrings/10 3.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
EncodeStrings/100 3.00 ± 0% 1.00 ± 0% -66.67% (p=0.000 n=10+10)
EncodeStrings/1000 3.00 ± 0% 1.00 ± 0% -66.67% (p=0.000 n=10+10)
2018-10-16 12:05:52 +01:00
d1b7e02483
Batch oriented timestamp encoders
...
This commit adds a tsm1 function for encoding a batch of timestamps into a
provided buffer.
The following benchmarks compare the performance of the existing
iterator based encoders, and the new batch oriented encoders. They look
at a sequential input slice, a randomly generated input slice and a
duplicate slice. All slices are sorted.
name old time/op new time/op delta
EncodeTimestamps/10_seq 153ns ± 2% 104ns ± 2% -31.62% (p=0.000 n=9+10)
EncodeTimestamps/10_ran 191ns ± 2% 142ns ± 0% -25.73% (p=0.000 n=10+9)
EncodeTimestamps/10_dup 114ns ± 1% 68ns ± 4% -39.77% (p=0.000 n=8+10)
EncodeTimestamps/100_seq 704ns ± 2% 321ns ± 2% -54.44% (p=0.000 n=9+9)
EncodeTimestamps/100_ran 7.27µs ± 4% 7.01µs ± 2% -3.59% (p=0.000 n=10+10)
EncodeTimestamps/100_dup 756ns ± 3% 396ns ± 2% -47.57% (p=0.000 n=10+10)
EncodeTimestamps/1000_seq 6.32µs ± 1% 2.46µs ± 2% -61.01% (p=0.000 n=8+10)
EncodeTimestamps/1000_ran 108µs ± 0% 68µs ± 3% -37.57% (p=0.000 n=8+10)
EncodeTimestamps/1000_dup 7.26µs ± 1% 3.64µs ± 1% -49.80% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
EncodeTimestamps/10_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/10_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/10_dup 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/100_dup 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_seq 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_ran 0.00B 0.00B ~ (all equal)
EncodeTimestamps/1000_dup 0.00B 0.00B ~ (all equal)
name old allocs/op new allocs/op delta
EncodeTimestamps/10_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/10_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/10_dup 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/100_dup 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_seq 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_ran 0.00 0.00 ~ (all equal)
EncodeTimestamps/1000_dup 0.00 0.00 ~ (all equal)
2018-10-16 12:05:52 +01:00
Daniel Moran
LLThomas
Jonathan A. Sternberg
Yun Zhao
Sam Arnold
Tristan Su
Ben Johnson
jl
sans
Stuart Carnie
Brett Buddin
Mark Rushakoff
Edd Robinson
Jeff Wendling
David Norton