b1b076f600
refactor: delete old backup/restore CLI commands, replace with dependency on new CLI repo ( #21703 )
2021-06-17 09:18:55 -04:00
5251c85412
fix: Do not close connection twice in DigestWithOptions ( #21659 ) ( #21662 )
...
tsm1.DigestWithOptions closes its network connection
twice. This may cause broken pipe errors on concurrent
invocations of the same procedure, by closing a reused
i/o descriptor. This fix also captures errors from TSM
file closures, which were previously ignored.
Closes https://github.com/influxdata/influxdb/issues/21656
(cherry picked from commit bce6553459https://github.com/influxdata/influxdb/issues/21660
2021-06-10 13:32:36 -07:00
d747e7ec4e
feat: add config parameters to toggle WAL concurrency and timeouts ( #21621 )
...
* feat: add context parameter to Take() method on fixed limiter
* refactor: plumb context through to uses of Take()
* test: update tests to pass context as needed
* feat: add config toggles for setting WAL write concurrency & timeout
2021-06-09 11:03:53 -04:00
d10a727157
fix: avoid rewriting fields.idx unnecessarily ( #21592 ) ( #21610 )
...
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.
Closes #21577
(cherry picked from commit f64be286behttps://github.com/influxdata/influxdb/issues/21598
2021-06-04 13:17:53 -07:00
2116332950
fix(tsm1): fix calculation of tsmFullCompactionQueue statistic ( #20897 )
...
Co-authored-by: zhaoyun.248 <zhaoyun.248@bytedance.com>
2021-06-04 10:26:37 -04:00
ce536037dc
fix(tsm1): limit concurrent WAL encodings to reduce memory pressure under heavy write load ( #20814 )
...
Co-authored-by: zhaoyun.248 <zhaoyun.248@bytedance.com>
2021-06-03 16:11:36 -04:00
fc3beb7d0a
fix: last-modified of empty shard directory shouldn't be Unix epoch. ( #21481 )
...
Co-authored-by: davidby-influx <72418212+davidby-influx@users.noreply.github.com>
2021-05-17 13:36:36 -04:00
531c984d37
chore: fix typo ( #21284 )
2021-05-01 08:55:47 -04:00
7766672797
fix(storage): cursor requests are [start, stop] instead of [start, stop) ( #21318 )
...
The cursors were previously [start, stop) to be consistent with how flux
requests data, but the underlying storage file store was [start, stop]
because that's how influxql read data. This reverts back the cursor
behavior so that it is now [start, stop] everywhere and the conversion
from [start, stop) to [start, stop] is performed when doing the cursor
request to get the next cursor.
Co-authored-by: Sam Arnold <sarnold@influxdata.com>
2021-04-30 12:15:06 -04:00
7169df3b51
refactor(tsm1): delete unused Write method on cache ( #20890 )
2021-03-09 09:09:20 -05:00
265c1f311e
fix(tsm1): fix wal's totalOldDiskSize statistics ( #20811 )
2021-03-03 15:20:24 -05:00
1068d1de6f
refactor: Remove unused function add and unused variable keysHint ( #20803 )
2021-02-25 08:31:00 -05:00
3abd65a1a1
fix(tsm1): fix data race and validation in cache ring ( #20797 )
...
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
2021-02-24 16:28:30 -05:00
f7516e31fe
fix(tsm1): mark tombstone stats as loaded to enable caching ( #20782 )
2021-02-19 11:11:51 -05:00
efd766d60f
fix(tsm1): fix data race when accessing tombstone stats ( #20773 )
2021-02-18 20:23:57 -05:00
727a7b58c1
test: replace influxlogger with zaptest logger ( #20589 )
2021-02-11 10:12:39 -05:00
1a00f2f123
fix(tsm): should not check write-ahead-log size against default size ( #20585 )
...
it should check against the local saved SegmentSize instead of the
default const DefaultSegmentSize.
2021-02-10 10:32:53 -05:00
781fa0e846
chore: add goimports
2021-01-29 14:06:52 -05:00
743aef4a98
fix(tsdb): allow backups during snapshotting, and don't leak tmp files ( #20527 )
...
Co-authored-by: davidby-influx <dbyrne@influxdata.com>
2021-01-18 19:02:26 -08:00
7f3f562b67
test: bump timeouts and retry counts to try fixing flaky test failures. ( #20412 )
2020-12-23 11:12:11 -08:00
9aefa6f868
fix(tsdb): never use an inmem index ( #20313 )
...
And fix the logging setup for the TSDB storage engine
2020-12-23 07:46:57 -08:00
15b9531273
fix: correct various typos ( #19987 )
...
Co-authored-by: kumakichi <xyesan@gmail.com>
2020-11-11 13:54:21 -05:00
23679c2375
feat: Implement backup/restore CLI subcommands.
2020-11-05 10:05:01 -07:00
fbe85ef548
feat: enable window pushdowns
2020-10-30 18:09:38 -07:00
7dcaf5c639
fix: typos ( #19734 )
2020-10-13 09:50:32 -07:00
bde1209349
fix(tsdb): exclude the stop time from the array cursor ( #19622 )
...
This is a backport of #14262 to the 1.x storage engine. The 1.x storage
engine is now the primary engine for open source so when we switched we
regressed to the old behavior.
This also fixes `go generate` for the tsm1 package by running `tmpl`
with `go run` instead of assuming the correct one is installed in the
path.
2020-09-28 12:04:30 -05:00
a24edb2b1c
chore: Skip tests on circleci
...
This is derived from 2fd82644f850b5
2020-08-31 12:14:27 -07:00
b917d8d9b0
chore(influxdb): Placate the linter.
2020-08-27 15:46:32 -04:00
dee8977d2c
chore: move v2/v1/tsdb → v2/tsdb
2020-08-26 10:46:47 -07:00
f2898d1992
Wipe out workspace in preparation for v2 merge
...
"Knock knock."
"Who's there?"
"InfluxDB Veet."
...
2019-01-11 10:38:50 -08:00
348dac1672
Add repro test case for UTF-8 issue
2018-12-19 14:38:31 +00:00
40db64d0b9
Limit force-full and cold compaction size.
...
This commit limits the number of files that can be compacted in
a single group when forcing a full compaction or when a shard
becomes cold. This is to prevent too many files being compacted
at the same time.
2018-12-05 10:18:56 -07:00
39a3d2335e
chore(flux): Update to Flux 0.7.1
...
Resolve breaking API changes
2018-11-30 10:38:56 -07:00
0a2f6191a6
tsdb: clean up fields index for every kind of delete
...
Before this, if you deleted everything with `delete where true`
for example, then you would be left with all of your measurements
in the fields index. That would cause ghost fields to reappear
if someone reinserted to the measurement.
This fixes that by making it so the deepest most delete code
checks if the measurement was removed from the index, and if so
cleaning it up out of the fields index.
Additionally, it fixes bugs in that cleanup code where if you had
a measurement like "m1" and "m10", when iterating over the cache
or file store, "m1" would match "m10" due to it only checking the
prefix. This also has it check the character right after the
measurement to be either a comma because tags started, or the first
character of the field separator.
2018-11-27 16:12:06 -07:00
c3d7f3de2b
fix: Allow compactor to make progress if v.MaxTime() != entry.MaxTime
2018-11-14 09:13:13 -07:00
5d083887a5
chore: Compactor test which replicates issue #10465
...
Due to an encoding bug with simple8b, it is possible that the
MaxTime for a TSM index entry does not match the last encoded timestamp.
2018-11-14 09:13:13 -07:00
a16096cbc4
Merge pull request #9943 from michaelyou/hotfix-typo
...
Some typo and Wrong position of comment
2018-11-05 12:36:05 -06:00
be662a5853
Fix TSM index maxtime modification
2018-10-29 15:44:31 +00:00
3ad44c0ff4
error if manifest is read/written more than once
...
This change makes the shard digest writer and reader return an error if
the manifest is written or read more than once.
2018-10-22 14:42:05 -04:00
3d01051dfc
make digest reader skip manifest if needed
...
This change makes the digest reader read and discard the manifest if
needed. Not all readers of a digest are interested in the manifest.
This change also makes it a requirement for the writer to write a
manifest because it is a non-optional part of a digest file.
2018-10-22 13:14:35 -04:00
0734f6fe21
feat(tsm1): Improve performance of Gorilla float block decoding
...
```
name old time/op new time/op delta
FloatArrayDecodeAll/1-8 45.9ns ± 1% 13.8ns ± 1% -70.00% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 686ns ± 0% 232ns ± 1% -66.10% (p=0.000 n=9+8)
FloatArrayDecodeAll/550-8 5.78µs ± 0% 2.22µs ± 1% -61.61% (p=0.000 n=9+9)
FloatArrayDecodeAll/1000-8 10.2µs ± 2% 4.0µs ± 5% -60.47% (p=0.000 n=10+10)
name old speed new speed delta
FloatArrayDecodeAll/1-8 414MB/s ± 1% 1383MB/s ± 1% +233.76% (p=0.000 n=9+9)
FloatArrayDecodeAll/55-8 144MB/s ± 0% 424MB/s ± 1% +194.19% (p=0.000 n=9+9)
FloatArrayDecodeAll/550-8 133MB/s ± 0% 346MB/s ± 1% +160.09% (p=0.000 n=9+10)
FloatArrayDecodeAll/1000-8 135MB/s ± 2% 340MB/s ± 5% +153.03% (p=0.000 n=10+10)
```
2018-10-16 17:28:36 -07:00
4dccba29c3
chore(tsm1): go fmt file
2018-10-16 17:07:19 -07:00
a989b01356
Merge pull request #10249 from hpbieker/hpb-delete-from-prevent-rebuild-series
...
Prevent DELETE FROM to rebuild series files for shards where nothing is deleted
2018-10-16 14:53:09 -06:00
5054d6fae4
Address PR feedback
2018-10-16 13:37:49 +01:00
a792fbbdfa
fix(encoding): Improve array string encoding perf a little more
...
Encode the compressed data at the start internal buffer. This ensures
the returned slice maintains the entire capacity and is available for
subsequent use.
When we pool / reuse string buffers, this will help considerably.
Improvements over previous commit:
```
name old time/op new time/op delta
EncodeStrings/10/batch-8 542ns ± 1% 355ns ± 2% -34.53% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 5.29µs ± 1% 3.58µs ± 2% -32.20% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 48.6µs ± 0% 36.2µs ± 2% -25.40% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
EncodeStrings/10/batch-8 704B ± 0% 0B -100.00% (p=0.008 n=5+5)
EncodeStrings/100/batch-8 9.47kB ± 0% 0.00kB -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 90.1kB ± 0% 0.0kB -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
EncodeStrings/10/batch-8 0.00 0.00 ~ (all equal)
EncodeStrings/100/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
EncodeStrings/1000/batch-8 1.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
964bc3c19e
fix(encoding): Improve simple8b another 6%; fix inconsequential bug
...
simple8b encodes deltas[1:], thus deltas[0] >= simple8b.MaxValue is
invalid.
Also changed loop calculating deltas, RLE and max to be similar to
batch timestamp, for greater consistency.
Improvements over previous commit:
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 1.50µs ± 1% 1.48µs ± 1% -1.40% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 6.10µs ± 0% 5.69µs ± 2% -6.58% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 1.50µs ± 1% 1.49µs ± 0% -1.21% (p=0.008 n=5+5)
```
Improvements overall:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.48µs ± 1% -27.25% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 5.69µs ± 2% -35.29% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.49µs ± 0% -26.93% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
43f96a6ddf
feat(encoding): Improve timestamp encoding
...
Timestamp improvements prior to any improvements to simple8b
```
name old time/op new time/op delta
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 1.36µs ± 1% -48.25% (p=0.008 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 32.2µs ± 1% -49.64% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 1.30µs ± 1% -86.06% (p=0.008 n=5+5)
```
2018-10-16 12:08:12 +01:00
e9531b7830
feat(encoding): Improve integer and simple8b encoding performance
...
simple8b EncodeAll improvements should
```
name old time/op new time/op delta
EncodeAll/1_bit-8 28.5µs ± 1% 28.6µs ± 1% ~ (p=0.133 n=9+10)
EncodeAll/2_bits-8 28.9µs ± 2% 28.7µs ± 0% ~ (p=0.068 n=10+8)
EncodeAll/3_bits-8 29.3µs ± 1% 28.8µs ± 0% -1.70% (p=0.000 n=10+10)
EncodeAll/4_bits-8 29.6µs ± 1% 29.1µs ± 1% -1.85% (p=0.000 n=10+10)
EncodeAll/5_bits-8 30.6µs ± 1% 29.8µs ± 2% -2.70% (p=0.000 n=10+10)
EncodeAll/6_bits-8 31.3µs ± 1% 30.0µs ± 1% -4.08% (p=0.000 n=9+9)
EncodeAll/7_bits-8 32.6µs ± 1% 30.8µs ± 0% -5.49% (p=0.000 n=9+9)
EncodeAll/8_bits-8 33.6µs ± 2% 31.0µs ± 1% -7.77% (p=0.000 n=10+9)
EncodeAll/10_bits-8 34.9µs ± 0% 31.9µs ± 2% -8.55% (p=0.000 n=9+10)
EncodeAll/12_bits-8 36.8µs ± 1% 32.6µs ± 1% -11.35% (p=0.000 n=9+10)
EncodeAll/15_bits-8 39.8µs ± 1% 34.1µs ± 2% -14.40% (p=0.000 n=10+10)
EncodeAll/20_bits-8 45.2µs ± 3% 36.2µs ± 1% -19.97% (p=0.000 n=10+9)
EncodeAll/30_bits-8 55.0µs ± 0% 40.9µs ± 1% -25.62% (p=0.000 n=9+9)
EncodeAll/60_bits-8 86.2µs ± 1% 55.2µs ± 1% -35.92% (p=0.000 n=10+10)
EncodeAll/combination-8 582µs ± 2% 502µs ± 1% -13.80% (p=0.000 n=9+9)
```
EncodeIntegers:
```
name old time/op new time/op delta
EncodeIntegers/1000_seq/batch-8 2.04µs ± 0% 1.50µs ± 1% -26.22% (p=0.008 n=5+5)
EncodeIntegers/1000_ran/batch-8 8.80µs ± 2% 6.10µs ± 0% -30.73% (p=0.008 n=5+5)
EncodeIntegers/1000_dup/batch-8 2.03µs ± 1% 1.50µs ± 1% -26.04% (p=0.008 n=5+5)
```
EncodeTimestamps (ran is improved due to simple8b improvements)
```
name old time/op new time/op delta
EncodeTimestamps/1000_seq/batch-8 2.64µs ± 1% 2.65µs ± 2% ~ (p=0.310 n=5+5)
EncodeTimestamps/1000_ran/batch-8 64.0µs ± 1% 33.8µs ± 1% -47.23% (p=0.008 n=5+5)
EncodeTimestamps/1000_dup/batch-8 9.32µs ± 0% 9.28µs ± 1% ~ (p=0.087 n=5+5)
```
2018-10-16 12:08:12 +01:00
91d0a8c3d2
Fix index bug in float encoder
2018-10-16 12:08:12 +01:00
09da18c08e
Add TSM batch key iterator
...
The batch focussed TSM key iterator iterates TSM blocks, decoding and
merging blocks where appropriate using the the batch focussed
approaches.
2018-10-16 12:08:12 +01:00
Daniel Moran
davidby-influx
Yun Zhao
LLThomas
Jonathan A. Sternberg
Sam Arnold
Tristan Su
Ben Johnson
jl
sans
Stuart Carnie
Brett Buddin
Mark Rushakoff
Edd Robinson
Jeff Wendling
David Norton