Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
35,236 Commits
268 Branches
456 Tags
360 MiB
Commit Graph

1411 Commits (ec7fdd3a58d3a0b2386af045cb41057c47113a4c)

Author SHA1 Message Date
Sam Arnold 4de89afd37
refactor: remove dead iterator code (#23887) 
* fix: codegen without needing goimports

* refactor: remove dead code
 2022-11-09 19:26:12 -05:00
davidby-influx b72848d436
feat: optimize saving changes to fields.idx (#23701) (#23728) 
Instead of writing out the complete fields.idx
file when it changes, write out incremental
changes that will be applied to the file on
close and startup.

closes https://github.com/influxdata/influxdb/issues/23653

(cherry picked from commit 80c10c8c04)

closes https://github.com/influxdata/influxdb/issues/23703
 2022-09-15 12:15:14 -07:00
Abirdcfly c433342830
chore: remove duplicate word in comments (#23685) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-09-13 11:00:52 -05:00
Dane Strandboge 82d1123e78
build: upgrade to Go 1.18.1 (#23252) 2022-04-13 15:24:27 -05:00
Sam Arnold b970e359dc
feat: remaining storage metrics from OSS engine (#22938) 
* fix: simplify disk size tracking

* refactor: EngineTags in tsdb package

* fix: fewer compaction buckets and dead code removal

* feat: shard metrics

* chore: formatting

* feat: tsdb store metrics

* feat: retention check metrics

* chore: fix go vet

* fix: review comments
 2021-12-02 09:01:46 -05:00
Sam Arnold edb21abe91
feat: metrics for wal subsystem (#22918) 
https://github.com/influxdata/influxdb/issues/20026
 2021-11-23 12:17:52 -05:00
Sam Arnold feb459c785
feat: metrics for cache subsystem (#22915) 
* fix: drop complicated cache metrics and document remaining

* feat: metrics for cache
 2021-11-23 10:11:22 -05:00
Sam Arnold a74e05177c
feat: disk size metrics per shard (#22912) 2021-11-22 16:53:55 -05:00
Sam Arnold dece95d1dd
feat: tsm compaction metrics via prometheus (#22904) 
* feat: tsm compaction metrics via prometheus

* chore: fix formatting

* chore: make activeCompactions a pointer
 2021-11-19 14:51:22 -05:00
Dane Strandboge f4e9ae94ca
build: upgrade protobuf library (#22654) 2021-11-02 16:00:54 -05:00
davidby-influx 5e6b0d539b
fix: extend snapshot copy to filesystems that cannot link (#22703) (#22732) 
If os.Link fails with syscall.ENOTSUP, then the file
system does not support links, and we must make copies
to snapshot files for backup. We also automatically make
copies instead of link on Windows, because although it
makes links, their semantics are different from Linux.

closes https://github.com/influxdata/influxdb/issues/16739

(cherry picked from commit d9b9e86db9)

closes https://github.com/influxdata/influxdb/issues/22701
 2021-10-22 08:59:41 -07:00
Dane Strandboge ca992e9fff
chore: use io/os over ioutil (#22656) 2021-10-12 16:55:07 -05:00
davidby-influx 47007f6988
fix: for Windows, copy snapshot files being backed up (#22551) (#22562) 
On Windows, make copies of files for snapshots, because
Go does not support the FILE_SHARE_DELETE flag which
allows files (and links) to be deleted while open. This
causes temporary directories to be left behind after
backups.

closes https://github.com/influxdata/influxdb/issues/16289

(cherry picked from commit 3702fe8e76)

closes https://github.com/influxdata/influxdb/issues/22557
 2021-09-22 13:06:28 -07:00
Daniel Moran e06e34aa56
test: sleep between 2nd and 3rd checks of engine last-modified time (#22554) 
And rewrite the test to use testify
 2021-09-22 14:14:46 -04:00
Sam Arnold 5015297d40
fix: more expressive errors (#22448) 
* fix: more expressive errors

Closes #22446

* fix: server only logging for untyped errors

* chore: fix formatting
 2021-09-13 15:12:35 -04:00
davidby-influx 7ad612b0d7
fix: discard excessive errors (#22379) (#22391) 
The tsmBatchKeyIterator discards excessive errors to avoid
out-of-memory crashes when compacting very corrupt files.
Any error beyond DefaultMaxSavedErrors (100) will be
discarded instead of appended to the error slice.

closes https://github.com/influxdata/influxdb/issues/22328

(cherry picked from commit e53f75e06d)

closes https://github.com/influxdata/influxdb/issues/22381
 2021-09-03 14:57:36 -07:00
Daniel Moran 1c2d68b0cb
build: upgrade to go1.17 (#22363) 2021-09-01 16:09:02 -04:00
Daniel Moran 37088e8f53
feat(influxql): Add hyper log log operators (#22322) 
In addition to helping with normal queries, this can improve the 'SHOW CARDINALITY'
meta-queries.


Co-authored-by: Sam Arnold <sarnold@influxdata.com>
 2021-08-30 15:46:46 -04:00
Daniel Moran df448c654b
feat(tsi): optimize series iteration (#22316) 
When using queries like 'select count(_seriesKey) from bigmeasurement`, we
should iterate over the tsi structures to serve the query instead of loading
all the series into memory up front.


Co-authored-by: Sam Arnold <sarnold@influxdata.com>
 2021-08-27 09:59:23 -04:00
Jonathan A. Sternberg f94783e016
build(flux): update flux to master and change renamed structs (#22281) 2021-08-26 10:07:02 -05:00
davidby-influx 9923d2e8d5
fix: avoid compaction queue stats flutter (#22235) 
When the compaction planner runs, if it cannot acquire
a lock on the files it plans to compact, it returns a
nil list of compaction groups. This, in turn, sets the
engine statistics for compactions queues to zero,
which is incorrect. Instead, use the length of pending
files which would have been returned.

closes https://github.com/influxdata/influxdb/issues/22138

(cherry picked from commit 7d3efe1e9e)

closes https://github.com/influxdata/influxdb/issues/22141
 2021-08-17 14:03:54 -07:00
davidby-influx a78729b2ff
chore: add logging to compaction (#21707) (#21900) 
Compaction logging will generate intermediate information on
volume of data written and output files created, as well as
improve some of the anti-entropy messages related to compaction.

Closes https://github.com/influxdata/influxdb/issues/21704

(cherry picked from commit 73bdb2860e)

Closes https://github.com/influxdata/influxdb/issues/21706
 2021-07-21 09:43:21 -07:00
davidby-influx dd34f5fd9d
chore: add more logging 
tsdb.Engine.IsIdle and tsdb.Engine.Digest now return a reason string for why the engine & shard are not idle.
Callers can then use this string for logging, if desired. The returned reason does not allocate memory, so the
caller may want to add the shard ID and path for more information in the log. This is intended to be used in
calls from the anti-entropy service in Enterprise.

(cherry picked from commit bf45841359)

fixes https://github.com/influxdata/influxdb/issues/21448

(cherry picked from commit c8da9bafbf)

closes https://github.com/influxdata/influxdb/issues/21894
 2021-07-20 11:57:52 -07:00
Daniel Moran b1b076f600
refactor: delete old backup/restore CLI commands, replace with dependency on new CLI repo (#21703) 2021-06-17 09:18:55 -04:00
davidby-influx 5251c85412
fix: Do not close connection twice in DigestWithOptions (#21659) (#21662) 
tsm1.DigestWithOptions closes its network connection
twice. This may cause broken pipe errors on concurrent
invocations of the same procedure, by closing a reused
i/o descriptor. This fix also captures errors from TSM
file closures, which were previously ignored.

Closes https://github.com/influxdata/influxdb/issues/21656

(cherry picked from commit bce6553459)

Closes https://github.com/influxdata/influxdb/issues/21660
 2021-06-10 13:32:36 -07:00
Daniel Moran d747e7ec4e
feat: add config parameters to toggle WAL concurrency and timeouts (#21621) 
* feat: add context parameter to Take() method on fixed limiter
* refactor: plumb context through to uses of Take()
* test: update tests to pass context as needed
* feat: add config toggles for setting WAL write concurrency & timeout
 2021-06-09 11:03:53 -04:00
davidby-influx d10a727157
fix: avoid rewriting fields.idx unnecessarily (#21592) (#21610) 
Under heavy write load creating new fields and measurements
the rewrite of the fields.idx file is a bottleneck. This
enhancement combines multiple writes into a single one and
shares any error return value with all of the combined
invocations. MeasurementFieldSet and the new
MeasurementFieldSetWriter must both now be explicitly
closed.

Closes #21577

(cherry picked from commit f64be286be)

Closes https://github.com/influxdata/influxdb/issues/21598
 2021-06-04 13:17:53 -07:00
Yun Zhao 2116332950
fix(tsm1): fix calculation of tsmFullCompactionQueue statistic (#20897) 
Co-authored-by: zhaoyun.248 <zhaoyun.248@bytedance.com>
 2021-06-04 10:26:37 -04:00
Yun Zhao ce536037dc
fix(tsm1): limit concurrent WAL encodings to reduce memory pressure under heavy write load (#20814) 
Co-authored-by: zhaoyun.248 <zhaoyun.248@bytedance.com>
 2021-06-03 16:11:36 -04:00
Daniel Moran fc3beb7d0a
fix: last-modified of empty shard directory shouldn't be Unix epoch. (#21481) 
Co-authored-by: davidby-influx <72418212+davidby-influx@users.noreply.github.com>
 2021-05-17 13:36:36 -04:00
LLThomas 531c984d37
chore: fix typo (#21284) 2021-05-01 08:55:47 -04:00
Jonathan A. Sternberg 7766672797
fix(storage): cursor requests are [start, stop] instead of [start, stop) (#21318) 
The cursors were previously [start, stop) to be consistent with how flux
requests data, but the underlying storage file store was [start, stop]
because that's how influxql read data. This reverts back the cursor
behavior so that it is now [start, stop] everywhere and the conversion
from [start, stop) to [start, stop] is performed when doing the cursor
request to get the next cursor.

Co-authored-by: Sam Arnold <sarnold@influxdata.com>
 2021-04-30 12:15:06 -04:00
Daniel Moran 7169df3b51
refactor(tsm1): delete unused Write method on cache (#20890) 2021-03-09 09:09:20 -05:00
Yun Zhao 265c1f311e
fix(tsm1): fix wal's totalOldDiskSize statistics (#20811) 2021-03-03 15:20:24 -05:00
Sam Arnold 1068d1de6f
refactor: Remove unused function add and unused variable keysHint (#20803) 2021-02-25 08:31:00 -05:00
Daniel Moran 3abd65a1a1
fix(tsm1): fix data race and validation in cache ring (#20797) 
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
 2021-02-24 16:28:30 -05:00
Daniel Moran f7516e31fe
fix(tsm1): mark tombstone stats as loaded to enable caching (#20782) 2021-02-19 11:11:51 -05:00
Daniel Moran efd766d60f
fix(tsm1): fix data race when accessing tombstone stats (#20773) 2021-02-18 20:23:57 -05:00
Daniel Moran 727a7b58c1
test: replace influxlogger with zaptest logger (#20589) 2021-02-11 10:12:39 -05:00
Tristan Su 1a00f2f123
fix(tsm): should not check write-ahead-log size against default size (#20585) 
it should check against the local saved SegmentSize instead of the
default const DefaultSegmentSize.
 2021-02-10 10:32:53 -05:00
Sam Arnold 781fa0e846 chore: add goimports 2021-01-29 14:06:52 -05:00
Daniel Moran 743aef4a98
fix(tsdb): allow backups during snapshotting, and don't leak tmp files (#20527) 
Co-authored-by: davidby-influx <dbyrne@influxdata.com>
 2021-01-18 19:02:26 -08:00
Daniel Moran 7f3f562b67
test: bump timeouts and retry counts to try fixing flaky test failures. (#20412) 2020-12-23 11:12:11 -08:00
Daniel Moran 9aefa6f868
fix(tsdb): never use an inmem index (#20313) 
And fix the logging setup for the TSDB storage engine
 2020-12-23 07:46:57 -08:00
Daniel Moran 15b9531273
fix: correct various typos (#19987) 
Co-authored-by: kumakichi <xyesan@gmail.com>
 2020-11-11 13:54:21 -05:00
Ben Johnson 23679c2375 feat: Implement backup/restore CLI subcommands. 2020-11-05 10:05:01 -07:00
jl fbe85ef548 feat: enable window pushdowns 2020-10-30 18:09:38 -07:00
sans 7dcaf5c639
fix: typos (#19734) 2020-10-13 09:50:32 -07:00
Jonathan A. Sternberg bde1209349
fix(tsdb): exclude the stop time from the array cursor (#19622) 
This is a backport of #14262 to the 1.x storage engine. The 1.x storage
engine is now the primary engine for open source so when we switched we
regressed to the old behavior.

This also fixes `go generate` for the tsm1 package by running `tmpl`
with `go run` instead of assuming the correct one is installed in the
path.
 2020-09-28 12:04:30 -05:00
Stuart Carnie a24edb2b1c
chore: Skip tests on circleci 
This is derived from 2fd8264 and 4f850b5, which skips tests on appveyor
 2020-08-31 12:14:27 -07:00