Commit Graph

182 Commits (jdstrand/update-golang-jwt-1.10)

Author SHA1 Message Date
Ben Tasker a9bf1d54c1
fix: don't write skipped shard messages to the line protocol output destination (#23727)
This switches so that the message

    skipped missing file: /path/to/tsm.tsm

is written to stdErr instead of stdout (or the output file if `-out` has been provided)
2022-11-07 16:54:22 +00:00
davidby-influx 4aa7cd88e8
feat: add type conflict checker to influx_inspect (#23616) (#23617)
adds two commands "check-schema" and
"merge-schema" to influx_inspect.
These test for field type conflicts
in all fields.idx beneath a directory
and merges the derived schemas if
"check-schema" has been run multiple
times on different directories

(cherry picked from commit 84c4f676b0)
2022-08-10 15:14:05 -07:00
davidby-influx bf34bce6ea
fix: add reporttsi to the help text (#23566) (#23574)
reporttsi was not listed as a command in the influx_inspect help text.

(cherry picked from commit 5c22d6c729)

closes https://github.com/influxdata/influxdb/issues/23567
2022-07-20 13:51:21 -07:00
davidby-influx ec38d1688c
feat: estimate Cloud2 cardinality on 1.X databases (#23351) (#23417)
feat: estimate Cloud2 cardinality on 1.X databases

To ease migrations to Cloud 2 installations from
1.X databases, estimate Cloud 2 cardinality for
a data node (or OSS system).

closes https://github.com/influxdata/influxdb/issues/23356

(cherry picked from commit ef90bc830f)

closes https://github.com/influxdata/influxdb/issues/23416
2022-06-10 10:40:05 -07:00
Shiwen Cheng 476111c40f
fix(influx_inspect): multiple retention policies bug in influx_inspect export (#23197) 2022-04-14 12:59:06 -05:00
Dane Strandboge 0574163566
build: upgrade to go1.18 (#23250) 2022-03-31 16:17:57 -05:00
Dane Strandboge 06d1df22a2
chore: fix deadlock in `influx_inspect dumptsi` (#22661) 2021-10-20 12:48:59 -05:00
Sam Arnold b64c2c3dcf
fix: tsi index should compact old or too-large log files (#21943)
* fix: tsi index should compact old log files that are too large

* chore: run automated formatter

* chore: update changelog

* fix: review comments
2021-07-26 17:40:15 -04:00
Tristan Su 108e2600b3
fix(tsi): clean up FileSet fields (#18961) 2021-07-12 10:42:38 -04:00
Sam Arnold db38959b7a
fix: fix help test for influx_inspect (#21052)
Closes #21007
2021-03-23 19:15:26 -04:00
davidby-influx 642726e898
feat: influx_inspect export to standard out (#20977)
Add a special value to the -out flag, a hyphen, to write to stdout.
While writing to stdout, send status messages to stderr instead of
stdout (the current behavior).

Closes https://github.com/influxdata/influxdb/issues/20974
2021-03-17 10:01:40 -07:00
Sam Arnold 21823db00b
feat: series creation ingress metrics (#20700)
After turning this on and testing locally, note the 'seriesCreated' metric

"localStore": {"name":"localStore","tags":null,"values":{"pointsWritten":2987,"seriesCreated":58,"valuesWritten":23754}},
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":4}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":46}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":14,"seriesCreated":14,"valuesWritten":42}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":10}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":30}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":22}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":18}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":58}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":8}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":18}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1342,"seriesCreated":13,"valuesWritten":13420}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"seriesCreated":6,"valuesWritten":4494}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":2,"valuesWritten":2354}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":963}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":856}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":1,"valuesWritten":642}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"seriesCreated":1,"valuesWritten":749}},

Closes: https://github.com/influxdata/influxdb/issues/20613
2021-02-05 14:52:43 -04:00
Sam Arnold ec40d5c380 chore: Fix spaces 2021-01-29 13:20:36 -05:00
Sam Arnold 8a16bf0531 chore: run goimports -w ./ 2021-01-29 11:40:02 -05:00
Sam Arnold 3a31e2370e fix(inspect): bad pattern matching 2021-01-29 10:55:20 -05:00
Sam Arnold a6152e8ac1 feat(inspect): Add report-disk for disk usage by measurement 2021-01-29 10:51:42 -05:00
Ayan George 431f073b9e
feat: Add -lponly flag to export sub-command (#19609)
When applied, this patch will add the -lponly flag to the export command
which instructs influx_inspect to only output line protocol without
comments and other out-of-band data.
2020-09-22 10:09:09 -04:00
Tristan Su 17d192e062
chore(dumptsm): clean up dead code (#17381) 2020-06-22 15:02:53 -07:00
David Norton 25381f97c8
Merge pull request #15952 from influxdata/er-verify-tombstone
feat(inspect): add influx_inspect verify-tombstone tool
2020-03-11 15:56:37 -04:00
David Norton fca2ee03a7 fix: update package comment 2020-03-11 15:37:02 -04:00
David Norton 0fcb240ebb fix: use highest verbosity level specified 2020-03-11 15:32:34 -04:00
David Norton b93bac197c fix: typo in error message 2020-03-11 15:28:45 -04:00
Ben Johnson 7a9eb1420c fix(tsdb): Fix -compact-series-file flag 2020-02-06 13:40:19 -07:00
Edd Robinson 34c0fdafc0 feat(storage): Offline series file compaction 2020-02-03 13:57:31 -07:00
Edd Robinson 92b9bdef14 feat(inspect): add verify-tombstone 2019-11-17 14:53:39 +00:00
Adam Perlin 23bb49981a fix(storage): replace panic with error in influx_inspect verify 2019-06-25 12:11:42 -07:00
Jeff Wendling 6833c4d082 fix(influx_inspect): verify-seriesfile with deleted index entries
The series index looks at a set of tombstones when querying the id for
a given key, but it does not look when asking for the offset for some
id, even if that id is deleted.

Update the verify tooling to check that the index agrees with the
deleted status of the id, but skip doing the extra checks if the
id is deleted.
2019-04-17 12:24:23 -06:00
Stuart Carnie 8aba6f73a5
feat(influx_inspect): Add -check-utf8 to verify sub-command 2018-12-21 10:44:43 -07:00
Jeff Wendling c4a3ecf021 fix verify seriesfile in presence of tombstones
tombstone entries don't contain a key, and parsing the empty
key panics.
2018-11-21 11:07:38 -07:00
Ben Johnson 76df4371ee
Merge pull request #9884 from influxdata/bj-dumptsi-series-file-help
Add -series-file flag to dumptsi command help.
2018-10-17 08:52:50 -06:00
Edd Robinson c57c1744ed Set cache size when building TSI index
If there is a significant amount of data in the WAL, then building the
TSI index can be problematic without being able to set the max cache
size to something larger.

This commit adds an option to se the maximum cache size.
2018-10-08 16:26:37 +01:00
Ben Johnson 2cb97146f0
Fix series file tombstoning.
This commit fixes an issue with the series file compaction process
where tombstones are lost after compaction and series existence
checks are not correct. This commit also fixes some smaller flushing
issues within the series file that mainly related to testing.
2018-10-05 08:23:25 -06:00
Jacob Marble 5c8f350dd8 influx_inspect: Make buildtsi code resusable 2018-08-07 11:33:21 -07:00
Ben Johnson fdfd038401
Add roaring bitmaps to TSI index files. 2018-07-24 17:59:23 +01:00
Edd Robinson ad388a8fd8 Address PR feedback 2018-07-09 11:51:48 +01:00
Edd Robinson 52e291eb62 uncomment profiling 2018-07-09 11:51:48 +01:00
Edd Robinson cd0294ea70 Enable concurrent index building 2018-07-09 11:51:48 +01:00
Edd Robinson ac0f125db9 Provide an API for re-using tags 2018-07-09 11:51:48 +01:00
Edd Robinson 9c2e7635d5 Avoid allocating a string for measurment names 2018-07-09 11:51:48 +01:00
Edd Robinson 3cf20823e9 Allow LogFile buffer size to be changed
When adding many series using offline tooling, it's likely that every
series involves an entry being appended to a LogFile. Typically an entry
is 11 or 12 bytes, but the default bufio.Writer buffer size is only 4K.

This means by default a write of 10,000 new series would involve ~30
buffer flushes.

This commit makes the buffer configurable, and sets the value in
`buildtsi` such that it reflects the number of series being written to
the LogFile.
2018-07-09 11:51:48 +01:00
Edd Robinson 4a90d142a7 Batch series being created 2018-07-09 11:51:15 +01:00
Edd Robinson 32ed165d9e Don't flush/sync when building index 2018-07-09 11:51:15 +01:00
Edd Robinson 2124b5e48e Logging format 2018-07-09 11:49:26 +01:00
Edd Robinson af88db51c1 Add profiling 2018-07-09 11:49:26 +01:00
Edd Robinson 08c3800a8a Address PR feedback 2018-07-04 10:57:40 +01:00
Edd Robinson cef4c399c5 Sort shard results 2018-07-03 14:20:10 +01:00
Edd Robinson e959fd49a9 Fix build 2018-07-03 14:15:32 +01:00
Edd Robinson 07ca0378bf Implementing -tag-keys is a TODO 2018-07-03 13:52:04 +01:00
Edd Robinson 89b1a49d38 Fine tune limit for when to use bitmap 2018-07-03 13:39:33 +01:00
Edd Robinson 5bce912a39 Fix panic when no series for measurement: 2018-07-03 12:39:20 +01:00