Commit Graph

1908 Commits (d73df4ef53d769186ce63e84c5e725212889f748)

Author SHA1 Message Date
davidby-influx 642726e898
feat: influx_inspect export to standard out (#20977)
Add a special value to the -out flag, a hyphen, to write to stdout.
While writing to stdout, send status messages to stderr instead of
stdout (the current behavior).

Closes https://github.com/influxdata/influxdb/issues/20974
2021-03-17 10:01:40 -07:00
Sam Arnold 6b8ec8cfe0
chore: remove deprecated tool influx_stress (#20892)
https://github.com/influxdata/influxdb/issues/20891
2021-03-09 10:29:24 -05:00
Sam Arnold bf7dddaec5
fix: Add back flux CLI (#20889)
Closes:
https://github.com/influxdata/influxdb/issues/18947
https://github.com/influxdata/influxdb/issues/20852

Similarly to 2.x, the flux CLI now only supports the -execute argument
or commands on stdin. Examples:

```
➜ echo 'import "influxdata/influxdb/v1" v1.databases()' | ./influx -type flux
Result: _result
Table: keys: [organizationID]
 organizationID:string     databaseName:string  retentionPolicy:string         retentionPeriod:int  default:bool         bucketId:string
----------------------  ----------------------  ----------------------  --------------------------  ------------  ----------------------
                                     _internal                 monitor             604800000000000          true
                                      telegraf                 autogen                           0          true
                                            db                 autogen                           0          true

➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1" v1.databases()'
Result: _result
Table: keys: [organizationID]
 organizationID:string     databaseName:string  retentionPolicy:string         retentionPeriod:int  default:bool         bucketId:string
----------------------  ----------------------  ----------------------  --------------------------  ------------  ----------------------
                                     _internal                 monitor             604800000000000          true
                                      telegraf                 autogen                           0          true
                                            db                 autogen                           0          true

➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1"'
Error (500): error in query specification while starting program: this Flux script returns no streaming data. Consider adding a "yield" or invoking streaming functions directly, without performing an assignment
```

Note that tty input is explicitly forbidden, unlike 2.x:

```
➜ ./influx -type flux
Connected to http://localhost:8086 version unknown
InfluxDB shell version: unknown
Interactive flux is not supported. Provide your flux script via stdin or the -execute argument.
```
2021-03-09 09:56:52 -05:00
Sam Arnold 7210fa6355
chore: Upgrade 1.x to the latest flux (#20767)
* chore: Update flux to 0.67

* chore: Builds against 0.68 flux

* chore: Builds against 0.80.0

* chore: Builds against 0.90.0

* chore: Everything builds on latest flux

* chore: goimports fixed

* chore: fix tests locally

* chore: fix CI dockerfiles

* chore: clean up some unused code

* chore: remove flux repl and Spec in flux query json

* chore: port flux end to end tests from 2.x

* chore: fix up goimports

* chore: remove 32 bit build support
2021-03-03 16:24:49 -05:00
davidby-influx 092c7a9976 feat: Make meta queries respect QueryTimeout values
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.

Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries

https://github.com/influxdata/influxdb/issues/20736
2021-02-23 12:52:44 -08:00
Sam Arnold 21823db00b
feat: series creation ingress metrics (#20700)
After turning this on and testing locally, note the 'seriesCreated' metric

"localStore": {"name":"localStore","tags":null,"values":{"pointsWritten":2987,"seriesCreated":58,"valuesWritten":23754}},
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":4}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":46}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":14,"seriesCreated":14,"valuesWritten":42}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":10}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":30}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":22}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":18}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":58}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":8}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":18}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1342,"seriesCreated":13,"valuesWritten":13420}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"seriesCreated":6,"valuesWritten":4494}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":2,"valuesWritten":2354}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":963}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":856}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":1,"valuesWritten":642}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"seriesCreated":1,"valuesWritten":749}},

Closes: https://github.com/influxdata/influxdb/issues/20613
2021-02-05 14:52:43 -04:00
Sam Arnold dd3baf6d4a
feat: measurement metrics by login (#20687)
After turning on authentication and both forms of ingress metrics:

"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":874}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":534,"valuesWritten":1068}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":190}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":570}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":836}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":114}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":684}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":2204}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":304}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":342}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"admin","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1316,"valuesWritten":13160}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"valuesWritten":4494}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":2354}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":963}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":856}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":642}},
"ingress:21": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"valuesWritten":749}},

Only by login:

"ingress": {"name":"ingress","tags":{"login":"_systemuser_monitor"},"values":{"pointsWritten":42,"valuesWritten":354}},
"ingress:1": {"name":"ingress","tags":{"login":"admin"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:2": {"name":"ingress","tags":{"login":"telegraf"},"values":{"pointsWritten":3547,"valuesWritten":28246}},

Notice writes by users 'telegraf', '_systemuser_monitor', and 'admin'.
2021-02-04 11:52:53 -05:00
Sam Arnold ec40d5c380 chore: Fix spaces 2021-01-29 13:20:36 -05:00
Sam Arnold 8a16bf0531 chore: run goimports -w ./ 2021-01-29 11:40:02 -05:00
Sam Arnold 3a31e2370e fix(inspect): bad pattern matching 2021-01-29 10:55:20 -05:00
Sam Arnold a6152e8ac1 feat(inspect): Add report-disk for disk usage by measurement 2021-01-29 10:51:42 -05:00
Sam Arnold 32612313df fix: minor test fixes for go1.15 and also flaky timeouts
Also run gofmt
2021-01-08 14:59:33 -05:00
davidby-influx 9e33be2619
fix(error): SELECT INTO doesn't return error with unsupported value (#20429)
When a SELECT INTO query generates an illegal value that cannot be inserted,
like +/- Inf, it should return an error, rather than failing silently.
This adds a boolean parameter to the [data] section of influxdb.conf:
* strict-error-handling
When false, the default, the old behavior is preserved.  When true,
unsupported values will return an error from SELECT INTO queries

Fixes https://github.com/influxdata/influxdb/issues/20426
2020-12-30 18:22:43 -08:00
Ayan George 431f073b9e
feat: Add -lponly flag to export sub-command (#19609)
When applied, this patch will add the -lponly flag to the export command
which instructs influx_inspect to only output line protocol without
comments and other out-of-band data.
2020-09-22 10:09:09 -04:00
Tristan Su f24f644510
chore: fix code format (#18013) 2020-06-22 15:23:36 -07:00
Tristan Su 17d192e062
chore(dumptsm): clean up dead code (#17381) 2020-06-22 15:02:53 -07:00
Ben Johnson 848ca48225 fix(tsdb): Revert "fix: remove some unsafe marshalling to reduce risk of segfault"
This reverts commit 30dab03310.
2020-04-13 13:22:22 -06:00
David Norton 25381f97c8
Merge pull request #15952 from influxdata/er-verify-tombstone
feat(inspect): add influx_inspect verify-tombstone tool
2020-03-11 15:56:37 -04:00
David Norton fca2ee03a7 fix: update package comment 2020-03-11 15:37:02 -04:00
David Norton 0fcb240ebb fix: use highest verbosity level specified 2020-03-11 15:32:34 -04:00
David Norton b93bac197c fix: typo in error message 2020-03-11 15:28:45 -04:00
docmerlin (j. Emrys Landivar) 30dab03310 fix: remove some unsafe marshalling to reduce risk of segfault
We were seing segfaults in Roaring bitmaps sometimes, under very
high load with networked drives.  This may reduce risk of segfault by
forcing marshalling to copy the data.
2020-03-11 13:47:29 -04:00
Stuart Carnie f71caf5c02
Merge pull request #16709 from influxdata/sgc/18/flux-cost
fix(flux): Reduce influxd and influx startup time if Flux unused
2020-03-02 15:16:13 -07:00
Ben Johnson 7a9eb1420c fix(tsdb): Fix -compact-series-file flag 2020-02-06 13:40:19 -07:00
Ayan George 85b5efc86e
refactor(influxdb): Refactor trace code for clarity and reliability
refactor(influxdb): Refactor trace code for clarity and reliability

* Make startProfile a method of Server.  startProfile() is only used one
  place.  We bother to copy the Server.CPUProfile and Server.MemProfile
  values out of our Server struct into it's parameters.  It makes more
  sense to make startProfile() a method of Server and have it access
  those members directory via its receiver value.

* Have startProfile() return an error instead of log.Fatal()ing.  We can
  simply propagate the error up the stack and let the caller handle the
  error -- we shouldn't be exiting deep in the bowels of a non-main
  package.

* Capture and return errors from pprof.StartCPUProfile(). Currently
  there is only one possible error it can return but if it returns an
  error, we should handle it.

* add CPUProfileWriteCloser and MemProfileWriteCloser to Server struct.

* make stopProfile() a method of Server

* remove prof variable

* fix captialization of log messages.
2020-02-04 11:31:15 -05:00
Edd Robinson 34c0fdafc0 feat(storage): Offline series file compaction 2020-02-03 13:57:31 -07:00
Stuart Carnie 5b58a81700
fix(flux): Flux startup costs should only be paid if Flux is used
This PR lazily initializes Flux built-in functions when Flux is used.

It significantly reduces the startup time of the `influxd` and `influx`
binaries.

Before (4.66s):

```
↳ time bin/18/influx
bin/18/influx  4.66s user 0.19s system 198% cpu 2.441 total
```

After (10ms):

```
↳ time bin/18/influx
bin/18/influx  0.01s user 0.01s system 88% cpu 0.021 total
```
2020-02-03 12:39:00 -07:00
Brandon Farmer c2c864d611 feat(influxdb): Add test around CLI proxy path 2020-01-22 14:13:25 -08:00
Brandon Farmer 6a621bf28b fix(influx): fixes parsing of url on connect 2020-01-21 12:02:32 -08:00
Brandon Farmer fe517fd9ce feat(influxdb): Add proxy path to cli 2020-01-21 10:23:46 -08:00
Edd Robinson 92b9bdef14 feat(inspect): add verify-tombstone 2019-11-17 14:53:39 +00:00
Edd Robinson f7f19c5904 refactor(storage): add tombstone extension 2019-11-17 14:43:38 +00:00
Jonathan A. Sternberg 40a7a577fc
Update flux version to v0.50.2
This upgrades the flux version to v0.50.2.

The secret service, which is used for alerts, is not included. The
`to()` function is also still not included.
2019-10-29 16:19:14 -05:00
Adam Perlin b8668c325a
Merge pull request #14201 from influxdata/influx-verify-panic-fix
fix(storage): replace panic with error in influx_inspect verify
2019-07-08 09:00:17 -07:00
Jonathan A. Sternberg 7ca4e644f1
Update flux version to v0.33.2 (#14208)
The flux in influxdb has been upgraded to use v0.33.2. A lot of
interfaces for the storage engine were changed during this so code had
to change to accomodate the new interfaces and remove the old ones.

Included in this commit is a patch file for the changes that were made.
A patch was generated for the following packages:

* `flux/stdlib/influxdata/influxdb`
* `storage/reads`
* `tsdb/cursors`

These are the three packages that are in common with version 2 of the
database and the first of these packages contains the specific
implementations that are used for version 1.

It is very possible that the next time we upgrade this, the patch will
not apply cleanly just like it wouldn't have applied cleanly to this
update. The patch is mostly meant to document exactly what changed
during the copy over to help ensure we don't forget things when adapting
the interfaces.

Add a patch file to hopefully make this easier in the future
2019-06-27 13:52:02 -05:00
Adam Perlin 23bb49981a fix(storage): replace panic with error in influx_inspect verify 2019-06-25 12:11:42 -07:00
Diego Rabatone Oliveira decbf4e62d Remove query language from version command
There is no need to print such messages in the CLI output of the version
flag. Besides being unnecessary, it makes harder to automate some tests
for those installing influxDB CE by themselves and/or automating
dockerfiles.

Fix influxdata/influxdb/issues/10451
2019-04-29 13:51:30 -03:00
Jeff Wendling 6833c4d082 fix(influx_inspect): verify-seriesfile with deleted index entries
The series index looks at a set of tombstones when querying the id for
a given key, but it does not look when asking for the offset for some
id, even if that id is deleted.

Update the verify tooling to check that the index agrees with the
deleted status of the id, but skip doing the extra checks if the
id is deleted.
2019-04-17 12:24:23 -06:00
Jonathan A. Sternberg 31501c9dcf
Upgrade flux to the latest version and remove the platform dependency
This integrates the influxdb 1.x series to the latest version of Flux
and updates the code to use it. It also removes the dependency on
platform and copies the necessary code from storage into the 1.x series
so the dependency is unneeded.

The flux functions specific to 1.x have been moved to the same structure
that flux changed to with having a `stdlib` directory instead of a
`functions` directory. It also adds a `databases()` function that
returns the databases from the meta client.
2019-04-04 10:55:09 -05:00
Stuart Carnie 09a9d34430
fix(influx): Ensure credentials are passed for Flux queries
Fixes #11710
2019-03-29 16:39:34 -07:00
Stuart Carnie a43852958d
chore(gen): Back port improvements and changes from OSS 2.0
Specifically:

* renamed files for consistency between versions
* added `time-interval` schema option
* updated schema example documentation

Back port of improvements from #12710
2019-03-20 11:01:50 -07:00
Stuart Carnie 9b7ffd36ad feat(influx_tools): Add support for describing schema via a TOML file
The `influx_tools generate` command has a new option, `-schema`, for
specifying a TOML file which describes a desired schema.
2019-02-26 16:51:19 -07:00
Edd Robinson 42267331bd
Merge pull request #10206 from rda0/master
Fix legacy restore bug #10072
2019-02-11 10:23:15 +00:00
Edd Robinson 4356c06a71
Merge pull request #10336 from wrrn/error-golint
Fix linting errors in the errlist package
2019-02-01 07:58:05 -08:00
Stuart Carnie 8aba6f73a5
feat(influx_inspect): Add -check-utf8 to verify sub-command 2018-12-21 10:44:43 -07:00
Stuart Carnie 216cfa329f feat(flux): Add user authentication and authorization support
* Add AuthorizeDatabase API to QueryAuthorizer to verify a user has
  appropriate access to the specified database
* Update serverFluxQuery handler to require a meta.User when auth is
  enabled
* update Flux createFromSource and createBucketsSource dependencies to
  require Authorizer when auth is enabled in configuration
* update createFromSource to verify read permissions for each bucket
  specified in a Flux query
* update BucketsDecoder, which implements the buckets() Flux function,
  to return buckets that the user has read or write permissions to
* add unit tests to verify authentication is required for Flux HTTP
  requests when auth is enabled in configuration
2018-12-19 14:46:12 -07:00
Stuart Carnie 23d1281b2a feat(influx_tools): Add gen-init and gen-exec commands
* gen-init initializes a database based on the provided CLI spec
* gen-exec generates the data for the target database based on the same
  CLI spec as gen-init
2018-12-12 07:38:21 -07:00
Stuart Carnie 39a3d2335e chore(flux): Update to Flux 0.7.1
Resolve breaking API changes
2018-11-30 10:38:56 -07:00
Jeff Wendling c4a3ecf021 fix verify seriesfile in presence of tombstones
tombstone entries don't contain a key, and parsing the empty
key panics.
2018-11-21 11:07:38 -07:00
Stuart Carnie 6affa7c7f5 chore(flux): Remove Flux gRPC handler and update config.sample.toml 2018-11-05 14:26:37 -07:00