6dd2d58163
build: fix/add `go generate` calls for flux-related `.proto` files ( #21103 )
2021-03-30 17:14:15 -04:00
b9ee9e5b44
feat: add memory and concurrency limits in flux controller ( #21100 )
...
* chore: pull in controller from 2.x
* chore: fix up 2.x controller to work with 1.x
* feat: Default query limits in flux code
Partial fix of https://github.com/influxdata/influxdb/issues/17212
* chore: update changelog
* chore: refactor to remove panic and reformat code
2021-03-30 15:02:46 -04:00
78724e5c20
chore: Add kit ( #21086 )
...
* chore: pull in unchanged kit from v2
* chore: remove v2 from import paths
* chore: update module paths and go.mod for kit
* chore: remove kit/cli again, not needed in 1.x
2021-03-30 13:09:04 -04:00
fbfd4b4651
test: add fluxtest harness ( #21074 )
...
* test: add script to run flux tests
* feat(flux): enable test capabilities in Flux controller
* feat(flux): add MergeFiltersRule
* build: bump existing Dockerfiles to go 1.15
* build: add flux tests to CI
* refactor: allow for overriding tcp.Mux logger
* build: upgrade to Flux v0.111.0
2021-03-30 11:18:18 -04:00
6edfe99edf
build: set 'noasm' tag when building ARM64 binaries ( #21071 )
2021-03-25 11:35:55 -04:00
db38959b7a
fix: fix help test for influx_inspect ( #21052 )
...
Closes #21007
2021-03-23 19:15:26 -04:00
adee278c6e
build: fix windows build for go1.15 ( #21041 )
...
Closes: https://github.com/influxdata/influxdb/issues/21018
See https://github.com/golang/go/issues/40795
2021-03-23 09:08:19 -04:00
d1a5c06596
chore: Upgrade to go1.15.10 and flux 0.108.1 ( #21015 )
2021-03-22 14:11:53 -04:00
70755bf42c
feat: Log query text for POST requests ( #20993 )
...
The HTTP handler logs URLs, but not body values for POST requests.
This means that queries sent by GET are logged, because the query
is in the URL, but queries sent by POST have no query text in the
log. This feature prints all the key-value pairs in the post body,
which includes the query text, except passwords, which are redacted.
Closes https://github.com/influxdata/influxdb/issues/20653
2021-03-22 10:31:56 -07:00
76777680b0
build: make windows and arm64 builds possible ( #21002 )
2021-03-22 11:47:08 -04:00
b7e7de24d6
refactor: separate coarse and fine permission interfaces ( #20996 )
2021-03-22 09:52:33 -04:00
d73df4ef53
chore: updated CONTRIBUTING.md for new package config script ( #20990 )
2021-03-17 14:20:53 -07:00
642726e898
feat: influx_inspect export to standard out ( #20977 )
...
Add a special value to the -out flag, a hyphen, to write to stdout.
While writing to stdout, send status messages to stderr instead of
stdout (the current behavior).
Closes https://github.com/influxdata/influxdb/issues/20974
2021-03-17 10:01:40 -07:00
274f33b3f4
chore: add back hashing of build tarball ( #20981 )
2021-03-17 10:13:34 -04:00
52f939eadf
chore: fix oss unit tests ( #20972 )
2021-03-16 11:56:28 -04:00
1e53bf13e0
chore: fix release build scripts ( #20970 )
...
Note: this commit fixes linux/amd64 and darwin/amd64 builds. We are still
working on fixes for linux/arm64 and windows/amd64.
Partial fix of https://github.com/influxdata/influxdb/issues/20854
2021-03-16 09:23:55 -04:00
178a1d0c3b
chore: 1.9.0 rc0 changelog ( #20948 )
...
* chore: reset changelog to 1.8.5
* chore: update changelog in preparation for 1.9.0rc0
2021-03-15 12:34:04 -04:00
dd6abe6f19
fix(flux): add durations to Flux logging ( #19697 ) ( #20940 )
...
(cherry picked from commit 25fb1077e2
2021-03-15 12:33:57 -04:00
532a23da84
chore: remove Jenkinsfile ( #20951 )
...
The unit tests are run by circleci and the changelog updates are not working
currently.
2021-03-15 12:33:29 -04:00
849221b709
fix: storage read service should not return a nil cursor ( #17359 ) ( #20941 )
...
The storage read service would return a nil cursor if there was no data
and no shards to read from. This modifies it to return an empty cursor
instead.
(cherry picked from commit 2852bf0399
2021-03-12 16:36:08 -05:00
df46a86326
fix(tls): Enable configuration of TLS 1.3 ( #20939 )
...
Co-authored-by: Gunnar Aasen <gunnar.r.aasen@gmail.com>
2021-03-12 16:35:28 -05:00
aa2fcd295f
fix(logger): Ensure default level value is info ( #17403 ) ( #20937 )
...
(cherry picked from commit 4d5a688a1e
2021-03-12 16:34:51 -05:00
5ef0d38bc7
fix(flux): buckets call no longer panics ( #17319 ) ( #20936 )
...
The buckets call had a column removed and the backport did not remove
that line. This modifies the buckets command to not append a row to a
column that no longer exists.
(cherry picked from commit 637ca24d5e
2021-03-12 16:34:40 -05:00
b0cf385491
fix(storage): ArrayFilterCursor truncation for multi-block data ( #19439 ) ( #20935 )
...
Backport abfe5a54a077d1a4180b
2021-03-12 16:34:31 -05:00
04f4817aae
fix(services/storage): multi measurement queries return all applicable series ( #19592 ) ( #20934 )
...
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.
This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.
This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.
Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.
This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
Backport of #19566 .
(cherry picked from commit ceead88bd5
2021-03-12 16:34:14 -05:00
3eb4fdaf33
fix(tsm1): fix data race when accessing tombstone stats ( #20903 )
2021-03-09 15:20:40 -05:00
6b8ec8cfe0
chore: remove deprecated tool influx_stress ( #20892 )
...
https://github.com/influxdata/influxdb/issues/20891
2021-03-09 10:29:24 -05:00
bf7dddaec5
fix: Add back flux CLI ( #20889 )
...
Closes:
https://github.com/influxdata/influxdb/issues/18947
https://github.com/influxdata/influxdb/issues/20852
Similarly to 2.x, the flux CLI now only supports the -execute argument
or commands on stdin. Examples:
```
➜ echo 'import "influxdata/influxdb/v1" v1.databases()' | ./influx -type flux
Result: _result
Table: keys: [organizationID]
organizationID:string databaseName:string retentionPolicy:string retentionPeriod:int default:bool bucketId:string
---------------------- ---------------------- ---------------------- -------------------------- ------------ ----------------------
_internal monitor 604800000000000 true
telegraf autogen 0 true
db autogen 0 true
➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1" v1.databases()'
Result: _result
Table: keys: [organizationID]
organizationID:string databaseName:string retentionPolicy:string retentionPeriod:int default:bool bucketId:string
---------------------- ---------------------- ---------------------- -------------------------- ------------ ----------------------
_internal monitor 604800000000000 true
telegraf autogen 0 true
db autogen 0 true
➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1"'
Error (500): error in query specification while starting program: this Flux script returns no streaming data. Consider adding a "yield" or invoking streaming functions directly, without performing an assignment
```
Note that tty input is explicitly forbidden, unlike 2.x:
```
➜ ./influx -type flux
Connected to http://localhost:8086 version unknown
InfluxDB shell version: unknown
Interactive flux is not supported. Provide your flux script via stdin or the -execute argument.
```
2021-03-09 09:56:52 -05:00
7a992dac77
feat: add arguments to flux to function ( #20873 )
...
Also remove some dead code
Closes #20853
2021-03-05 11:25:52 -05:00
d6f7716924
fix: infinite recursion bug ( #20862 )
...
* Revert "fix(error): unsupported value: +Inf" error not handled gracefully (#20250 )"
This reverts commit 6ac0bb3fe3https://github.com/influxdata/influxdb/issues/20249
2021-03-04 16:31:06 -05:00
7210fa6355
chore: Upgrade 1.x to the latest flux ( #20767 )
...
* chore: Update flux to 0.67
* chore: Builds against 0.68 flux
* chore: Builds against 0.80.0
* chore: Builds against 0.90.0
* chore: Everything builds on latest flux
* chore: goimports fixed
* chore: fix tests locally
* chore: fix CI dockerfiles
* chore: clean up some unused code
* chore: remove flux repl and Spec in flux query json
* chore: port flux end to end tests from 2.x
* chore: fix up goimports
* chore: remove 32 bit build support
2021-03-03 16:24:49 -05:00
de491dab97
refactor: Remove unused function add and unused variable keysHint ( #20803 ) ( #20805 )
...
(cherry picked from commit 1068d1de6f
2021-02-25 15:15:32 -05:00
17b9ea8723
feat: Add WITH KEY to show tag keys ( #20793 )
...
* fix: Change from RewriteExpr to PartitionExpr
Also remove some dead code
* feat: WITH KEY implementation
* feat: query rewriting for WITH KEY in SHOW TAG KEYS
2021-02-25 08:38:29 -05:00
e85a10248d
fix(tsm1): fix data race and validation in cache ring ( #20802 )
...
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
2021-02-24 17:17:08 -05:00
a8b2129df5
chore: move context.Context to first argument in methods per convention
2021-02-23 12:52:44 -08:00
092c7a9976
feat: Make meta queries respect QueryTimeout values
...
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.
Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries
https://github.com/influxdata/influxdb/issues/20736
2021-02-23 12:52:44 -08:00
de1a0eb2a9
feat: use count_hll for 'show series cardinality' queries ( #20745 )
...
Closes: https://github.com/influxdata/influxdb/issues/20614
Also fix nil pointer for seriesKey iterator
Fix for bug in: https://github.com/influxdata/influxdb/issues/20543
Also add a test for ingress metrics
2021-02-10 16:00:16 -05:00
903b8cd0ea
feat(query): Hyper log log operators in influxql ( #20603 )
...
* feat(query): hyper log log counting in query engine
In addition to helping with normal queries, this can improve the 'SHOW CARDINALITY'
meta-queries:
time influx -database mydb -execute 'select count_hll(sum_hll(_seriesKey)) from big'
name: big
time count_hll
---- ---------
0 200767781
influx -database mydb -execute 0.06s user 0.12s system 0% cpu 8:49.99 total
2021-02-08 08:38:14 -05:00
21823db00b
feat: series creation ingress metrics ( #20700 )
...
After turning this on and testing locally, note the 'seriesCreated' metric
"localStore": {"name":"localStore","tags":null,"values":{"pointsWritten":2987,"seriesCreated":58,"valuesWritten":23754}},
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":4}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":46}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":14,"seriesCreated":14,"valuesWritten":42}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":10}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":30}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":22}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":18}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":58}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":8}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":18}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1342,"seriesCreated":13,"valuesWritten":13420}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"seriesCreated":6,"valuesWritten":4494}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":2,"valuesWritten":2354}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":963}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":856}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":1,"valuesWritten":642}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"seriesCreated":1,"valuesWritten":749}},
Closes: https://github.com/influxdata/influxdb/issues/20613
2021-02-05 14:52:43 -04:00
dd3baf6d4a
feat: measurement metrics by login ( #20687 )
...
After turning on authentication and both forms of ingress metrics:
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":874}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":534,"valuesWritten":1068}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":190}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":570}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":836}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":114}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":684}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":2204}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":304}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":342}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"admin","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1316,"valuesWritten":13160}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"valuesWritten":4494}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":2354}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":963}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":856}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":642}},
"ingress:21": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"valuesWritten":749}},
Only by login:
"ingress": {"name":"ingress","tags":{"login":"_systemuser_monitor"},"values":{"pointsWritten":42,"valuesWritten":354}},
"ingress:1": {"name":"ingress","tags":{"login":"admin"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:2": {"name":"ingress","tags":{"login":"telegraf"},"values":{"pointsWritten":3547,"valuesWritten":28246}},
Notice writes by users 'telegraf', '_systemuser_monitor', and 'admin'.
2021-02-04 11:52:53 -05:00
b3e763d96f
fix: consistent error for missing shard ( #20694 )
2021-02-04 09:49:14 -05:00
bb27966fc2
Merge pull request #20677 from lesam/ingress-metrics-measurement-points
...
feat: Ingress metrics by measurement
2021-02-02 16:25:10 -05:00
eb92c997cd
feat: Ingress metrics by measurement
...
Partial implementation of https://github.com/influxdata/influxdb/issues/20612
Implements per-measurement points written metric. Next step: Also support per-login.
2021-02-02 15:58:28 -05:00
7a9a0ec1bf
Merge pull request #20664 from lesam/measurement-metric-ingress
...
refactor: do not use context value anti-pattern
2021-02-02 11:12:43 -05:00
117341fb0f
fix: Move value metric down to tsdb store
...
Previously we tracked values on the http ingress, but the tsdb store is the correct
place to track total values written for the instance.
2021-02-02 10:58:47 -05:00
6795ec6c01
refactor: do not use context value anti-pattern
...
Extending the context instead of fixing the API breaks type safety.
For tracking the number of points / values written, it is much clearer
to pass an explicit tracker.
2021-02-01 14:34:11 -05:00
a483fb5068
Merge pull request #20649 from lesam/add-goimports
...
chore: add goimports to ci checks
2021-01-29 14:19:39 -05:00
ec40d5c380
chore: Fix spaces
2021-01-29 13:20:36 -05:00
053fae914b
chore: update docs with install instructions for goimports
2021-01-29 11:45:17 -05:00
8a16bf0531
chore: run goimports -w ./
2021-01-29 11:40:02 -05:00
Daniel Moran
Sam Arnold
davidby-influx
davidby-influx