274f33b3f4
chore: add back hashing of build tarball ( #20981 )
2021-03-17 10:13:34 -04:00
52f939eadf
chore: fix oss unit tests ( #20972 )
2021-03-16 11:56:28 -04:00
1e53bf13e0
chore: fix release build scripts ( #20970 )
...
Note: this commit fixes linux/amd64 and darwin/amd64 builds. We are still
working on fixes for linux/arm64 and windows/amd64.
Partial fix of https://github.com/influxdata/influxdb/issues/20854
2021-03-16 09:23:55 -04:00
178a1d0c3b
chore: 1.9.0 rc0 changelog ( #20948 )
...
* chore: reset changelog to 1.8.5
* chore: update changelog in preparation for 1.9.0rc0
2021-03-15 12:34:04 -04:00
dd6abe6f19
fix(flux): add durations to Flux logging ( #19697 ) ( #20940 )
...
(cherry picked from commit 25fb1077e2
2021-03-15 12:33:57 -04:00
532a23da84
chore: remove Jenkinsfile ( #20951 )
...
The unit tests are run by circleci and the changelog updates are not working
currently.
2021-03-15 12:33:29 -04:00
849221b709
fix: storage read service should not return a nil cursor ( #17359 ) ( #20941 )
...
The storage read service would return a nil cursor if there was no data
and no shards to read from. This modifies it to return an empty cursor
instead.
(cherry picked from commit 2852bf0399
2021-03-12 16:36:08 -05:00
df46a86326
fix(tls): Enable configuration of TLS 1.3 ( #20939 )
...
Co-authored-by: Gunnar Aasen <gunnar.r.aasen@gmail.com>
2021-03-12 16:35:28 -05:00
aa2fcd295f
fix(logger): Ensure default level value is info ( #17403 ) ( #20937 )
...
(cherry picked from commit 4d5a688a1e
2021-03-12 16:34:51 -05:00
5ef0d38bc7
fix(flux): buckets call no longer panics ( #17319 ) ( #20936 )
...
The buckets call had a column removed and the backport did not remove
that line. This modifies the buckets command to not append a row to a
column that no longer exists.
(cherry picked from commit 637ca24d5e
2021-03-12 16:34:40 -05:00
b0cf385491
fix(storage): ArrayFilterCursor truncation for multi-block data ( #19439 ) ( #20935 )
...
Backport abfe5a54a077d1a4180b
2021-03-12 16:34:31 -05:00
04f4817aae
fix(services/storage): multi measurement queries return all applicable series ( #19592 ) ( #20934 )
...
This fixes multi measurement queries that go through the storage service
to correctly pick up all series that apply with the filter. Previously,
negative queries such as `!=`, `!~`, and predicates attempting to match
empty tags did not work correctly with the storage service when multiple
measurements or `OR` conditions were included.
This was because these predicates would be categorized as "multiple
measurements" and then it would attempt to use the field keys iterator
to find the fields for each measurement. The meta queries for these did
not correctly account for negative equality operators or empty tags when
finding appropriate measurements and those could not be changed because
it would cause a breaking change to influxql too.
This modifies the storage service to use new methods that correctly
account for the above situations rather than the field keys iterator.
Some queries that appeared to be single measurement queries also get
considered as multiple measurement queries. Any query with an `OR`
condition will be considered a multiple measurement query.
This bug did not apply to single measurement queries where one
measurement was selected and all of the logical operators were `AND`
values. This is because it used a different code path that correctly
handled these situations.
Backport of #19566 .
(cherry picked from commit ceead88bd5
2021-03-12 16:34:14 -05:00
3eb4fdaf33
fix(tsm1): fix data race when accessing tombstone stats ( #20903 )
2021-03-09 15:20:40 -05:00
6b8ec8cfe0
chore: remove deprecated tool influx_stress ( #20892 )
...
https://github.com/influxdata/influxdb/issues/20891
2021-03-09 10:29:24 -05:00
bf7dddaec5
fix: Add back flux CLI ( #20889 )
...
Closes:
https://github.com/influxdata/influxdb/issues/18947
https://github.com/influxdata/influxdb/issues/20852
Similarly to 2.x, the flux CLI now only supports the -execute argument
or commands on stdin. Examples:
```
➜ echo 'import "influxdata/influxdb/v1" v1.databases()' | ./influx -type flux
Result: _result
Table: keys: [organizationID]
organizationID:string databaseName:string retentionPolicy:string retentionPeriod:int default:bool bucketId:string
---------------------- ---------------------- ---------------------- -------------------------- ------------ ----------------------
_internal monitor 604800000000000 true
telegraf autogen 0 true
db autogen 0 true
➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1" v1.databases()'
Result: _result
Table: keys: [organizationID]
organizationID:string databaseName:string retentionPolicy:string retentionPeriod:int default:bool bucketId:string
---------------------- ---------------------- ---------------------- -------------------------- ------------ ----------------------
_internal monitor 604800000000000 true
telegraf autogen 0 true
db autogen 0 true
➜ ./influx -type flux -execute 'import "influxdata/influxdb/v1"'
Error (500): error in query specification while starting program: this Flux script returns no streaming data. Consider adding a "yield" or invoking streaming functions directly, without performing an assignment
```
Note that tty input is explicitly forbidden, unlike 2.x:
```
➜ ./influx -type flux
Connected to http://localhost:8086 version unknown
InfluxDB shell version: unknown
Interactive flux is not supported. Provide your flux script via stdin or the -execute argument.
```
2021-03-09 09:56:52 -05:00
7a992dac77
feat: add arguments to flux to function ( #20873 )
...
Also remove some dead code
Closes #20853
2021-03-05 11:25:52 -05:00
d6f7716924
fix: infinite recursion bug ( #20862 )
...
* Revert "fix(error): unsupported value: +Inf" error not handled gracefully (#20250 )"
This reverts commit 6ac0bb3fe3https://github.com/influxdata/influxdb/issues/20249
2021-03-04 16:31:06 -05:00
7210fa6355
chore: Upgrade 1.x to the latest flux ( #20767 )
...
* chore: Update flux to 0.67
* chore: Builds against 0.68 flux
* chore: Builds against 0.80.0
* chore: Builds against 0.90.0
* chore: Everything builds on latest flux
* chore: goimports fixed
* chore: fix tests locally
* chore: fix CI dockerfiles
* chore: clean up some unused code
* chore: remove flux repl and Spec in flux query json
* chore: port flux end to end tests from 2.x
* chore: fix up goimports
* chore: remove 32 bit build support
2021-03-03 16:24:49 -05:00
de491dab97
refactor: Remove unused function add and unused variable keysHint ( #20803 ) ( #20805 )
...
(cherry picked from commit 1068d1de6f
2021-02-25 15:15:32 -05:00
17b9ea8723
feat: Add WITH KEY to show tag keys ( #20793 )
...
* fix: Change from RewriteExpr to PartitionExpr
Also remove some dead code
* feat: WITH KEY implementation
* feat: query rewriting for WITH KEY in SHOW TAG KEYS
2021-02-25 08:38:29 -05:00
e85a10248d
fix(tsm1): fix data race and validation in cache ring ( #20802 )
...
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
Co-authored-by: Yun Zhao <zhaoyun2316@gmail.com>
2021-02-24 17:17:08 -05:00
a8b2129df5
chore: move context.Context to first argument in methods per convention
2021-02-23 12:52:44 -08:00
092c7a9976
feat: Make meta queries respect QueryTimeout values
...
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.
Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries
https://github.com/influxdata/influxdb/issues/20736
2021-02-23 12:52:44 -08:00
de1a0eb2a9
feat: use count_hll for 'show series cardinality' queries ( #20745 )
...
Closes: https://github.com/influxdata/influxdb/issues/20614
Also fix nil pointer for seriesKey iterator
Fix for bug in: https://github.com/influxdata/influxdb/issues/20543
Also add a test for ingress metrics
2021-02-10 16:00:16 -05:00
903b8cd0ea
feat(query): Hyper log log operators in influxql ( #20603 )
...
* feat(query): hyper log log counting in query engine
In addition to helping with normal queries, this can improve the 'SHOW CARDINALITY'
meta-queries:
time influx -database mydb -execute 'select count_hll(sum_hll(_seriesKey)) from big'
name: big
time count_hll
---- ---------
0 200767781
influx -database mydb -execute 0.06s user 0.12s system 0% cpu 8:49.99 total
2021-02-08 08:38:14 -05:00
21823db00b
feat: series creation ingress metrics ( #20700 )
...
After turning this on and testing locally, note the 'seriesCreated' metric
"localStore": {"name":"localStore","tags":null,"values":{"pointsWritten":2987,"seriesCreated":58,"valuesWritten":23754}},
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":4}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":46}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":14,"seriesCreated":14,"valuesWritten":42}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":10}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":30}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":22}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":6}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":18}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":58}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":4}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":2,"valuesWritten":8}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":2,"seriesCreated":1,"valuesWritten":18}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1342,"seriesCreated":13,"valuesWritten":13420}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"seriesCreated":6,"valuesWritten":4494}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":2,"valuesWritten":2354}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":963}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"seriesCreated":1,"valuesWritten":856}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"seriesCreated":1,"valuesWritten":642}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"_systemuser_unknown","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"seriesCreated":1,"valuesWritten":749}},
Closes: https://github.com/influxdata/influxdb/issues/20613
2021-02-05 14:52:43 -04:00
dd3baf6d4a
feat: measurement metrics by login ( #20687 )
...
After turning on authentication and both forms of ingress metrics:
"ingress": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"cq","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:1": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"database","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:2": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"httpd","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":874}},
"ingress:3": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"ingress","rp":"monitor"},"values":{"pointsWritten":534,"valuesWritten":1068}},
"ingress:4": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"localStore","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":76}},
"ingress:5": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"queryExecutor","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":190}},
"ingress:6": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"runtime","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":570}},
"ingress:7": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"shard","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":836}},
"ingress:8": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"subscriber","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":114}},
"ingress:9": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_cache","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":684}},
"ingress:10": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_engine","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":2204}},
"ingress:11": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_filestore","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":152}},
"ingress:12": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"tsm1_wal","rp":"monitor"},"values":{"pointsWritten":76,"valuesWritten":304}},
"ingress:13": {"name":"ingress","tags":{"db":"_internal","login":"_systemuser_monitor","measurement":"write","rp":"monitor"},"values":{"pointsWritten":38,"valuesWritten":342}},
"ingress:14": {"name":"ingress","tags":{"db":"telegraf","login":"admin","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:15": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"cpu","rp":"autogen"},"values":{"pointsWritten":1316,"valuesWritten":13160}},
"ingress:16": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"disk","rp":"autogen"},"values":{"pointsWritten":642,"valuesWritten":4494}},
"ingress:17": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"diskio","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":2354}},
"ingress:18": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"mem","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":963}},
"ingress:19": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"processes","rp":"autogen"},"values":{"pointsWritten":107,"valuesWritten":856}},
"ingress:20": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"swap","rp":"autogen"},"values":{"pointsWritten":214,"valuesWritten":642}},
"ingress:21": {"name":"ingress","tags":{"db":"telegraf","login":"telegraf","measurement":"system","rp":"autogen"},"values":{"pointsWritten":321,"valuesWritten":749}},
Only by login:
"ingress": {"name":"ingress","tags":{"login":"_systemuser_monitor"},"values":{"pointsWritten":42,"valuesWritten":354}},
"ingress:1": {"name":"ingress","tags":{"login":"admin"},"values":{"pointsWritten":1,"valuesWritten":1}},
"ingress:2": {"name":"ingress","tags":{"login":"telegraf"},"values":{"pointsWritten":3547,"valuesWritten":28246}},
Notice writes by users 'telegraf', '_systemuser_monitor', and 'admin'.
2021-02-04 11:52:53 -05:00
b3e763d96f
fix: consistent error for missing shard ( #20694 )
2021-02-04 09:49:14 -05:00
bb27966fc2
Merge pull request #20677 from lesam/ingress-metrics-measurement-points
...
feat: Ingress metrics by measurement
2021-02-02 16:25:10 -05:00
eb92c997cd
feat: Ingress metrics by measurement
...
Partial implementation of https://github.com/influxdata/influxdb/issues/20612
Implements per-measurement points written metric. Next step: Also support per-login.
2021-02-02 15:58:28 -05:00
7a9a0ec1bf
Merge pull request #20664 from lesam/measurement-metric-ingress
...
refactor: do not use context value anti-pattern
2021-02-02 11:12:43 -05:00
117341fb0f
fix: Move value metric down to tsdb store
...
Previously we tracked values on the http ingress, but the tsdb store is the correct
place to track total values written for the instance.
2021-02-02 10:58:47 -05:00
6795ec6c01
refactor: do not use context value anti-pattern
...
Extending the context instead of fixing the API breaks type safety.
For tracking the number of points / values written, it is much clearer
to pass an explicit tracker.
2021-02-01 14:34:11 -05:00
a483fb5068
Merge pull request #20649 from lesam/add-goimports
...
chore: add goimports to ci checks
2021-01-29 14:19:39 -05:00
ec40d5c380
chore: Fix spaces
2021-01-29 13:20:36 -05:00
053fae914b
chore: update docs with install instructions for goimports
2021-01-29 11:45:17 -05:00
8a16bf0531
chore: run goimports -w ./
2021-01-29 11:40:02 -05:00
b0d26fe412
chore: add goimports to ci checks
2021-01-29 11:39:42 -05:00
b2f0d05ecc
Merge pull request #20647 from lesam/add-diskusage-inspect
...
feat(inspect): Add report-disk for disk usage by measurement
2021-01-29 11:36:31 -05:00
3a31e2370e
fix(inspect): bad pattern matching
2021-01-29 10:55:20 -05:00
a6152e8ac1
feat(inspect): Add report-disk for disk usage by measurement
2021-01-29 10:51:42 -05:00
d28bcb8e27
Merge pull request #20544 from lesam/series-iteration-optimization
...
feat(tsi): optimize series iteration
2021-01-25 18:17:18 -04:00
98a76a11a0
feat(tsi): optimize series iteration
...
When using queries like 'select count(_seriesKey) from bigmeasurement`, we
should iterate over the tsi structures to serve the query instead of loading
all the series into memory up front.
Closes #20543
2021-01-25 14:27:31 -05:00
fe3af66c54
fix(tsdb): minimize lock contention when adding new fields or measurements ( #20504 )
...
fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written
in memory and when the new file is being renamed.
Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.
Fixes https://github.com/influxdata/influxdb/issues/20500
2021-01-15 08:31:45 -08:00
415361e1eb
Merge pull request #20481 from influxdata/fix-tests
...
fix: minor test fixes for go1.15 and also flaky timeouts
2021-01-08 15:21:19 -05:00
32612313df
fix: minor test fixes for go1.15 and also flaky timeouts
...
Also run gofmt
2021-01-08 14:59:33 -05:00
9e33be2619
fix(error): SELECT INTO doesn't return error with unsupported value ( #20429 )
...
When a SELECT INTO query generates an illegal value that cannot be inserted,
like +/- Inf, it should return an error, rather than failing silently.
This adds a boolean parameter to the [data] section of influxdb.conf:
* strict-error-handling
When false, the default, the old behavior is preserved. When true,
unsupported values will return an error from SELECT INTO queries
Fixes https://github.com/influxdata/influxdb/issues/20426
2020-12-30 18:22:43 -08:00
2e26dc62cb
build: switch tested centos base images ( #20417 )
2020-12-23 21:17:55 -08:00
8a8b25ec4f
fix(prometheus): regexp handling should comply with PromQL ( #19832 ) ( #20388 )
...
(cherry picked from commit 5296fe990f
2020-12-18 14:58:29 -08:00
5b98166b05
fix: cp.Mux.Serve() closes all net.Listener instances silently on error. ( #20278 )
...
A customer has seen a rash of "connection refused" errors to the meta node.
This fix ensures that when net.Listener instances are closed because of an
error in Accept(), influxdb logs the error which caused the closures, as well
as any errors in closing the Listeners.
Fixes https://github.com/influxdata/influxdb/issues/20256
2020-12-08 16:17:59 -08:00
Sam Arnold
Daniel Moran
davidby-influx
davidby-influx