* build: upgrade to go1.18 (#23250)
* chore: upgrade to Go 1.18.6
* fix: remove unused directory
* chore: upgrade to Go 1.18.7
Co-authored-by: Dane Strandboge <dstrandboge@influxdata.com>
Dump all active queries to the log when a SIGTERM
is received and the termination-query-log flag is
true in the coordinator section of the config. The
default is false.
(cherry picked from commit fd1d51690d)
closes https://github.com/influxdata/influxdb/issues/22637
Closes#22031
This updates the flux integration to use the `flux/array` package rather
than directly using the `arrow/array` package.
Flux recently switched to wrapping the array types from arrow and
creating its own array package to be used for table columns instead of
directly referencing the arrow package. This allows us to keep a
consistent interface, but potentially change internal implementations
without changing downstream consumers. Most recently, the
`*array.String` type has some of its own optimizations for certain array
patterns.
This change updates the flux integration to use the new API.
(cherry picked from commit 8fae484a66)
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
Closes#22033
* fix: tsi index should compact old log files that are too large
* chore: run automated formatter
* chore: update changelog
* fix: review comments
(cherry picked from commit b64c2c3dcf)
* chore: harmonize OSS and Enterprise subscriber initialization
* refactor: chanwriter close blocks until chanwriter is flushed
* refactor: remove redundant state
* test: add test for blocked subscriber during subscription update
* fix: blocked subscriber does not fail subscription update
* fix: only iterate matching subscriptions
* fix: use serialized points in subscriber queues
* feat: add total-buffer-bytes config parameter to subscriptions
* fix: put subscription serialization on the write path
* fix: subscription service only needs regular mutex, not RWMutex
* fix: review comments
* chore: update changelog
(cherry picked from commit 85abb3af36)
* feat: make flux controller limits configurable
A sample of the new config:
```
[flux-controller]
query-concurrency = 0
query-initial-memory-bytes = 0
query-max-memory-bytes = 0
total-max-memory-bytes = 0
query-queue-size = 0
```
Also use the prometheus metrics in debug/vars, here is a sample:
```
"query_control_all_active": {"name":"query_control_all_active","tags":null,"values":{"gauge":0}},
"query_control_all_duration_seconds": {"name":"query_control_all_duration_seconds","tags":null,"values":{"0.001":0,"0.005":0,"0.025":0,"0.125":0,"0.625":0,"15.625":2,"3.125":2,"count":2,"sum":2.9953034240000003}},
"query_control_compiling_active": {"name":"query_control_compiling_active","tags":null,"values":{"gauge":0}},
"query_control_compiling_duration_seconds": {"name":"query_control_compiling_duration_seconds","tags":null,"values":{"0.001":2,"0.005":2,"0.025":2,"0.125":2,"0.625":2,"15.625":2,"3.125":2,"count":2,"sum":0.0010411650000000001}},
"query_control_executing_active": {"name":"query_control_executing_active","tags":null,"values":{"gauge":0}},
"query_control_executing_duration_seconds": {"name":"query_control_executing_duration_seconds","tags":null,"values":{"0.001":0,"0.005":0,"0.025":0,"0.125":0,"0.625":0,"15.625":2,"3.125":2,"count":2,"sum":2.994032791}},
"query_control_memory_unused_bytes": {"name":"query_control_memory_unused_bytes","tags":null,"values":{"gauge":0}},
"query_control_queueing_active": {"name":"query_control_queueing_active","tags":null,"values":{"gauge":0}},
"query_control_queueing_duration_seconds": {"name":"query_control_queueing_duration_seconds","tags":null,"values":{"0.001":2,"0.005":2,"0.025":2,"0.125":2,"0.625":2,"15.625":2,"3.125":2,"count":2,"sum":0.000087963}},
"query_control_requests_total": {"name":"query_control_requests_total","tags":null,"values":{"counter":1}},
"query_control_requests_total:1": {"name":"query_control_requests_total","tags":null,"values":{"counter":1}}
```
* chore: update changelog
* fix: shorten metric names for query control
* fix: zaptest logger and goimports
* fix: races in the query controller
Previously some tests were failing due to logging after the end of the test.
* chore: pull in controller from 2.x
* chore: fix up 2.x controller to work with 1.x
* feat: Default query limits in flux code
Partial fix of https://github.com/influxdata/influxdb/issues/17212
* chore: update changelog
* chore: refactor to remove panic and reformat code
* test: add script to run flux tests
* feat(flux): enable test capabilities in Flux controller
* feat(flux): add MergeFiltersRule
* build: bump existing Dockerfiles to go 1.15
* build: add flux tests to CI
* refactor: allow for overriding tcp.Mux logger
* build: upgrade to Flux v0.111.0
Add a special value to the -out flag, a hyphen, to write to stdout.
While writing to stdout, send status messages to stderr instead of
stdout (the current behavior).
Closes https://github.com/influxdata/influxdb/issues/20974
* chore: Update flux to 0.67
* chore: Builds against 0.68 flux
* chore: Builds against 0.80.0
* chore: Builds against 0.90.0
* chore: Everything builds on latest flux
* chore: goimports fixed
* chore: fix tests locally
* chore: fix CI dockerfiles
* chore: clean up some unused code
* chore: remove flux repl and Spec in flux query json
* chore: port flux end to end tests from 2.x
* chore: fix up goimports
* chore: remove 32 bit build support
Meta queries (SHOW TAG VALUES, SHOW TAG KEYS, SHOW SERIES CARDINALITY, etc.) do not respect
the QueryTimeout config parameter. Meta queries should check the query context when possible
to allow cancellation and timeout. This will not be as frequent as regular queries, which
use iterators, because meta queries return data in batches.
Add a context.Context to
(*Store).MeasurementNames()
(*Store).MeasurementsCardinality()
(*Store).SeriesCardinality()
(*Store).TagValues()
(*Store).TagKeys()
(*Store).SeriesSketches()
(*Store).MeasurementsSketches()
which is tested for timeout or cancellation
to allow limitation of time spent in meta queries
https://github.com/influxdata/influxdb/issues/20736
When a SELECT INTO query generates an illegal value that cannot be inserted,
like +/- Inf, it should return an error, rather than failing silently.
This adds a boolean parameter to the [data] section of influxdb.conf:
* strict-error-handling
When false, the default, the old behavior is preserved. When true,
unsupported values will return an error from SELECT INTO queries
Fixes https://github.com/influxdata/influxdb/issues/20426
When applied, this patch will add the -lponly flag to the export command
which instructs influx_inspect to only output line protocol without
comments and other out-of-band data.
We were seing segfaults in Roaring bitmaps sometimes, under very
high load with networked drives. This may reduce risk of segfault by
forcing marshalling to copy the data.
refactor(influxdb): Refactor trace code for clarity and reliability
* Make startProfile a method of Server. startProfile() is only used one
place. We bother to copy the Server.CPUProfile and Server.MemProfile
values out of our Server struct into it's parameters. It makes more
sense to make startProfile() a method of Server and have it access
those members directory via its receiver value.
* Have startProfile() return an error instead of log.Fatal()ing. We can
simply propagate the error up the stack and let the caller handle the
error -- we shouldn't be exiting deep in the bowels of a non-main
package.
* Capture and return errors from pprof.StartCPUProfile(). Currently
there is only one possible error it can return but if it returns an
error, we should handle it.
* add CPUProfileWriteCloser and MemProfileWriteCloser to Server struct.
* make stopProfile() a method of Server
* remove prof variable
* fix captialization of log messages.
Jeffrey Smith II
Brandon Pfeifer
davidby-influx
Dane Strandboge
Sam Arnold
Daniel Moran
davidby-influx
Ayan George
Tristan Su
Ben Johnson
David Norton
docmerlin (j. Emrys Landivar)
Stuart Carnie