Commit Graph

169 Commits (db/4201/compaction-bugs)

Author SHA1 Message Date
Shiwen Cheng f01181a7a1
fix: fix typo 'exceeed' with 'exceeded' (#25292) 2024-09-09 15:15:25 -07:00
Geoffrey Wossum 2cf2103cc4
feat: add hook for optimizing series reads based on authorizer (#25207) 2024-08-02 15:03:44 -05:00
davidby-influx d60741b506
fix: GROUP BY queries with offset that crosses a DST boundary fail. (#25082)
This is actually the second fix for
https://github.com/influxdata/influxdb/issues/20238
for when the time zone falls back in autumn.

closes https://github.com/influxdata/influxdb/issues/25078
2024-06-24 10:24:34 -07:00
davidby-influx 5fda409f39
fix: return MergeIterator.Close errors (#24975)
Ensure that errors from closing the
iterators underneath a MergeIterator
are returned up the stack.
2024-05-13 15:10:56 -07:00
Brandon Pfeifer d4b16dcd98
chore: upgrade protocol buffers to v5.26.1 (#24949) 2024-05-01 11:00:26 -07:00
Jakub Bednář dbbe4611c0
build(deps): upgrade google.golang.org/protobuf to v1.33.0 (master-1.x) (#24818) 2024-03-26 14:07:28 +01:00
Geoffrey Wossum 8648298012
chore: upgrade to influxdata/influxql v1.2.0 (#24764)
Upgrade to influxdata/influxql v1.2.0. While it does not fix any
known issues in InfluxDB OSS 1.x, it is necessary for upstream
projects impacted by https://github.com/influxdata/influxql/issues/65.

In addition to upgrading influxdata/influxql, this also updates test
cases that relied on the erroneous precision handling when stringifying
InfluxQL ASTs.

Visible impacts to InfluxDB OSS 1.x:
- Changes precision of floating point numbers in error messages
  related to InfluxQL
- Changes precision of floating point numbers in "EXPLAIN" and
  "EXPLAIN ANALYZE" output
- Changes precision of floating point numbers from InfluxQL
  expressions included in tracing spans

closes: #24763
2024-03-14 12:42:03 -05:00
davidby-influx c05b340b72
chore: upgrade flux (#24504)
* chore: upgrade flux

* chore: execute "go generate" inside cross-builder (#24582)

---------

Co-authored-by: Brandon Pfeifer <bpfeifer@influxdata.com>
2024-01-19 17:40:48 -05:00
Jeffrey Smith II 88d2760bc8
feat: add the ability to log queries killed by `query-timeout` (#23978)
* feat: add the ability to log queries killed by `query-timeout`

* chore: update example config

* chore: improve logging details
2023-01-10 13:37:11 -05:00
Brandon Pfeifer e484c4d871
chore: upgrade Go to v1.19.3 (1.x) (#23941)
* chore: upgrade Go to 1.19.3

This re-runs ./generate.sh and ./checkfmt.sh to format and update
source code (this is primarily responsible for the huge diff.)

* fix: update tests to reflect sorting algorithm change
2022-11-28 12:15:47 -05:00
Brandon Pfeifer 5976e41d54
feat: upgrade flux to v0.188.0 (#23911)
* feat: upgrade flux to 0.171.0

Tests failing, safety commit

First step in https://github.com/influxdata/influxdb/issues/23815

* fix: remove "org" parameter" from writeOptSource

I attempted to implement the "orgOpt" argument in a similar fashion
to f6669f7512. However, it looks like Flux doesn't accept "org" as
a parameter to "load". It responds with:

Error calling function \"load\" @113:16-113:30: error calling function \"to\" @6:19-6:47: unused arguments [org]

This brings us from 194 passing to 570 passing.

* fix: temporarily disable broken flux tests

These tests expect rows to be stored in a certain order. However,
nothing is specifying the sort order. This has been fixed in a
later update to flux: (see 3d6f47ded).

Temporarily disable these tests until we include a fixed
version of the flux tests.

* chore: add tests from a492993012

This fixes "test-flux.sh" so it runs tests within the "flux/"
directory. This uncovered some other issues with the tests
located within "flux/". These also needed to be updated
to match the newer flux API.

* feat: upgrade flux to 0.172.0

This includes changes made in "cbbf4b27da". Since "test.go" in 2.x
diverged from 1.x, some modifications were required to make this
compatible.

* feat: upgrade flux to 0.173.0

* feat: upgrade flux to v0.174.0

* fix: Update the condition when reseting cursor (#23522)

Filters that contain `or` may change between cursor resets so we must remember to update the condition in the read cursor.

```flux
|> filter(fn: (r) => ((r["_field"] == "field1" and r["_value"]==true) or (r["_field"] == "field2" and r["_value"] == false)))
```

Closes https://github.com/influxdata/flux/issues/4804

* feat: upgrade flux to 0.174.1

* feat: upgrade flux to 0.175.0

* chore: remove end-to-end tests

These were removed in a492993 for 2.x. These tests prevent "go test ./..."
from completing. As stated in the original commit, these tests should now be
handled by the "fluxtest" harness.

* feat: upgrade flux to 0.176.0

Some tests needed to be disabled within the flux harness. This is a
result of enabling "Optimize Aggregate Window" in flux@05a1065f.
These tests are not present in 2.x. Therefore, I am unsure if
the breakage is resolved in a later commit.

* feat: upgrade flux to 0.177.0

* feat: upgrade flux to 0.178.0

* feat: upgrade flux to v0.179.0

This removes all invocations of "flux.RegisterOpSpec". According
to flux@e39096d5, "flux.RegisterOpSpec" does nothing in the
current version of flux and was removed.

* chore: update fluxtest skip list (#23633)

* chore: manually backport 785a465e9a

This removes the reference to "flux.Spec".

* build(flux): update flux to v0.181.0 (#23682)

* build(flux): update flux to v0.184.2

* chore: skip more Flux acceptance tests

There are issues for each skip detailed in test-flux.sh.

* feat: upgrade flux to v0.185.0

This adds "FluxTesting" to the "HTTPD" configuration. This option is
hidden and disabled by default. When "FluxTesting" is set, it
enables the default testing flags for "Flux".

These flags allow the "vectorized float tests" and tests requiring
the "removeRedundantSortNodes" and "labelPolymorphism" flag
enabled to work. These changes are based off of d8553c002e.

flux@3d6f47ded is included within this version of Flux. Therefore
we can now include the "group_*" tests.

* feat: upgrade flux to 0.186.0

* feat: upgrade flux to 0.187.0

* feat: upgrade flux to 0.188.0

* fix: re-run ./generate.sh with updated protoc

* fix: restrict cores to match CircleCI documentation

Co-authored-by: davidby-influx <dbyrne@influxdata.com>
Co-authored-by: Markus Westerlind <marwes91@gmail.com>
Co-authored-by: Sean Brickley <sean@wabr.io>
Co-authored-by: Jonathan A. Sternberg <jonathan@influxdata.com>
Co-authored-by: Christopher M. Wolff <chris.wolff@influxdata.com>
2022-11-15 15:20:27 -05:00
davidby-influx f5da0f50f4
fix: Optimize SHOW FIELD KEY CARDINALITY (#23871)
Use the _fieldKeys system iterator

closes https://github.com/influxdata/influxdb/issues/23840
2022-11-08 08:32:10 -08:00
Brandon Pfeifer 3074e62784
chore: upgrade flux to v1.167.0 for 1.x (#23349)
* chore: backport 8334dd0a23 to 1.x

* chore: upgrade flux to v0.167.0

* chore: update flux to latest version (#23249)

* chore: update flux to latest version

* fix: backport "convert allocator to interface"

* fix: construct `span` during dependency injection
2022-05-17 15:59:24 -04:00
Dane Strandboge 8b38d0e2bf
build: upgrade protobuf library (#22606) 2021-10-15 11:42:47 -05:00
davidby-influx fd1d51690d
feat: optionally dump queries to log on SIGTERM (#22638)
Dump all active queries to the log when a SIGTERM
is received and the termination-query-log flag is
true in the coordinator section of the config. The
default is false.
2021-10-07 14:02:38 -07:00
Sam Arnold 50d3bcad99
build: upgrade flux to 0.127.3 (#22375) 2021-09-02 14:21:56 -04:00
Sam Arnold 38de69cc1c
fix: flux error properly read by cloud (#22348) 2021-08-31 17:43:12 -04:00
davidby-influx 0090c5b111
fix: return correct count of ErrNotExecuted (#22273)
executeQuery() iterates over statements until each is
processed or if an error is encountered that causes
the loop to exit pre-maturely. It should return
ErrNotExecuted for each remaining statement in the
query

closes https://github.com/influxdata/influxdb/issues/19136
2021-08-24 11:27:10 -07:00
Sam Arnold 90b4c80af2
test: compile time only needs testing once (#22123)
* test: compile time only needs testing once

* test: fix review comments for Cody
2021-08-09 16:58:07 -04:00
Sam Arnold 733a6b4245
feat: add subscription buffer size usage metric (#22020) 2021-08-03 14:32:41 -04:00
davidby-influx 519e23b86a
feat: SHOW TAG VALUES should produce results from one specific RP (#21983)
Ensure that the Sources field of the ShowTagValuesStatement is
filled in. Then use the sources to limit the retention policies,
and thus the shards from which tag values are collected.

This fix only works on TSI databases; INMEM shards share
indices, so restricting shard indices used does not restrict the
tag values returned.

This will not permit multiple retention policies to be specified in 
a query; either all RPs or one are permitted.

Closes https://github.com/influxdata/influxdb/issues/21981
2021-08-02 11:11:39 -07:00
Sam Arnold eef37b96b9
test: ensure compile takes time (#21973) 2021-07-29 17:22:17 -04:00
Sam Arnold b64c2c3dcf
fix: tsi index should compact old or too-large log files (#21943)
* fix: tsi index should compact old log files that are too large

* chore: run automated formatter

* chore: update changelog

* fix: review comments
2021-07-26 17:40:15 -04:00
Sam Arnold 23c3d35aab
chore: update protobuf library versions and remove influx_tsm (#21882)
* chore: update protobufs

* fix: run codegen during build

* fix: fully remove influx_tsm
2021-07-20 09:42:52 -04:00
Sam Arnold 98361e2073
fix: error instead of panic for statement rewrite failure (#21792) 2021-07-06 11:05:21 -04:00
davidby-influx acc4105b8c
fix: do not panic on cleaning up failed iterators (#21666)
We have seen occasional panics in Iterators.Close()
when cleaning up after failed iterator creation.
This commit checks for nil on any iterator to be
closed, and now returns any errors generated by
that Close().

Closes https://github.com/influxdata/influxdb/issues/19579
Closes https://github.com/influxdata/influxdb/issues/19476
2021-06-11 11:28:59 -07:00
Sam Arnold 894f54e6ac
fix: group by returns multiple results per group in some circumstances (#21631)
* fix: Revert performance improvement for sorted merge iterator

This reverts commit af8e66cd25.

* test: add end to end regression test for broken group-by

* chore: update changelog
2021-06-08 10:41:58 -04:00
Sam Arnold f1aa051787
feat: make flux controller limits configurable (#21108)
* feat: make flux controller limits configurable

A sample of the new config:

```
[flux-controller]
  query-concurrency = 0
  query-initial-memory-bytes = 0
  query-max-memory-bytes = 0
  total-max-memory-bytes = 0
  query-queue-size = 0
```

Also use the prometheus metrics in debug/vars, here is a sample:

```
"query_control_all_active": {"name":"query_control_all_active","tags":null,"values":{"gauge":0}},
"query_control_all_duration_seconds": {"name":"query_control_all_duration_seconds","tags":null,"values":{"0.001":0,"0.005":0,"0.025":0,"0.125":0,"0.625":0,"15.625":2,"3.125":2,"count":2,"sum":2.9953034240000003}},
"query_control_compiling_active": {"name":"query_control_compiling_active","tags":null,"values":{"gauge":0}},
"query_control_compiling_duration_seconds": {"name":"query_control_compiling_duration_seconds","tags":null,"values":{"0.001":2,"0.005":2,"0.025":2,"0.125":2,"0.625":2,"15.625":2,"3.125":2,"count":2,"sum":0.0010411650000000001}},
"query_control_executing_active": {"name":"query_control_executing_active","tags":null,"values":{"gauge":0}},
"query_control_executing_duration_seconds": {"name":"query_control_executing_duration_seconds","tags":null,"values":{"0.001":0,"0.005":0,"0.025":0,"0.125":0,"0.625":0,"15.625":2,"3.125":2,"count":2,"sum":2.994032791}},
"query_control_memory_unused_bytes": {"name":"query_control_memory_unused_bytes","tags":null,"values":{"gauge":0}},
"query_control_queueing_active": {"name":"query_control_queueing_active","tags":null,"values":{"gauge":0}},
"query_control_queueing_duration_seconds": {"name":"query_control_queueing_duration_seconds","tags":null,"values":{"0.001":2,"0.005":2,"0.025":2,"0.125":2,"0.625":2,"15.625":2,"3.125":2,"count":2,"sum":0.000087963}},
"query_control_requests_total": {"name":"query_control_requests_total","tags":null,"values":{"counter":1}},
"query_control_requests_total:1": {"name":"query_control_requests_total","tags":null,"values":{"counter":1}}
```

* chore: update changelog

* fix: shorten metric names for query control

* fix: zaptest logger and goimports

* fix: races in the query controller

Previously some tests were failing due to logging after the end of the test.
2021-04-01 09:55:59 -04:00
Sam Arnold b9ee9e5b44
feat: add memory and concurrency limits in flux controller (#21100)
* chore: pull in controller from 2.x

* chore: fix up 2.x controller to work with 1.x

* feat: Default query limits in flux code

Partial fix of https://github.com/influxdata/influxdb/issues/17212

* chore: update changelog

* chore: refactor to remove panic and reformat code
2021-03-30 15:02:46 -04:00
Sam Arnold b7e7de24d6
refactor: separate coarse and fine permission interfaces (#20996) 2021-03-22 09:52:33 -04:00
Sam Arnold 7210fa6355
chore: Upgrade 1.x to the latest flux (#20767)
* chore: Update flux to 0.67

* chore: Builds against 0.68 flux

* chore: Builds against 0.80.0

* chore: Builds against 0.90.0

* chore: Everything builds on latest flux

* chore: goimports fixed

* chore: fix tests locally

* chore: fix CI dockerfiles

* chore: clean up some unused code

* chore: remove flux repl and Spec in flux query json

* chore: port flux end to end tests from 2.x

* chore: fix up goimports

* chore: remove 32 bit build support
2021-03-03 16:24:49 -05:00
Sam Arnold 17b9ea8723
feat: Add WITH KEY to show tag keys (#20793)
* fix: Change from RewriteExpr to PartitionExpr

Also remove some dead code

* feat: WITH KEY implementation

* feat: query rewriting for WITH KEY in SHOW TAG KEYS
2021-02-25 08:38:29 -05:00
davidby-influx a8b2129df5 chore: move context.Context to first argument in methods per convention 2021-02-23 12:52:44 -08:00
Sam Arnold de1a0eb2a9
feat: use count_hll for 'show series cardinality' queries (#20745)
Closes: https://github.com/influxdata/influxdb/issues/20614

Also fix nil pointer for seriesKey iterator

Fix for bug in: https://github.com/influxdata/influxdb/issues/20543

Also add a test for ingress metrics
2021-02-10 16:00:16 -05:00
Sam Arnold 903b8cd0ea
feat(query): Hyper log log operators in influxql (#20603)
* feat(query): hyper log log counting in query engine

In addition to helping with normal queries, this can improve the 'SHOW CARDINALITY'
meta-queries:

time influx -database mydb -execute 'select count_hll(sum_hll(_seriesKey)) from big'
name: big
time count_hll
---- ---------
0    200767781
influx -database mydb -execute   0.06s user 0.12s system 0% cpu 8:49.99 total
2021-02-08 08:38:14 -05:00
Sam Arnold 8a16bf0531 chore: run goimports -w ./ 2021-01-29 11:40:02 -05:00
davidby-influx df39b1e71c
fix(query): Group By queries with offset that crosses a DST boundary can fail (#20230)
* fix(query): Group By queries with offset that crosses a DST boundary can fail

Customer reported that a GROUP BY query with an offset that caused an interval
to cross a daylight savings change inserted an extra output row off by one hour.
This fix ensured that the start time for the interval of a GROUP BY operator is
correctly set before calculating the time zone offset for that date and time.

Add TestGroupByIterator_DST() in query/iterator_test.go
for regression testing of this bug.

Fixes https://github.com/influxdata/influxdb/issues/20238
2020-12-04 09:40:43 -08:00
David Norton 78a05d1119
Merge pull request #17596 from foobar/optimize-sorted-merge-iterator
improvement(query): performance improvement for sorted merge iterator
2020-06-23 12:50:10 -04:00
Ben Johnson 5263070632 feat(query): Parallelize field iterator planning 2020-06-11 08:01:14 -06:00
Ayan George 9dae8c9802
fix(query): address staticcheck warning S1020 (#18136)
Quiet staicccheck warning "should merge variable declaration with
assignment on next line (S1021)" by updating point.gen.go.tmpl and
re-generating point.gen.go.
2020-05-18 20:20:54 -04:00
Ayan George 2529799abb
refactor: Change ToLower comparisons to EqualFold (#18147)
When comparing strings in a case-insensitive way, strings.EqualFold() is
(almost?) always faster than comparing the results of strings.ToLower().

In addition, strings.EqualFold() never causes an allocation.

This patch replaces case-insensitive string comparisons that use
strings.ToLower() with a strings.EqualFold() call.
2020-05-18 19:46:59 -04:00
Ayan George 6808702703
refactor(query): reuse matchAllRegex (#18146)
matchAllRegex is a global variable containing the precompiled regex that
matches ".+".

Prior to this commit, it was used in only one place and we called its
.Copy() method.

According to the docs, .Copy() is no longer needed for safe concurrent
access:

  Deprecated: In earlier releases, when using a Regexp in multiple
  goroutines, giving each goroutine its own copy helped to avoid lock
  contention. As of Go 1.12, using Copy is no longer necessary to avoid
  lock contention. Copy may still be appropriate if the reason for its
  use is to make two copies with different Longest settings.

Since we require Go 1.13 or later now and we're not calling the
Longest() method, this patch removes the .Copy() call.

Now that we have a reusable matchAllRegex value, this patch then
replaces all instances of regexp.MustCompile(`.+`) with matchAllRegex.
This will elminate runtime regex compilations.
2020-05-18 18:45:33 -04:00
Ayan George 3b9be0145c
fix: address static check warning s1039 (#18135)
This commit quiets staticcheck's warnings about "unnecessary use of
fmt.Sprintf" and "unnecessary use of fmt.Sprint".

Prior to this commit we were wrapping simple constant strings without
any formatting verbs with fmt.Sprintf().
2020-05-18 13:55:05 -04:00
Tristan Su af8e66cd25 improvement(query): performance improvement for sorted merge iterator
Sorted merge iterator has cpu-intensive operations to sort the points
from multiple inputs. Typical queries like `SELECT * FROM m GROUP BY *`
do not behave well due to the comparison of points though in many cases
it doesn't necessarily have to use the slow path.

This patch adds a shortcut.  If each input has a single and unique
series we can just return the points input by input.
The detection of the shortcut introduces slight overhead but the gains
are significant in many slow queries.
2020-04-20 21:06:45 +08:00
Ayan George f51709f09c
refactor(http): Simplify Authorizer (#17704)
Have AuthorizerIsOpen() assert if a given authizer has an
AuthorizeUnrestricted() method and if so, call that to provide the
result of AuthorizerIsOpen().

Otherwise we check if the supplied Authorizer is nil.

This preserves the fast-path for checking tag-level (and other) tsdb
operations.

This simplifies how we handle such authorizers by handling this case in
only one place.
2020-04-15 12:59:52 +00:00
Ayan George b1ea8eff3e
refactor(query): save an indent level when checking r.prev.Nil (#17125) 2020-03-06 13:48:30 -05:00
Brandon Farmer fe517fd9ce feat(influxdb): Add proxy path to cli 2020-01-21 10:23:46 -08:00
Adam ec32b3a4bb fix(query/compile.go): time range was exceeding min/max bounds under certain conditions
added test for missing lower bound
2019-08-14 09:59:17 -04:00
Adam Perlin 7cdebbe422 fix(query): add additional unit tests for series cardinality query rewriting 2019-07-24 17:06:14 -07:00
Adam Perlin d6e34bb3e0 fix(query): make show series exact cardinality count only distinct series 2019-07-24 16:39:05 -07:00