Commit Graph

12029 Commits (60931631b807403db7f22a807658b9a6c582e4e8)

Author SHA1 Message Date
Phil Bracikowski 420e3d0a70 chore: Merge branch 'main' into pjb-17451-gc-s3-connection-refused
* fix conflicts
2023-04-21 11:45:57 -07:00
Phil Bracikowski f0b9a0b315 chore: respond to pr feedback
* remove dry-run catalog method
* improve info and debug messages
2023-04-21 10:59:25 -07:00
wiedld 781d6c040d
fix: process query param for token, even when header is not present. (#7619)
* Move the or_else conditional out of the Some() chain
2023-04-21 17:44:59 +00:00
dependabot[bot] 1d95e520c3
chore(deps): Bump libc from 0.2.141 to 0.2.142 (#7614)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.141 to 0.2.142.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.141...0.2.142)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-21 15:07:05 +00:00
dependabot[bot] 91ecf3e820
chore(deps): Bump regex from 1.7.3 to 1.8.0 (#7616)
* chore(deps): Bump regex from 1.7.3 to 1.8.0

Bumps [regex](https://github.com/rust-lang/regex) from 1.7.3 to 1.8.0.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/commits)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-21 14:58:46 +00:00
dependabot[bot] 8450efdbce
chore(deps): Bump rustix from 0.37.12 to 0.37.13 (#7617)
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.12 to 0.37.13.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.12...v0.37.13)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-21 14:50:38 +00:00
kodiakhq[bot] 4785ba2e33
Merge pull request #7613 from influxdata/sgc/issue/time_range_7610
fix: Rewrite time range expressions so they apply to entire result set
2023-04-21 03:55:04 +00:00
Stuart Carnie 0cc6147d6f
fix: Need to handle Regex operators
Ignoring LIKE for now as they aren't used by the InfluxQL planner
2023-04-21 13:39:34 +10:00
Stuart Carnie 7a49c3484e
feat: Documentation to describe time range behaviour 2023-04-21 09:31:54 +10:00
Stuart Carnie 79fcb9bfb9
chore: Use idiomatic initialisation
Co-authored-by: Christopher M. Wolff <chris.wolff@influxdata.com>
2023-04-21 09:25:14 +10:00
Stuart Carnie 9416fcfb88
Merge branch 'main' into sgc/issue/time_range_7610 2023-04-21 07:53:34 +10:00
Stuart Carnie b2279abb59
feat: Rewrite time range expressions to apply to entire result set 2023-04-21 07:53:19 +10:00
kodiakhq[bot] 8cce0baab1
Merge pull request #7612 from influxdata/cn/remove-partitions_with_recent_created_files
fix: Remove unused partitions_with_recent_created_files method
2023-04-20 19:09:38 +00:00
Carol (Nichols || Goulding) 0ac2494c5d
fix: Remove unused partitions_with_recent_created_files method
And PartitionParam type.
2023-04-20 14:54:03 -04:00
Chunchun Ye 7a1b0b6fc2
chore(flightsql): rename Namespace to Database in error message (#7608)
* chore(flightsql): rename Namespace to Database in error message

* chore(flightsql): rename Namespace to Database in test error msg

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-20 13:11:11 +00:00
Marco Neumann 370413b2d2
feat: `SHOW TAG KEYS ... WHERE <condition>` (#7601)
Ref https://github.com/influxdata/idpe/issues/17474 .
2023-04-20 09:16:35 +00:00
dependabot[bot] 4cb79a9b19
chore(deps): Bump clap from 4.2.3 to 4.2.4 (#7609)
Bumps [clap](https://github.com/clap-rs/clap) from 4.2.3 to 4.2.4.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.3...v4.2.4)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-20 08:24:34 +00:00
Marco Neumann 9eac89ebfb
fix: also include metadata in empty InfluxQL select results (#7604)
At least the schema metdata and the measurements column should be there.
2023-04-20 08:12:07 +00:00
wiedld d648a7cdda
Merge pull request #7594 from influxdata/chore/add-cst-mode-to-allInOne
chore: all cst mode for router to the all-in-one commands
2023-04-19 11:42:12 -07:00
wiedld a8ab9040c9
Merge branch 'main' into chore/add-cst-mode-to-allInOne 2023-04-19 11:30:54 -07:00
kodiakhq[bot] 63b51fdd50
Merge pull request #7530 from influxdata/cn/cold-vs-hot-cli
feat: Cold compaction CLI that only selects partitions and logs them
2023-04-19 16:40:50 +00:00
kodiakhq[bot] 45be11bd2c
Merge branch 'main' into cn/cold-vs-hot-cli 2023-04-19 16:33:54 +00:00
Carol (Nichols || Goulding) 3604f141dc
test: Add a unit test for logging that uses cold compaction 2023-04-19 12:33:00 -04:00
Christopher M. Wolff 8d3b3be2c4
test: add tests for gap filling with nanosecond precision (#7596) 2023-04-19 15:53:31 +00:00
wiedld c123971113 chore: all cst mode for router to the all-in-one commands
Passes the bool to the generated router config, which enables us to run in cst mdoe when we don't use the clapblock flags for the router config
2023-04-19 08:30:17 -07:00
wiedld 1d2003d385
feat(idpe-17265): cst write authorization (#7527)
* feat(idpe-17265): authorization should occur as part of the single_tenant specific mod
* authz service is accessed only through the single_tenant mod handler
* authz service is wrapped in auth mod
* move auth integration test into auth mod
* push down the authorize() call into the query params parser call, in order to access query params in the extract_token
* provide configuration error when authz or single_tenant mode are not co-presented
* update authz e2e fixtures

* feat(idpe-17265): extract tokens based upon preferred ordering in spec, and write tests to verify behavior.

* chore(idpe-17265): update naming conventions for a unifying parser

* test: make MockAuthorizer have default, and add a test_delegate_to_authz for CST

* chore: record authz duration metric, and include in delegation test.

* chore: use authz terminology instead of auth_service

* chore: more explicit naming

* Revert "chore: record authz duration metric, and include in delegation test."

This reverts commit 05c36888ca7247b6953343d759a5185098fae679.

* refactor: extract_header_token versus the else condition

* refactor: make single_tenant mod and move auth within

* chore: make unreachable explicitly panic in the build

* test: make token values be const, to be consumed when MockAuthorizer is used

* test: use locking for calls_counter in test

* fix: add base64 encoding as expected for Basic header

* fix: merge conflict resolution. The AuthorizationHeaderExtension is now under the authz::http mod, which is a required feature for router package.

* chore: run rustfmt nightly with preferred import handling, on files with modified imports

* chore: code cleanup, to have minimal code needed
2023-04-19 15:28:10 +00:00
Christopher M. Wolff 7a6862ee3a
refactor: let date_bin_gapfill allow omitted origin (#7595)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 14:56:52 +00:00
Dom 842e49b75a
Merge pull request #7603 from influxdata/dom/router-msg-size
feat: configurable internal RPC write message size
2023-04-19 15:19:40 +01:00
Dom Dwyer 2783b00907
feat(ingester): configurable RPC write message size
Provide a configuration item for the ingester2 that controls the maximum
incoming RPC message size.

Raises the maximum from the default 4MiB to a more reasonable 100MiB.
2023-04-19 14:57:53 +02:00
Dom Dwyer 03c5ea5488
feat(router): configurable RPC write message size
Provide a configuration item for the router (in RPC mode) that controls
the maximum outgoing RPC message size when communicating with an
Ingester.

Raises the maximum from the default 4MiB to 100MiB. This does not
increase exposure to memory-based DOS, as writes are size-limited by the
HTTP layer to 10MiB, preventing a user from submitting a write this
large (or larger!) across the RPC boundary.
2023-04-19 14:57:53 +02:00
Dom Dwyer cf38e3bae5
chore: use http in router authz deps
The router should be using the "http" feature - this prevents
crate-specific tests from compiling otherwise.
2023-04-19 14:57:53 +02:00
Marco Neumann ce273b6006
feat: `SHOW MEASUREMENTS ... WHERE <condition>` (#7590)
Ref https://github.com/influxdata/idpe/issues/17474 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 08:59:54 +00:00
dependabot[bot] 8930da33a4
chore(deps): Bump rustix from 0.37.11 to 0.37.12 (#7598)
Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.11 to 0.37.12.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.37.11...v0.37.12)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-19 08:43:16 +00:00
dependabot[bot] 79c2d86740
chore(deps): Bump clap from 4.2.2 to 4.2.3 (#7597)
Bumps [clap](https://github.com/clap-rs/clap) from 4.2.2 to 4.2.3.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v4.2.2...v4.2.3)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 08:33:58 +00:00
Armin Primadi dd54d8b7fe
fix: Garbage collector hangs indefinitely on shutdown (#7567)
* fix: Garbage collector hangs indefinitely on shutdown

* style(garbage_collector): conform to linter and fmt

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 08:27:38 +00:00
Marco Neumann 53abef0447
feat: `SHOW TAG VALUES ... WHERE <condition>` (#7589)
Ref https://github.com/influxdata/idpe/issues/17474 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 08:10:56 +00:00
Dom 711b69a23f
Merge pull request #7593 from influxdata/dom/coalesce-drive-completion
fix(ingester): always drive partition resolve futures to completion
2023-04-18 18:16:43 +01:00
Dom Dwyer b0cc2bda0d
fix(ingester): always complete partition fetches
In #7559 I introduced code to coalesce N queries for a given partition
down to a single query, with the other callers (async) blocking to
obtain the shared result.

This is usually fine, but it is hypothetically possible for all callers
to start the partition lookup and abort (cancelling their shared poll
future) before the partition resolves, leaving it half complete.

The problem is this half-complete future MAY be holding a connection
from the catalog connection pool (internally a semaphore). This means
this connection is "stuck", lent out to this future that isn't making
progress (and may never make progress!). Given sufficient occurrences,
it's possible this could lead to starving the system of catalog
connections.

This fix is simple - spawn the resolve future into the runtime so it
completes irrespective of any caller polling it (they only poll for the
result, not to drive it forward).

NOTE: this isn't practically possible, because writes that traverse the
WAL DmlSink currently become uncancellable - this may not always be the
case however, so this underlying foot gun should be patched over!
2023-04-18 17:09:02 +02:00
Dom Dwyer 5e42c5be61
perf(ingester): avoid optimistic RMW operation
Changes the CoalescePartitionResolver state cleanup path to avoid an
optimistic RMW operation against the atomic "has been cleaned up" flag.
This gives callers who are going to fail the opportunity to do so
quickly and cheaply, where all but one caller will fail.
2023-04-18 16:35:34 +02:00
Dom Dwyer 23a6dfedaa
test(ingester): use constants in coalesce test
Clean up the CoalescePartitionResolver tests by using constants across
tests - I should have done this before but was rushing around!
2023-04-18 16:34:13 +02:00
Fraser Savage 851eda92a3
Merge branch 'main' into savage/additive-namespace-schema-caching 2023-04-18 14:25:52 +01:00
Fraser Savage ae99a8725f
feat(router): Additively merge tables in NamespaceCache
This commit adds additive merge behaviour for tables missing
from the new NamespaceCache entry, as well as moving calculation
of change stat metrics down to the in-memory implementation.

The metrics no longer distinguish between insert and update
caches OPs as a result of the change to the `put_schema()` interface.
2023-04-18 14:18:48 +01:00
kodiakhq[bot] d68597b977
Merge pull request #7591 from influxdata/dom/create-ns-encoding
fix: consistent NamespaceName validation
2023-04-18 13:15:07 +00:00
Dom Dwyer c4a802cea8
fix: apply namespace name validation on create
Prior to this commit, the gRPC NamespaceService would allow namespaces
to be created with invalid names due to a missing call to NamespaceName
that performs validation.
2023-04-18 14:50:51 +02:00
Dom Dwyer 54d4ca8729
test: invalid namespace name creation
Adds tests that attempt to create namespaces via the gRPC
NamespaceService with a variety of invalid names.

NOTE: this test currently fails!
2023-04-18 14:48:44 +02:00
Dom Dwyer 92b1055e41
refactor: error message should not end in .
Change the "bad character" namespace name error to not include a
trailing "."
2023-04-18 14:46:22 +02:00
Marco Neumann d7dc305972
feat: allow overwriting DataFusion's default config (#7586)
This is helpful to test changes in our defaults but also for testing.

Required for https://github.com/influxdata/idpe/issues/17474 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-18 11:28:45 +00:00
kodiakhq[bot] b6c2b00353
Merge pull request #7588 from influxdata/dom/router-deps
refactor(router): remove unused deps
2023-04-18 11:00:09 +00:00
kodiakhq[bot] f8f57ceeec
Merge branch 'main' into dom/router-deps 2023-04-18 10:53:43 +00:00
kodiakhq[bot] 393bcf245a
Merge pull request #7587 from influxdata/dom/tidy-authz
refactor: move HTTP authz helpers into authz
2023-04-18 10:53:27 +00:00