Commit Graph

184 Commits (c77c4b3d238b858c31733c717cf58632a71739bc)

Author SHA1 Message Date
Carol (Nichols || Goulding) 35e8c52702
fix: Rename remaining test helper methods with a 2 suffix to not 2023-05-09 22:07:46 -04:00
Carol (Nichols || Goulding) e60f703e95
fix: Rename router2 to router
Including an alias and a test for continuing to support `influxdb_iox
run router2`.
2023-05-09 22:01:39 -04:00
Carol (Nichols || Goulding) 6daf5660b4
fix: Rename querier2 to querier
Just some test helpers; there never was an `influxdb_iox run querier2`
command so no alias or test needed.
2023-05-09 22:01:06 -04:00
Carol (Nichols || Goulding) dd9c5d1b13
fix: Rename compactor2 to compactor 2023-05-09 10:58:55 +02:00
Carol (Nichols || Goulding) 56916cf942
fix: Rename ingester2 to ingester 2023-05-08 12:03:05 -04:00
Christopher M. Wolff 55b35367ac
test: add test for gap fill query missing time bounds (#7747)
* test: add test for gap fill query missing time bounds

* chore: update unit test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 21:01:45 +00:00
Carol (Nichols || Goulding) b0959667d5
fix: Move topic and query pool within iox catalog (#7734)
Still insert them into the database and associate them with namespaces,
but don't ever query them back out.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 13:45:56 +00:00
Andrew Lamb 667da5eea4
chore: Add test for `dsn-file://` catalog urls (#7735)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 10:48:18 +00:00
Dom Dwyer c9cfe05f8d
revert: PR #7708
This reverts commit 61abb58933.
2023-05-02 13:51:30 +02:00
Andrew Lamb 61abb58933
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres (#7708)
* refactor: Change catalog configuration so it is entirely dsn based

* docs: Add documentation

* chore: update docs

* chore: review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 10:48:33 +00:00
dependabot[bot] bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667)
* chore(deps): Bump tokio from 1.27.0 to 1.28.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
Carol (Nichols || Goulding) 5e01261a87
fix: Change the normalization regex to new path format 2023-04-24 10:08:00 -04:00
Carol (Nichols || Goulding) a9081fc8a8
fix: Remove uses and tests for RPC write env var 2023-04-24 10:00:09 -04:00
dependabot[bot] 91ecf3e820
chore(deps): Bump regex from 1.7.3 to 1.8.0 (#7616)
* chore(deps): Bump regex from 1.7.3 to 1.8.0

Bumps [regex](https://github.com/rust-lang/regex) from 1.7.3 to 1.8.0.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/commits)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-21 14:58:46 +00:00
Marco Neumann 9eac89ebfb
fix: also include metadata in empty InfluxQL select results (#7604)
At least the schema metdata and the measurements column should be there.
2023-04-20 08:12:07 +00:00
wiedld 1d2003d385
feat(idpe-17265): cst write authorization (#7527)
* feat(idpe-17265): authorization should occur as part of the single_tenant specific mod
* authz service is accessed only through the single_tenant mod handler
* authz service is wrapped in auth mod
* move auth integration test into auth mod
* push down the authorize() call into the query params parser call, in order to access query params in the extract_token
* provide configuration error when authz or single_tenant mode are not co-presented
* update authz e2e fixtures

* feat(idpe-17265): extract tokens based upon preferred ordering in spec, and write tests to verify behavior.

* chore(idpe-17265): update naming conventions for a unifying parser

* test: make MockAuthorizer have default, and add a test_delegate_to_authz for CST

* chore: record authz duration metric, and include in delegation test.

* chore: use authz terminology instead of auth_service

* chore: more explicit naming

* Revert "chore: record authz duration metric, and include in delegation test."

This reverts commit 05c36888ca7247b6953343d759a5185098fae679.

* refactor: extract_header_token versus the else condition

* refactor: make single_tenant mod and move auth within

* chore: make unreachable explicitly panic in the build

* test: make token values be const, to be consumed when MockAuthorizer is used

* test: use locking for calls_counter in test

* fix: add base64 encoding as expected for Basic header

* fix: merge conflict resolution. The AuthorizationHeaderExtension is now under the authz::http mod, which is a required feature for router package.

* chore: run rustfmt nightly with preferred import handling, on files with modified imports

* chore: code cleanup, to have minimal code needed
2023-04-19 15:28:10 +00:00
Marco Neumann 53abef0447
feat: `SHOW TAG VALUES ... WHERE <condition>` (#7589)
Ref https://github.com/influxdata/idpe/issues/17474 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-19 08:10:56 +00:00
Marco Neumann 87ecdc5eaa
feat: InfluxQl `SHOW TAG KEYS` planner+exec (#7451)
Closes https://github.com/influxdata/idpe/issues/17363 .
2023-04-17 13:31:33 +00:00
dependabot[bot] 567ce82df2
chore(deps): Bump assert_cmd from 2.0.10 to 2.0.11 (#7553)
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.10 to 2.0.11.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.10...v2.0.11)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-14 09:42:31 +00:00
Andrew Lamb 3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` (#7516)
* chore: Update datafusion

* chore: Update for change in PhysicalSortExpr

* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
Stuart Carnie 5965cee179
feat: Update InfluxQL end-to-end test output (#7512)
* feat: Specialises test output formatting for each language

* Also fixes an error uncovered in the `write_columnar` when tag
  columns are `NULL`

Closes #7145

* chore: Run cargo hakari tasks

* chore: Add sorted output until #7513 is addressed

* chore: clippy 📋

* feat: Add `options` to `write_columnar`

* Added ability to configure border rendering, including removing
  borders. This helps avoid variable width issues with EXPLAIN output,
  which tends to vary and cause flaky test failures.

* chore: rustfmt 🧹

* chore: update expected output

* chore: clarify what "this" is

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-13 09:58:12 +00:00
Andrew Lamb 20e9c91866
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc (#7515)
* refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
Marco Neumann f97e1765d8
fix: do not loose schema for empty query responses (#7506)
Within our query tests and our CLI, we've used to print out empty
query responses as:

```text
++
++
```

This is pretty misleading. Why are there no columns?! The reason is that
while Flight provides us with schema information, we often have zero
record batches (because why would the querier send an empty batch). Now
lets fix this by creating an empty batch on the client side based on the
schema data we've received. This way, people know that there are columns
but no rows:

```text
+-------+--------+------+------+
| count | system | time | town |
+-------+--------+------+------+
+-------+--------+------+------+
```

An alternative fix would be to pass the schema in addition to
`Vec<RecordBatch>` to the formatting code, but that seemed to be more
effort.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-11 14:35:29 +00:00
Martin Hilton d2585002fe
chore(authz): Change "namespace" to "database" (#7502)
Part of the wider effort to consistently use tht term "database"
for the user-facing terminology, update the authorization system.
Whilst this system is technically user-facing, it is unlikely many
users will see it. It is however new enough that the change is
relatively little effort.
2023-04-11 11:04:51 +00:00
Martin Hilton 1a42155077
chore: end to end tests for authorization (#7441)
* chore: end-to-end tests for authorization

Add tests to validate the behaviour of the authorization machinery
in the write and query paths.

In order to facilitate this an authorizer implentation has been
added to the the test helpers that runs an authorizer gRPC service
for the use of tests. The gRPC service is started in the process
that is running the test and listens on a OS-assigned port number.
The authorization service cannot be shared between tests so a
non-shared cluster must be used when the authorizer is configured.

The influxdb_iox_client has been enhanced so that the user can
configure additional headers in the flight client, which is used
for SQL and InfluxQL queries. This uses the same interface as the
Flight SQL client has for the same job.

* chore: fix lint errors

* chore: review suggestion

Consolate the authorization tests into fewer tests in order to avoid
repeating set-up and tear-down unnecessarily.
2023-04-11 09:56:45 +00:00
dependabot[bot] 86c55918ec
chore(deps): Bump tempfile from 3.4.0 to 3.5.0 (#7376)
* chore(deps): Bump tempfile from 3.4.0 to 3.5.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:15:54 +00:00
dependabot[bot] 9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 (#7373)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
Carol (Nichols || Goulding) 621672b38f
test: Add an e2e test for write replication and fix it because it didn't work 😜 (#7344)
* test: Add an e2e test for write replication

* fix: Pass through rpc_write_replicas configuration to RpcWrite handler

---------

Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-28 09:31:16 +00:00
Stuart Carnie 08ef689d21
feat: Teach InfluxQL how to plan an aggregate query (#7230)
* feat: Display failed query

Allows a user to immediately identify the failed query.

* feat: API improvements to InfluxQL parser

* feat: Extend `SchemaProvider` trait to query for UDFs

* fix: We don't want the parser to panic on overflows

* fix: ensure `map_type` maps the timestamp data type

* feat: API to map a InfluxQL duration expression to a DataFusion interval

* chore: Copied APIs from DataFusion SQL planner

These APIs are private but useful for InfluxQL planning.

* feat: Initial aggregate query support

* feat: Add an API to fetch a field by name

* chore: Fixes to handling NULLs in aggregates

* chore: Add ability to test expected failures for InfluxQL

* chore: appease rustfmt and clippy 😬

* chore: produce same error as InfluxQL

* chore: appease clippy

* chore: Improve docs

* chore: Simplify aggregate and raw planning

* feat: Add support for GROUP BY TIME(stride, offset)

* chore: Update docs

* chore: remove redundant `is_empty` check

Co-authored-by: Christopher M. Wolff <chris.wolff@influxdata.com>

* chore: PR feedback to clarify purpose of function

* chore: The series_sort can't be empty, as `time` is always added

This was originally intended as an optimisation when executing an
aggregate query that did not group by time or tags, as it will produce
N rows, where N is the number of measurements queried.

* chore: update comment for clarity

---------

Co-authored-by: Christopher M. Wolff <chris.wolff@influxdata.com>
2023-03-23 01:13:15 +00:00
Carol (Nichols || Goulding) d8bd185fa0
test: Add compactor shard e2e tests 2023-03-20 15:52:47 -04:00
Carol (Nichols || Goulding) 01a609af45
fix: Actually use the bind addrs when running the compactor once 2023-03-20 13:21:25 -04:00
Carol (Nichols || Goulding) 528d83e989
test: Re-enable running compactor once in step tests
Putting back the test that I removed in
<https://github.com/influxdata/influxdb_iox/pull/6850>

now that they can use `run compactor2 --compaction-process-once
--compaction-process-all-partitions`

thanks to <https://github.com/influxdata/influxdb_iox/pull/6886>!
2023-03-20 13:20:43 -04:00
dependabot[bot] f0fc79ee3b
chore(deps): Bump assert_cmd from 2.0.9 to 2.0.10 (#7248)
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.9 to 2.0.10.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.9...v2.0.10)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-17 10:35:10 +00:00
dependabot[bot] 2dde0658c6
chore(deps): Bump assert_cmd from 2.0.8 to 2.0.9 (#7228)
* chore(deps): Bump assert_cmd from 2.0.8 to 2.0.9

Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.8 to 2.0.9.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.8...v2.0.9)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-16 08:52:27 +00:00
dependabot[bot] c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 (#7107)
* chore(deps): Bump tokio from 1.25.0 to 1.26.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
Carol (Nichols || Goulding) 3bf0f2779e
refactor: Move query plan normalizer to arrow_util 2023-03-01 15:44:22 -05:00
Carol (Nichols || Goulding) bbfff8699c
fix: Use the same normalization code for explain tests as e2e tests do
The regex for replacing UUIDs needed to be changed like the normalizer's
regex did, so keep them in sync by using the same code.

This might point to the normalizer needing to be moved somewhere else,
or changing these tests to be e2e?
2023-03-01 13:00:04 -05:00
Carol (Nichols || Goulding) faae5eb438 chore: Rerun cargo hakari manage-deps 2023-02-27 11:56:15 +01:00
dependabot[bot] 4cf066680e
chore(deps): Bump tempfile from 3.3.0 to 3.4.0 (#7069)
* chore(deps): Bump tempfile from 3.3.0 to 3.4.0

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 10:02:21 +00:00
dependabot[bot] 77ee4d512a
chore(deps): Bump http from 0.2.8 to 0.2.9
Bumps [http](https://github.com/hyperium/http) from 0.2.8 to 0.2.9.
- [Release notes](https://github.com/hyperium/http/releases)
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/http/compare/v0.2.8...v0.2.9)

---
updated-dependencies:
- dependency-name: http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 02:54:56 +00:00
Carol (Nichols || Goulding) b11228c72e
test: Write an e2e test of namespace soft-deletion (#7002)
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-16 09:59:14 +00:00
Carol (Nichols || Goulding) 9e27709c84
test: Add an e2e test of just the ingester's API 2023-02-13 16:36:59 -05:00
Carol (Nichols || Goulding) 7b80b03b28
test: Test with multiple ingesters, like in production 2023-02-09 17:31:28 -05:00
Carol (Nichols || Goulding) 8eace0d4d8
docs: Try to clarify what MiniCluster does in regard to compactor 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) fcb7f4c61d
fix: Remove unused "other server" minicluster code 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) b8360e7710
fix: Remove old test ServerTypes 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) c4fa0cc7ad
fix: Remove test configs for old server types 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) 2204630d90
fix: Remove test steps and helper functions only used in Kafkaful arch 2023-02-09 13:56:07 -05:00
Carol (Nichols || Goulding) 38f8e18357
fix: Only run end-to-end tests against Kafkaless 2023-02-09 13:56:06 -05:00
Carol (Nichols || Goulding) d7c59da46b
fix: Switch more tests over to the kafkaless architecture 2023-02-09 13:56:06 -05:00