Commit Graph

1139 Commits (7cb30480350b26af8596eb62192a19c3afbb6e0d)

Author SHA1 Message Date
Stuart Carnie ad3e6aadea
Merge branch 'main' into sgc/issue/last_7538
# Conflicts:
#	iox_query_influxql/src/plan/planner.rs
2023-04-18 06:18:02 +10:00
kodiakhq[bot] da96239605
Merge branch 'main' into cn/delete-tombstones 2023-04-17 13:59:49 +00:00
Marco Neumann 87ecdc5eaa
feat: InfluxQl `SHOW TAG KEYS` planner+exec (#7451)
Closes https://github.com/influxdata/idpe/issues/17363 .
2023-04-17 13:31:33 +00:00
Marco Neumann e7511c0f33
feat: basic InfluxQL `SHOW MEASUREMENTS` (#7504)
No WHERE-clauses hat result in actual data checks yet, just pure metdata
queries.

Ref https://github.com/influxdata/idpe/issues/17358 .
2023-04-17 11:28:01 +00:00
Marco Neumann bda9c07c0b
feat: basic InfluxQL `SHOW TAG KEYS` support (#7526)
- No `ON` clause
- No `WHERE` clause
- No time restriction yet
- No `FROM <db>.<retention>`

Ref https://github.com/influxdata/idpe/issues/17360 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-17 10:01:41 +00:00
Stuart Carnie 3f4e38aa78
refactor: tidy up the implementation of the function_to_df_expr function
* Also fixes an issue that `FILL(<number>)` would fail when using the
  `struct_selector_<name>` functions
2023-04-16 10:04:53 +10:00
Stuart Carnie ccfd334834
chore: correct typo 🔨 2023-04-16 08:32:01 +10:00
Stuart Carnie acd6cff631
chore: validate single-selector with tags or fields is not implemented 2023-04-16 08:21:19 +10:00
Stuart Carnie 69d75745cc
feat: add limited `last`, `first`, `min` and `max` selector functions
Returns a `NotImplemented` error when attempting to execute a
selector query, which projects a single selector function and additional
tags or fields until #7533 is implemented.

Introduced `error` module to simplify error handling and ensure
consistency of error messages.
2023-04-16 07:59:28 +10:00
Carol (Nichols || Goulding) 5e6dbec909
fix: Remove tombstones as they aren't functional currently 2023-04-14 13:36:08 -04:00
Andrew Lamb f46d06d56f
chore: Update DataFusion + arrow ecosystem to 37 (#7544)
* chore: Update datafusion and arrow/parquet to 37, tonic to 0.9.1

* refactor: Update for FieldRef and other API changes

* fix: Update field size calculation

* fix: Use `NullBuffer` directly

* fix: remove outdated comment

* chore: Update test for tonic

* chore: Run cargo hakari tasks

* chore: cargo update

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-14 12:43:01 +00:00
dependabot[bot] 567ce82df2
chore(deps): Bump assert_cmd from 2.0.10 to 2.0.11 (#7553)
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.10 to 2.0.11.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.10...v2.0.11)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-14 09:42:31 +00:00
dependabot[bot] 072d685f94
chore(deps): Bump predicates from 3.0.2 to 3.0.3 (#7552)
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v3.0.2...v3.0.3)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-14 09:25:19 +00:00
Chunchun Ye 69da3c2495
feat(flightsql): Support `GetCrossReference` metadata endpoint with an empty RecordBatch (#7548)
* feat: support CommandGetCrossReference metadata endpoint with tests

* chore: create two tables in the test for GetCrossReference endpoint

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 19:53:37 +00:00
Chunchun Ye 5182ab0037
chore: add more test and logic to grpc database header names (#7529)
* chore: support returning the database name if all the keys refer to the same database

* test: add test cases to check for same, different, and no database in request header

* chore: lint

* chore: more lint

* refactor: replace empty string with None for database_name

* refactor: simplify logic for NoFlightSQLDatabase error

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 18:28:29 +00:00
Chunchun Ye 8bf47df621
feat(flightsql): Support `GetImportedKeys` metadata endpoint with an empty RecordBatch (#7546)
* feat: support CommandGetImportedKeys metadata endpoint with tests

* chore: remove comments that is no longer valid

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 18:06:47 +00:00
kodiakhq[bot] 0020660b53
Merge branch 'main' into cn/remove-old-querier 2023-04-13 13:59:52 +00:00
Chunchun Ye e86e8de1a9
feat: support CommandGetExportedKeys metadata endpoint with tests (#7532)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 13:27:09 +00:00
Andrew Lamb 3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` (#7516)
* chore: Update datafusion

* chore: Update for change in PhysicalSortExpr

* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
dependabot[bot] e811a69a1e
chore(deps): Bump serde_json from 1.0.95 to 1.0.96 (#7535)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.95 to 1.0.96.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.95...v1.0.96)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 10:07:32 +00:00
Stuart Carnie 5965cee179
feat: Update InfluxQL end-to-end test output (#7512)
* feat: Specialises test output formatting for each language

* Also fixes an error uncovered in the `write_columnar` when tag
  columns are `NULL`

Closes #7145

* chore: Run cargo hakari tasks

* chore: Add sorted output until #7513 is addressed

* chore: clippy 📋

* feat: Add `options` to `write_columnar`

* Added ability to configure border rendering, including removing
  borders. This helps avoid variable width issues with EXPLAIN output,
  which tends to vary and cause flaky test failures.

* chore: rustfmt 🧹

* chore: update expected output

* chore: clarify what "this" is

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-13 09:58:12 +00:00
Christopher M. Wolff cbd747db44
feat: update gap fill planner rule to use `interpolate` (#7494)
* feat: add INTERPOLATE fn and update planner gap-fill planner rule

* test: add an end-to-end test for interpolate()
2023-04-12 21:51:44 +00:00
Carol (Nichols || Goulding) acf857816e
fix: Remove old querier 2023-04-12 13:18:23 -04:00
Chunchun Ye fb6b3f66da
feat(flightsql): support `database`, `bucket`, and `bucket-name` as grpc header names (#7511)
* feat: support `database`, `bucket`, and `bucket-name` as grpc header names

* chore: lint

* chore: update doc to accept `database`, `bucket`, and `bucket-name` as parameter names

* chore: update doc to only show `database` as the parameter name

* refactor: consolidate header names into a const vec and update comments on database

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:58:44 +00:00
Andrew Lamb 20e9c91866
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc (#7515)
* refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
Marco Neumann c7197a289e
fix: gRPC error code for invalid InfluQL (#7519)
This was "internal". The mapping works like this: we take the
`DataFusionError` and call `find_root` which should traverse the
`External(...)` chain (even through Arrow) to find the last error that
is not within the Arrow/DataFusion land. This is then mapped by us.

`DataFusionError::External(...)` is no further inspected and mapped
straight to "internal". I think this if fine because in the end we're
mostly dealing w/ DataFusion stuff anyways.

I've slightly changed the error mapping in the planner to emit
`DataFusionError::Plan(...)` instead which we map to "invalid argument".
I think this is way better for the user.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:00:38 +00:00
Dom 13572e36fa
Merge branch 'main' into dom/remove-explicit-replica 2023-04-12 16:29:18 +01:00
Carol (Nichols || Goulding) d025362ce0
fix: Remove old router 2023-04-12 10:15:48 -04:00
Dom Dwyer ed06d8e760
chore: remove ingest_replica
The replication implementation uses a different design - the
ingest_replica is no longer needed.
2023-04-12 16:06:31 +02:00
Marco Neumann e7fcb25904
feat: `SHOW FIELD KEYS` InfluxQL plan+exec (#7405)
Closes https://github.com/influxdata/idpe/issues/17362
2023-04-12 09:09:29 +00:00
Marco Neumann f97e1765d8
fix: do not loose schema for empty query responses (#7506)
Within our query tests and our CLI, we've used to print out empty
query responses as:

```text
++
++
```

This is pretty misleading. Why are there no columns?! The reason is that
while Flight provides us with schema information, we often have zero
record batches (because why would the querier send an empty batch). Now
lets fix this by creating an empty batch on the client side based on the
schema data we've received. This way, people know that there are columns
but no rows:

```text
+-------+--------+------+------+
| count | system | time | town |
+-------+--------+------+------+
+-------+--------+------+------+
```

An alternative fix would be to pass the schema in addition to
`Vec<RecordBatch>` to the formatting code, but that seemed to be more
effort.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-11 14:35:29 +00:00
Chunchun Ye b131895fc2
feat(flightsql): Support `GetPrimaryKeys` metadata endpoint with an empty RecordBatch (#7496)
* feat: add CommandGetPrimaryKeys metadata endpoint and tests

* chore: update schema for the returned record batch

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-11 14:13:44 +00:00
Dom 78f2b3a36a
Merge branch 'main' into dom/req-mode-parsing 2023-04-11 11:00:22 +01:00
Martin Hilton 1a42155077
chore: end to end tests for authorization (#7441)
* chore: end-to-end tests for authorization

Add tests to validate the behaviour of the authorization machinery
in the write and query paths.

In order to facilitate this an authorizer implentation has been
added to the the test helpers that runs an authorizer gRPC service
for the use of tests. The gRPC service is started in the process
that is running the test and listens on a OS-assigned port number.
The authorization service cannot be shared between tests so a
non-shared cluster must be used when the authorizer is configured.

The influxdb_iox_client has been enhanced so that the user can
configure additional headers in the flight client, which is used
for SQL and InfluxQL queries. This uses the same interface as the
Flight SQL client has for the same job.

* chore: fix lint errors

* chore: review suggestion

Consolate the authorization tests into fewer tests in order to avoid
repeating set-up and tear-down unnecessarily.
2023-04-11 09:56:45 +00:00
Dom Dwyer 7fed2ba456 feat(router): single tenancy operational mode
Adds a single-tenant mode (CST) to the IOx routers.

Single-tenancy mode differs in two main ways:

    * V1 write endpoint is partially supported
    * V2 write endpoint ignores "org" parameter

The "normal" mode is "multi tenant" which is the default operational
mode, and all existing behaviour remains unchanged. Single tenant mode
can be enabled by specifying INFLUXDB_IOX_SINGLE_TENANCY=true.

Request parsing is delegated to two implementations of the
WriteParamExtractor trait, one each for CST and MT - the logic of each
"mode" is defined within these files and all other functionality is
common between the two.

This commit also renames some of the error types for clarity
(NoSpecified -> NoOrgBucketSpecified, other NotSpecified ->
NoQueryParams, etc).

Note: single tenant code requires testing
2023-04-10 12:59:20 -07:00
kodiakhq[bot] 3a8b9d9ae2
Merge branch 'main' into cn/remove-old-ingester 2023-04-10 13:37:01 +00:00
Stuart Carnie b629023f45
feat: Add `FILL(linear)` support (#7466)
* feat: Add FILL(linear) support

Closes #6916

* chore: Remove old test output

* chore: Fix `FILL(linear)` test
2023-04-08 21:34:30 +00:00
Stuart Carnie b9479a2b3c
refactor: `limit` function supports multiple measurements and tag sets (#7464)
* fix: Add sort operator after window aggregate operator

Closes #7460

* fix: Refactor `LIMIT` and `OFFSET` implementation

These changes should allow the `limit` function to be used
generically with any plan following the same conventions.

* chore: No need to reorder this

* chore: Add documentation to the `limit` function
2023-04-08 21:22:54 +00:00
Carol (Nichols || Goulding) 6c2ce01f1e
fix: Remove old ingester and ioxd_ingester 2023-04-07 11:06:37 -04:00
Carol (Nichols || Goulding) 004a264d93
test: Remove a moot test for removed compactor command 2023-04-06 16:58:04 -04:00
Carol (Nichols || Goulding) a5769a461b
fix: Remove old compactor and ioxd_compactor 2023-04-06 16:49:52 -04:00
Stuart Carnie 9f5fec42b8
feat: Teach InfluxQL how to plan `LIMIT` and `OFFSET` with `GROUP BY` (#7447)
* feat: Support LIMIT and OFFSET with GROUP BY

* fix: Compile error

* chore: Improve function name and comment

* chore: rustfmt

* chore: fix clippy warnings

Allowing the too-many-arguments warning for project_select,
as it will require some refactoring after this PR has already
been reviewed. It may be refactored in the future when subqueries are
implemented
2023-04-05 23:01:56 +00:00
Fraser Savage 44478348c7
refactor(cli): Simplify namespace update-limit command code 2023-04-05 15:25:37 +01:00
Fraser Savage 9fcaee9ac9
Merge branch 'main' into savage/6824-update-namespace-service-limit-api-call 2023-04-05 14:52:44 +01:00
Christopher M. Wolff d57a4f8947
refactor: make null-as-missing default behavior for LOCF (#7443)
* refactor: make null-as-missing default behavior for LOCF

* test: update InfluxQL test

---------

Co-authored-by: Christopher Wolff <cwolff@athena.tail244ec.ts.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 18:03:09 +00:00
dependabot[bot] 66982f988b
chore(deps): Bump object_store from 0.5.5 to 0.5.6 (#7433)
Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.5 to 0.5.6.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md)
- [Commits](https://github.com/apache/arrow-rs/commits)

---
updated-dependencies:
- dependency-name: object_store
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 08:43:34 +00:00
Stuart Carnie a1b29b3ebb
fix: Incorrect results when planning aggregate expressions (#7431)
* fix: Incorrect results when mixing non-existent fields in aggregates

* chore: Improve comments around aggregates
2023-04-04 00:19:50 +00:00
Marco Neumann f04962d630
feat: new query planning (#7250)
Closes #6098.
2023-04-03 10:31:03 +00:00
Carol (Nichols || Goulding) 5efc59d08a
docs: Fix out of date docs for test ChunkStage 2023-03-31 16:00:53 -04:00
Carol (Nichols || Goulding) 89d5cb7fb3
feat: Clarify test setup logging 2023-03-31 16:00:53 -04:00