Commit Graph

1206 Commits (68117aee0fd3f080236c7fe47ae541e8b831c255)

Author SHA1 Message Date
Carol (Nichols || Goulding) d8b56888a7
feat: Add a command alias so that 'influxdb_iox run ingester2' runs ingester 2023-05-08 12:03:05 -04:00
Carol (Nichols || Goulding) 56916cf942
fix: Rename ingester2 to ingester 2023-05-08 12:03:05 -04:00
Andrew Lamb be6bcdef45
fix: do not panic on unimplemented (#7765) 2023-05-08 15:13:54 +00:00
Andrew Lamb 2860d87fe1
chore: Update DataFusion (#7756)
* chore: Update DataFusion pin

* chore: Update explain plans

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-05 18:58:18 +00:00
dependabot[bot] da6e123e9d
chore(deps): Bump serde from 1.0.160 to 1.0.162 (#7751)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.160 to 1.0.162.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.160...1.0.162)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-05 13:29:50 +00:00
Stuart Carnie f8b1d37d5a
fix: Consistent alias names with InfluxQL OG
Fixes #7750
2023-05-05 10:11:37 +10:00
Christopher M. Wolff 55b35367ac
test: add test for gap fill query missing time bounds (#7747)
* test: add test for gap fill query missing time bounds

* chore: update unit test

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 21:01:45 +00:00
Chunchun Ye a98952bff6
feat(flightsql): Support `CommandGetXdbcTypeInfo` metadata endpoint with actual data type info (#7696)
* chore: add expected xdbc type info value to jdbc test

* chore: add query skeleton to kick off plan_get_xdbc_type_info()`

* chore: implement a minimun version query for type info

* chore: rewrite `plan_get_xdbc_type_info` to use a static recrod batch

* chore: construct create_params as a string list

* chore: add create_params column in e2e test result

* chore: re-define create_params list items to be non-nullable

* chore: remove comment

* chore: refactor TYPE_INFO_RECORD_BATCH using XdbcTypeInfo struct and rewrite metadata for character types

chore: lint

chore: lint doc

chore: lint doc use automatic link

* chore: add unimplemented error msg

* chore: add `INTEGER`, `FLOAT`, `TIMESTAMP`, `INTERVAL` and  remove `CHAR`, `TEXT`, `STRING`

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 17:22:57 +00:00
Carol (Nichols || Goulding) b0959667d5
fix: Move topic and query pool within iox catalog (#7734)
Still insert them into the database and associate them with namespaces,
but don't ever query them back out.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 13:45:56 +00:00
Andrew Lamb 667da5eea4
chore: Add test for `dsn-file://` catalog urls (#7735)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-04 10:48:18 +00:00
Stuart Carnie 60931631b8
Merge branch 'main' into sgc/issue/subquery_6891_03 2023-05-04 08:58:05 +10:00
Nga Tran 2d601bf211
test: num files to read exceed the max_parquet_fanout and all sorted files are resorted (#7737)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-03 22:29:26 +00:00
Stuart Carnie 54d50bb6cf
Merge branch 'main' into sgc/issue/subquery_6891_03 2023-05-04 07:07:02 +10:00
Joe-Blount 6de18b6544
chore: conditionally parse shard_id from HOSTNAME (#7733)
* chore: conditionally parse shard_id from HOSTNAME

* chore: remove HOSTNAME env from test case relying on it not being there.
2023-05-03 20:09:00 +00:00
Christopher M. Wolff 05688799c4
fix: handle aliases in gapfill aggregate columns (#7725)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-03 15:20:14 +00:00
Marco Neumann 231e0f48ab
test: add test for InfluxQL md queries w/ `FROM ""` (#7728)
See https://github.com/influxdata/idpe/issues/17559 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-03 12:46:51 +00:00
Stuart Carnie 72397f63e5
Merge branch 'main' into sgc/issue/subquery_6891_03 2023-05-03 12:06:18 +10:00
Stuart Carnie 43baecbb1a
chore: handle aliased time column in sort expression 2023-05-03 12:04:24 +10:00
Stuart Carnie 19ea80390a
chore: ignore time column when processing SELECT 2023-05-03 11:04:54 +10:00
Andrew Lamb 2b1f8b56e2
chore: Update DataFusion (#7719)
* chore: Update DataFusion

* chore: update for API change

* chore: update some tests

* fix: Update plans in optimizer

* chore: Update plans

* chore: Update error messages

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-05-02 17:55:04 +00:00
Andrew Lamb 530ee94558
fix: use correct sort key in projection_pushdown (#7718)
* fix: use correct sort key in projection_pushdown

* fix: tabs in docs

* refactor: Use Serde to format test results
2023-05-02 16:50:04 +00:00
Utkarsh Gupta 18b02a03cf
chore: move to the smaller, cargo-team maintained `home` crate (#7716)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 16:10:22 +00:00
Dom Dwyer c9cfe05f8d
revert: PR #7708
This reverts commit 61abb58933.
2023-05-02 13:51:30 +02:00
Andrew Lamb 61abb58933
refactor: Change catalog configuration so it is entirely dsn based / support end to end testing without postgres (#7708)
* refactor: Change catalog configuration so it is entirely dsn based

* docs: Add documentation

* chore: update docs

* chore: review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-02 10:48:33 +00:00
Christopher M. Wolff 493b26831d
fix: make influx RPC interface break up series into multiple frames (#7691)
* fix: make influx RPC interface break up series into multiple frames

* refactor: code review feedback

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-01 20:18:05 +00:00
Andrew Lamb 7a77ba2e48
fix: Remove `remote store pull` command (#7688)
* fix: Remove  `remote store pull` command

* fix: fmt

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-01 15:03:27 +00:00
Andrew Lamb 2217799256
feat: Add --partition_filter to `remote get table` command (#7685)
* feat: Add --partition_filter to `remote get table` command

* fix: update command description

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-01 11:15:13 +00:00
Joe-Blount 49e19dace5
chore: renaming compaction job concurrency to df concurrency (#7686)
This PR renames the CLI option and variables throughout the compactor to be `df_concurrency` (and similar) instead of `job_concurrency` (and similar).

From the perspective of the compactor, the most noteworthy consequence of this limit is on concurrency within Data Fusion.  This is the limit on the number of DataFusion jobs the compactor may start concurrently.
2023-04-27 18:43:39 +00:00
Dom Dwyer c76129a7e8
refactor: fix lint failures 2023-04-27 13:19:06 +02:00
Martin Hilton f87f4c1869
fix: racy JDBC tests (#7678)
* fix: racing JDBC tests

The JDBC tests have been flaky since adding additional tests. Use
the makefile to build the client to avoid the clients racing.

* chore: pre-download JDBC drive in integration test

* fix: remove stray lockfile
2023-04-27 09:07:06 +00:00
Carol (Nichols || Goulding) 038f8e9ce0
fix: Move shard concepts into only the catalog
This still inserts the shard id into the database, always set to the
TRANSITION_SHARD_ID, but never reads it back out again.
2023-04-26 11:42:32 -04:00
Martin Hilton 4b24c988ad
feat(service_grpc_flight): JDBC compatible Handshake (#7660)
* refactor(authz): move extract_header_token into authz

Move the extract_header_token method into the authz package so that
it can be shared by the query path. The method is renamed to reflect
the fact that it can now also extract a token from gRPC metadata.

The extract_token function is now a little more generic to allow
it to be used with HTTP header values and gRPC metadata values.

* feat(service_grpc_flight): JDBC compatible Handshake

While testing some JDBC based clients we found that some, Tableau
in this case,  cannot be configured with authoriztion tokens. In
these cases we need to be able to support username/password. The
approach taken is to ignore the username and make the token the
password. This is the same approach being taken throughout the
product.

To facilitate this the Flight RPC Handshake command has been extended
to look for Basic authorization credentials and respond with the
appropriate Bearer authorization header.

While adding end-to-end tests the subprocess commands were causing
a deadlock. These have been changed to using the tonic::process
module.

There are also some small changes to the JDBC test application where
the hardcoded values were clashing with the authorization parameters.

* fix: lint

* chore: apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: review suggestion

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 13:52:49 +00:00
dependabot[bot] bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667)
* chore(deps): Bump tokio from 1.27.0 to 1.28.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
Andrew Lamb 3f69f5eea6
fix: Fix `The data type inlist should be same` error (#7645)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 10:19:03 +00:00
dependabot[bot] 0b9240cbbe
chore(deps): Bump tokio-util from 0.7.7 to 0.7.8 (#7665)
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.7 to 0.7.8.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.7...tokio-util-0.7.8)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:24:39 +00:00
Chunchun Ye eca86a0f06
feat(flightsql): Support `CommandGetXdbcTypeInfo` metadata endpoint with an empty RecordBatch (#7576)
* feat: WIP support CommandGetXdbcTypeInfo metadata endpoint with tests

* chore: update test case and add jdbc test

* chore: uncomment jdbc getCoumns test

* chore: lint

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 21:06:27 +00:00
Nga Tran b74a81e792
test: add tests on month and year date_bin (#7648)
* test: add tests on month and year date_bin

* fix: add IOX_COMPARE: uuid to get deterministics name for output parquet_file in the explain

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 10:16:21 +00:00
wiedld 148c103c70
refactor: push authz into specific configs (#7607)
* refactor: move authz-addr flag into router-specific config

* refactor: move authz-addr flag into querier-specific config

* refactor: remove global AuthzConfig which is now redundant with the pushdown to individual configs. Keep constant the env vars used universally.

* chore: make errors lowercase, and use the required bool for the authz-addr flag

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 00:14:28 +00:00
Christopher M. Wolff 2d592a0295
feat: teach parser to allow arithmetic functions (#7620)
* feat: teach parser to allow arithmetic functions

* refactor: remove unnecessary lowercasing of fn names

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-24 17:09:22 +00:00
Carol (Nichols || Goulding) 8f17d217fb
fix: Move an error condition earlier, before we try to do catalog work
If the command is going to fail anyway, may as well stop earlier
2023-04-24 10:08:00 -04:00
Carol (Nichols || Goulding) 8d4c2bfabb
fix: Only ever create the transition shard in the in-memory catalog
Tests that use the in-memory catalog are creating different shards that
then creates old-style Parquet file paths, but in production, everything
uses the transition shard now. To make the tests more like production,
only ever create and use the transition shard, and stop checking for
different shard IDs.
2023-04-24 10:08:00 -04:00
Carol (Nichols || Goulding) 5e01261a87
fix: Change the normalization regex to new path format 2023-04-24 10:08:00 -04:00
Carol (Nichols || Goulding) a9081fc8a8
fix: Remove uses and tests for RPC write env var 2023-04-24 10:00:09 -04:00
Marco Neumann 9730c52c8a
fix: InfluxQL output sorting (#7633)
* test: remove #7513 workarounds from `issue_6112.influxql`

* fix: `SHOW TAG VALUES` output sorting

* fix: `SHOW MEASUREMENTS` output sorting

* fix: `SHOW TAG KEYS` output sorting
2023-04-24 13:37:25 +00:00
Marco Neumann f411f5d6d5
chore: update DF to `181e5ccf2816ccaa05d8aaef0b375d4b7bbceece` (#7630)
I need the following two PRs:

- https://github.com/apache/arrow-datafusion/pull/6045
- https://github.com/apache/arrow-datafusion/pull/6085

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-24 09:46:54 +00:00
Stuart Carnie 0cc6147d6f
fix: Need to handle Regex operators
Ignoring LIKE for now as they aren't used by the InfluxQL planner
2023-04-21 13:39:34 +10:00
Stuart Carnie 9416fcfb88
Merge branch 'main' into sgc/issue/time_range_7610 2023-04-21 07:53:34 +10:00
Stuart Carnie b2279abb59
feat: Rewrite time range expressions to apply to entire result set 2023-04-21 07:53:19 +10:00
Chunchun Ye 7a1b0b6fc2
chore(flightsql): rename Namespace to Database in error message (#7608)
* chore(flightsql): rename Namespace to Database in error message

* chore(flightsql): rename Namespace to Database in test error msg

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-20 13:11:11 +00:00
Marco Neumann 370413b2d2
feat: `SHOW TAG KEYS ... WHERE <condition>` (#7601)
Ref https://github.com/influxdata/idpe/issues/17474 .
2023-04-20 09:16:35 +00:00