Commit Graph

12215 Commits (c9cd1fdc44ea03920ed986959c9894a2b6d57ca9)

Author SHA1 Message Date
Stuart Carnie 44ac3ca960
chore: Better not to clone 2023-04-27 08:06:06 +10:00
Stuart Carnie 19c5aa9ac8
feat: Add API to find the field type by a name 2023-04-27 08:05:42 +10:00
Stuart Carnie c89250ff20
feat: Add `take` and `replace` API to allow ownership transfer
This can reduce allocations, as it allows us to mutate the inner vector
2023-04-27 08:05:09 +10:00
Stuart Carnie a133c56a9c
feat: Evaluate data types of binary expressions (#7670)
* feat: Evaluate data types of binary expressions

This is necessary to ensure column data types of projections of a
subquery are accurately determined.

* chore: rustfmt 🧹
2023-04-26 21:05:25 +00:00
kodiakhq[bot] 92f1016e13
Merge pull request #7640 from influxdata/cn/hide-shard
fix: Move shard concepts into only the catalog
2023-04-26 16:23:15 +00:00
Carol (Nichols || Goulding) 721bb2661e
fix: Remove ShardService that is no longer used 2023-04-26 11:42:32 -04:00
Carol (Nichols || Goulding) 038f8e9ce0
fix: Move shard concepts into only the catalog
This still inserts the shard id into the database, always set to the
TRANSITION_SHARD_ID, but never reads it back out again.
2023-04-26 11:42:32 -04:00
Andrew Lamb 3263934db1
chore: Run end to end integration tests as part of a separate circle CI Job (#7658)
* chore: Run end to end integration tests as part of a separate circle CI Job

* fix: but

* feat: update job name

* chore: Update resources

---------

Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 15:22:13 +00:00
Dom 0a32daf922
Merge pull request #7672 from influxdata/dom/proptest-cache
test: property test namespace schema merging
2023-04-26 15:18:45 +01:00
Dom be3256d1a7
Merge branch 'main' into dom/proptest-cache 2023-04-26 14:55:58 +01:00
Martin Hilton 4b24c988ad
feat(service_grpc_flight): JDBC compatible Handshake (#7660)
* refactor(authz): move extract_header_token into authz

Move the extract_header_token method into the authz package so that
it can be shared by the query path. The method is renamed to reflect
the fact that it can now also extract a token from gRPC metadata.

The extract_token function is now a little more generic to allow
it to be used with HTTP header values and gRPC metadata values.

* feat(service_grpc_flight): JDBC compatible Handshake

While testing some JDBC based clients we found that some, Tableau
in this case,  cannot be configured with authoriztion tokens. In
these cases we need to be able to support username/password. The
approach taken is to ignore the username and make the token the
password. This is the same approach being taken throughout the
product.

To facilitate this the Flight RPC Handshake command has been extended
to look for Basic authorization credentials and respond with the
appropriate Bearer authorization header.

While adding end-to-end tests the subprocess commands were causing
a deadlock. These have been changed to using the tonic::process
module.

There are also some small changes to the JDBC test application where
the hardcoded values were clashing with the authorization parameters.

* fix: lint

* chore: apply suggestions from code review

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

* chore: review suggestion

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 13:52:49 +00:00
Dom Dwyer 9cc58eb4e4
test: property test namespace schema merging
This commit adds a randomised property test, that compares the results
of the new namespace cache schema merging (#7555) with a known-good
stdlib HashSet union (the cache implementation is effectively a more
specialised set union operation).

This property test also validates the "last writer wins" semantics for
other, non-schema data within the namespace.

Additionally the ChangeSet values returned over a pair of updates are
asserted to reflect the actual values added to the cache (but not each
call individually) to ensure accurate metrics are reported.
2023-04-26 15:45:54 +02:00
kodiakhq[bot] 94d308114c
Merge pull request #7555 from influxdata/savage/additive-namespace-schema-caching
feat(router): Retain tables and columns not present in new schema placed in `NamespaceCache`
2023-04-26 13:33:47 +00:00
Fraser Savage 8e1372c4ed
Merge branch 'main' into savage/additive-namespace-schema-caching 2023-04-26 14:26:05 +01:00
kodiakhq[bot] 8c23b06abd
Merge pull request #7671 from influxdata/savage/update-tokio-stream
chore: Bump tokio-stream as 0.1.13 was yanked
2023-04-26 13:25:30 +00:00
Fraser Savage 2ad126c0eb
chore: Bump tokio-stream as 0.1.13 was yanked 2023-04-26 14:17:55 +01:00
Fraser Savage 66068f5026
Merge branch 'main' into savage/additive-namespace-schema-caching 2023-04-26 14:10:40 +01:00
Fraser Savage ffe4747cf2
fix(router): Fix new_columns calculation for namespace cache table merges
This commit adds logic to ensure that all pre-existing columns are
counted when no merge takes place and a test covering that.
2023-04-26 14:00:10 +01:00
dependabot[bot] bdf7f316d7
chore(deps): Bump tokio from 1.27.0 to 1.28.0 (#7667)
* chore(deps): Bump tokio from 1.27.0 to 1.28.0

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.27.0...tokio-1.28.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:53:26 +00:00
Fraser Savage d9111e2a1a
Merge branch 'main' into savage/additive-namespace-schema-caching 2023-04-26 12:30:52 +01:00
Fraser Savage 2921a79ac3
refactor(router): Use sum to count new_columns instead of fold
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 12:21:45 +01:00
Fraser Savage 41ee990d68
fix(router): Re-introduce cache put metric insert/update attribute 2023-04-26 12:04:43 +01:00
Fraser Savage c837a6e8dc
docs(router): Explicitly document use of get() and insert() for schema merge
Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-26 11:55:55 +01:00
Andrew Lamb 3f69f5eea6
fix: Fix `The data type inlist should be same` error (#7645)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 10:19:03 +00:00
dependabot[bot] 09d6b4ae50
chore(deps): Bump tokio-stream from 0.1.12 to 0.1.13 (#7666)
Bumps [tokio-stream](https://github.com/tokio-rs/tokio) from 0.1.12 to 0.1.13.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.12...tokio-stream-0.1.13)

---
updated-dependencies:
- dependency-name: tokio-stream
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:31:04 +00:00
dependabot[bot] 0b9240cbbe
chore(deps): Bump tokio-util from 0.7.7 to 0.7.8 (#7665)
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.7 to 0.7.8.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.7...tokio-util-0.7.8)

---
updated-dependencies:
- dependency-name: tokio-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-26 09:24:39 +00:00
Dom 108e72e3a0
Merge pull request #7668 from influxdata/dependabot/cargo/tracing-0.1.38
chore(deps): Bump tracing from 0.1.37 to 0.1.38
2023-04-26 10:17:15 +01:00
CircleCI[bot] d72179fbfc chore: Run cargo hakari tasks 2023-04-26 02:03:57 +00:00
dependabot[bot] d36c9dced5
chore(deps): Bump tracing from 0.1.37 to 0.1.38
Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.37 to 0.1.38.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.37...tracing-0.1.38)

---
updated-dependencies:
- dependency-name: tracing
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 02:02:42 +00:00
Chunchun Ye eca86a0f06
feat(flightsql): Support `CommandGetXdbcTypeInfo` metadata endpoint with an empty RecordBatch (#7576)
* feat: WIP support CommandGetXdbcTypeInfo metadata endpoint with tests

* chore: update test case and add jdbc test

* chore: uncomment jdbc getCoumns test

* chore: lint

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 21:06:27 +00:00
Andrew Lamb 2341306c26
refactor: Use upstream copy of FlightDataEncoder (#7656) 2023-04-25 20:16:47 +00:00
Andrew Lamb bc324c4ac2
chore: Update DataFusion and arrow/parquet/arrow-flight to `38.0.0` (#7662)
* chore: Update DataFusion and arrow/parquet/arrow-flight to `38.0.0`

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-25 14:29:23 -04:00
Jeffrey Smith II 0b39e6738f chore: add method for creating transition shard using transactions 2023-04-25 13:40:57 -04:00
Jeffrey Smith II 60eaa71191 test: add test for create_upgrade_delete 2023-04-25 13:40:55 -04:00
Jeffrey Smith II 6920d23a5b chore: remove transactions from Catalog callers 2023-04-25 13:39:18 -04:00
Jeffrey Smith II a01ae8f23d feat: remove transactions from Catalog 2023-04-25 13:39:16 -04:00
wiedld 523fd0cabf
Merge pull request #7621 from influxdata/idpe-17434/namespace-validation-alignment
chore(idpe-17434): remove utf8-percent encoding from onWrite path
2023-04-25 09:52:50 -07:00
wiedld 3990dcd519
Merge branch 'main' into idpe-17434/namespace-validation-alignment 2023-04-25 09:41:45 -07:00
Phil Bracikowski 06fbab2649
fix(garbage collector): limit catalog update for files to delete (#7623)
* fix(garbage collector): limit catalog update for files to delete

Impose a 1000 LIMIT on flag_for_delete_by_retention so the garbage
collector's load on the catalog is limited. 1000 is used as the fixed
limit in another catalog DML.

* follow up to requests in #7562

* chore: add test for limit on update

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 08:57:23 -07:00
Andrew Lamb 159a254ada
fix: Lower message about unsupported predicates on the ingester to debug level (#7655) 2023-04-25 10:46:03 -04:00
Dom 068da64c65
Merge pull request #7651 from influxdata/cn/partition-data
refactor: Extract commonly duplicated test code into helpers
2023-04-25 15:20:26 +01:00
Dom ba5de6b21d
Merge branch 'main' into cn/partition-data 2023-04-25 14:37:22 +01:00
Nga Tran b74a81e792
test: add tests on month and year date_bin (#7648)
* test: add tests on month and year date_bin

* fix: add IOX_COMPARE: uuid to get deterministics name for output parquet_file in the explain

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 10:16:21 +00:00
Carol (Nichols || Goulding) 740269839d
fix: Extract test helper functions 2023-04-24 20:14:59 -04:00
wiedld 148c103c70
refactor: push authz into specific configs (#7607)
* refactor: move authz-addr flag into router-specific config

* refactor: move authz-addr flag into querier-specific config

* refactor: remove global AuthzConfig which is now redundant with the pushdown to individual configs. Keep constant the env vars used universally.

* chore: make errors lowercase, and use the required bool for the authz-addr flag

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-25 00:14:28 +00:00
Carol (Nichols || Goulding) 2aaa431d5a
fix: You can't just call everything bananas, Dom
Remove the MockNamespaceNameProvider::default implementation that
hardcodes all namespace names to "bananas"; I just tore some of my hair
out trying to figure out why I was getting bananas WHEN THERE SHOULD
HAVE BEEN NO BANANAS
2023-04-24 20:13:54 -04:00
Carol (Nichols || Goulding) 38262f37cb
refactor: Take an Arc as an argument to be more consistent with similar types
NamespaceData::new doesn't take an Arc of the namespace name loader, but
PartitionData::new does. If they both take Arcs, then making test
constants and helpers they can share is easier. Same deal for
TableData::new.
2023-04-24 20:13:54 -04:00
wiedld b3cdea25ba chore: update term conventions to allowlist 2023-04-24 17:07:25 -07:00
Stuart Carnie 06bc2d6272
feat: Add `pre` and `post` visit events for `Literal` (#7629) 2023-04-24 21:15:44 +00:00
Joe-Blount ebf0b34523
feat: add compactor concurrency within single partition's plans (#7642)
* feat: trying to get concurrency within partition compaction

* feat: Create a stream then map and try collect

* feat: set compactor concurrency within single partition's plans

* chore: add log about plan concurrency

---------

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2023-04-24 20:48:35 +00:00