Commit Graph

11924 Commits (0b9240cbbe86fe687c1ed0a8c43f9bda61de212a)

Author SHA1 Message Date
Chunchun Ye 5182ab0037
chore: add more test and logic to grpc database header names (#7529)
* chore: support returning the database name if all the keys refer to the same database

* test: add test cases to check for same, different, and no database in request header

* chore: lint

* chore: more lint

* refactor: replace empty string with None for database_name

* refactor: simplify logic for NoFlightSQLDatabase error

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 18:28:29 +00:00
Chunchun Ye 8bf47df621
feat(flightsql): Support `GetImportedKeys` metadata endpoint with an empty RecordBatch (#7546)
* feat: support CommandGetImportedKeys metadata endpoint with tests

* chore: remove comments that is no longer valid

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 18:06:47 +00:00
dependabot[bot] b4003a70fe
chore(deps): Bump h2 from 0.3.16 to 0.3.17 (#7547)
Bumps [h2](https://github.com/hyperium/h2) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.3.16...v0.3.17)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 17:37:47 +00:00
Andrew Lamb 134ff2ef83
chore: update DataFusion pin (right before arrow 37 update) (#7540)
* chore: update DataFusion pin

* refactor: Update for deprecated API

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 17:25:24 +00:00
kodiakhq[bot] 08274b12d2
Merge pull request #7541 from influxdata/dom/remove-bad-comments
docs: remove misleading API comments
2023-04-13 14:31:13 +00:00
Dom Dwyer 3a8803c43c
docs: remove misleading API comments
These fields are very much in use now!
2023-04-13 16:17:48 +02:00
kodiakhq[bot] b7096bdad4
Merge pull request #7524 from influxdata/cn/remove-old-querier
fix: Remove old querier
2023-04-13 14:05:53 +00:00
kodiakhq[bot] 0020660b53
Merge branch 'main' into cn/remove-old-querier 2023-04-13 13:59:52 +00:00
Chunchun Ye e86e8de1a9
feat: support CommandGetExportedKeys metadata endpoint with tests (#7532)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 13:27:09 +00:00
Marco Neumann 0679cae4c6
chore: reduce prod image layer count (#7539)
- fuse two RUN commands into one, they're not cached anyways
- remove pointless `RUN ls ...`

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 12:06:59 +00:00
Andrew Lamb 3ebd07358b
chore: Update DataFusion pin, upgrade `date_bin` and `InfluxQL` to use `Interval(MonthDayNano)` (#7516)
* chore: Update datafusion

* chore: Update for change in PhysicalSortExpr

* refactor: Update date_bin_gapfill to take IntervalMonthDayNano, fix FlightSQL

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-13 10:43:32 +00:00
dependabot[bot] bf0bcb1184
chore(deps): Bump prost from 0.11.8 to 0.11.9 (#7537)
* chore(deps): Bump prost from 0.11.8 to 0.11.9

Bumps [prost](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: prost
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): Bump prost-build from 0.11.8 to 0.11.9

Bumps [prost-build](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: prost-build
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): Bump prost-types from 0.11.8 to 0.11.9

Bumps [prost-types](https://github.com/tokio-rs/prost) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: prost-types
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 10:27:15 +00:00
dependabot[bot] e811a69a1e
chore(deps): Bump serde_json from 1.0.95 to 1.0.96 (#7535)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.95 to 1.0.96.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.95...v1.0.96)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 10:07:32 +00:00
Stuart Carnie 5965cee179
feat: Update InfluxQL end-to-end test output (#7512)
* feat: Specialises test output formatting for each language

* Also fixes an error uncovered in the `write_columnar` when tag
  columns are `NULL`

Closes #7145

* chore: Run cargo hakari tasks

* chore: Add sorted output until #7513 is addressed

* chore: clippy 📋

* feat: Add `options` to `write_columnar`

* Added ability to configure border rendering, including removing
  borders. This helps avoid variable width issues with EXPLAIN output,
  which tends to vary and cause flaky test failures.

* chore: rustfmt 🧹

* chore: update expected output

* chore: clarify what "this" is

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-04-13 09:58:12 +00:00
Christopher M. Wolff cbd747db44
feat: update gap fill planner rule to use `interpolate` (#7494)
* feat: add INTERPOLATE fn and update planner gap-fill planner rule

* test: add an end-to-end test for interpolate()
2023-04-12 21:51:44 +00:00
Christopher M. Wolff 0937615dba
fix: make interpolate() fill null values in input (#7490)
* fix: make interpolate() fill null values in input

* chore: cargo doc
2023-04-12 21:41:11 +00:00
Christopher M. Wolff 3e60369eff
refactor: input buffering for gap filling interpolate null-as-missing (#7478)
* refactor: move logic for knowing how much to buffer into GapFiller

* chore: clippy

* chore: add some clarifying comments

* refactor: clean up relationships between gap filling types

* refactor: remove use of RefCell from BufferedInput

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 21:08:51 +00:00
Carol (Nichols || Goulding) f0f74bae02
fix: Treat empty ingester info differently than not having ingester info
When pre-warming the catalog cache before the ingester responses have
returned, we don't have any ingester parquet file counts. This is
different than asking the ingesters for the parquet file counts and not
getting any. So keep the Option to be able to treat "not present"
differently from "present but empty".
2023-04-12 14:50:18 -04:00
Andrew Lamb b9c9ee6cab
docs: note that Span::child() does not automatically emit spans (#7525)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 18:40:30 +00:00
Carol (Nichols || Goulding) acf857816e
fix: Remove old querier 2023-04-12 13:18:23 -04:00
Chunchun Ye fb6b3f66da
feat(flightsql): support `database`, `bucket`, and `bucket-name` as grpc header names (#7511)
* feat: support `database`, `bucket`, and `bucket-name` as grpc header names

* chore: lint

* chore: update doc to accept `database`, `bucket`, and `bucket-name` as parameter names

* chore: update doc to only show `database` as the parameter name

* refactor: consolidate header names into a const vec and update comments on database

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:58:44 +00:00
Andrew Lamb 171113b922
feat: Add process/service names to spans, fix ingester span propagation (#7507)
* fix: Add IOx process / service name to tracing spans

* fix: Fix ingester span context propagation

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:26:35 +00:00
kodiakhq[bot] a28afbcd68
Merge pull request #7522 from influxdata/cn/remove-write-summary
fix: Remove the write_summary crate and write info service
2023-04-12 16:16:00 +00:00
kodiakhq[bot] 53ddca45d8
Merge branch 'main' into cn/remove-write-summary 2023-04-12 16:07:35 +00:00
Andrew Lamb 20e9c91866
refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc (#7515)
* refactor: Use workspace dependencies for `tonic`, `tonic-build`, etc

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:07:19 +00:00
Marco Neumann c7197a289e
fix: gRPC error code for invalid InfluQL (#7519)
This was "internal". The mapping works like this: we take the
`DataFusionError` and call `find_root` which should traverse the
`External(...)` chain (even through Arrow) to find the last error that
is not within the Arrow/DataFusion land. This is then mapped by us.

`DataFusionError::External(...)` is no further inspected and mapped
straight to "internal". I think this if fine because in the end we're
mostly dealing w/ DataFusion stuff anyways.

I've slightly changed the error mapping in the planner to emit
`DataFusionError::Plan(...)` instead which we map to "invalid argument".
I think this is way better for the user.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 16:00:38 +00:00
Andrew Lamb f3de4e39f0
docs: Improve documentation on query API (#7520)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 15:53:42 +00:00
Carol (Nichols || Goulding) 2e656766f8
fix: Remove docs about testing object_store crate
This crate has its own repo now.
2023-04-12 11:53:29 -04:00
Carol (Nichols || Goulding) f561d599d2
fix: Remove uses of KAFKA_CONNECT, redpanda, and some talk of Kafka 2023-04-12 11:53:23 -04:00
Joe-Blount 7dd221aee0
chore: add logging around compaction job semaphore (#7523)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-12 15:47:02 +00:00
kodiakhq[bot] 0cdeef531f
Merge pull request #7517 from influxdata/dom/remove-explicit-replica
chore: remove ingest_replica
2023-04-12 15:39:26 +00:00
Carol (Nichols || Goulding) 6387a9576a
fix: Remove the write_summary crate and write info service 2023-04-12 11:31:23 -04:00
Dom 13572e36fa
Merge branch 'main' into dom/remove-explicit-replica 2023-04-12 16:29:18 +01:00
kodiakhq[bot] ffe8714956
Merge pull request #7417 from influxdata/cn/cold-compaction-selection
feat: Add the ability to specify a max threshold duration on CatalogToCompactPartitionsSource
2023-04-12 15:26:21 +00:00
kodiakhq[bot] 14d7098ddd
Merge branch 'main' into cn/cold-compaction-selection 2023-04-12 15:19:06 +00:00
kodiakhq[bot] b9a49907fe
Merge pull request #7521 from influxdata/cn/remove-write-buffer
fix: Remove the write buffer crate
2023-04-12 15:18:11 +00:00
Carol (Nichols || Goulding) 3199d65c2f
feat: Add the ability to specify a max threshold duration on CatalogToCompactPartitionsSource 2023-04-12 11:08:51 -04:00
Carol (Nichols || Goulding) a244e5b078
test: Add some tests for CatalogToCompactPartitionsSource's existing behavior 2023-04-12 11:07:43 -04:00
Carol (Nichols || Goulding) 65c90b0281
fix: Remove the write buffer crate 2023-04-12 11:01:26 -04:00
kodiakhq[bot] 110a2112fe
Merge pull request #7518 from influxdata/cn/remove-old-router
fix: Remove old router
2023-04-12 14:59:00 +00:00
Carol (Nichols || Goulding) d025362ce0
fix: Remove old router 2023-04-12 10:15:48 -04:00
Dom 92bf602f69
Merge branch 'main' into dom/remove-explicit-replica 2023-04-12 15:06:44 +01:00
Dom Dwyer ed06d8e760
chore: remove ingest_replica
The replication implementation uses a different design - the
ingest_replica is no longer needed.
2023-04-12 16:06:31 +02:00
kodiakhq[bot] 1c24440305
Merge pull request #7503 from influxdata/savage/router-read-through-namespace-cache
feat(router): Use a read-through decorator for `NamespaceCache` usage
2023-04-12 13:50:17 +00:00
Fraser Savage a9d1a3a5f9
Merge branch 'main' into savage/router-read-through-namespace-cache 2023-04-12 14:40:56 +01:00
Fraser Savage dc6053bfba
refactor(router): Apply further code review changes, clean up docs 2023-04-12 14:40:02 +01:00
Fraser Savage 8a2b88398f
refactor(router): Apply suggestions from code review
Assert an invariant, document existing edge cases and a little cleanup.

Co-authored-by: Dom <dom@itsallbroken.com>
2023-04-12 14:12:12 +01:00
Dom 28f931db56
Merge pull request #7514 from influxdata/dom/ingester-bench-context
test(bench): ingester2 write benchmarks
2023-04-12 13:42:18 +01:00
Dom Dwyer 5c6d51c808
refactor: constify defaults
Constify default TestContext config params.
2023-04-12 14:28:06 +02:00
Dom Dwyer dd92459c41
test(bench): ingester2 write benchmarks
Adds a single-line & batched write request benchmarks to ingester2.

When using the WAL, a floor of 10ms is added to all writes (the WAL
linger time, configurable). With the WAL disabled in code, benchmarked
writes complete in less than a millisecond:

    single row/write
        time:   [1.9138 µs 1.9168 µs 1.9197 µs]
        thrpt:  [520.92 Kelem/s 521.70 Kelem/s 522.51 Kelem/s]

    batched/write/1000
        time:   [129.15 µs 129.97 µs 131.28 µs]
        thrpt:  [7.6173 Melem/s 7.6941 Melem/s 7.7429 Melem/s]

Note these benchmarks exclude network I/O, and measure single-threaded,
synchronous write client performance.
2023-04-12 12:51:47 +02:00