ace497d47c
fix: Rename database to namespace in the commands I just added
2022-10-27 10:40:39 -04:00
d65a6a86dd
fix: Make error output less repetitive/wordy
2022-10-27 10:30:58 -04:00
de2ae6f557
feat: MVP of remote store get-table command
2022-10-26 13:50:03 -04:00
88c3a1f5e7
feat: Use workspace dep inheritance for the arrow-flight crate
2022-10-26 10:34:54 -04:00
3145e2c05b
feat: Use workspace dep inheritance for the arrow crate
2022-10-26 10:34:29 -04:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
fa7fe2e9cf
feat: Add a gRPC endpoint to delete a skipped compaction
...
Also add a CLI usage of it for convenience
2022-10-21 15:12:20 -04:00
0132a33946
fix: Rename SkippedCompactionService to CompactionService
...
To make a good place for other compactor-related gRPC actions in the
future.
2022-10-21 13:40:37 -04:00
ba25300b01
feat: Create compactor service to list skipped compactions
2022-10-21 13:40:31 -04:00
5f04acc9d5
refactor: Alphabetize module declarations
2022-10-21 13:40:30 -04:00
2179aeb785
fix: Allow a clippy false positive for mutable_key_type
2022-10-20 08:55:30 -04:00
59e1c1d5b9
feat: Pass trace id through Flight requests from querier to ingester
...
Fixes #5723 .
2022-10-20 08:55:30 -04:00
48fa18f346
fix: Alphabetize deps in client Cargo.toml whether they're in the workspace or not
2022-10-20 08:55:30 -04:00
d706f8221d
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 ( #5900 )
...
* chore: Update datafusion and `arrow` / `parquet` / `arrow-flight` 25.0.0
* chore: Update for structure changes
* chore: Update for new projection pushdown
* chore: Run cargo hakari tasks
* fix: fmt
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
c28ac4a3c3
fix: Return an error for unsupported SQL queries ( #5876 )
...
* test: Failing tests for unsupported queries
* fix: Catch unsupported SQL operations and error rather than return nothing
* test: Document a few more error messages that come through DataFusion
* refactor: Extract a Step to make query error tests nicer to read and write
* fix: update tests for new error codes
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
7202dddab6
chore(deps): Bump tokio-stream from 0.1.10 to 0.1.11 ( #5838 )
...
Bumps [tokio-stream](https://github.com/tokio-rs/tokio ) from 0.1.10 to 0.1.11.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.10...tokio-stream-0.1.11 )
---
updated-dependencies:
- dependency-name: tokio-stream
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 12:37:24 +00:00
82d5c7f336
feat: support parallel, chunked upload via `influxdb_iox write` of line protocol, gzip'd line protocol, and parquet ( #5757 )
...
* feat: Upload in small chunks and in parallel
* fix: doclink
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Update influxdb_iox_client/src/client/write.rs
* fix: fixup error handling and fmt
* fix: Make default chunk sizes the same and add docs
* fix: clippy
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-30 21:02:38 +00:00
8dbeef25ce
Merge branch 'main' into dom/remove-dbg
2022-09-29 12:14:19 +01:00
56a1c579a1
refactor: Change influxdb_iox client to use http rather than grpc for write ( #5756 )
...
* refactor: Change influxdb_iox client to use http rather than grpc for write
* refactor: remove custom variants
* refactor: consolidate more
2022-09-29 11:12:51 +00:00
cd4087e00d
style: add no todo!() or dbg!() lints
...
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
227dde1dfc
chore(deps): Bump thiserror from 1.0.36 to 1.0.37 ( #5753 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.36 to 1.0.37.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.36...1.0.37 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 10:37:14 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
b1740f45d6
chore(deps): Bump thiserror from 1.0.35 to 1.0.36 ( #5737 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.35 to 1.0.36.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.35...1.0.36 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-26 14:44:36 +00:00
b4a25fdb0e
chore(deps): Bump thiserror from 1.0.34 to 1.0.35 ( #5629 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.34 to 1.0.35.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.34...1.0.35 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 12:54:12 +00:00
1fd31ee3bf
chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0 ( #5591 )
...
* chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0
* fix: enable dynamic comparison flag
* chore: derive Eq for clippy
* chore: update explain plans
* chore: Update sizes for ReadBuffer encoding
* chore: update more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 17:45:03 +00:00
adeacf416c
ci: fix ( #5569 )
...
* ci: use same feature set in `build_dev` and `build_release`
* ci: also enable unstable tokio for `build_dev`
* chore: update tokio to 1.21 (to fix console-subscriber 0.1.8
* fix: "must use"
2022-09-06 14:13:28 +00:00
9f0b0328f7
chore(deps): Bump thiserror from 1.0.33 to 1.0.34 ( #5556 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.33 to 1.0.34.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.33...1.0.34 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 09:18:41 +00:00
00ed79ff1b
chore(deps): Bump thiserror from 1.0.32 to 1.0.33 ( #5524 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.32 to 1.0.33.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.32...1.0.33 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 09:11:31 +00:00
6669d85fb4
chore: Update datafusion + arrow/parquet to `21.0.0` ( #5519 )
...
* chore: Update arrow/arrow-flight/parquet to 21.0.0
* chore: Update datafusion pin
* chore: Fix arrow update script
* chore: Update Cargo.lock
* chore: Update for new API
2022-08-31 13:30:47 +00:00
74c9529062
fix: Rename KafkaPartition to ShardIndex
2022-08-29 14:07:18 -04:00
b982bdaf2f
fix: Derive Eq when we derive PartialEq and members can derive Eq
...
Allow this in generated code that we don't control, though.
Recommended by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq
2022-08-11 15:04:06 -04:00
16ddc5efc6
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem ( #5360 )
...
* chore: Update datafusion and arrow
* chore: Update Cargo.lock
* chore: update to Decimal128
* chore: Update tonic/prost/pbjson/etc
* chore: Run cargo hakari tasks
* fix: doctest in generated types
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
55e1e2ec2b
chore(deps): Bump thiserror from 1.0.31 to 1.0.32 ( #5294 )
...
* chore(deps): Bump thiserror from 1.0.31 to 1.0.32
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.31 to 1.0.32.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.31...1.0.32 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 16:20:36 +00:00
9215a534d0
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0` ( #5229 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0`
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 08:10:47 +00:00
278a7f91af
chore(deps): Bump bytes from 1.1.0 to 1.2.0 ( #5156 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:00:08 +00:00
e2d871b00b
chore: Update datafusion and arrow/parquet/arrow-flight to `18.0.0` ( #5079 )
...
* chore: Update datafusion to 10.0.0, arrow/parquet/arrow-flight to 18
* chore: Run cargo hakari tasks
* fix: update cargo pin
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 15:01:03 +00:00
9b67de2f43
chore(deps): Bump tokio from 1.19.2 to 1.20.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.19.2 to 1.20.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.19.2...tokio-1.20.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-14 01:21:43 +00:00
c46e1c6347
chore: Update datafusion + arrow/parquet/arrow-flight to `17.0.0` ( #5021 )
...
* fix: correct nullability declaration of system tables
* chore: Update datafusion and arrow/parquet/arrow-flight
* chore: Run cargo hakari tasks
* fix: Update tests
* fix: Update tests
* fix: predicate pruning
* fix: add some tests
* fix: query_functions
* fix: fix read_buffer test
* fix: fix clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-07 19:22:15 +00:00
77a4246432
docs: Readability improvements ( #4946 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-27 21:46:18 +00:00
66c7d95312
refactor: use new ingester<>querier wire protocol ( #4867 )
...
* refactor: use new ingester<>querier wire protocol
Use and document the new and more flexible ingester<>querier wire
protocol.
Note that the ingester does NOT stream the response data yet, but the
internal data structures would allow that. A follow-up change will
adjust the ingester code to stream the data.
Ref #4849 .
* fix: typos
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: clarify naming and public interface
* test: add schema assertion to `ingester_response_to_record_batches`
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-16 08:02:28 +00:00
4df2964566
refactor: store PartitionKey in DmlWrite
...
Carry the PartitionKey in the DmlWrite, allowing the batch to be
associated with a specific partition key.
2022-06-15 15:48:54 +01:00
3bd24b67ba
feat: extend flight client to accept multiple (changing) schemas ( #4853 )
...
* feat: extend flight client to accept multiple (changing) schemas
See #4849 .
Originally I intended not to use Flight at all for the new
ingester<>querier protocol. However since flight also deals with
dictionary batches and multiple batches and the gRPC protocol that I
would write would look very similar, I will use Flight with a bit more
flexible message types.
The rough idea for the protocol is the following stream:
- for each partition:
1. "none" message with partition metadata
2. for each chunk (can have different schemas under certain
circumstances):
1. "schema" message (resets dictionary state)
2. (optional) dictionary batch messages
3. one or more "record batch" message
The nice thing about it is that the same arrow client works also for the
existing client<>querier protocol since there we just send:
1. "schema" message (no app metadata)
2. (optional) dictionary batch messages
3. zero, one or more "record batch" message (no app metadata)
* refactor: separate high- and low-level flight client
It is very unlikely that a user will use the high-level batch-producing
functionality and the low-level stuff within the same session. So let's
split this into to clients (high-level uses the low-level one
internally) to avoid confusion.
Also add documentation on our protocol handling.
* refactor: enumerate all variants in match statement to better catch errors in the future
2022-06-15 11:38:08 +00:00
e91d00b10c
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `16.0.0 ( #4851 )
...
* chore: TEMP Update DataFusion to pre-release
* chore: update arrow et al to 16.0.0
* chore: Run cargo hakari tasks
* fix: update reader read_dictionary API
* chore: Update to real Datafusion release
* fix: Update parquet API
* fix: update test
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-06-14 16:31:40 +00:00
aa97c918b3
docs: fix README for influxdb_iox_client ( #4825 )
...
Closes #4816
2022-06-10 09:13:17 +00:00
e03bf94420
chore(deps): Bump tokio from 1.18.2 to 1.19.1 ( #4783 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.2...tokio-1.19.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:15:12 +00:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
9c1ffc2b0d
test: panic handling, add compactor to end to end test harness ( #4677 )
...
* feat: add test gRPC client
* test: start compactor in mini cluster
* test: assert panic handling
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-24 14:55:26 +00:00
2029bd16ba
feat: enable debugging of failed querier->ingester requests ( #4659 )
...
* feat: enable debugging of failed querier->ingester requests
- extend `query-ingester` CLI to allow usage of predicates
- on failed requests: log all information that required for the CLI
- test the "ingester fails" scenario
* test: explain
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* docs: improve
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: move b64 pred. serde into a single crate
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-05-23 15:37:31 +00:00
3a33e806c7
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `14.0.0` ( #4619 )
...
* chore: Update datafusion deps
* chore: update arrow/parquet/arrow flight deps
* chore: Run cargo hakari tasks
* chore: Update location of utils
* chore: Update some more APIs
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-17 14:13:03 +00:00
2ee4a6669a
refactor: Move the code merging write infos to generated_types to share
2022-05-11 14:07:42 -04:00
e07bcd40c2
refactor: Remove unused dependencies
...
These were found by iterating over all of the dependencies of each
Cargo.toml, then grepping that crate for the dependency's name. If it
didn't show up, I attempted to remove it.
I left a few dependencies that this process flagged:
* generated_types
- `pbjson`,`serde`. Apparently used by the generated code.
* grpc-router-test-gen
- `prost`. Apparently used by the generated code.
* influxdb_iox
- `heappy`. Doesn't appear used, but is behind enough feature
flags that I don't care to reason about and it's already optional.
- `tikv_jemalloc_sys`. Appears to be setting a feature flag of an
indirect dependency.
* iox_gitops_adapter
- `k8s_openapi`. Appears to be setting a feature flag of an indirect
dependency.
2022-05-06 15:57:58 -04:00
02893e598c
chore: Update datafusion and upgrade arrow/parquet/arrow-flight to 13 ( #4516 )
...
* chore: Tool for automating arrow version update
* chore: Update datafusion and arrow/parquet/arrow-flight
* fix: update for changes in Arrow API
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-05 00:21:02 +00:00
590a71010d
chore(deps): Bump serde_json from 1.0.80 to 1.0.81 ( #4514 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.80 to 1.0.81.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.80...v1.0.81 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-04 14:10:35 +00:00
c05383bbf5
chore(deps): Bump serde_json from 1.0.79 to 1.0.80 ( #4501 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.80 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-02 15:33:47 +00:00
e09157e81a
Merge branch 'main' into cn/bye-og-grpc-client
2022-05-02 14:54:54 +00:00
e1355f69af
chore(deps): Bump serde from 1.0.136 to 1.0.137 ( #4504 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.136 to 1.0.137.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.136...v1.0.137 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-02 14:11:18 +00:00
c59eaada5a
chore(deps): Bump thiserror from 1.0.30 to 1.0.31 ( #4498 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.30 to 1.0.31.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.30...1.0.31 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-02 13:43:45 +00:00
a4443e4c31
fix: Remove OG gRPC client code and APIs
2022-04-29 16:29:49 -04:00
8e48fcd620
feat: add remote pull partition ( #4433 )
...
Add lookup of partitions by table id to catalog.
Add API to catalog to return partitions by table id.
Add to client to return partitions by table id.
Add CLI to pull remote schema, partition, and parquet files into a local catalog and object store.
2022-04-28 21:04:27 +00:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
e2ebeff39c
feat: Add write tokens to the gRPC write endpoint response
...
As a header in the same way the HTTP write endpoint does.
Fixes #4208 .
2022-04-25 09:46:13 -04:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
0642ec0b82
docs: add note about write_info API being internal ( #4356 )
...
* docs: add note about write_info API being internal
* fix: update doc urls
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 09:25:14 +00:00
5ea676d3f7
feat: add per kafka partition durability reporting to write info response ( #4341 )
...
* feat: add per kafka partition durability reporting to write info response
* fix: buf lint + test cleanup
* fix: clean up protobuf
* refactor: pull out conversion of KafkaPartitionStatus into a function
* fix: fmt
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:46:20 +00:00
103629b01d
feat: add client and CLI to get file from object store ( #4343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:16:27 +00:00
197a3818d3
feat: add catalog client and remote command ( #4329 )
...
* feat: add catalog client and remote command
Adds the catalog gRPC service to influxdb_iox_client.
Adds a new remote command to execute commands against a remote IOx host.
Adds partition subcommand to remote to get the details of a partition by id.
* test: add end to end test for `remote partition` CLI (#4336 )
* chore: cleanup partition CLI PR feedback
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-15 13:59:49 +00:00
380cd9bbff
refactor: use a single flight client implementation ( #4273 )
...
"end-user -> querier" and "querier -> ingester" should use a single
Flight client implementation. The difference is just the request and
response metadata.
This changes our default Flight client to use protobuf instead of JSON
for the ticket format.
2022-04-12 09:08:25 +00:00
a30a85e62c
feat: Add get_write_info service ( #4227 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:24:58 +00:00
a1df864283
feat: Support 'SHOW NAMESPACES' in sql repl ( #4164 )
...
* feat: Support `SHOW NAMESPACES` in sql repl
* feat: add basic support to clients
* fix: add get_namespaces service test
* fix: proper error handling
* test: end to end test for namespace client
* refactor: Use QuerierDatabase rather than Catalog
* refactor: remove unused function
2022-03-31 12:57:33 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
e109fa4987
feat: schema client and CLI ( #4105 )
...
* feat: schema client and CLI
* chore: clarification in comment in schema command
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-23 13:49:24 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
677a272095
refactor: Clean up some future clippy warnings from nightly ( #3892 )
...
* refactor: clean up new clippy lints
* refactor: complete other cleanups
* fix: ignore overzealous clippy
* fix: re-remove old code
2022-03-03 19:14:27 +00:00
5a79b3a68b
chore(deps): Bump arrow-flight from 9.0.2 to 9.1.0 ( #3829 )
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:03:22 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
4e3f66ed07
feat: CLI and gRPC APIs for shutting down and restarting databases ( #3720 )
...
* feat: allow catalog wipe and rebuild whilst shutdown
* feat: CLI and gRPC APIs for shutting down and restarting databases
* feat: add ability to skip replay on restart
* fix: test_wipe_persisted_catalog_error_db_exists
* fix: wipe_preserved_catalog
2022-02-11 10:14:43 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
16d584b2ff
feat: Add db_name/namespace to DmlWrite and DmlDelete ( #3531 )
...
* feat: Add db_name/namespace to DmlWrite and DmlDelete
This is required for the new ingester to be able to work with the write buffer. The protobuf that gets serialized over Kafka already includes the database name, it just wasn't getting carried through to the marshaled Dml operation.
* fix: database != namespace, propagation through write buffer
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:12:20 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
758b65dd29
feat: Add database initialization state and errors to CLI and remove list_databases_detailed gRPC ( #3377 )
...
* feat: Add database initialization state and errors to CLI:
* fix: do not use optional in protobuf
* fix: clippy
* fix: correct check I broke appeasing clippy
2021-12-15 12:18:41 +00:00
c0ba69f09e
chore: marge main to branch and resolve conflict
2021-12-09 15:40:33 -05:00
3cda6b6c0f
refactor: Remove collect_query and replication ( #3348 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-09 19:58:19 +00:00
efbfbb1a0b
feat: compact all object store chunks of a given partition
2021-12-08 16:06:03 -05:00
c6a3765d76
feat: Add force flag to RebuildCatalog ( #3292 )
...
* feat: Add force flag to RebuildCatalog
* fix: small cleanups
* docs: Update comments and add WARNING
2021-12-08 15:36:07 +00:00
c17a6c10c1
feat: add `GetRouter` gRPC method
2021-12-08 13:45:55 +01:00
75cd1d24f4
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-07 03:26:06 +00:00
3834435620
chore: Merge branch 'ntran/grpc_compact_os_chunks' of https://github.com/influxdata/influxdb_iox into ntran/grpc_compact_os_chunks
2021-12-06 12:06:22 -05:00
d8ed8a3bf7
refactor: address review comments
2021-12-06 12:06:07 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
7a241882c2
fix: Make version reqs on futures-related crates consistent
2021-12-06 09:37:14 -05:00
65660dace5
Merge branch 'main' into ntran/grpc_compact_os_chunks
2021-12-03 18:16:42 -05:00
5f3706e0ee
feat: grpc call for copact object store chunks
2021-12-03 18:01:28 -05:00
7dea2ef362
feat: Add force flag to ClaimDatabase ( #3284 )
...
* feat: Add force flag to ClaimDatabase
* fix: Update name of test
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update influxdb_iox/tests/end_to_end_cases/management_cli.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-12-03 21:11:23 +00:00
Carol (Nichols || Goulding)
Jake Goulding
Andrew Lamb
Carol (Nichols || Goulding)
dependabot[bot]
Andrew Lamb
Dom
Marco Neumann
Ryan Russell
Luke Bond
kodiakhq[bot]
Paul Dix
Raphael Taylor-Davies
Paul Dix
Nga Tran