a38995ca0f
feat: add `MeasurementTagKeys` support to storage CLI ( #6039 )
...
Needed this to debug something.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 18:08:07 +00:00
3ba0458653
feat: Add object_store handler to querier so `remote get-table` works ( #6014 )
...
* feat: Add object_store handler to querier
* test: end to end test for get-table from querier
* fix: doc links
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 14:20:26 +00:00
fba4408d05
refactor: move `influxdb_iox debug namespace` command to `influxdb_iox namespace` ( #6031 )
...
* refactor: move command to
* docs: update the doc accordingly
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 10:57:58 +00:00
b1572c50a6
chore(deps): Bump once_cell from 1.15.0 to 1.16.0 ( #6009 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.15.0...v1.16.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-31 16:23:40 +00:00
69a2e6b871
feat: Last 2 bonus features of remote store get-table ( #5991 )
...
* feat: Only get files that aren't already on disk with the reported size
* feat: Stream Parquet file bytes to file on disk
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-28 11:03:08 +00:00
ace497d47c
fix: Rename database to namespace in the commands I just added
2022-10-27 10:40:39 -04:00
d65a6a86dd
fix: Make error output less repetitive/wordy
2022-10-27 10:30:58 -04:00
47faca6843
feat: Allow specifying output dir for get-table
2022-10-27 10:30:57 -04:00
dc4adfeefb
feat: Add the partition ID to fetched parquet files
2022-10-27 10:30:57 -04:00
f720dcee36
docs: Clarifications suggested in code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-27 10:10:28 -04:00
de2ae6f557
feat: MVP of remote store get-table command
2022-10-26 13:50:03 -04:00
8697ef4967
feat: Set up CLI args for new get-table command
2022-10-26 11:19:00 -04:00
71770486af
refactor: Extract influxdb_iox remote CLI tests to their own file
2022-10-26 11:19:00 -04:00
3145e2c05b
feat: Use workspace dep inheritance for the arrow crate
2022-10-26 10:34:29 -04:00
44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-26 10:33:56 -04:00
48d806b326
Merge branch 'main' into cn/workspace-inheritance
2022-10-24 20:10:08 +00:00
335dafa3f7
fix: Do not truncate data retrieved from `remote store get` command ( #5966 )
2022-10-24 17:45:56 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
1d440ddb2d
refactor: `IOxReadFilterNode` can always accumulate statistics ( #5954 )
...
* refactor: `IOxReadFilterNode` can always accumulate statistics
`IOxReadFilterNode` used to not emit statistics if one chunk has
duplicates or delete predicates. This is wrong (or at least overly
conservative), because the node itself (or the chunks themselves) do NOT
perform dedup or delete predicate filtering. Instead this is done is
done by parent nodes (`DeduplicateExec` and `FilterExec`) and its their
job to propagate statistics correctly.
Helps w/ #5897 .
* test: explain setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-24 13:34:22 +00:00
e0062f2d40
refactor: do NOT use fake DF context for parquet reading ( #5942 )
...
Use the proper top-level DataFusion context and register the object
store there.
Note that we still hide the `ParquetExec` behind an opaque record batch
stream. Fixing that is next on my list.
Helps with #5897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 08:20:26 +00:00
fe98e7a65c
fix: Compactor skipped_at is in seconds, not nanoseconds oops
2022-10-21 16:41:55 -04:00
fa7fe2e9cf
feat: Add a gRPC endpoint to delete a skipped compaction
...
Also add a CLI usage of it for convenience
2022-10-21 15:12:20 -04:00
68e310f45d
feat: Display skipped compactions in a table instead of JSON
2022-10-21 13:59:19 -04:00
0132a33946
fix: Rename SkippedCompactionService to CompactionService
...
To make a good place for other compactor-related gRPC actions in the
future.
2022-10-21 13:40:37 -04:00
ba25300b01
feat: Create compactor service to list skipped compactions
2022-10-21 13:40:31 -04:00
9b67db3c06
Merge branch 'main' into cn/ingester-tracing
2022-10-21 13:13:13 +00:00
6e6e180aad
chore(deps): Bump assert_cmd from 2.0.4 to 2.0.5 ( #5937 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.4...v2.0.5 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:50:06 +00:00
59e1c1d5b9
feat: Pass trace id through Flight requests from querier to ingester
...
Fixes #5723 .
2022-10-20 08:55:30 -04:00
3fd0c5e4c2
fix: improve error message for storage read group command ( #5915 )
...
* fix: improve error message for storage read group command
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:31:50 +00:00
bebb15d30f
chore(deps): Bump serde_json from 1.0.86 to 1.0.87
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.86 to 1.0.87.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.86...v1.0.87 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-20 07:52:33 +00:00
82d6fc3bda
feat: support queries via influxrpc with periods in field names ( #5919 )
...
* feat: support queries via influxrpc with periods in field names
* fix: update comments
* fix: more tests
* fix: more tests
2022-10-19 20:09:55 +00:00
1df7a0d4fb
refactor: remove outdated observer sql repl mode ( #5918 )
...
* refactor: remove Observer mode from repl
* chore: remove outdated SQL docs
* fix: more update of sql docs
2022-10-19 18:39:05 +00:00
d706f8221d
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 ( #5900 )
...
* chore: Update datafusion and `arrow` / `parquet` / `arrow-flight` 25.0.0
* chore: Update for structure changes
* chore: Update for new projection pushdown
* chore: Run cargo hakari tasks
* fix: fmt
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
c28ac4a3c3
fix: Return an error for unsupported SQL queries ( #5876 )
...
* test: Failing tests for unsupported queries
* fix: Catch unsupported SQL operations and error rather than return nothing
* test: Document a few more error messages that come through DataFusion
* refactor: Extract a Step to make query error tests nicer to read and write
* fix: update tests for new error codes
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
f3c27c5c71
chore(deps): Bump dotenvy from 0.15.5 to 0.15.6 ( #5881 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.5 to 0.15.6.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.5...v0.15.6 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-18 07:06:40 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
266b8f2a58
Merge branch 'main' into dependabot/cargo/clap-4.0.2
2022-10-12 14:01:28 +00:00
c4f542bbe2
refactor(ingester): remove tombstone support
...
This commit removes tombstone support from the ingester, and deletes
associated code/helpers/tests. This commit does NOT remove tombstone
support from any other service, but MAY include removing overlapping
test coverage.
This also removes the tombstone support from the Ingester -> Querier RPC
response message.
This has the nice side effect of removing a whole lot of thread spawning
in the ingester tests for the Executor, speeding everything up!
2022-10-11 13:10:04 +02:00
933493fab3
chore(deps): Bump object_store from 0.5.0 to 0.5.1
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.0...object_store_0.5.1 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-11 01:19:10 +00:00
2277fcf08a
chore(deps): Bump serde_json from 1.0.85 to 1.0.86
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.85 to 1.0.86.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.85...v1.0.86 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 01:42:37 +00:00
c4c83e0840
fix: query error propagation ( #5801 )
...
- treat OOM protection as "resource exhausted"
- use `DataFusionError` in more places instead of opaque `Box<dyn Error>`
- improve conversion from/into `DataFusionError` to preserve more
semantics
Overall, this improves our error handling. DF can now return errors like
"resource exhausted" and gRPC should now automatically generate a
sensible status code for it.
Fixes #5799 .
2022-10-06 08:54:01 +00:00
a11aafe25b
chore: Update SQL repl to refer to `namespace` rather than `database` ( #5788 )
2022-10-04 12:53:17 +00:00
82d5c7f336
feat: support parallel, chunked upload via `influxdb_iox write` of line protocol, gzip'd line protocol, and parquet ( #5757 )
...
* feat: Upload in small chunks and in parallel
* fix: doclink
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Update influxdb_iox_client/src/client/write.rs
* fix: fixup error handling and fmt
* fix: Make default chunk sizes the same and add docs
* fix: clippy
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-30 21:02:38 +00:00
627f617284
fix: Skip creating an ArgGroup for the All-in-One `Config`
...
As this type is flattened into other types also called `Config`, the
reused name would cause a conflict.
2022-09-30 16:59:29 -04:00
b2377a117a
fix: Restore --help flag
2022-09-30 16:59:28 -04:00
576d629ce4
fix: Remove leading `--` from long option names
2022-09-30 16:59:28 -04:00
50f84906e2
fix: Remove multiple_values = true; it's now implied because of Vec
...
See <https://docs.rs/clap/4.0.2/clap/_derive/index.html#arg-types >
> clap assumes some intent based on the type used:
>
> ...
>
> Vec<T> 0.. occurrences of argument .action(ArgAction::Append).required(false).num_args(1..)
2022-09-30 16:59:03 -04:00
73d7105f20
fix: Update from clap ArgEnum to ValueEnum
...
See <https://github.com/clap-rs/clap/pull/4127 >
2022-09-30 16:59:03 -04:00
199e47721a
chore(deps): Bump clap from 3.2.22 to 4.0.7
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.22 to 4.0.7.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...v4.0.7 )
2022-09-30 16:46:56 -04:00
d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… ( #5772 )
...
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently
* fix: comment
* fix: test output
* refactor: using var instead of literal
* fix: consider deleted L0s for recent writes
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
Marco Neumann
Andrew Lamb
Nga Tran
dependabot[bot]
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]
Jake Goulding
Dom Dwyer
Andrew Lamb