44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-26 10:33:56 -04:00
48d806b326
Merge branch 'main' into cn/workspace-inheritance
2022-10-24 20:10:08 +00:00
335dafa3f7
fix: Do not truncate data retrieved from `remote store get` command ( #5966 )
2022-10-24 17:45:56 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
1d440ddb2d
refactor: `IOxReadFilterNode` can always accumulate statistics ( #5954 )
...
* refactor: `IOxReadFilterNode` can always accumulate statistics
`IOxReadFilterNode` used to not emit statistics if one chunk has
duplicates or delete predicates. This is wrong (or at least overly
conservative), because the node itself (or the chunks themselves) do NOT
perform dedup or delete predicate filtering. Instead this is done is
done by parent nodes (`DeduplicateExec` and `FilterExec`) and its their
job to propagate statistics correctly.
Helps w/ #5897 .
* test: explain setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-10-24 13:34:22 +00:00
e0062f2d40
refactor: do NOT use fake DF context for parquet reading ( #5942 )
...
Use the proper top-level DataFusion context and register the object
store there.
Note that we still hide the `ParquetExec` behind an opaque record batch
stream. Fixing that is next on my list.
Helps with #5897 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-24 08:20:26 +00:00
fe98e7a65c
fix: Compactor skipped_at is in seconds, not nanoseconds oops
2022-10-21 16:41:55 -04:00
fa7fe2e9cf
feat: Add a gRPC endpoint to delete a skipped compaction
...
Also add a CLI usage of it for convenience
2022-10-21 15:12:20 -04:00
68e310f45d
feat: Display skipped compactions in a table instead of JSON
2022-10-21 13:59:19 -04:00
0132a33946
fix: Rename SkippedCompactionService to CompactionService
...
To make a good place for other compactor-related gRPC actions in the
future.
2022-10-21 13:40:37 -04:00
ba25300b01
feat: Create compactor service to list skipped compactions
2022-10-21 13:40:31 -04:00
9b67db3c06
Merge branch 'main' into cn/ingester-tracing
2022-10-21 13:13:13 +00:00
6e6e180aad
chore(deps): Bump assert_cmd from 2.0.4 to 2.0.5 ( #5937 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.4...v2.0.5 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 10:50:06 +00:00
59e1c1d5b9
feat: Pass trace id through Flight requests from querier to ingester
...
Fixes #5723 .
2022-10-20 08:55:30 -04:00
3fd0c5e4c2
fix: improve error message for storage read group command ( #5915 )
...
* fix: improve error message for storage read group command
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-20 09:31:50 +00:00
bebb15d30f
chore(deps): Bump serde_json from 1.0.86 to 1.0.87
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.86 to 1.0.87.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.86...v1.0.87 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-20 07:52:33 +00:00
82d6fc3bda
feat: support queries via influxrpc with periods in field names ( #5919 )
...
* feat: support queries via influxrpc with periods in field names
* fix: update comments
* fix: more tests
* fix: more tests
2022-10-19 20:09:55 +00:00
1df7a0d4fb
refactor: remove outdated observer sql repl mode ( #5918 )
...
* refactor: remove Observer mode from repl
* chore: remove outdated SQL docs
* fix: more update of sql docs
2022-10-19 18:39:05 +00:00
d706f8221d
chore: Update datafusion and arrow / parquet / arrow-flight 25.0.0 ( #5900 )
...
* chore: Update datafusion and `arrow` / `parquet` / `arrow-flight` 25.0.0
* chore: Update for structure changes
* chore: Update for new projection pushdown
* chore: Run cargo hakari tasks
* fix: fmt
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 20:58:47 +00:00
c28ac4a3c3
fix: Return an error for unsupported SQL queries ( #5876 )
...
* test: Failing tests for unsupported queries
* fix: Catch unsupported SQL operations and error rather than return nothing
* test: Document a few more error messages that come through DataFusion
* refactor: Extract a Step to make query error tests nicer to read and write
* fix: update tests for new error codes
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-18 19:27:29 +00:00
f3c27c5c71
chore(deps): Bump dotenvy from 0.15.5 to 0.15.6 ( #5881 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.5 to 0.15.6.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.5...v0.15.6 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-18 07:06:40 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
266b8f2a58
Merge branch 'main' into dependabot/cargo/clap-4.0.2
2022-10-12 14:01:28 +00:00
c4f542bbe2
refactor(ingester): remove tombstone support
...
This commit removes tombstone support from the ingester, and deletes
associated code/helpers/tests. This commit does NOT remove tombstone
support from any other service, but MAY include removing overlapping
test coverage.
This also removes the tombstone support from the Ingester -> Querier RPC
response message.
This has the nice side effect of removing a whole lot of thread spawning
in the ingester tests for the Executor, speeding everything up!
2022-10-11 13:10:04 +02:00
933493fab3
chore(deps): Bump object_store from 0.5.0 to 0.5.1
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.0...object_store_0.5.1 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-11 01:19:10 +00:00
2277fcf08a
chore(deps): Bump serde_json from 1.0.85 to 1.0.86
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.85 to 1.0.86.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.85...v1.0.86 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 01:42:37 +00:00
c4c83e0840
fix: query error propagation ( #5801 )
...
- treat OOM protection as "resource exhausted"
- use `DataFusionError` in more places instead of opaque `Box<dyn Error>`
- improve conversion from/into `DataFusionError` to preserve more
semantics
Overall, this improves our error handling. DF can now return errors like
"resource exhausted" and gRPC should now automatically generate a
sensible status code for it.
Fixes #5799 .
2022-10-06 08:54:01 +00:00
a11aafe25b
chore: Update SQL repl to refer to `namespace` rather than `database` ( #5788 )
2022-10-04 12:53:17 +00:00
82d5c7f336
feat: support parallel, chunked upload via `influxdb_iox write` of line protocol, gzip'd line protocol, and parquet ( #5757 )
...
* feat: Upload in small chunks and in parallel
* fix: doclink
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Update influxdb_iox_client/src/client/write.rs
* fix: fixup error handling and fmt
* fix: Make default chunk sizes the same and add docs
* fix: clippy
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-30 21:02:38 +00:00
627f617284
fix: Skip creating an ArgGroup for the All-in-One `Config`
...
As this type is flattened into other types also called `Config`, the
reused name would cause a conflict.
2022-09-30 16:59:29 -04:00
b2377a117a
fix: Restore --help flag
2022-09-30 16:59:28 -04:00
576d629ce4
fix: Remove leading `--` from long option names
2022-09-30 16:59:28 -04:00
50f84906e2
fix: Remove multiple_values = true; it's now implied because of Vec
...
See <https://docs.rs/clap/4.0.2/clap/_derive/index.html#arg-types >
> clap assumes some intent based on the type used:
>
> ...
>
> Vec<T> 0.. occurrences of argument .action(ArgAction::Append).required(false).num_args(1..)
2022-09-30 16:59:03 -04:00
73d7105f20
fix: Update from clap ArgEnum to ValueEnum
...
See <https://github.com/clap-rs/clap/pull/4127 >
2022-09-30 16:59:03 -04:00
199e47721a
chore(deps): Bump clap from 3.2.22 to 4.0.7
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.22 to 4.0.7.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...v4.0.7 )
2022-09-30 16:46:56 -04:00
d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… ( #5772 )
...
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently
* fix: comment
* fix: test output
* refactor: using var instead of literal
* fix: consider deleted L0s for recent writes
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
39a810ec3a
feat: Generate parquet files instead of line protocol
2022-09-29 16:22:37 -04:00
56a1c579a1
refactor: Change influxdb_iox client to use http rather than grpc for write ( #5756 )
...
* refactor: Change influxdb_iox client to use http rather than grpc for write
* refactor: remove custom variants
* refactor: consolidate more
2022-09-29 11:12:51 +00:00
227dde1dfc
chore(deps): Bump thiserror from 1.0.36 to 1.0.37 ( #5753 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.36 to 1.0.37.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.36...1.0.37 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 10:37:14 +00:00
8d90181e67
fix: Make sure generated level 1 files don't overlap
...
The data generator is inclusive in start and end times, so don't use the
same end time as the next level 1 file's start time to prevent overlap.
2022-09-28 14:41:00 -04:00
72203b3b31
feat: Generate the number of rows across the cold time range
2022-09-28 14:34:27 -04:00
21488ac360
feat: Generate the number of rows across the "hot" time range
2022-09-28 14:34:27 -04:00
2c70cd0cd0
refactor: Only pass along the one relevant config field
2022-09-28 14:34:27 -04:00
13ed1c089a
feat: use /api/v2 upload for write command rather than grpc `write_service` ( #5749 )
...
* feat: use /api/v2 upload for write command rather than grpc service
* fix: Update influxdb_iox/src/commands/write.rs
2022-09-28 11:16:51 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
84b10b28b2
feat: send only needed projection columns from querier to ingester in… ( #5678 )
...
* feat: send only needed projection columns from querier to ingester in case of normal SQL queries
* refactor: push column index down until we need to convert them strings
* fix: make the test deterministic
* test: test for the projection pushdown
* test: add asserts for the proj pushdown test
* test: implement projection pushdown for partitions of MockIngesterConnection
* chore: cleanup
* chore: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: address review comments
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-26 17:19:20 +00:00
87d0d19b0f
Merge branch 'main' into cn/generate-parquet-to-compact
2022-09-26 16:15:17 +00:00
e1210439bd
fix: Clear out data generation directory on every run
2022-09-26 12:14:36 -04:00
b1740f45d6
chore(deps): Bump thiserror from 1.0.35 to 1.0.36 ( #5737 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.35 to 1.0.36.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.35...1.0.36 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-26 14:44:36 +00:00
65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol ( #5734 )
...
* feat: add `influxdb_iox debug parquet_to_lp` command
* chore: Run cargo hakari tasks
* fix: update command description
* fix: remove unecessary Result import
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
8dfdd73533
feat: Write the specified number of line protocol files
2022-09-23 11:45:48 -04:00
e752684378
feat: Run the data generator with the generated spec
2022-09-23 11:45:48 -04:00
ea39e760e8
feat: Write out a data generator spec for one file of line protocol
2022-09-23 11:21:20 -04:00
5c458d439a
feat: Create a data generation spec file
2022-09-23 11:16:37 -04:00
febf46f3b5
refactor: Extract compactor generate to its own module file
2022-09-23 11:16:36 -04:00
61df629265
test: Failing test for actually creating parquet files
2022-09-23 11:16:31 -04:00
d2dd38c2cf
feat: Set up CLI for compactor generate
2022-09-23 10:49:04 -04:00
c8108f01e7
chore: Upgrade to Rust 1.64 ( #5727 )
...
* chore: Upgrade to Rust 1.64
* fix: Use iter find instead of a for loop, thanks clippy
* fix: Remove some needless borrows, thanks clippy
* fix: Use then_some rather than then with a closure, thanks clippy
* fix: Use iter retain rather than filter collect, thanks clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 18:04:00 +00:00
e3deb23bcc
feat: add minimum row_count per file in estimating compacting memory… ( #5715 )
...
* feat: add minimum row_count per file in estiumating compacting memory budget and limit number files per compaction
* chore: cleanup
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* test: add test per review comments
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* test: add one more test that has limit num files larger than total input files
* fix: make the L1 files in tests not overlapped
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 14:37:39 +00:00
365a246f8d
refactor: do not run de-dup in ingester for querier requests ( #5626 )
...
* refactor: do not run de-dup in ingester for querier requests
This removes the entire de-dup logic from the inegster for querier
requests. Furthermore, it even removes the entire datafusion execution
from the querier and just dumps the in-memory record batches as quickly
as possible. No filters are applied. Note that even prior to this PR,
we've never applied projections (tracked by #5624 ).
**Pros:**
- speed up query planning within the querier (since we need the ingester
response for state reconciling)
- lowered ingester CPU load
**Cons:**
- more querier<>ingester network traffic
Closes #5602 .
* test: extend query test case
* fix: ingester tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 07:33:54 +00:00
ea1e822e3b
chore(deps): Bump itertools from 0.10.4 to 0.10.5 ( #5707 )
...
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.4 to 0.10.5.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/commits )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 08:15:59 +00:00
0d18943ad2
chore(deps): Bump once_cell from 1.14.0 to 1.15.0 ( #5701 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 05:20:55 +00:00
5e7fd55a42
refactor: retry querier->ingester requests ( #5695 )
...
* refactor: retry querier->ingester requests
Esp. for InfluxRPC requests that scan multiple tables, it may be that
one ingester requests fails. We shall retry that request instead of
failing the entire query.
* refactor: improve docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* fix: less foo
* docs: remove outdated TODO
* test: assert that panic happened
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-09-20 15:51:02 +00:00
7fbd1a4d0c
chore: Expose all DSN flags in all-in-one ( #5682 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-20 07:22:40 +00:00
b6fb481b0f
chore(deps): Bump dotenvy from 0.15.3 to 0.15.5 ( #5689 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.3 to 0.15.5.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.3...v0.15.5 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-20 05:28:47 +00:00
46ab254dfb
fix: Make us log logs from the log crate again ( #5680 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-19 23:04:28 +00:00
a2a864100d
chore: replace `ansi_style` with `nu-ansi-style` ( #5669 )
...
See <https://rustsec.org/advisories/RUSTSEC-2021-0139.html >.
This follows the migration that `tracing-subscriber` will perform:
https://github.com/tokio-rs/tracing/pull/2287
Note that `tracing-subscriber` is not released yet, so for the time
being we will have `ansi_style` and `nu-ansi-style` in our dependency
list. It's likely that `tracing-subscriber` will backport and and
release this change rather soon though, because people are getting
annoyed by RUSTSEC warnings.
2022-09-19 08:17:01 +00:00
7e1f013346
chore(deps): Bump itertools from 0.10.3 to 0.10.4 ( #5631 )
...
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.3 to 0.10.4.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.3...v0.10.4 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 14:02:14 +00:00
b4a25fdb0e
chore(deps): Bump thiserror from 1.0.34 to 1.0.35 ( #5629 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.34 to 1.0.35.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.34...1.0.35 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 12:54:12 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
1fd31ee3bf
chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0 ( #5591 )
...
* chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0
* fix: enable dynamic comparison flag
* chore: derive Eq for clippy
* chore: update explain plans
* chore: Update sizes for ReadBuffer encoding
* chore: update more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 17:45:03 +00:00
3cea67fe36
test: Compaction of 1 file now upgrades the level, so the ID doesn't change
2022-09-12 13:13:29 -04:00
dfd7255c46
fix: Remove now-unused cold_input_file_count_threshold
2022-09-12 13:13:28 -04:00
3a368c02c2
fix: Remove now-unused cold_input_size_threshold_bytes
2022-09-12 13:13:28 -04:00
eefc71ac90
fix: Remove now unused max_cold_concurrent_size_bytes
2022-09-12 13:13:28 -04:00
6436afc3d9
fix: Remove cold max bytes CLI option; use existing max bytes CLI option
...
As discussed in https://github.com/influxdata/influxdb_iox/issues/5330#issuecomment-1218170063
2022-09-12 13:13:27 -04:00
10ba3fef47
feat: Compact cold partitions completely
...
Fixes #5330 .
2022-09-12 13:13:26 -04:00
15b3705f9a
feat: add "read group" support to storage CLI ( #5601 )
...
* fix: do not panic if measurement name is not the first tag
* feat: add "read group" support to storage CLI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 08:04:09 +00:00
6eaa971a52
chore: Allow running all-in-one with external object store ( #5600 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-10 12:03:54 +00:00
786ce75e26
chore(deps): Bump tokio-util from 0.7.3 to 0.7.4 ( #5596 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.3 to 0.7.4.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.3...tokio-util-0.7.4 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 07:40:16 +00:00
267a53a9e8
chore: update `tracing-subscriber`, fix trogging, fix CLI test port allocation ( #5581 )
...
* test: use dedicated ports for CLI tests
* chore: update `tracing-subscriber`
* fix: work around tracing-subscriber weirdness
It seems that trogging with tracing-subscriber >= 0.3.14 does not
produce any output at all. I suspect we are hitting
<https://github.com/tokio-rs/tracing/issues/2265 >. Let's change the
construct to not use multiple optional layers but a single dyn-dispatch
layer. Logging shouldn't have such a high throughput that his makes any
difference, esp. because the dyn-dispatch happens AFTER the filter.
2022-09-08 09:37:37 +00:00
52ae60bf2e
chore: fix comment typo ( #5551 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-07 08:49:29 +00:00
a280acb860
Merge branch 'main' into alamb/guilio-python-main
2022-09-06 16:57:00 +01:00
adeacf416c
ci: fix ( #5569 )
...
* ci: use same feature set in `build_dev` and `build_release`
* ci: also enable unstable tokio for `build_dev`
* chore: update tokio to 1.21 (to fix console-subscriber 0.1.8
* fix: "must use"
2022-09-06 14:13:28 +00:00
b494c73cb3
chore(deps): Bump console-subscriber from 0.1.7 to 0.1.8 ( #5558 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/tokio-console-v0.1.7...console-subscriber-v0.1.8 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 12:46:07 +00:00
9f0b0328f7
chore(deps): Bump thiserror from 1.0.33 to 1.0.34 ( #5556 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.33 to 1.0.34.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.33...1.0.34 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 09:18:41 +00:00
366c4d9965
chore(deps): Bump once_cell from 1.13.1 to 1.14.0 ( #5555 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.13.1 to 1.14.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.13.1...v1.14.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-06 09:02:28 +00:00
8b419ecd84
refactor: changed iox_shared to iox-shared
...
changed io_shared to iox-shared in the following files: update_catalog.rs, partition.rs, lib.rs (in the service_grpc_catalog folder) and lib.rs (in the service_grpc_object_store folder).
2022-09-04 07:59:07 -04:00
92656edddd
chore: upgrade jemalloc to version 5.3.0 ( #5542 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-02 10:45:13 +00:00
00ed79ff1b
chore(deps): Bump thiserror from 1.0.32 to 1.0.33 ( #5524 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.32 to 1.0.33.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.32...1.0.33 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 09:11:31 +00:00
ed2490deb2
Merge branch 'main' into dom/ingester-row-limit
2022-08-31 14:56:42 +01:00
2a19606456
feat(ingester): restrict partition row count
...
This limit restricts a single partition to containing at most N rows
before it is marked for persistence (note: being marked for persistence
does not currently prevent further ingest for that partition.)
2022-08-31 15:48:18 +02:00
6669d85fb4
chore: Update datafusion + arrow/parquet to `21.0.0` ( #5519 )
...
* chore: Update arrow/arrow-flight/parquet to 21.0.0
* chore: Update datafusion pin
* chore: Fix arrow update script
* chore: Update Cargo.lock
* chore: Update for new API
2022-08-31 13:30:47 +00:00
cb10a7c6d8
feat: More accurate memory estimate for compaction ( #5471 )
...
* feat: initial implementation of memory estimation for a compaction
* feat: estimate size of files and have the right actions for the needed budget
* feat: run candidates in parallel
* fix: have the right name for the column field of the output struct
* feat: add metrics for estimated budgets
* chore: cleanup
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fix syntax after applying review's suggestions
* refactor: Convert a Vec to VecDeque to go well with pop and push
* chore: remove max_concurrent_size_bytes and input_size_threshold_bytes
* chore: remove input_file_count_threshold
* test: tests for estimate_arrow_bytes_for_file
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-30 13:44:44 +00:00
dbd27f648f
refactor: Rename more mentions of Kafka to their other name where appropriate
2022-08-29 14:27:02 -04:00
1b49ad25f7
refactor: Rename KafkaTopicId to TopicId
2022-08-29 14:27:02 -04:00
58f0b63cdc
refactor: Rename KafkaTopic to Topic or TopicMetadata or topic name as appropriate
2022-08-29 14:27:02 -04:00
3aa3ae2ba5
docs: Add more comments about why to use ShardIndex or ShardId
2022-08-29 14:07:20 -04:00
74c9529062
fix: Rename KafkaPartition to ShardIndex
2022-08-29 14:07:18 -04:00
6443858870
fix: Rename compactor option from sequencer to shard
2022-08-29 14:06:45 -04:00
698f1a47ff
refactor: Rename test structures from sequencer to shard where appropriate
2022-08-29 14:06:44 -04:00
4abf21c724
refactor: Rename Sequencer (and its entourage) to Shard
2022-08-29 14:06:43 -04:00
8bc7606cb5
refactor: provide process-wide static strings (version, UUID) ( #5487 )
...
We currently only use the human-readable version string for the CLI
help, but for #5464 I want to use the GIT hash and a process-time UUID.
This is the prep work for that.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-29 12:25:19 +00:00
3950ca3a17
feat: upsert partition & update sort key for each day in bulk ingest ( #5447 )
...
* feat: upsert partition & update sort key for each day in bulk ingest
feat: import schema now supports earliest/latest time merging
chore: tests & tidying up for bulk ingest catalog update
* fix: always sort time last in PK in import schema update catalog
* chore: additional test for computing sort key in bulk ingest
* chore: bulk import catalog update gets sequencer from sharder service
chore: import update schema tests refactor using sharder svc mock
* chore: dead code fix
* chore: import schema sequencer lookup test
* chore: clarifying comment in import schema catalog update
2022-08-25 10:47:12 +00:00
99daa13897
test: Test dotenvy regression ( #5461 )
2022-08-24 09:39:55 +00:00
4beb721a9a
fix: Revert Bump dotenvy from 0.15.1 to 0.15.2 ( #5450 ) ( #5455 )
...
This reverts commit 84acbd2fadCloses #5454
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-24 09:10:09 +00:00
3220c6f88b
feat: add file_count_threshold for comapcting cold partitions ( #5456 )
...
* feat: file file_count_threshold for comapcting cold partitions to make it consistent with the hot case and help set up to avoid oom easier
* chore: remove unecessary commments
2022-08-23 20:12:21 +00:00
84acbd2fad
chore(deps): Bump dotenvy from 0.15.1 to 0.15.2 ( #5450 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.1 to 0.15.2.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/commits/v0.15.2 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-23 11:24:42 +00:00
f4443f0b3a
feat: import schema override ( #5420 )
...
* chore: struct for overrides of import schema conflicts
* chore: import schema override shouldn't support tags
* feat: import schema merge can take an override schema
* fix: schema override in test had superfluous tag
* chore: test for batch schema merge with override in import schema
* feat: import schema merge now takes override schema
2022-08-17 14:59:50 +00:00
7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 ( #5419 )
...
* chore: update object_store
* chore: update hakari config
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-17 13:44:48 +00:00
78665d3092
chore(deps): Bump once_cell from 1.13.0 to 1.13.1 ( #5413 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.13.0...v1.13.1 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 08:31:46 +00:00
10fee5535a
feat: import schema updates iox catalog ( #5385 )
...
* feat: import schema updates iox catalog
- renamed import/schema module to aggregate_tsm_schema to not conflic
with schema crate
- fetch schema from iox catalog, and validate/merge/create as needed
chore: add catalog dsn config to import schema command
chore: import schema command connects to catalog
chore: import schema merge validation errors return non-zero code
chore: simplified and tidies import update catalog code
chore: tests and refactoring of import schema catalog update
* chore: require retention on ns creation in import
* chore: fixed bad test in import schema validation
* chore: friendlier errors & more tests in import schema catalog update
2022-08-16 11:05:27 +00:00
549a267e3c
fix: Use Self instead of unnecessary structure name repetition
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#use_self
2022-08-11 15:21:02 -04:00
b982bdaf2f
fix: Derive Eq when we derive PartialEq and members can derive Eq
...
Allow this in generated code that we don't control, though.
Recommended by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq
2022-08-11 15:04:06 -04:00
ae6ac27960
chore(deps): Bump console-subscriber from 0.1.6 to 0.1.7 ( #5374 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.6 to 0.1.7.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.6...tokio-console-v0.1.7 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-11 11:08:10 +00:00
9321c96aaf
test: Add a step for running compaction in e2e tests
2022-08-10 11:30:22 -04:00
3915841a53
feat: Introduce a separate config for the compactor command
2022-08-10 11:30:21 -04:00
7787c51b57
feat: add new CLI command to run the compactor once
2022-08-10 11:28:51 -04:00
463a13b814
test: Remove the compactor from the test MiniCluster
2022-08-10 11:28:51 -04:00
45f8e567ed
fix: Revert adjustment of e2e test to expect compaction
...
This was part of
"feat: Different branch to hook up new compaction algorithm (#5194 )"
and will be added back in a new test for compaction specifically in the
next commit.
This reverts part of commit 69640c0ba5
2022-08-10 11:28:50 -04:00
7e9918f067
chore: import validate merged schema ( #5367 )
...
* feat: import schema merge now outputs validation results
chore: refactor import crate
chore: renamed some structs for clarity in import crate
* chore: tests for import schema merge validation
* chore: Run cargo hakari tasks
* chore: clippy
* chore: make hashmap loop easier to read in import schema validation
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 12:15:37 +00:00
c5f062bba0
feat: initial commit of schema merge bulk import tool ( #5344 )
...
* feat: initial commit of schema merge bulk import tool
* chore: use observability depds instead of tracing-*
* chore: removed debug printlns
* chore: fix feature decls for cloud providers for import crate
* chore: use println instead of info in import- no need for a simple CLI
* chore: tidy whitespace
* chore: remove unused dep in import
* chore: Run cargo hakari tasks
* chore: removed unimpld import job subcommand
* chore: clarifying comment about custom serialisation code
* chore: clarifying comment about schema merge code in import
* chore: fix wrong comment in import command
* chore: bump object store dep to get bugfix
* chore: rename import schema struct for clarity
* chore: run `cargo hakari generate`
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:07:38 +00:00
16ddc5efc6
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem ( #5360 )
...
* chore: Update datafusion and arrow
* chore: Update Cargo.lock
* chore: update to Decimal128
* chore: Update tonic/prost/pbjson/etc
* chore: Run cargo hakari tasks
* fix: doctest in generated types
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
da0b031c44
feat: Add parameters to limit total memory usage of cold partition compaction
2022-08-04 16:55:48 -04:00
d55f45a5c2
feat: Run compaction of hot partitions a configurable number of times more than cold
2022-08-04 16:55:48 -04:00
e8231b2986
chore(deps): Bump serde_json from 1.0.82 to 1.0.83 ( #5297 )
...
* chore(deps): Bump serde_json from 1.0.82 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.82 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.82...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 14:28:29 +00:00
b3958321d3
Merge branch 'main' into crepererum/issue5272
2022-08-03 21:06:15 +00:00
55e1e2ec2b
chore(deps): Bump thiserror from 1.0.31 to 1.0.32 ( #5294 )
...
* chore(deps): Bump thiserror from 1.0.31 to 1.0.32
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.31 to 1.0.32.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.31...1.0.32 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 16:20:36 +00:00
4bd8977d55
refactor: add some main function debug logs
2022-08-03 18:00:28 +02:00
840e4801b8
feat: make querier RAM pool split a proper feature ( #5283 )
...
* feat: make querier RAM pool split a proper feature
- use propre pool names
- expose sizing via CLI/env
Closes https://github.com/influxdata/conductor/issues/1102 .
* refactor: improve naming and docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:27:23 +00:00
663a20d743
refactor: remove `--ingster-address` ( #5255 )
...
Closes #5002 .
2022-08-03 15:05:01 +00:00
273b3cc165
chore: replace `dotenv` with `dotenvy` ( #5285 )
...
The latter one is a maintained fork. This avoids having both crates
after #5282 .
2022-08-03 12:41:38 +00:00
9215a534d0
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0` ( #5229 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0`
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 08:10:47 +00:00
9a9a1a4777
feat: limit per-table chunk data for every query ( #5223 )
...
* feat: `QueryChunk::as_any`
* feat: allo `ChunkPruner::prune_chunks` to fail
* feat: limit per-table chunk data for every query
Closes #5211 .
* fix: address review comments
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-27 13:20:05 +00:00
e4dc8c2067
refactor: rename garbage collector crates for consistency ( #5196 )
...
* refactor: rename garbage collector crates for consistency
* fix: cargo fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
69640c0ba5
feat: Different branch to hook up new compaction algorithm ( #5194 )
...
* chore: cherry pick the first 3 commits of branch cn/connect-new-compaction
* fix: modify the test to work correctly with compactor running
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 19:29:47 +00:00
5a0af921c8
chore: Roll forward: Sync ReadWindowAggregate API: TagKeyMetaNames ( #5186 )
...
This reverts commit 5d02c755687ef041f5f45dbfc3e633a833284edb.
2022-07-22 10:44:06 +00:00
07cdb99192
chore: Revert "Sync ReadWindowAggregate API: TagKeyMetaNames" ( #5184 )
...
We're noticing a possible regression (OOMs) in our testing cluster that roughly correlates with this.
2022-07-22 09:26:42 +00:00
69cb3f2b19
refactor: remove min_sequence_number from Compactor and Querier, add `count_by_overlaps_with_level_0` and `count_by_overlaps_with_level_1` to catalog ( #5151 )
...
* refactor: remove min_sequnce_number
* fix: typos
* fix: remove min_sequencer_number from new files from merging main
* fix: add back throwing error if the compactor compacts files persisted by the ingester after the ingester sends max seq_num back to querier
* test: add test_compactor_collision back but modify the input to make it work woth new changes
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-21 13:51:54 +00:00
21d033eafd
fix: Sync ReadWindowAggregate API: TagKeyMetaNames
...
The storage API has been updated in https://github.com/influxdata/idpe/pull/12868
in January, but since we forked the `.proto` files we never noticed.
2022-07-21 15:07:04 +02:00
efe39033b3
fix: tab complete from hints in `sql` relp (itch scratch) ( #5155 )
...
* fix: tab complete from hints
* fix: remove default impl
* docs: Update influxdb_iox/src/commands/sql/repl.rs
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 11:41:01 +00:00
c20288f60e
fix: Add TagKeyMetaNamesCapability capability ( #5160 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:52:40 +00:00
278a7f91af
chore(deps): Bump bytes from 1.1.0 to 1.2.0 ( #5156 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:00:08 +00:00
b8236e2b9d
fix: Fix SeriesKey sort order for special _measurement and _field ( #5150 )
...
* fix: Fix SeriesKey sort order for special _measurement and _field
* fix: Update expected test output
* fix: Update more tests
* fix: Re-sort tag key when using binary encoding
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-20 08:45:17 +00:00
f7a0fd43d2
feat: make object store garbage collector into a long-running service ( #5135 )
...
* refactor: remove unused logging config
* chore: remove the object store garbage collector CLI tool
* refactor: accept an object store and catalog
* refactor: make Result type alias public like the error
* refactor: remove public modifier from modules
* refactor: allow shutting down the object store garbage collector
* feat: Introduce the object-store garbage collection server
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
07e10852a8
feat: Add an input file count threshold to the compactor settings
2022-07-18 15:41:17 -04:00
128833e7d9
fix: Change placeholder new_param to input_size_threshold_bytes
2022-07-18 15:16:43 -04:00
d62b1ed7ee
feat: Select a subset of parquet files for a partition to compact
...
Fixes #5120 .
2022-07-18 15:14:22 -04:00
4416f1ce37
fix: Remove max number of level 0 files configuration option
2022-07-18 15:08:16 -04:00
57c70fcec5
fix: Remove redundant 'compaction' naming from CompactorConfig fields
2022-07-18 15:03:33 -04:00
Carol (Nichols || Goulding)
kodiakhq[bot]
Andrew Lamb
Marco Neumann
Jake Goulding
dependabot[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Nga Tran
Marko Mikulicic
YIXIAO SHI
Luke Bond
Juul Christiaens