73d7105f20
fix: Update from clap ArgEnum to ValueEnum
...
See <https://github.com/clap-rs/clap/pull/4127 >
2022-09-30 16:59:03 -04:00
199e47721a
chore(deps): Bump clap from 3.2.22 to 4.0.7
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.22 to 4.0.7.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.22...v4.0.7 )
2022-09-30 16:46:56 -04:00
d171697fd7
feat: always pick cold partitions in next cycle even if it has been pa… ( #5772 )
...
* fix: always pick cold partitions in next cycle even if it has been partially compacted recently
* fix: comment
* fix: test output
* refactor: using var instead of literal
* fix: consider deleted L0s for recent writes
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-30 15:54:00 +00:00
39a810ec3a
feat: Generate parquet files instead of line protocol
2022-09-29 16:22:37 -04:00
56a1c579a1
refactor: Change influxdb_iox client to use http rather than grpc for write ( #5756 )
...
* refactor: Change influxdb_iox client to use http rather than grpc for write
* refactor: remove custom variants
* refactor: consolidate more
2022-09-29 11:12:51 +00:00
227dde1dfc
chore(deps): Bump thiserror from 1.0.36 to 1.0.37 ( #5753 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.36 to 1.0.37.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.36...1.0.37 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-29 10:37:14 +00:00
8d90181e67
fix: Make sure generated level 1 files don't overlap
...
The data generator is inclusive in start and end times, so don't use the
same end time as the next level 1 file's start time to prevent overlap.
2022-09-28 14:41:00 -04:00
72203b3b31
feat: Generate the number of rows across the cold time range
2022-09-28 14:34:27 -04:00
21488ac360
feat: Generate the number of rows across the "hot" time range
2022-09-28 14:34:27 -04:00
2c70cd0cd0
refactor: Only pass along the one relevant config field
2022-09-28 14:34:27 -04:00
13ed1c089a
feat: use /api/v2 upload for write command rather than grpc `write_service` ( #5749 )
...
* feat: use /api/v2 upload for write command rather than grpc service
* fix: Update influxdb_iox/src/commands/write.rs
2022-09-28 11:16:51 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
84b10b28b2
feat: send only needed projection columns from querier to ingester in… ( #5678 )
...
* feat: send only needed projection columns from querier to ingester in case of normal SQL queries
* refactor: push column index down until we need to convert them strings
* fix: make the test deterministic
* test: test for the projection pushdown
* test: add asserts for the proj pushdown test
* test: implement projection pushdown for partitions of MockIngesterConnection
* chore: cleanup
* chore: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: address review comments
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-26 17:19:20 +00:00
87d0d19b0f
Merge branch 'main' into cn/generate-parquet-to-compact
2022-09-26 16:15:17 +00:00
e1210439bd
fix: Clear out data generation directory on every run
2022-09-26 12:14:36 -04:00
b1740f45d6
chore(deps): Bump thiserror from 1.0.35 to 1.0.36 ( #5737 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.35 to 1.0.36.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.35...1.0.36 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-26 14:44:36 +00:00
65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol ( #5734 )
...
* feat: add `influxdb_iox debug parquet_to_lp` command
* chore: Run cargo hakari tasks
* fix: update command description
* fix: remove unecessary Result import
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
8dfdd73533
feat: Write the specified number of line protocol files
2022-09-23 11:45:48 -04:00
e752684378
feat: Run the data generator with the generated spec
2022-09-23 11:45:48 -04:00
ea39e760e8
feat: Write out a data generator spec for one file of line protocol
2022-09-23 11:21:20 -04:00
5c458d439a
feat: Create a data generation spec file
2022-09-23 11:16:37 -04:00
febf46f3b5
refactor: Extract compactor generate to its own module file
2022-09-23 11:16:36 -04:00
61df629265
test: Failing test for actually creating parquet files
2022-09-23 11:16:31 -04:00
d2dd38c2cf
feat: Set up CLI for compactor generate
2022-09-23 10:49:04 -04:00
c8108f01e7
chore: Upgrade to Rust 1.64 ( #5727 )
...
* chore: Upgrade to Rust 1.64
* fix: Use iter find instead of a for loop, thanks clippy
* fix: Remove some needless borrows, thanks clippy
* fix: Use then_some rather than then with a closure, thanks clippy
* fix: Use iter retain rather than filter collect, thanks clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 18:04:00 +00:00
e3deb23bcc
feat: add minimum row_count per file in estimating compacting memory… ( #5715 )
...
* feat: add minimum row_count per file in estiumating compacting memory budget and limit number files per compaction
* chore: cleanup
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* test: add test per review comments
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* test: add one more test that has limit num files larger than total input files
* fix: make the L1 files in tests not overlapped
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 14:37:39 +00:00
365a246f8d
refactor: do not run de-dup in ingester for querier requests ( #5626 )
...
* refactor: do not run de-dup in ingester for querier requests
This removes the entire de-dup logic from the inegster for querier
requests. Furthermore, it even removes the entire datafusion execution
from the querier and just dumps the in-memory record batches as quickly
as possible. No filters are applied. Note that even prior to this PR,
we've never applied projections (tracked by #5624 ).
**Pros:**
- speed up query planning within the querier (since we need the ingester
response for state reconciling)
- lowered ingester CPU load
**Cons:**
- more querier<>ingester network traffic
Closes #5602 .
* test: extend query test case
* fix: ingester tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 07:33:54 +00:00
ea1e822e3b
chore(deps): Bump itertools from 0.10.4 to 0.10.5 ( #5707 )
...
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.4 to 0.10.5.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/commits )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 08:15:59 +00:00
0d18943ad2
chore(deps): Bump once_cell from 1.14.0 to 1.15.0 ( #5701 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 05:20:55 +00:00
5e7fd55a42
refactor: retry querier->ingester requests ( #5695 )
...
* refactor: retry querier->ingester requests
Esp. for InfluxRPC requests that scan multiple tables, it may be that
one ingester requests fails. We shall retry that request instead of
failing the entire query.
* refactor: improve docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* fix: less foo
* docs: remove outdated TODO
* test: assert that panic happened
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-09-20 15:51:02 +00:00
7fbd1a4d0c
chore: Expose all DSN flags in all-in-one ( #5682 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-20 07:22:40 +00:00
b6fb481b0f
chore(deps): Bump dotenvy from 0.15.3 to 0.15.5 ( #5689 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.3 to 0.15.5.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/compare/v0.15.3...v0.15.5 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-20 05:28:47 +00:00
46ab254dfb
fix: Make us log logs from the log crate again ( #5680 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-19 23:04:28 +00:00
a2a864100d
chore: replace `ansi_style` with `nu-ansi-style` ( #5669 )
...
See <https://rustsec.org/advisories/RUSTSEC-2021-0139.html >.
This follows the migration that `tracing-subscriber` will perform:
https://github.com/tokio-rs/tracing/pull/2287
Note that `tracing-subscriber` is not released yet, so for the time
being we will have `ansi_style` and `nu-ansi-style` in our dependency
list. It's likely that `tracing-subscriber` will backport and and
release this change rather soon though, because people are getting
annoyed by RUSTSEC warnings.
2022-09-19 08:17:01 +00:00
7e1f013346
chore(deps): Bump itertools from 0.10.3 to 0.10.4 ( #5631 )
...
Bumps [itertools](https://github.com/rust-itertools/itertools ) from 0.10.3 to 0.10.4.
- [Release notes](https://github.com/rust-itertools/itertools/releases )
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.10.3...v0.10.4 )
---
updated-dependencies:
- dependency-name: itertools
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 14:02:14 +00:00
b4a25fdb0e
chore(deps): Bump thiserror from 1.0.34 to 1.0.35 ( #5629 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.34 to 1.0.35.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.34...1.0.35 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-14 12:54:12 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
1fd31ee3bf
chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0 ( #5591 )
...
* chore: Update datafusion / `arrow` / `arrow-flight` / `parquet` to version 22.0.0
* fix: enable dynamic comparison flag
* chore: derive Eq for clippy
* chore: update explain plans
* chore: Update sizes for ReadBuffer encoding
* chore: update more tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 17:45:03 +00:00
3cea67fe36
test: Compaction of 1 file now upgrades the level, so the ID doesn't change
2022-09-12 13:13:29 -04:00
dfd7255c46
fix: Remove now-unused cold_input_file_count_threshold
2022-09-12 13:13:28 -04:00
3a368c02c2
fix: Remove now-unused cold_input_size_threshold_bytes
2022-09-12 13:13:28 -04:00
eefc71ac90
fix: Remove now unused max_cold_concurrent_size_bytes
2022-09-12 13:13:28 -04:00
6436afc3d9
fix: Remove cold max bytes CLI option; use existing max bytes CLI option
...
As discussed in https://github.com/influxdata/influxdb_iox/issues/5330#issuecomment-1218170063
2022-09-12 13:13:27 -04:00
10ba3fef47
feat: Compact cold partitions completely
...
Fixes #5330 .
2022-09-12 13:13:26 -04:00
15b3705f9a
feat: add "read group" support to storage CLI ( #5601 )
...
* fix: do not panic if measurement name is not the first tag
* feat: add "read group" support to storage CLI
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-12 08:04:09 +00:00
6eaa971a52
chore: Allow running all-in-one with external object store ( #5600 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-10 12:03:54 +00:00
786ce75e26
chore(deps): Bump tokio-util from 0.7.3 to 0.7.4 ( #5596 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.3 to 0.7.4.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.3...tokio-util-0.7.4 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-09 07:40:16 +00:00
267a53a9e8
chore: update `tracing-subscriber`, fix trogging, fix CLI test port allocation ( #5581 )
...
* test: use dedicated ports for CLI tests
* chore: update `tracing-subscriber`
* fix: work around tracing-subscriber weirdness
It seems that trogging with tracing-subscriber >= 0.3.14 does not
produce any output at all. I suspect we are hitting
<https://github.com/tokio-rs/tracing/issues/2265 >. Let's change the
construct to not use multiple optional layers but a single dyn-dispatch
layer. Logging shouldn't have such a high throughput that his makes any
difference, esp. because the dyn-dispatch happens AFTER the filter.
2022-09-08 09:37:37 +00:00
52ae60bf2e
chore: fix comment typo ( #5551 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2022-09-07 08:49:29 +00:00
a280acb860
Merge branch 'main' into alamb/guilio-python-main
2022-09-06 16:57:00 +01:00
adeacf416c
ci: fix ( #5569 )
...
* ci: use same feature set in `build_dev` and `build_release`
* ci: also enable unstable tokio for `build_dev`
* chore: update tokio to 1.21 (to fix console-subscriber 0.1.8
* fix: "must use"
2022-09-06 14:13:28 +00:00
b494c73cb3
chore(deps): Bump console-subscriber from 0.1.7 to 0.1.8 ( #5558 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.7 to 0.1.8.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/tokio-console-v0.1.7...console-subscriber-v0.1.8 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 12:46:07 +00:00
9f0b0328f7
chore(deps): Bump thiserror from 1.0.33 to 1.0.34 ( #5556 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.33 to 1.0.34.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.33...1.0.34 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-06 09:18:41 +00:00
366c4d9965
chore(deps): Bump once_cell from 1.13.1 to 1.14.0 ( #5555 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.13.1 to 1.14.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.13.1...v1.14.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-06 09:02:28 +00:00
8b419ecd84
refactor: changed iox_shared to iox-shared
...
changed io_shared to iox-shared in the following files: update_catalog.rs, partition.rs, lib.rs (in the service_grpc_catalog folder) and lib.rs (in the service_grpc_object_store folder).
2022-09-04 07:59:07 -04:00
92656edddd
chore: upgrade jemalloc to version 5.3.0 ( #5542 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-02 10:45:13 +00:00
00ed79ff1b
chore(deps): Bump thiserror from 1.0.32 to 1.0.33 ( #5524 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.32 to 1.0.33.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.32...1.0.33 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 09:11:31 +00:00
ed2490deb2
Merge branch 'main' into dom/ingester-row-limit
2022-08-31 14:56:42 +01:00
2a19606456
feat(ingester): restrict partition row count
...
This limit restricts a single partition to containing at most N rows
before it is marked for persistence (note: being marked for persistence
does not currently prevent further ingest for that partition.)
2022-08-31 15:48:18 +02:00
6669d85fb4
chore: Update datafusion + arrow/parquet to `21.0.0` ( #5519 )
...
* chore: Update arrow/arrow-flight/parquet to 21.0.0
* chore: Update datafusion pin
* chore: Fix arrow update script
* chore: Update Cargo.lock
* chore: Update for new API
2022-08-31 13:30:47 +00:00
cb10a7c6d8
feat: More accurate memory estimate for compaction ( #5471 )
...
* feat: initial implementation of memory estimation for a compaction
* feat: estimate size of files and have the right actions for the needed budget
* feat: run candidates in parallel
* fix: have the right name for the column field of the output struct
* feat: add metrics for estimated budgets
* chore: cleanup
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fix syntax after applying review's suggestions
* refactor: Convert a Vec to VecDeque to go well with pop and push
* chore: remove max_concurrent_size_bytes and input_size_threshold_bytes
* chore: remove input_file_count_threshold
* test: tests for estimate_arrow_bytes_for_file
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-30 13:44:44 +00:00
dbd27f648f
refactor: Rename more mentions of Kafka to their other name where appropriate
2022-08-29 14:27:02 -04:00
1b49ad25f7
refactor: Rename KafkaTopicId to TopicId
2022-08-29 14:27:02 -04:00
58f0b63cdc
refactor: Rename KafkaTopic to Topic or TopicMetadata or topic name as appropriate
2022-08-29 14:27:02 -04:00
3aa3ae2ba5
docs: Add more comments about why to use ShardIndex or ShardId
2022-08-29 14:07:20 -04:00
74c9529062
fix: Rename KafkaPartition to ShardIndex
2022-08-29 14:07:18 -04:00
6443858870
fix: Rename compactor option from sequencer to shard
2022-08-29 14:06:45 -04:00
698f1a47ff
refactor: Rename test structures from sequencer to shard where appropriate
2022-08-29 14:06:44 -04:00
4abf21c724
refactor: Rename Sequencer (and its entourage) to Shard
2022-08-29 14:06:43 -04:00
8bc7606cb5
refactor: provide process-wide static strings (version, UUID) ( #5487 )
...
We currently only use the human-readable version string for the CLI
help, but for #5464 I want to use the GIT hash and a process-time UUID.
This is the prep work for that.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-29 12:25:19 +00:00
3950ca3a17
feat: upsert partition & update sort key for each day in bulk ingest ( #5447 )
...
* feat: upsert partition & update sort key for each day in bulk ingest
feat: import schema now supports earliest/latest time merging
chore: tests & tidying up for bulk ingest catalog update
* fix: always sort time last in PK in import schema update catalog
* chore: additional test for computing sort key in bulk ingest
* chore: bulk import catalog update gets sequencer from sharder service
chore: import update schema tests refactor using sharder svc mock
* chore: dead code fix
* chore: import schema sequencer lookup test
* chore: clarifying comment in import schema catalog update
2022-08-25 10:47:12 +00:00
99daa13897
test: Test dotenvy regression ( #5461 )
2022-08-24 09:39:55 +00:00
4beb721a9a
fix: Revert Bump dotenvy from 0.15.1 to 0.15.2 ( #5450 ) ( #5455 )
...
This reverts commit 84acbd2fadCloses #5454
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-24 09:10:09 +00:00
3220c6f88b
feat: add file_count_threshold for comapcting cold partitions ( #5456 )
...
* feat: file file_count_threshold for comapcting cold partitions to make it consistent with the hot case and help set up to avoid oom easier
* chore: remove unecessary commments
2022-08-23 20:12:21 +00:00
84acbd2fad
chore(deps): Bump dotenvy from 0.15.1 to 0.15.2 ( #5450 )
...
Bumps [dotenvy](https://github.com/allan2/dotenvy ) from 0.15.1 to 0.15.2.
- [Release notes](https://github.com/allan2/dotenvy/releases )
- [Changelog](https://github.com/allan2/dotenvy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/allan2/dotenvy/commits/v0.15.2 )
---
updated-dependencies:
- dependency-name: dotenvy
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-23 11:24:42 +00:00
f4443f0b3a
feat: import schema override ( #5420 )
...
* chore: struct for overrides of import schema conflicts
* chore: import schema override shouldn't support tags
* feat: import schema merge can take an override schema
* fix: schema override in test had superfluous tag
* chore: test for batch schema merge with override in import schema
* feat: import schema merge now takes override schema
2022-08-17 14:59:50 +00:00
7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 ( #5419 )
...
* chore: update object_store
* chore: update hakari config
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-17 13:44:48 +00:00
78665d3092
chore(deps): Bump once_cell from 1.13.0 to 1.13.1 ( #5413 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.13.0...v1.13.1 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 08:31:46 +00:00
10fee5535a
feat: import schema updates iox catalog ( #5385 )
...
* feat: import schema updates iox catalog
- renamed import/schema module to aggregate_tsm_schema to not conflic
with schema crate
- fetch schema from iox catalog, and validate/merge/create as needed
chore: add catalog dsn config to import schema command
chore: import schema command connects to catalog
chore: import schema merge validation errors return non-zero code
chore: simplified and tidies import update catalog code
chore: tests and refactoring of import schema catalog update
* chore: require retention on ns creation in import
* chore: fixed bad test in import schema validation
* chore: friendlier errors & more tests in import schema catalog update
2022-08-16 11:05:27 +00:00
549a267e3c
fix: Use Self instead of unnecessary structure name repetition
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#use_self
2022-08-11 15:21:02 -04:00
b982bdaf2f
fix: Derive Eq when we derive PartialEq and members can derive Eq
...
Allow this in generated code that we don't control, though.
Recommended by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq
2022-08-11 15:04:06 -04:00
ae6ac27960
chore(deps): Bump console-subscriber from 0.1.6 to 0.1.7 ( #5374 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.6 to 0.1.7.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.6...tokio-console-v0.1.7 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-11 11:08:10 +00:00
9321c96aaf
test: Add a step for running compaction in e2e tests
2022-08-10 11:30:22 -04:00
3915841a53
feat: Introduce a separate config for the compactor command
2022-08-10 11:30:21 -04:00
7787c51b57
feat: add new CLI command to run the compactor once
2022-08-10 11:28:51 -04:00
463a13b814
test: Remove the compactor from the test MiniCluster
2022-08-10 11:28:51 -04:00
45f8e567ed
fix: Revert adjustment of e2e test to expect compaction
...
This was part of
"feat: Different branch to hook up new compaction algorithm (#5194 )"
and will be added back in a new test for compaction specifically in the
next commit.
This reverts part of commit 69640c0ba5
2022-08-10 11:28:50 -04:00
7e9918f067
chore: import validate merged schema ( #5367 )
...
* feat: import schema merge now outputs validation results
chore: refactor import crate
chore: renamed some structs for clarity in import crate
* chore: tests for import schema merge validation
* chore: Run cargo hakari tasks
* chore: clippy
* chore: make hashmap loop easier to read in import schema validation
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 12:15:37 +00:00
c5f062bba0
feat: initial commit of schema merge bulk import tool ( #5344 )
...
* feat: initial commit of schema merge bulk import tool
* chore: use observability depds instead of tracing-*
* chore: removed debug printlns
* chore: fix feature decls for cloud providers for import crate
* chore: use println instead of info in import- no need for a simple CLI
* chore: tidy whitespace
* chore: remove unused dep in import
* chore: Run cargo hakari tasks
* chore: removed unimpld import job subcommand
* chore: clarifying comment about custom serialisation code
* chore: clarifying comment about schema merge code in import
* chore: fix wrong comment in import command
* chore: bump object store dep to get bugfix
* chore: rename import schema struct for clarity
* chore: run `cargo hakari generate`
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:07:38 +00:00
16ddc5efc6
chore: Update datafusion / arrow/parquet/arrow-flight and prost/tonic ecosystem ( #5360 )
...
* chore: Update datafusion and arrow
* chore: Update Cargo.lock
* chore: update to Decimal128
* chore: Update tonic/prost/pbjson/etc
* chore: Run cargo hakari tasks
* fix: doctest in generated types
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 17:30:44 +00:00
da0b031c44
feat: Add parameters to limit total memory usage of cold partition compaction
2022-08-04 16:55:48 -04:00
d55f45a5c2
feat: Run compaction of hot partitions a configurable number of times more than cold
2022-08-04 16:55:48 -04:00
e8231b2986
chore(deps): Bump serde_json from 1.0.82 to 1.0.83 ( #5297 )
...
* chore(deps): Bump serde_json from 1.0.82 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.82 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.82...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 14:28:29 +00:00
b3958321d3
Merge branch 'main' into crepererum/issue5272
2022-08-03 21:06:15 +00:00
55e1e2ec2b
chore(deps): Bump thiserror from 1.0.31 to 1.0.32 ( #5294 )
...
* chore(deps): Bump thiserror from 1.0.31 to 1.0.32
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.31 to 1.0.32.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.31...1.0.32 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 16:20:36 +00:00
4bd8977d55
refactor: add some main function debug logs
2022-08-03 18:00:28 +02:00
840e4801b8
feat: make querier RAM pool split a proper feature ( #5283 )
...
* feat: make querier RAM pool split a proper feature
- use propre pool names
- expose sizing via CLI/env
Closes https://github.com/influxdata/conductor/issues/1102 .
* refactor: improve naming and docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:27:23 +00:00
663a20d743
refactor: remove `--ingster-address` ( #5255 )
...
Closes #5002 .
2022-08-03 15:05:01 +00:00
273b3cc165
chore: replace `dotenv` with `dotenvy` ( #5285 )
...
The latter one is a maintained fork. This avoids having both crates
after #5282 .
2022-08-03 12:41:38 +00:00
9215a534d0
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0` ( #5229 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0`
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 08:10:47 +00:00
9a9a1a4777
feat: limit per-table chunk data for every query ( #5223 )
...
* feat: `QueryChunk::as_any`
* feat: allo `ChunkPruner::prune_chunks` to fail
* feat: limit per-table chunk data for every query
Closes #5211 .
* fix: address review comments
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-27 13:20:05 +00:00
e4dc8c2067
refactor: rename garbage collector crates for consistency ( #5196 )
...
* refactor: rename garbage collector crates for consistency
* fix: cargo fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
69640c0ba5
feat: Different branch to hook up new compaction algorithm ( #5194 )
...
* chore: cherry pick the first 3 commits of branch cn/connect-new-compaction
* fix: modify the test to work correctly with compactor running
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 19:29:47 +00:00
5a0af921c8
chore: Roll forward: Sync ReadWindowAggregate API: TagKeyMetaNames ( #5186 )
...
This reverts commit 5d02c755687ef041f5f45dbfc3e633a833284edb.
2022-07-22 10:44:06 +00:00
07cdb99192
chore: Revert "Sync ReadWindowAggregate API: TagKeyMetaNames" ( #5184 )
...
We're noticing a possible regression (OOMs) in our testing cluster that roughly correlates with this.
2022-07-22 09:26:42 +00:00
69cb3f2b19
refactor: remove min_sequence_number from Compactor and Querier, add `count_by_overlaps_with_level_0` and `count_by_overlaps_with_level_1` to catalog ( #5151 )
...
* refactor: remove min_sequnce_number
* fix: typos
* fix: remove min_sequencer_number from new files from merging main
* fix: add back throwing error if the compactor compacts files persisted by the ingester after the ingester sends max seq_num back to querier
* test: add test_compactor_collision back but modify the input to make it work woth new changes
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-21 13:51:54 +00:00
21d033eafd
fix: Sync ReadWindowAggregate API: TagKeyMetaNames
...
The storage API has been updated in https://github.com/influxdata/idpe/pull/12868
in January, but since we forked the `.proto` files we never noticed.
2022-07-21 15:07:04 +02:00
efe39033b3
fix: tab complete from hints in `sql` relp (itch scratch) ( #5155 )
...
* fix: tab complete from hints
* fix: remove default impl
* docs: Update influxdb_iox/src/commands/sql/repl.rs
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 11:41:01 +00:00
c20288f60e
fix: Add TagKeyMetaNamesCapability capability ( #5160 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:52:40 +00:00
278a7f91af
chore(deps): Bump bytes from 1.1.0 to 1.2.0 ( #5156 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:00:08 +00:00
b8236e2b9d
fix: Fix SeriesKey sort order for special _measurement and _field ( #5150 )
...
* fix: Fix SeriesKey sort order for special _measurement and _field
* fix: Update expected test output
* fix: Update more tests
* fix: Re-sort tag key when using binary encoding
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-20 08:45:17 +00:00
f7a0fd43d2
feat: make object store garbage collector into a long-running service ( #5135 )
...
* refactor: remove unused logging config
* chore: remove the object store garbage collector CLI tool
* refactor: accept an object store and catalog
* refactor: make Result type alias public like the error
* refactor: remove public modifier from modules
* refactor: allow shutting down the object store garbage collector
* feat: Introduce the object-store garbage collection server
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
07e10852a8
feat: Add an input file count threshold to the compactor settings
2022-07-18 15:41:17 -04:00
128833e7d9
fix: Change placeholder new_param to input_size_threshold_bytes
2022-07-18 15:16:43 -04:00
d62b1ed7ee
feat: Select a subset of parquet files for a partition to compact
...
Fixes #5120 .
2022-07-18 15:14:22 -04:00
4416f1ce37
fix: Remove max number of level 0 files configuration option
2022-07-18 15:08:16 -04:00
57c70fcec5
fix: Remove redundant 'compaction' naming from CompactorConfig fields
2022-07-18 15:03:33 -04:00
c8f4000f04
feat: Select compaction candidates ( #5131 )
...
* feat: initial implementation for selecting compaction candidates
* feat: 2 catalog functions to choose the most thorughput partitions to compact and the selecting candidate function itself
* test: tests for the new 2 queries
* feat: more tests and metrics for chooing compaction candidates
* chore: Apply self suggestions from self review
* chore: cleanup
* chore: fix doc comment
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: address review comments
* fix: get the right time provider for the tests
* refactor: remove the left over compaction_
* fix: typos
* fix: make the param name and env name consistent
* refactor: make relevant iSomething to uSomething
* fix: typo
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-07-18 18:05:13 +00:00
e2d871b00b
chore: Update datafusion and arrow/parquet/arrow-flight to `18.0.0` ( #5079 )
...
* chore: Update datafusion to 10.0.0, arrow/parquet/arrow-flight to 18
* chore: Run cargo hakari tasks
* fix: update cargo pin
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 15:01:03 +00:00
1eeee9809c
chore(deps): Bump rustyline from 9.1.2 to 10.0.0 ( #5139 )
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 9.1.2 to 10.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v9.1.2...v10.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-18 08:10:22 +00:00
9b67de2f43
chore(deps): Bump tokio from 1.19.2 to 1.20.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.19.2 to 1.20.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.19.2...tokio-1.20.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-14 01:21:43 +00:00
61c023139b
refactor: Switch compaction levels to an enum with values rather than separate consts
...
Bonuses:
- Type checking
- Validation
- Less casting
- Exhaustiveness checking
- Less use of the numerical value
2022-07-13 11:30:36 -04:00
5c5c964dfe
feat: config params for Compactor ( #5108 )
...
* feat: config params for Compactor
* refactor: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-13 13:50:07 +00:00
c46e1c6347
chore: Update datafusion + arrow/parquet/arrow-flight to `17.0.0` ( #5021 )
...
* fix: correct nullability declaration of system tables
* chore: Update datafusion and arrow/parquet/arrow-flight
* chore: Run cargo hakari tasks
* fix: Update tests
* fix: Update tests
* fix: predicate pruning
* fix: add some tests
* fix: query_functions
* fix: fix read_buffer test
* fix: fix clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-07 19:22:15 +00:00
6b4325642f
refactor: Rename inner_main to main as it's now public
2022-07-07 09:48:06 -04:00
9a681e75cc
fix: Box a large enum variant
2022-07-07 09:48:06 -04:00
70dd6009e8
fix: Integrate the object store garbage collector into the main binary
2022-07-07 09:48:06 -04:00
89f5091546
refactor: Don't require DSN for test config
...
This enables writing a test for all-in-one's ephemeral mode, which
currently isn't working
2022-07-06 11:00:29 -04:00
1de022136c
feat: add max desired file size config param ( #5025 )
...
* feat: add max desired file size config param
* fix: comment typos
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-05 15:32:45 +00:00
68eff79594
chore(deps): Bump once_cell from 1.12.0 to 1.13.0 ( #5033 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.12.0...v1.13.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-05 08:54:51 +00:00
be53716e4d
refactor: use IDs for `parquet_file.column_set` ( #4965 )
...
* feat: `ColumnRepo::list_by_table_id`
* refactor: use IDs for `parquet_file.column_set`
Closes #4959 .
* refactor: introduce `TableSchema::column_id_map`
2022-06-30 15:08:41 +00:00
9438570ba1
test: document how to run tests ( #4982 )
...
* test: document how to run tests
Also fix a few issues for local runs.
* docs: add back one-liner for running end to end tests
* docs: add comment for clap_blocks test
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* docs: add comment in influxdb_iox/tests/end_to_end_cases/cli.rs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-30 14:01:35 +00:00
3049479b78
feat: Implement new querier to ingester config design
2022-06-30 08:26:50 -04:00
59da2dccb8
feat: Assert if no ingester addresses are found
...
Temporarily support `--ingester-addresses` (and always return all
ingesters) so that this PR can be deployed during the switchover.
2022-06-30 08:22:47 -04:00
0e450deca8
feat: Support a sequencer being mapped to multiple ingesters
2022-06-30 08:22:47 -04:00
7965bda42f
fix: Accept JSON ingester/shard config as CLI param value or env var value
2022-06-30 08:22:47 -04:00
4e91121e29
feat: Allow specification of sequencer to ingester mappings in a JSON file
2022-06-30 08:22:46 -04:00
40a8525520
chore(deps): Bump serde_json from 1.0.81 to 1.0.82 ( #4992 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.81 to 1.0.82.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.82 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-30 09:54:08 +00:00
835e1c91c7
chore: update object_store to 0.3.0 ( #4707 )
...
* chore: update object_store to 0.3.0
* chore: review feedback
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-29 21:44:03 +00:00
cfcc4b8426
refactor: change level 1 to level 2 preparing for next design changes ( #4954 )
...
* refactor: change level 1 to level 2 preparing for next design changes
* fix: make level-2 consistent everywhere
* chore: remove unused comments
* refactor: change all the name level_1 to level_2 to completely replace 1 with 2 to amke everything consistent
* chore: add correspinding constants for the comapction levels in the comments
Co-authored-by: Dom <dom@itsallbroken.com>
2022-06-29 14:08:58 +00:00
bfddb032ce
docs: improve docs for `persist_partition_size_threshold_bytes` / `INFLUXDB_IOX_PERSIST_PARTITION_SIZE_THRESHOLD_BYTES` ( #4877 )
...
* docs: improve docs for `persist_partition_size_threshold_bytes` / `INFLUXDB_IOX_PERSIST_PARTITION_SIZE_THRESHOLD_BYTES`
* docs: improve comments about LifecycleConfig::partition_size_threshold
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-27 21:52:40 +00:00
c22aed4347
Merge branch 'main' into dom/schema-api
2022-06-27 21:34:07 +00:00
215f297162
refactor: parquet file metadata from catalog ( #4949 )
...
* refactor: remove `ParquetFileWithMetadata`
* refactor: remove `ParquetFileRepo::parquet_metadata`
* refactor: parquet file metadata from catalog
Closes #4124 .
2022-06-27 15:38:39 +00:00
75c425f375
refactor(schema-api): column data type enum
...
Previously the column data type was exposed using an internal i32 value.
This commit changes the Schema API to use a self-descriptive proto enum
for the column data type.
2022-06-27 16:14:49 +01:00
087dbd3eca
fix: fix heappy + update docs ( #4917 )
...
* docs: Update heap profiling documentation
* fix: fix heappy builds
* fix: do not run cli tests with heappy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-21 19:53:28 +00:00
c3912e34e9
refactor: store per-file column set in catalog ( #4908 )
...
* refactor: store per-file column set in catalog
Together with the table-wide schema and the partition-wide sort key, this should
be everything we need to read a parquet file directly into memory
without peeking any file-level metadata.
The querier will use this to directly load parquet files into the read
buffer.
**WARNING: This requires a catalog wipe!**
Ref #4124 .
* refactor: use proper `ColumnSet` type
2022-06-21 10:26:12 +00:00
0fbff981ec
chore(deps): Bump sqlx to 0.6.0 and uuid to 1 ( #4894 )
...
Closes #4889 .
Closes #4890 .
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-17 10:28:28 +00:00
66c7d95312
refactor: use new ingester<>querier wire protocol ( #4867 )
...
* refactor: use new ingester<>querier wire protocol
Use and document the new and more flexible ingester<>querier wire
protocol.
Note that the ingester does NOT stream the response data yet, but the
internal data structures would allow that. A follow-up change will
adjust the ingester code to stream the data.
Ref #4849 .
* fix: typos
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: clarify naming and public interface
* test: add schema assertion to `ingester_response_to_record_batches`
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-16 08:02:28 +00:00
e9cdaffe74
fix: Create querier sharder from catalog sequencer info
...
Panic if there are no sharders in the catalog.
2022-06-15 10:18:54 -04:00
874ef89daa
feat: Make specifying the write buffer, and thus getting a sharder, optional in querier
2022-06-15 10:01:45 -04:00
Carol (Nichols || Goulding)
dependabot[bot]
Nga Tran
Andrew Lamb
kodiakhq[bot]
Carol (Nichols || Goulding)
Marco Neumann
Marko Mikulicic
YIXIAO SHI
Luke Bond
Juul Christiaens
Dom
Jake Goulding
Sam Arnold
Raphael Taylor-Davies