e4dc8c2067
refactor: rename garbage collector crates for consistency ( #5196 )
...
* refactor: rename garbage collector crates for consistency
* fix: cargo fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
278a7f91af
chore(deps): Bump bytes from 1.1.0 to 1.2.0 ( #5156 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-20 10:00:08 +00:00
f7a0fd43d2
feat: make object store garbage collector into a long-running service ( #5135 )
...
* refactor: remove unused logging config
* chore: remove the object store garbage collector CLI tool
* refactor: accept an object store and catalog
* refactor: make Result type alias public like the error
* refactor: remove public modifier from modules
* refactor: allow shutting down the object store garbage collector
* feat: Introduce the object-store garbage collection server
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 21:27:38 +00:00
e2d871b00b
chore: Update datafusion and arrow/parquet/arrow-flight to `18.0.0` ( #5079 )
...
* chore: Update datafusion to 10.0.0, arrow/parquet/arrow-flight to 18
* chore: Run cargo hakari tasks
* fix: update cargo pin
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-18 15:01:03 +00:00
1eeee9809c
chore(deps): Bump rustyline from 9.1.2 to 10.0.0 ( #5139 )
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 9.1.2 to 10.0.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v9.1.2...v10.0.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-18 08:10:22 +00:00
9b67de2f43
chore(deps): Bump tokio from 1.19.2 to 1.20.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.19.2 to 1.20.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.19.2...tokio-1.20.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-07-14 01:21:43 +00:00
c46e1c6347
chore: Update datafusion + arrow/parquet/arrow-flight to `17.0.0` ( #5021 )
...
* fix: correct nullability declaration of system tables
* chore: Update datafusion and arrow/parquet/arrow-flight
* chore: Run cargo hakari tasks
* fix: Update tests
* fix: Update tests
* fix: predicate pruning
* fix: add some tests
* fix: query_functions
* fix: fix read_buffer test
* fix: fix clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-07 19:22:15 +00:00
70dd6009e8
fix: Integrate the object store garbage collector into the main binary
2022-07-07 09:48:06 -04:00
68eff79594
chore(deps): Bump once_cell from 1.12.0 to 1.13.0 ( #5033 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.12.0...v1.13.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-05 08:54:51 +00:00
40a8525520
chore(deps): Bump serde_json from 1.0.81 to 1.0.82 ( #4992 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.81 to 1.0.82.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.82 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-30 09:54:08 +00:00
835e1c91c7
chore: update object_store to 0.3.0 ( #4707 )
...
* chore: update object_store to 0.3.0
* chore: review feedback
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-29 21:44:03 +00:00
087dbd3eca
fix: fix heappy + update docs ( #4917 )
...
* docs: Update heap profiling documentation
* fix: fix heappy builds
* fix: do not run cli tests with heappy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-21 19:53:28 +00:00
0fbff981ec
chore(deps): Bump sqlx to 0.6.0 and uuid to 1 ( #4894 )
...
Closes #4889 .
Closes #4890 .
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-17 10:28:28 +00:00
127467b5c4
feat: Create a sharder in the querier
2022-06-15 10:01:45 -04:00
e91d00b10c
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `16.0.0 ( #4851 )
...
* chore: TEMP Update DataFusion to pre-release
* chore: update arrow et al to 16.0.0
* chore: Run cargo hakari tasks
* fix: update reader read_dictionary API
* chore: Update to real Datafusion release
* fix: Update parquet API
* fix: update test
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-06-14 16:31:40 +00:00
3ecb1ee056
chore(deps): Bump http from 0.2.7 to 0.2.8 ( #4796 )
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.7...v0.2.8 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 13:35:01 +00:00
04c685b3b7
chore(deps): Bump tokio-util from 0.7.2 to 0.7.3 ( #4784 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.2...tokio-util-0.7.3 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 14:46:27 +00:00
e03bf94420
chore(deps): Bump tokio from 1.18.2 to 1.19.1 ( #4783 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.2...tokio-1.19.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:15:12 +00:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
ca49820a0f
chore(deps): Bump console-subscriber from 0.1.5 to 0.1.6 ( #4670 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.5 to 0.1.6.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.5...console-subscriber-v0.1.6 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-24 08:24:12 +00:00
76f7043417
chore(deps): Bump once_cell from 1.11.0 to 1.12.0 ( #4666 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.11.0 to 1.12.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.11.0...v1.12.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-24 08:14:03 +00:00
6bc0c74c7d
chore(deps): Bump once_cell from 1.10.0 to 1.11.0 ( #4646 )
...
* chore(deps): Bump once_cell from 1.10.0 to 1.11.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.10.0...v1.11.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-20 07:40:38 +00:00
52346642a0
ci: fix cargo deny ( #4629 )
...
* ci: fix cargo deny
* chore: downgrade `socket2`, version 0.4.5 was yanked
* chore: rename `query` to `iox_query`
`query` is already taken on crates.io and yanked and I am getting tired
of working around that.
2022-05-18 09:38:35 +00:00
3a33e806c7
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `14.0.0` ( #4619 )
...
* chore: Update datafusion deps
* chore: update arrow/parquet/arrow flight deps
* chore: Run cargo hakari tasks
* chore: Update location of utils
* chore: Update some more APIs
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-17 14:13:03 +00:00
259d2486c1
chore(deps): Bump tokio-util from 0.7.1 to 0.7.2 ( #4605 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.1...tokio-util-0.7.2 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-16 11:42:31 +00:00
f2bb0fdf77
feat: update to crates.io object_store version ( #4595 )
...
* feat: update to crates.io object_store version
* chore: Run cargo hakari tasks
* fix: tests
* chore: remove object store integration test plumbing
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-13 16:26:07 +00:00
48e6e5713d
fix: Rename test_helpers_end_to_end_ng to test_helpers_end_to_end
...
Connects to #4450 .
2022-05-12 16:09:07 -04:00
381ad3b81d
chore: Update heappy
2022-05-11 09:49:10 -04:00
84d60ce56e
fix: feature flags ( #4550 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-10 13:42:51 +00:00
99b1a9b83f
refactor: split out ObjectStoreMetrics ( #4547 )
...
* refactor: split out ObjectStoreMetrics
* chore: add workspace hack
* fix: compile
2022-05-10 10:56:28 +00:00
1759443a13
fix: Remove unused dependencies in influxdb_iox found by manual inspection
2022-05-06 14:51:54 -04:00
fcd4815645
fix: Rename router2 to router
2022-05-06 14:51:52 -04:00
0650a9bb77
fix: Rename ioxd_router2 to ioxd_router
2022-05-06 14:45:39 -04:00
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
0541c6e40f
fix: Remove data_types crate where it's no longer used
2022-05-06 14:45:39 -04:00
ea46830954
fix: Remove iox_object_store crate; move ParquetFilePath to parquet_file
2022-05-06 14:45:36 -04:00
f8bdb022bc
fix: Remove job_registry crate
2022-05-06 11:35:11 -04:00
b88d071ce7
fix: Remove server
2022-05-06 11:30:36 -04:00
e0bc1801ac
fix: Remove router
2022-05-06 11:30:36 -04:00
cae32209da
fix: Remove parquet_catalog
2022-05-06 11:30:26 -04:00
2d8656e2e1
fix: Remove mutable_buffer
2022-05-06 11:27:33 -04:00
e7de16732d
fix: Remove internal_types
2022-05-06 11:27:33 -04:00
7286b4391a
fix: Remove db crate
2022-05-06 11:27:33 -04:00
c765455c73
feat: Remove the `server` command
2022-05-06 09:48:30 -04:00
ece38417bc
feat: Remove the `run database` command
2022-05-06 09:48:30 -04:00
b939d80db8
feat: Remove the `run router` command
2022-05-06 09:48:30 -04:00
02893e598c
chore: Update datafusion and upgrade arrow/parquet/arrow-flight to 13 ( #4516 )
...
* chore: Tool for automating arrow version update
* chore: Update datafusion and arrow/parquet/arrow-flight
* fix: update for changes in Arrow API
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-05 00:21:02 +00:00
590a71010d
chore(deps): Bump serde_json from 1.0.80 to 1.0.81 ( #4514 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.80 to 1.0.81.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.80...v1.0.81 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-04 14:10:35 +00:00
c05383bbf5
chore(deps): Bump serde_json from 1.0.79 to 1.0.80 ( #4501 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.79 to 1.0.80.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.79...v1.0.80 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-02 15:33:47 +00:00
e3adff2281
chore(deps): Bump console-subscriber from 0.1.4 to 0.1.5 ( #4503 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.4 to 0.1.5.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.4...console-subscriber-v0.1.5 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-02 14:19:11 +00:00
c59eaada5a
chore(deps): Bump thiserror from 1.0.30 to 1.0.31 ( #4498 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.30 to 1.0.31.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.30...1.0.31 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-02 13:43:45 +00:00
bb7910f689
fix: Remove influxdb_iox unused dev deps
2022-04-29 10:32:40 -04:00
01a6cf1c2a
chore(deps): Bump http from 0.2.6 to 0.2.7 ( #4470 )
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.6...v0.2.7 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-29 09:56:48 +00:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
2984578aac
chore: Update pprof, remove old versions of prost ( #4374 )
...
* chore: Upgrade pprof to 0.8
* chore: Update heappy
2022-04-21 10:52:23 +00:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
5ea676d3f7
feat: add per kafka partition durability reporting to write info response ( #4341 )
...
* feat: add per kafka partition durability reporting to write info response
* fix: buf lint + test cleanup
* fix: clean up protobuf
* refactor: pull out conversion of KafkaPartitionStatus into a function
* fix: fmt
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:46:20 +00:00
f08c8373aa
feat: tokio-console support
...
This commit adds an optional feature (disabled by default) to the IOx
server binary named "tokio_console". When enabled, it adds support for
the tokio-console to IOx:
https://github.com/tokio-rs/console
Enabling this feature drops the compile-time log filter to TRACE, and
enables the necessary dependencies to support the instrumentation needed
for the console to function.
Unfortunately, this feature uses tonic 0.7 (latest) while we use tonic
0.6, so we wind up with two tonic versions being compiled in when this
feature is enabled.
2022-04-12 13:02:31 +01:00
2cdd1951d9
fix: fix pprof ( #4261 )
2022-04-08 17:46:01 +00:00
edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` ( #4210 )
...
* refactor: Extract test, compactor, ingester, and test
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-03 10:42:22 +00:00
d37af1a7f5
fix: include git sha (again) in release build ( #4193 )
...
* fix: error if git-sha can not be found
* refactor: move main to influxdb_iox
* fix: fmt
2022-03-31 19:14:21 +00:00
532d227d11
refactor: extract router2 into ioxd_router2 ( #4183 )
...
* refactor: extract router2 from ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-31 17:39:05 +00:00
367e926d35
refactor: extract querier into ioxd_querier ( #4182 )
...
* refactor: extract querier into ioxd_querier
* fix: dep
2022-03-31 16:03:31 +00:00
f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` ( #4162 )
...
* refactor: rename influxdb_ioxd to ioxd
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:31:27 +00:00
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 ( #4004 )
...
* chore(deps): Bump pprof from 0.6.2 to 0.7.0
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: `pprof` features for version 0.7
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 11:08:59 +00:00
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
aaec1c7828
build: propagate feature flags to influxdb_ioxd
...
When you did a "cargo build --no-default-features" before this commit,
the influxdb_iox crate was compiled without the default features
(namely jemalloc stuff) but the influxdb_ioxd crate was compiled with
jemalloc (it was not possible to compile without jemalloc support).
This commit fixes propagation of the jemalloc flag
("jemalloc_replacing_malloc") into the influxdb_ioxd crate.
2022-03-07 15:44:13 +00:00
48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 ( #3955 )
...
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-07 08:54:20 +00:00
d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate ( #3908 )
...
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox
split out serving readiness, get compiling
* fix: hakari
* fix: hakari again
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
04a7a957fe
Merge branch 'main' into dom/rustls
2022-03-03 11:59:40 +00:00
c0ebe5a2f1
Merge branch 'main' into dom/fix-feature-flags
2022-03-03 11:48:35 +00:00
6204a9046d
fix: jemalloc feature flag
...
In #2435 we moved the jemalloc metrics around, and missed a few feature
gates to avoid pulling in jemalloc unless it is requested.
2022-03-03 11:48:24 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
b63f920d4c
chore(deps): Bump parquet from 9.0.2 to 9.1.0 ( #3828 )
...
* chore(deps): Bump parquet from 9.0.2 to 9.1.0
Bumps [parquet](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update chunk size test
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:25:15 +00:00
5a79b3a68b
chore(deps): Bump arrow-flight from 9.0.2 to 9.1.0 ( #3829 )
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:03:22 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
d0965bb0b2
Merge branch 'main' into dom/mb-partitioning
2022-02-16 11:30:42 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
e99922d518
refactor: parametrise DML handler input type
...
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
2022-02-15 11:23:45 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
d328b37803
feat: teach IOx to convert RPC frames into Recordbatches
2022-02-04 18:34:54 +00:00
ea0ece8b4b
feat: issue read_filter request
2022-02-04 18:34:54 +00:00
0bd7941a18
fix(REPL): Don't buffer lines until a trailing semicolon is found and add history hinting ( #3630 )
...
* fix(REPL): Don't buffer lines until a trailing semicolon is found
The repl would silently buffer all lines until a trailing semicolon were found which
resulted in some very confusing error messages as I would input invalid commands followed
by a command I thought were valid, except I'd still get an error due to the previous command being buffered.
This uses rustyline's helper feature to detect incomplete input (no trailing semicolon) and makes
it accept multiline input until the input is completed.
I also included some of rustyline's default hint and highlighting while I was at it.
* chore: cargo clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 17:11:01 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
ce46bbaada
feat: wire up the write buffer to the ingester process ( #3533 )
...
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.
I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.
There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
5441682207
feat: add support for parsing predicate
2022-02-02 11:02:33 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
Andrew Lamb
dependabot[bot]
Jake Goulding
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Marco Neumann
二手掉包工程师
Dom Dwyer
Nga Tran
kodiakhq[bot]
Luke Bond
Carol (Nichols || Goulding)
Edd Robinson
Markus Westerlind
Paul Dix