2a601c3099
fix: Revert "chore: Revert "fx: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )" ( #4327 )" ( #4328 )
...
* fix: Revert "chore: Revert "fx: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" (#4299 )" (#4303 )" (#4327 )"
This reverts commit 7e5d719027
2022-04-18 15:27:39 +00:00
b37a92307f
chore(deps): Bump clap from 3.1.8 to 3.1.9 ( #4340 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.8 to 3.1.9.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.8...v3.1.9 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:56:44 +00:00
053bcefaee
chore(deps): Bump sqlx from 0.5.12 to 0.5.13 ( #4339 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.12 to 0.5.13.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.12...v0.5.13 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-18 10:48:17 +00:00
52ffa4a482
chore(deps): Bump sqlx-core from 0.5.11 to 0.5.12 ( #4335 )
...
* chore(deps): Bump sqlx-core from 0.5.11 to 0.5.12
Bumps [sqlx-core](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:48:02 +00:00
db2903cd50
chore(deps): Bump sqlx-macros from 0.5.11 to 0.5.12 ( #4334 )
...
* chore(deps): Bump sqlx-macros from 0.5.11 to 0.5.12
Bumps [sqlx-macros](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx-macros
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:40:44 +00:00
5bf4550259
feat: add object store service to router ( #4338 )
...
Add method to catalog to get parquet file by object store id.
Add gRPC service for object store to get a file from by its uuid.
Add the object store service to router2 with object store config.
2022-04-16 17:58:31 +00:00
72a2a68eea
chore(deps): Bump sqlx from 0.5.11 to 0.5.12 ( #4333 )
...
* chore(deps): Bump sqlx from 0.5.11 to 0.5.12
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-15 15:03:08 +00:00
6ca40334ea
chore(deps): Bump toml from 0.5.8 to 0.5.9 ( #4332 )
...
Bumps [toml](https://github.com/alexcrichton/toml-rs ) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/alexcrichton/toml-rs/releases )
- [Commits](https://github.com/alexcrichton/toml-rs/compare/0.5.8...0.5.9 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-15 14:55:18 +00:00
f76cbb31ac
chore(deps): Bump tracing from 0.1.33 to 0.1.34 ( #4330 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.33 to 0.1.34.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.33...tracing-0.1.34 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-15 14:44:07 +00:00
197a3818d3
feat: add catalog client and remote command ( #4329 )
...
* feat: add catalog client and remote command
Adds the catalog gRPC service to influxdb_iox_client.
Adds a new remote command to execute commands against a remote IOx host.
Adds partition subcommand to remote to get the details of a partition by id.
* test: add end to end test for `remote partition` CLI (#4336 )
* chore: cleanup partition CLI PR feedback
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-15 13:59:49 +00:00
963086f5e9
Merge pull request #4320 from influxdata/crepererum/improve_ingester_grpc_testing
...
test: mock gRPC ingester response for querier
2022-04-14 21:21:12 +00:00
9df51e634e
Merge branch 'main' into crepererum/improve_ingester_grpc_testing
2022-04-14 21:13:18 +00:00
8e2d158a37
test: deadlock test and add more debug log ( #4319 )
...
* test: use Paul deadlock reproducer and add more debug log
* test: remove compare many output rows
* test: verify the test putput
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 18:06:22 +00:00
99cbb28a89
feat: add initial catalog service to router ( #4316 )
...
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.
The catalog service will be added to over time to provide access to the catalog over gRPC.
2022-04-14 17:39:18 +00:00
7e5d719027
chore: Revert "fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )" ( #4327 )
...
This reverts commit fe8d9948d5
2022-04-14 17:11:55 +00:00
f938d4d6e3
chore: Update datafusion (and point back to apache) ( #4322 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 15:51:06 +00:00
fe8d9948d5
fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )
...
This reverts commit 7ddbf7c025
2022-04-14 15:42:28 +00:00
fa00682945
Merge pull request #4325 from influxdata/dom/fix-compactor-deadlock
...
fix: compactor deadlock
2022-04-14 15:19:41 +00:00
31fdeaaabc
refactor: log split worker panics at error level
...
When the split background worker panics, it now causes an ERROR level
log to be emitted.
2022-04-14 15:39:35 +01:00
00b5c1b296
fix: compaction deadlock
...
This commit resolves the compaction deadlock described in #4306 .
The deadlock occurs during StreamSplitExec execution, where a background
worker is spawned to read input record batches and partition them into
two groups. This code pushes the resulting split record batches into two
channels - one for records that match a given predicate, and another
channel for those that do not. These channels buffer at most 2 record
batches each.
The compactor that executes this plan reads the resulting partitions
sequentially to completion. Completion is indicated by reading until the
results stream ends, which ends when the underlying channel is closed,
and therefore the split worker task must have finished and closed the
results channel for the partition to be successfully read.
While the compactor is reading from the first partition, the worker is
attempting to push record batches into the second partition and blocks
due to the channel capacity being reached. The worker never drops the
channel for the first partition, so the compactor never finishes reading
the first partition, and nothing is reading the second partition to
unblock the worker. Deadlock!
2022-04-14 15:39:35 +01:00
5ac4785e19
feat: unbounded channel support for AdaptorStream
...
Allows the AdaptorStream to work with either a bounded, or unbounded
channel internally.
2022-04-14 15:30:57 +01:00
de1241db85
test: mock gRPC ingester response for querier
...
Add infrastructure to test how the querier processes ingester gRPC
responses w/o performing a full query or end2end test.
2022-04-14 15:00:35 +02:00
351b0d0c15
fix: unknown namespace/table in querier<>ingester flight protocol ( #4307 )
...
* fix: return "not found" gRPC error instead of "internal" when ingester does not know table
* fix: properly handle "namespace not found" in ingester queries
* fix: make `initialize_db` work with async code
* test: add custom step for NG tests
* fix: handle "unknown table/namespace" resp. in querier
* docs: explain test setup
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-14 12:36:15 +00:00
136e052e33
Merge pull request #4321 from influxdata/er/test/read_buffer/transcoders
...
test: add property test coverage to Read Buffer transcoders
2022-04-14 11:05:39 +00:00
0a253abd57
chore: Run cargo hakari tasks
2022-04-14 10:42:13 +00:00
d7370de615
test: add property tests to RLE encoder
2022-04-14 11:40:33 +01:00
aa4ad56939
test: add test case
2022-04-14 11:35:19 +01:00
5ce11a37f8
test: add property tests for float byte trimmer
2022-04-14 11:35:19 +01:00
e548ba7b17
test: add property tests for byte trimmer
2022-04-14 11:35:19 +01:00
cf0d048037
refactor: rename helper
2022-04-14 11:35:19 +01:00
8bf2fbb7d3
fix: ingester min-unpersisted-sequence-number calc + doc ( #4302 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 07:05:06 +00:00
e97cec3f4d
chore: use Nga's DF branch with debug information for compactor ( #4314 )
2022-04-13 22:20:16 +00:00
3070d78e8c
chore: add more compactor debug info ( #4310 )
...
* chore: add more compactor debug info
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: fix format
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-13 19:22:19 +00:00
85f3e696e8
refactor: Use declarative steps to reduce duplication in end to end testing ( #4301 )
...
* refactor: Use declarative steps to reduce duplication in end to end testing,
* fix: improve whitespace formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-13 16:24:57 +00:00
2d75133ce8
Merge pull request #4308 from influxdata/crepererum/kube_071
...
chore: bump kube to version 0.71.0
2022-04-13 16:16:52 +00:00
58763bd575
Merge branch 'main' into crepererum/kube_071
2022-04-13 16:05:20 +00:00
06118b50be
Merge pull request #4304 from influxdata/crepererum/ng_end2end_tests_schema_names
...
test: use random catalog schema name for NG end2end tests
2022-04-13 16:05:05 +00:00
410b6d6db4
Merge branch 'main' into crepererum/ng_end2end_tests_schema_names
2022-04-13 15:54:28 +00:00
dcd56f89b1
Merge pull request #4305 from influxdata/crepererum/improve_ng_server_fixtures
...
test: improve NG server fixture
2022-04-13 15:54:05 +00:00
cb16348c5b
chore(deps): Bump kube-runtime from 0.70.0 to 0.71.0
...
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:48:08 +02:00
6e24d075a9
chore(deps): Bump kube from 0.70.0 to 0.71.0
...
Bumps [kube](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:47:21 +02:00
e7f89f336a
chore(deps): Bump kube-derive from 0.70.0 to 0.71.0
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:46:17 +02:00
94e0682d9f
Merge branch 'main' into crepererum/improve_ng_server_fixtures
2022-04-13 15:43:23 +00:00
3a78661ad7
chore: Run cargo hakari tasks
2022-04-13 15:39:26 +00:00
e46387db5f
Merge pull request #4262 from influxdata/cn/parquet-metadata-on-demand
...
feat: Store parquet_metadata in catalog, but don't fetch it by default
2022-04-13 15:38:09 +00:00
ab31f78172
test: improve server restarts
...
- Ensure that the restarted server uses the same log file as the old
one. This is important for debugging because we print out server logs
for failed tests.
- Try to be polite when shutting down or restarting servers (i.e. try to
use SIGTERM).
2022-04-13 17:33:09 +02:00
1b2ed9af9d
test: allow test framework to restart ingesters
2022-04-13 17:33:09 +02:00
ef1f252c6c
test: use random catalog schema name for NG end2end tests
...
Otherwise tests leak state to other tests or previous runs.
2022-04-13 17:24:42 +02:00
8c9b7b501b
fix: Update a signature to use ParquetFileWithMetadata
2022-04-13 11:09:06 -04:00
65b1a83419
fix: Use parquet_metadata less in compactor
...
Connects to #4124 .
The parquet metadata is still needed to create the ParquetChunk, but
iox_metadata isn't needed in QueryableParquetChunk.
2022-04-13 10:43:20 -04:00
Nga Tran
dependabot[bot]
Paul Dix
kodiakhq[bot]
Andrew Lamb
Carol (Nichols || Goulding)
Dom Dwyer
Marco Neumann
CircleCI[bot]
Edd Robinson
Carol (Nichols || Goulding)