383c0b328d
feat: Issue queries from querier to ingester in parallel ( #4359 )
...
* feat: Issue queries from querier to ingester in parallel
* refactor: complete Arc-ification
* refactor: use a named struct to pass the state
2022-04-20 10:55:14 +00:00
9cc7d73671
chore: Install custom protoc in perf-builder image ( #4358 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 10:35:05 +00:00
0642ec0b82
docs: add note about write_info API being internal ( #4356 )
...
* docs: add note about write_info API being internal
* fix: update doc urls
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 09:25:14 +00:00
ca464b5ab6
chore(deps): Bump clap from 3.1.9 to 3.1.10 ( #4361 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.9 to 3.1.10.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.9...v3.1.10 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-20 08:44:17 +00:00
dbddffee6f
chore: add semantic PR and commit message checks ( #4344 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 17:29:03 +00:00
5ea676d3f7
feat: add per kafka partition durability reporting to write info response ( #4341 )
...
* feat: add per kafka partition durability reporting to write info response
* fix: buf lint + test cleanup
* fix: clean up protobuf
* refactor: pull out conversion of KafkaPartitionStatus into a function
* fix: fmt
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:46:20 +00:00
7ac7fdcd5b
Merge pull request #4351 from influxdata/dom/revert-service-limits
...
fix: revert column service limits (#4179 )
2022-04-19 16:38:19 +00:00
752490fe34
Merge branch 'main' into dom/revert-service-limits
2022-04-19 16:30:09 +00:00
6088c1a588
refactor: rewrite schema command end to end tests to use new StepTest framework ( #4337 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:29:50 +00:00
103629b01d
feat: add client and CLI to get file from object store ( #4343 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 16:16:27 +00:00
f55319cdc2
Merge pull request #4353 from influxdata/dom/logfmt-quoting
...
fix(logfmt): quote strings containing '='
2022-04-19 16:08:32 +00:00
3788a6105e
Merge branch 'main' into dom/logfmt-quoting
2022-04-19 16:00:02 +00:00
d711816548
feat: add sequencer ID and correct partition key to `IngesterPartition` ( #4348 )
...
* feat: impl `Debug` for `TestCatalog`
* feat: add sequencer ID and correct partition key to `IngesterPartition`
- simplifies debugging (parquet chunks and ingester chunks now use the
same partition key naming)
- the sequencer ID is required to correctly reason about tombstones (to
be implemented in a later PR)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-19 15:59:49 +00:00
8ec9d0eac4
fix(logfmt): quote strings containing '='
...
Logfmt uses '=' as a key/value separator, and prior to this commit the
formatter did not escape strings containing '='.
2022-04-19 16:47:18 +01:00
320f1073e0
fix: revert column service limits ( #4179 )
...
This reverts commit ea865b63f4
2022-04-19 16:08:56 +01:00
e3d83fe757
chore: update datafusion ( #4342 )
...
* chore: update datafusion
* fix: Update imports for change in datafusion organization
2022-04-19 13:38:12 +00:00
3588a06647
feat: correctly dismantle ingester gRPC reponse in querier ( #4323 )
...
This now correctly processes record batches for the different
partitions. The actual code change is rather small but it requires some
substantial test infrastructure.
2022-04-19 11:09:40 +00:00
5b48675435
fix: actually transmit record-batch metadata from querier ( #4347 )
...
Attaching the "batch => partition" mapping via per-batch schema KV
metadata does NOT work because flight will transmit the schema once for
all batches (even though on the Rust side we have a schema ref attached
to every batch, probably for convenience). Instead we now use the same
global protobuf metadata that we also use for the "partition => max
sequence number" information. This somewhat limits our ability to create
record batches lazily on the ingester side (since the global metadata is
sent before any actual payload) but I think we should not modify the
usage of the flight protocol too much right now (e.g. by sending more
schema messages). If this becomes an issue, we can always find a more
complex solution in the future.
2022-04-19 10:54:23 +00:00
cfac332000
chore(deps): Bump backtrace from 0.3.64 to 0.3.65 ( #4346 )
...
Bumps [backtrace](https://github.com/rust-lang/backtrace-rs ) from 0.3.64 to 0.3.65.
- [Release notes](https://github.com/rust-lang/backtrace-rs/releases )
- [Commits](https://github.com/rust-lang/backtrace-rs/compare/0.3.64...0.3.65 )
---
updated-dependencies:
- dependency-name: backtrace
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-19 10:12:06 +00:00
3d5896d18d
chore(deps): Bump libc from 0.2.123 to 0.2.124 ( #4345 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.123 to 0.2.124.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.123...0.2.124 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-19 08:47:40 +00:00
9c3b145714
chore(deps): Bump tracing-core from 0.1.25 to 0.1.26 ( #4331 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.25 to 0.1.26.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.25...tracing-core-0.1.26 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 17:19:58 +00:00
2a601c3099
fix: Revert "chore: Revert "fx: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )" ( #4327 )" ( #4328 )
...
* fix: Revert "chore: Revert "fx: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" (#4299 )" (#4303 )" (#4327 )"
This reverts commit 7e5d719027
2022-04-18 15:27:39 +00:00
b37a92307f
chore(deps): Bump clap from 3.1.8 to 3.1.9 ( #4340 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.8 to 3.1.9.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.8...v3.1.9 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:56:44 +00:00
053bcefaee
chore(deps): Bump sqlx from 0.5.12 to 0.5.13 ( #4339 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.12 to 0.5.13.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.12...v0.5.13 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-18 10:48:17 +00:00
52ffa4a482
chore(deps): Bump sqlx-core from 0.5.11 to 0.5.12 ( #4335 )
...
* chore(deps): Bump sqlx-core from 0.5.11 to 0.5.12
Bumps [sqlx-core](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:48:02 +00:00
db2903cd50
chore(deps): Bump sqlx-macros from 0.5.11 to 0.5.12 ( #4334 )
...
* chore(deps): Bump sqlx-macros from 0.5.11 to 0.5.12
Bumps [sqlx-macros](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx-macros
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-18 10:40:44 +00:00
5bf4550259
feat: add object store service to router ( #4338 )
...
Add method to catalog to get parquet file by object store id.
Add gRPC service for object store to get a file from by its uuid.
Add the object store service to router2 with object store config.
2022-04-16 17:58:31 +00:00
72a2a68eea
chore(deps): Bump sqlx from 0.5.11 to 0.5.12 ( #4333 )
...
* chore(deps): Bump sqlx from 0.5.11 to 0.5.12
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.11...v0.5.12 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-15 15:03:08 +00:00
6ca40334ea
chore(deps): Bump toml from 0.5.8 to 0.5.9 ( #4332 )
...
Bumps [toml](https://github.com/alexcrichton/toml-rs ) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/alexcrichton/toml-rs/releases )
- [Commits](https://github.com/alexcrichton/toml-rs/compare/0.5.8...0.5.9 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-15 14:55:18 +00:00
f76cbb31ac
chore(deps): Bump tracing from 0.1.33 to 0.1.34 ( #4330 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.33 to 0.1.34.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.33...tracing-0.1.34 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-15 14:44:07 +00:00
197a3818d3
feat: add catalog client and remote command ( #4329 )
...
* feat: add catalog client and remote command
Adds the catalog gRPC service to influxdb_iox_client.
Adds a new remote command to execute commands against a remote IOx host.
Adds partition subcommand to remote to get the details of a partition by id.
* test: add end to end test for `remote partition` CLI (#4336 )
* chore: cleanup partition CLI PR feedback
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-15 13:59:49 +00:00
963086f5e9
Merge pull request #4320 from influxdata/crepererum/improve_ingester_grpc_testing
...
test: mock gRPC ingester response for querier
2022-04-14 21:21:12 +00:00
9df51e634e
Merge branch 'main' into crepererum/improve_ingester_grpc_testing
2022-04-14 21:13:18 +00:00
8e2d158a37
test: deadlock test and add more debug log ( #4319 )
...
* test: use Paul deadlock reproducer and add more debug log
* test: remove compare many output rows
* test: verify the test putput
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 18:06:22 +00:00
99cbb28a89
feat: add initial catalog service to router ( #4316 )
...
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.
The catalog service will be added to over time to provide access to the catalog over gRPC.
2022-04-14 17:39:18 +00:00
7e5d719027
chore: Revert "fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )" ( #4327 )
...
This reverts commit fe8d9948d5
2022-04-14 17:11:55 +00:00
f938d4d6e3
chore: Update datafusion (and point back to apache) ( #4322 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 15:51:06 +00:00
fe8d9948d5
fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )
...
This reverts commit 7ddbf7c025
2022-04-14 15:42:28 +00:00
fa00682945
Merge pull request #4325 from influxdata/dom/fix-compactor-deadlock
...
fix: compactor deadlock
2022-04-14 15:19:41 +00:00
31fdeaaabc
refactor: log split worker panics at error level
...
When the split background worker panics, it now causes an ERROR level
log to be emitted.
2022-04-14 15:39:35 +01:00
00b5c1b296
fix: compaction deadlock
...
This commit resolves the compaction deadlock described in #4306 .
The deadlock occurs during StreamSplitExec execution, where a background
worker is spawned to read input record batches and partition them into
two groups. This code pushes the resulting split record batches into two
channels - one for records that match a given predicate, and another
channel for those that do not. These channels buffer at most 2 record
batches each.
The compactor that executes this plan reads the resulting partitions
sequentially to completion. Completion is indicated by reading until the
results stream ends, which ends when the underlying channel is closed,
and therefore the split worker task must have finished and closed the
results channel for the partition to be successfully read.
While the compactor is reading from the first partition, the worker is
attempting to push record batches into the second partition and blocks
due to the channel capacity being reached. The worker never drops the
channel for the first partition, so the compactor never finishes reading
the first partition, and nothing is reading the second partition to
unblock the worker. Deadlock!
2022-04-14 15:39:35 +01:00
5ac4785e19
feat: unbounded channel support for AdaptorStream
...
Allows the AdaptorStream to work with either a bounded, or unbounded
channel internally.
2022-04-14 15:30:57 +01:00
de1241db85
test: mock gRPC ingester response for querier
...
Add infrastructure to test how the querier processes ingester gRPC
responses w/o performing a full query or end2end test.
2022-04-14 15:00:35 +02:00
351b0d0c15
fix: unknown namespace/table in querier<>ingester flight protocol ( #4307 )
...
* fix: return "not found" gRPC error instead of "internal" when ingester does not know table
* fix: properly handle "namespace not found" in ingester queries
* fix: make `initialize_db` work with async code
* test: add custom step for NG tests
* fix: handle "unknown table/namespace" resp. in querier
* docs: explain test setup
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-14 12:36:15 +00:00
136e052e33
Merge pull request #4321 from influxdata/er/test/read_buffer/transcoders
...
test: add property test coverage to Read Buffer transcoders
2022-04-14 11:05:39 +00:00
0a253abd57
chore: Run cargo hakari tasks
2022-04-14 10:42:13 +00:00
d7370de615
test: add property tests to RLE encoder
2022-04-14 11:40:33 +01:00
aa4ad56939
test: add test case
2022-04-14 11:35:19 +01:00
5ce11a37f8
test: add property tests for float byte trimmer
2022-04-14 11:35:19 +01:00
e548ba7b17
test: add property tests for byte trimmer
2022-04-14 11:35:19 +01:00
Andrew Lamb
dependabot[bot]
Jacob Marble
kodiakhq[bot]
Paul Dix
Marco Neumann
Dom Dwyer
Nga Tran
Carol (Nichols || Goulding)
CircleCI[bot]
Edd Robinson