7e5d719027
chore: Revert "fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )" ( #4327 )
...
This reverts commit fe8d9948d5
2022-04-14 17:11:55 +00:00
f938d4d6e3
chore: Update datafusion (and point back to apache) ( #4322 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 15:51:06 +00:00
fe8d9948d5
fix: Revert "fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )" ( #4303 )
...
This reverts commit 7ddbf7c025
2022-04-14 15:42:28 +00:00
fa00682945
Merge pull request #4325 from influxdata/dom/fix-compactor-deadlock
...
fix: compactor deadlock
2022-04-14 15:19:41 +00:00
31fdeaaabc
refactor: log split worker panics at error level
...
When the split background worker panics, it now causes an ERROR level
log to be emitted.
2022-04-14 15:39:35 +01:00
00b5c1b296
fix: compaction deadlock
...
This commit resolves the compaction deadlock described in #4306 .
The deadlock occurs during StreamSplitExec execution, where a background
worker is spawned to read input record batches and partition them into
two groups. This code pushes the resulting split record batches into two
channels - one for records that match a given predicate, and another
channel for those that do not. These channels buffer at most 2 record
batches each.
The compactor that executes this plan reads the resulting partitions
sequentially to completion. Completion is indicated by reading until the
results stream ends, which ends when the underlying channel is closed,
and therefore the split worker task must have finished and closed the
results channel for the partition to be successfully read.
While the compactor is reading from the first partition, the worker is
attempting to push record batches into the second partition and blocks
due to the channel capacity being reached. The worker never drops the
channel for the first partition, so the compactor never finishes reading
the first partition, and nothing is reading the second partition to
unblock the worker. Deadlock!
2022-04-14 15:39:35 +01:00
5ac4785e19
feat: unbounded channel support for AdaptorStream
...
Allows the AdaptorStream to work with either a bounded, or unbounded
channel internally.
2022-04-14 15:30:57 +01:00
351b0d0c15
fix: unknown namespace/table in querier<>ingester flight protocol ( #4307 )
...
* fix: return "not found" gRPC error instead of "internal" when ingester does not know table
* fix: properly handle "namespace not found" in ingester queries
* fix: make `initialize_db` work with async code
* test: add custom step for NG tests
* fix: handle "unknown table/namespace" resp. in querier
* docs: explain test setup
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-14 12:36:15 +00:00
136e052e33
Merge pull request #4321 from influxdata/er/test/read_buffer/transcoders
...
test: add property test coverage to Read Buffer transcoders
2022-04-14 11:05:39 +00:00
0a253abd57
chore: Run cargo hakari tasks
2022-04-14 10:42:13 +00:00
d7370de615
test: add property tests to RLE encoder
2022-04-14 11:40:33 +01:00
aa4ad56939
test: add test case
2022-04-14 11:35:19 +01:00
5ce11a37f8
test: add property tests for float byte trimmer
2022-04-14 11:35:19 +01:00
e548ba7b17
test: add property tests for byte trimmer
2022-04-14 11:35:19 +01:00
cf0d048037
refactor: rename helper
2022-04-14 11:35:19 +01:00
8bf2fbb7d3
fix: ingester min-unpersisted-sequence-number calc + doc ( #4302 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-14 07:05:06 +00:00
e97cec3f4d
chore: use Nga's DF branch with debug information for compactor ( #4314 )
2022-04-13 22:20:16 +00:00
3070d78e8c
chore: add more compactor debug info ( #4310 )
...
* chore: add more compactor debug info
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: fix format
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-13 19:22:19 +00:00
85f3e696e8
refactor: Use declarative steps to reduce duplication in end to end testing ( #4301 )
...
* refactor: Use declarative steps to reduce duplication in end to end testing,
* fix: improve whitespace formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-13 16:24:57 +00:00
2d75133ce8
Merge pull request #4308 from influxdata/crepererum/kube_071
...
chore: bump kube to version 0.71.0
2022-04-13 16:16:52 +00:00
58763bd575
Merge branch 'main' into crepererum/kube_071
2022-04-13 16:05:20 +00:00
06118b50be
Merge pull request #4304 from influxdata/crepererum/ng_end2end_tests_schema_names
...
test: use random catalog schema name for NG end2end tests
2022-04-13 16:05:05 +00:00
410b6d6db4
Merge branch 'main' into crepererum/ng_end2end_tests_schema_names
2022-04-13 15:54:28 +00:00
dcd56f89b1
Merge pull request #4305 from influxdata/crepererum/improve_ng_server_fixtures
...
test: improve NG server fixture
2022-04-13 15:54:05 +00:00
cb16348c5b
chore(deps): Bump kube-runtime from 0.70.0 to 0.71.0
...
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:48:08 +02:00
6e24d075a9
chore(deps): Bump kube from 0.70.0 to 0.71.0
...
Bumps [kube](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:47:21 +02:00
e7f89f336a
chore(deps): Bump kube-derive from 0.70.0 to 0.71.0
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.70.0 to 0.71.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.70.0...0.71.0 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-04-13 17:46:17 +02:00
94e0682d9f
Merge branch 'main' into crepererum/improve_ng_server_fixtures
2022-04-13 15:43:23 +00:00
3a78661ad7
chore: Run cargo hakari tasks
2022-04-13 15:39:26 +00:00
e46387db5f
Merge pull request #4262 from influxdata/cn/parquet-metadata-on-demand
...
feat: Store parquet_metadata in catalog, but don't fetch it by default
2022-04-13 15:38:09 +00:00
ab31f78172
test: improve server restarts
...
- Ensure that the restarted server uses the same log file as the old
one. This is important for debugging because we print out server logs
for failed tests.
- Try to be polite when shutting down or restarting servers (i.e. try to
use SIGTERM).
2022-04-13 17:33:09 +02:00
1b2ed9af9d
test: allow test framework to restart ingesters
2022-04-13 17:33:09 +02:00
ef1f252c6c
test: use random catalog schema name for NG end2end tests
...
Otherwise tests leak state to other tests or previous runs.
2022-04-13 17:24:42 +02:00
8c9b7b501b
fix: Update a signature to use ParquetFileWithMetadata
2022-04-13 11:09:06 -04:00
65b1a83419
fix: Use parquet_metadata less in compactor
...
Connects to #4124 .
The parquet metadata is still needed to create the ParquetChunk, but
iox_metadata isn't needed in QueryableParquetChunk.
2022-04-13 10:43:20 -04:00
94dcde4996
fix: Do fewer queries for metadata
...
By adding another _with_metadata catalog function. Also introduce a new
type rather than passing around tuples everywhere.
2022-04-13 10:43:20 -04:00
bba4251363
fix: Remove duplication in metric name
2022-04-13 10:43:19 -04:00
02fee3b84f
feat: Request parquet metadata from the catalog when needed only
2022-04-13 10:43:19 -04:00
ec25620b73
feat: Add a catalog method for requesting a parquet file's metadata
2022-04-13 10:43:19 -04:00
ee56ebf0e3
feat: Store metadata in catalog, but don't fetch by default
2022-04-13 10:43:19 -04:00
7ddbf7c025
fix: Revert "feat: Use the sort key stored in the catalog during compaction" ( #4299 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-13 14:11:10 +00:00
12800bd07e
Merge pull request #4300 from influxdata/crepererum/fix_querier_executor_shutdown
...
fix: proper executor shutdown in querier
2022-04-13 14:00:31 +00:00
f75d3b1f5d
fix: proper executor shutdown in querier
...
This is not a huge issue but might drain resources like file descriptors
during tests. The dedicated exuector also logs a warning.
2022-04-13 15:52:44 +02:00
d829db6e5d
Merge pull request #4254 from influxdata/cn/sort-in-compactor
...
feat: Use the sort key stored in the catalog during compaction
2022-04-13 12:51:45 +00:00
21f748062e
Merge branch 'main' into cn/sort-in-compactor
2022-04-13 12:43:31 +00:00
cb8a80dc47
chore(deps): Bump tracing-core from 0.1.24 to 0.1.25 ( #4293 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.24 to 0.1.25.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.24...tracing-core-0.1.25 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-13 10:39:10 +00:00
ec4f9e318e
chore(deps): Bump libc from 0.2.122 to 0.2.123 ( #4291 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.122 to 0.2.123.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.122...0.2.123 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-13 10:27:53 +00:00
bd4566cbe0
fix: Do not specify a querier address by default ( #4289 )
2022-04-12 20:52:18 +00:00
17e11d6139
Merge pull request #4287 from influxdata/dom/tracing-static-level
...
refactor: disable max_debug static tracing level
2022-04-12 17:09:07 +00:00
adfc05e741
Merge branch 'main' into dom/tracing-static-level
2022-04-12 17:01:53 +00:00
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)
kodiakhq[bot]
Dom Dwyer
Marco Neumann
CircleCI[bot]
Edd Robinson
dependabot[bot]
Carol (Nichols || Goulding)