d0903b11bb
refactor: reduce test duplication in `querier/src/table/mod.rs` ( #4698 )
...
* refactor: reduce test duplication in `querier/src/table/mod.rs`
* fix: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: Update querier/src/table/test_util.rs
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* fix: use now_nanos()
* refactor: Add TestQuerierTable
* refactor: rename functions for explicitness
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2022-05-30 12:56:09 +00:00
6af32b7750
feat: add concurrency limit for ingester queries ( #4703 )
...
I've defaulted it to 20, we can adjust as needed.
Closes #4657
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-30 10:22:17 +00:00
73168f7989
chore(deps): Bump flate2 from 1.0.23 to 1.0.24 ( #4726 )
...
Bumps [flate2](https://github.com/rust-lang/flate2-rs ) from 1.0.23 to 1.0.24.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases )
- [Commits](https://github.com/rust-lang/flate2-rs/commits )
---
updated-dependencies:
- dependency-name: flate2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-30 08:26:12 +00:00
29069be7d4
chore(deps): Bump hyper from 0.14.18 to 0.14.19 ( #4725 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.18 to 0.14.19.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.18...v0.14.19 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-30 07:58:12 +00:00
7d4670e171
chore(deps): Bump indexmap from 1.8.1 to 1.8.2 ( #4724 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/1.8.2/RELEASES.rst )
- [Commits](https://github.com/bluss/indexmap/compare/1.8.1...1.8.2 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-30 07:47:05 +00:00
cddd6d9b6d
chore: Update datafusion ( #4723 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-28 19:00:54 +00:00
b52a3586a7
fix: Turn cargo doc warnings into errors ( #4710 )
...
* fix: Correct intra-doc links
* fix: Turn cargo doc warnings into errors
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-28 11:24:22 +00:00
9f21512296
chore: reduce `debug!` log spew in `parquet_file` ( #4718 )
...
* chore: reduce log spew
* chore: trace another overly verbose message
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-27 20:57:10 +00:00
0a84727c72
Merge pull request #4709 from influxdata/cn/fetch-from-parquet-file
...
feat: Make a QuerierRBChunk wrapper that implement QueryChunk and QueryChunkMeta
2022-05-27 17:14:05 +00:00
842ef8e308
Merge branch 'main' into cn/fetch-from-parquet-file
2022-05-27 17:08:28 +00:00
55cd8d15be
fix: Update method name to specify the kind of chunk it makes
2022-05-27 13:04:24 -04:00
f0b4d71f47
docs: Update comment to reflect new implementation
2022-05-27 13:04:24 -04:00
5232594aab
docs: Fix grammar in a comment
...
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-05-27 13:04:13 -04:00
16e7a6d596
test: test that hits panic becasue of no column meta data ( #4719 )
...
* test: test that hits panic becasue of no column meta data
* chore: Apply suggestions from code review
* chore: run format after applying changes
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* chore: run clippy
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-05-27 15:27:03 +00:00
dde3c3922c
refactor: use consistent spelling of serialize ( #4717 )
2022-05-27 14:42:59 +00:00
ea81152fac
refactor: add partition ID into debug info and panic earlier to identify the bug easier ( #4716 )
...
* chore: point tests to the new ticket
* chore: cleanup
* refactor: add partition ID into debug info and panic earlier to identify the bug easier
2022-05-27 12:20:36 +00:00
09b55a209d
chore: point tests to the new ticket ( #4715 )
...
* chore: point tests to the new ticket
* chore: cleanup
2022-05-27 11:12:55 +00:00
372b262f37
test: parquet meta decoded tests and more debug info ( #4713 )
...
* test: reproducer for 4695
* chore: some debug info
* test: test with many columns and rows
* chore: cleanup and add debug info
* chore: cleanup
* chore: cleanup
* chore: more debug info
2022-05-27 09:53:07 +00:00
700a1de8f3
fix: fix at least one intermittent failure ( #4711 )
2022-05-26 21:24:37 +00:00
2cb351cd0d
feat: Make a QuerierRBChunk wrapper to handle traits and extra data
...
This brings back a bunch of code from OG from read buffer backed
DbChunks.
2022-05-26 16:52:14 -04:00
b2905650aa
refactor: Extract extract_range to be a method on TableSummary
...
So that other kinds of chunks can use this code too.
2022-05-26 16:52:14 -04:00
5fd3ffc17f
refactor: Rename ParquetChunkAdapter to only ChunkAdapter
...
It might be creating chunks of different kinds other than ParquetChunks.
2022-05-26 16:52:14 -04:00
633117e595
feat: avoid catalog access on each query ( #4650 )
...
* feat: cache catalog access on query
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-05-26 20:44:22 +00:00
05151d5c69
test: reproducer for 4695 ( #4706 )
...
* test: reproducer for 4695
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-26 15:32:30 +00:00
f645ec8a42
Merge pull request #4704 from influxdata/cn/welcome-back-read-buffer
...
feat: Start of a read buffer chunk cache
2022-05-26 13:53:29 +00:00
1043c98e17
Merge branch 'main' into cn/welcome-back-read-buffer
2022-05-26 13:47:27 +00:00
2d5a327bf4
fix: expire empty parquet_files cache and empty tombstones cache ( #4701 )
...
* fix: expire empty parquet_files cache
* fix: expire empty tombstones cache
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-26 11:08:15 +00:00
cddcca1e05
feat: Implement a method to get a read buffer chunk from a stream of record batches
2022-05-25 17:24:35 -04:00
f7bc551d9a
feat: Sketch out skeleton methods for RBChunk cache
2022-05-25 17:19:10 -04:00
04531e77dd
feat: Implement get on ReadBufferCache
2022-05-25 17:19:10 -04:00
25b8260b72
feat: Implement ReadBufferCache::new
2022-05-25 17:19:10 -04:00
ab9010d9a6
refactor: Rename QuerierParquetChunk::new_parquet to new
2022-05-25 17:19:10 -04:00
df10452e2e
refactor: Rename methods from new_querier_chunk to new_querier_parquet_chunk
2022-05-25 17:19:10 -04:00
4a90d0af32
refactor: Remove ChunkStorage enum; inline into QuerierParquetChunk instead
2022-05-25 17:19:10 -04:00
b2c62c6808
refactor: Rename QuerierChunk to QuerierParquetChunk
2022-05-25 17:19:10 -04:00
66823522f3
docs: Fix comment wrapping while reading through
2022-05-25 17:19:10 -04:00
6cc767efcc
feat: teach compactor to compact smaller number of files ( #4671 )
...
* refactor: split compact_partition into two functions to handle concurrency better
* feat: limit number of files to compact
* test: add test for limit num files
* chore: fix cipply
* feat: split group if over max size
* fix: split the overlapped group to limit size or file num
* chore: reduce config values
* test: add tests and clearer comments for the split_overlapped_groups and test_limit_size_and_num_files
* chore: more comments
* chore: cleanup
2022-05-25 19:54:34 +00:00
31d1b37d73
refactor: de-duplicate low-level arrow code ( #4697 )
...
It seems that during prototyping NG we've copied low level code (w/o
tests!) and never cleaned up. Let's not have this functionality twice.
2022-05-25 16:24:28 +00:00
9ddb0a816e
fix: Return panic message in internal error ( #4693 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-25 15:11:17 +00:00
eb815f2b30
Merge pull request #4664 from influxdata/cn/fix-metric-names
...
fix: Make Kafka Partition related metric names potentially less confusing
2022-05-25 14:33:58 +00:00
788e6eaf69
docs: Fix a comment that was very confused about what means kafka partition
2022-05-25 10:04:40 -04:00
6ce6a38094
fix: Make metric names potentially less confusing
2022-05-25 10:04:39 -04:00
a08a91c5ba
fix: ensure querier cache is refreshed for partition sort key ( #4660 )
...
* test: call `maybe_start_logging` in auto-generated cases
* fix: ensure querier cache is refreshed for partition sort key
Fixes #4631 .
* docs: explain querier sort key handling and test
* test: test another version of issue 4631
* fix: correctly invalidate partition sort keys
* fix: fix `table_not_found_on_ingester`
2022-05-25 10:44:42 +00:00
cdbe546e50
fix: return gRPC error on panic ( #4686 )
2022-05-25 07:06:25 +00:00
24ee251080
chore(deps): Bump prost from 0.10.3 to 0.10.4 ( #4688 )
...
Bumps [prost](https://github.com/tokio-rs/prost ) from 0.10.3 to 0.10.4.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.10.3...v0.10.4 )
---
updated-dependencies:
- dependency-name: prost
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-25 06:07:05 +00:00
a8d3fe619c
chore(deps): Bump prost-build from 0.10.3 to 0.10.4 ( #4687 )
...
Bumps [prost-build](https://github.com/tokio-rs/prost ) from 0.10.3 to 0.10.4.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.10.3...v0.10.4 )
---
updated-dependencies:
- dependency-name: prost-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-25 06:00:27 +00:00
935743b525
refactor: Implement `new_querier_chunk` and `new_querier_chunk_from_file_with_metadata` ( #4685 )
2022-05-24 21:58:27 +00:00
a8d5f7f5f7
test: add debug output to test ( #4684 )
2022-05-24 19:57:11 +00:00
95e6a8ed46
chore: Update datafusion (again) ( #4679 )
...
* chore: Update datafusion deps
* fix: fix for changes in ScalarValue
* fix: fix for using TableSource rather than TableProvider
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-24 15:54:39 +00:00
8c537c9467
Merge pull request #4682 from influxdata/dom/fix-build
...
build: remove iox_gitops_adapter from build
2022-05-24 15:37:31 +00:00
Andrew Lamb
Paul Dix
dependabot[bot]
Carol (Nichols || Goulding)
kodiakhq[bot]
Carol (Nichols || Goulding)
Nga Tran
Marco Neumann
Marko Mikulicic