4509e3db57
feat: wire up RB metrics for querier chunks
2022-06-07 15:31:49 +02:00
5d98988c9f
chore(deps): Bump tokio from 1.19.1 to 1.19.2 ( #4795 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.19.1 to 1.19.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 12:37:55 +00:00
1fa4ef9bac
chore(deps): Bump tower-http from 0.3.3 to 0.3.4 ( #4793 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.3...tower-http-0.3.4 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 11:03:00 +00:00
20e832a5d1
chore(deps): Bump sqlparser from 0.17.0 to 0.18.0 ( #4792 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.17.0...v0.18.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 10:00:53 +00:00
8e96a2721d
chore: Update datafusion (again) ( #4788 )
...
* chore: Update datafusion
* chore: Update imports
* refactor: update API usage
* refactor: clean up some uses of binary_expr
* fix: remove unused export
* fix: update explain output
* chore: update more explain tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-07 08:17:56 +00:00
b60e1be0cf
chore: remove irrelaevant comments ( #4791 )
2022-06-07 00:43:56 +00:00
3e89daa0d4
feat: compact all overlapped files no matter how large they are ( #4779 )
...
* feat: add an option to compact all overlapped files no matter how large they are
* chore: Apply suggestions from code review
* feat: always compact oerlapped files no matter how large they are
* chore: cleaup
2022-06-06 23:39:09 +00:00
04c685b3b7
chore(deps): Bump tokio-util from 0.7.2 to 0.7.3 ( #4784 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.2...tokio-util-0.7.3 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 14:46:27 +00:00
a1ea793e13
chore(deps): Bump tokio-stream from 0.1.8 to 0.1.9 ( #4785 )
...
Bumps [tokio-stream](https://github.com/tokio-rs/tokio ) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.8...tokio-stream-0.1.9 )
---
updated-dependencies:
- dependency-name: tokio-stream
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:21:54 +00:00
e03bf94420
chore(deps): Bump tokio from 1.18.2 to 1.19.1 ( #4783 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.2...tokio-1.19.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:15:12 +00:00
bd59c89103
Merge pull request #4767 from influxdata/cn/read-buffer-cache
...
feat: Actually hook up read buffer chunk cache into querier
2022-06-06 12:58:39 +00:00
412309e7b1
Merge branch 'main' into cn/read-buffer-cache
2022-06-06 12:52:48 +00:00
bc107aff1d
Merge pull request #4780 from influxdata/cn/chunk-updating
...
docs: Remove outdated references to the old MUB -> RUB -> OS chunk cycle
2022-06-06 12:52:34 +00:00
6070421245
Merge branch 'main' into cn/chunk-updating
2022-06-06 12:46:30 +00:00
55f5d0ae11
docs: Improving Readability r2 ( #4781 )
...
* docs: Improve TOML readability
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: Fix `jemalloc` link
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: Readability in `point_generation.rs`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-06-04 16:46:57 +00:00
bfd537c853
docs: Remove comments referencing number of test scenarios created
...
These comments aren't near the code that affects how many scenarios get
created, so they were incorrect and are likely to be incorrect in
different ways in the future.
2022-06-03 16:29:30 -04:00
5c6c086d26
docs: Improve description of ChunkStage in query test scenarios
...
Namely, that ChunkStage::Parquet probably doesn't correspond to
ParquetChunk; it means the data has been persisted to parquet and the
chunks are now managed by the querier.
2022-06-03 16:29:30 -04:00
c6cb594a6d
test: There are no more MUB chunk types, remove that from test helper fn
2022-06-03 16:29:29 -04:00
e1061ce623
docs: Don't attempt to list out chunk types exhaustively
2022-06-03 16:29:29 -04:00
63b59f6470
test: Document current possibly-incorrect behavior in the test
2022-06-03 14:33:04 -04:00
c73011d4c4
docs: update logging ( #4766 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 17:54:45 +00:00
7daf680e76
test: Add nonexistent column not equal; this currently fails
2022-06-03 12:51:12 -04:00
5af0cc6acf
fix: Handle read buffer column not existing for column_names in QueryChunk impl
2022-06-03 12:45:16 -04:00
7b080ca9e1
fix: Revert read_buffer table handling of column not existing
...
And change the test to a case that might be incorrect, but it at least
captures the current behavior
2022-06-03 11:41:33 -04:00
2e752157b9
docs: Clean up docs ( #4764 )
...
* docs: Remove outdated instructions for running OG
* docs: clarify docs / readme
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 14:21:14 +00:00
aa510ae4e6
fix: Remove test uses of parquet chunks and document as unused
...
The querier is now using read buffer chunks only, but we're leaving the
parquet chunk code around for the moment.
2022-06-03 09:16:04 -04:00
d3df9db1a6
test: Validate SQL referencing nonexistent column returns an error
2022-06-03 09:16:04 -04:00
6b882db671
fix: Don't return any column names if predicate references a column that doesn't exist
2022-06-03 09:16:04 -04:00
c7f52ce362
fix: Return a different error type to distinguish different situations
2022-06-03 09:16:04 -04:00
a4f51d99f6
feat: Use the read buffer chunk cache in the querier
2022-06-03 09:16:04 -04:00
40d3a09296
docs: Add some comments to InstrumentedAsyncOwnedSemaphorePermit ( #4775 )
2022-06-03 11:08:16 +00:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
9a21292db8
chore(deps): Bump async-trait from 0.1.53 to 0.1.56 ( #4774 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.53 to 0.1.56.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.53...0.1.56 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-03 09:10:40 +00:00
73a7e6f0a5
chore(deps): Bump syn from 1.0.95 to 1.0.96 ( #4773 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.95 to 1.0.96.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.95...1.0.96 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 08:02:56 +00:00
f7cbd5d490
test: query limits ( #4769 )
...
* test: query limits
This was left out of #4760 .
* test: additional debugging
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-03 07:30:30 +00:00
81730fd0ff
feat: add owned versions of instrumented semaphores ( #4770 )
...
Owned versions will be required to instrument the query concurrency
limiter.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 07:20:54 +00:00
d279deddad
docs(various): Improve Readability ( #4768 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-02 18:01:06 +00:00
79895b995c
chore: add debug info to see how many concurrent partitions being compacted in each cycle ( #4772 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-02 15:19:08 +00:00
1f87dde95f
Merge pull request #4722 from influxdata/cn/convert
...
feat: Cache read buffer chunks
2022-06-02 14:15:46 +00:00
9d9c5d3692
fix: Take backoff config as an argument to be consistent with the other caches
2022-06-02 09:50:48 -04:00
76b40ac6a1
refactor: Make the type alias into a struct
2022-06-02 09:26:11 -04:00
715c65dfef
docs: Clarify a comment about what is an Arc
2022-06-02 09:22:44 -04:00
879dd7cec4
test: LRU behavior of the read buffer chunk cache
2022-06-02 09:22:44 -04:00
9328ba8c45
feat: Use new extra loading info to load read buffer chunks into cache
2022-06-02 09:22:44 -04:00
054c25de50
refactor: Add more methods to DecodedParquetFile
...
I'm tired of trying to remember which info is on which metadata.
2022-06-02 09:22:44 -04:00
9e30a3eb29
refactor: rework querier concurrency limiting ( #4760 )
...
* refactor: rework querier concurrency limiting
With #4752 we introduced a concurrency limit into the querier. It works
by drawing permits from a central semaphore whenever we create a
`QuerierNamespace`. This however only limits concurrency during query
planning and not query execution, because the objects contained within
the plan (chunks and some metadata) neither reference the permit nor the
`QuerierNamespace`.
Now one approach to fix that would be to wire up the permit all the down
into all the query-related data structures. This however is very fiddly
and potentially will get lost at some point, because as soon as we
transform these data structures -- e.g. into streams -- the permit might
get lost again. This will be potentially query-dependent and very hard
to debug.
So instead we reverse the approach and track the permits at the upper
layer of the stack: the gRPC service entry points. There we also need to
be careful -- e.g. when we return streams to tonic -- but it's way
easier to review that then the deeply nested object hierarchy that is
involved with queries. Also the separation of concerns is a bit clearer,
because why would a "chunk" care about the "query concurrency" as a
whole.
* refactor: improve gRPC permit keeping and prepare tests
2022-06-02 09:49:58 +00:00
1472ec272f
refactor: consolidate duplicate testing logic ( #4708 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-01 20:02:13 +00:00
a37c553545
refactor: Split up rpc_predicate module a bit ( #4763 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-01 19:56:11 +00:00
7328cc6a9a
docs: Update readme ( #4765 )
...
* docs: Update readme
* fix: Update README.md
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-01 19:50:06 +00:00
b714269b13
Merge pull request #4754 from influxdata/cn/extra-cache-system
...
feat: Add an Extra type to Cacher Loader to specify extra information…
2022-06-01 18:11:46 +00:00
Marco Neumann
dependabot[bot]
Andrew Lamb
Nga Tran
kodiakhq[bot]
Ryan Russell
Carol (Nichols || Goulding)