2ec7764fdd
refactor: rename builder like predicate methods to be `with_` ( #4808 )
...
* refactor: rename builder like predicate methods to be `with_`
* fix: merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-09 11:26:03 +00:00
d8331e8679
fix: do not return 'readable' until a write is completely readable ( #4778 )
...
* fix: do not return readable until a write is completely readable
* docs: Add diagram with partially buffered write
* refactor: account for actively buffering during update rather than fixup
* fix: fixup
* fix: use checked_sub
Co-authored-by: Marco Neumann <marco@crepererum.net>
* fix: checked_sub calculation
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-06-09 11:15:15 +00:00
107e5f7284
docs: Add some docs about `StreamSplit` ( #4810 )
...
* docs: Add some docs about `StreamSplit`
* docs: fix struct name
2022-06-09 10:53:34 +00:00
5e4fcfaa4d
refactor: reduce mut usage in Predicate ( #4807 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-09 10:46:01 +00:00
2e3ba83795
refactor: expose `CacheGetStatus` (and improve tests) ( #4804 )
...
* refactor: expose `CacheGetStatus` (and improve tests)
- add a `CacheGetStatus` which tells the user if the request was a hit
or miss (or something inbetween)
- adapt some tests to use the status (only the tests where this could be
relevant)
- move the test suite from using `sleep` to proper barriers (more stable
under high load, more correct, potentially faster)
* refactor: improve `abort_and_wait` checks
* docs: typos and improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: `FutureExt2` -> `EnsurePendingExt`
* refactor: `Queried` -> `MissAlreadyLoading`
* docs: explain `abort_or_wait` more
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-09 07:32:46 +00:00
f34282be2c
fix: Do not run DataFusion optimizer pass twice ( #4809 )
...
* fix: Do not run DataFusion optimizer pass twice
* docs: improve docstring and logging
2022-06-08 21:01:22 +00:00
46de8d6cb3
refactor: remove redundant code in predicate ( #4805 )
2022-06-08 15:03:26 +00:00
afc1c12062
refactor: consolidate `PredicateBuilder` into `Predicate` ( #4799 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-08 12:21:24 +00:00
82f6696516
chore: remove some unused deps ( #4803 )
...
Found by `cargo udeps`.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-08 11:15:18 +00:00
317e9486df
refactor: make `Cache` a trait ( #4802 )
...
* refactor: make `Cache` a trait
To insert more high-level metrics (e.g. cache misses/hits) it would be
helpful if we could easily instrument the layer right above the cache
driver (that combines the backend and the loader). To do that without
polluting the types too much, let's introduce a trait that describes the
driver interface and that we could later wrap with intrumentation.
This also pulls out the test into a generic setup, similar to how this
is done for the cache storage backends.
This does NOT include any functionality changes.
* fix: typo
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-06-08 11:04:53 +00:00
5ff8749251
Merge pull request #4797 from influxdata/dom/ingester-streaming-compact
...
perf: streaming compaction in ingester
2022-06-07 14:49:29 +00:00
6d8b93e129
Merge branch 'main' into dom/ingester-streaming-compact
2022-06-07 14:43:24 +00:00
1a04573075
Merge pull request #4798 from influxdata/crepererum/querier_rb_metrics
...
feat: wire up RB metrics for querier chunks
2022-06-07 14:26:45 +00:00
f2be4b190f
Merge branch 'main' into crepererum/querier_rb_metrics
2022-06-07 14:21:02 +00:00
3ecb1ee056
chore(deps): Bump http from 0.2.7 to 0.2.8 ( #4796 )
...
Bumps [http](https://github.com/hyperium/http ) from 0.2.7 to 0.2.8.
- [Release notes](https://github.com/hyperium/http/releases )
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/http/compare/v0.2.7...v0.2.8 )
---
updated-dependencies:
- dependency-name: http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 13:35:01 +00:00
4509e3db57
feat: wire up RB metrics for querier chunks
2022-06-07 15:31:49 +02:00
5d98988c9f
chore(deps): Bump tokio from 1.19.1 to 1.19.2 ( #4795 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.19.1 to 1.19.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 12:37:55 +00:00
1fa4ef9bac
chore(deps): Bump tower-http from 0.3.3 to 0.3.4 ( #4793 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.3...tower-http-0.3.4 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 11:03:00 +00:00
1fc5596023
perf: streaming compaction in ingester
...
Reduces memory usage in the ingester during persist operations by
streaming the results of the snapshot merge/sort/dedupe directly to
the parquet file.
Prior to this commit the output of the compact was buffered in memory
before being wrote to the parquet file.
2022-06-07 12:01:26 +01:00
20e832a5d1
chore(deps): Bump sqlparser from 0.17.0 to 0.18.0 ( #4792 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.17.0...v0.18.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-07 10:00:53 +00:00
8e96a2721d
chore: Update datafusion (again) ( #4788 )
...
* chore: Update datafusion
* chore: Update imports
* refactor: update API usage
* refactor: clean up some uses of binary_expr
* fix: remove unused export
* fix: update explain output
* chore: update more explain tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-07 08:17:56 +00:00
b60e1be0cf
chore: remove irrelaevant comments ( #4791 )
2022-06-07 00:43:56 +00:00
3e89daa0d4
feat: compact all overlapped files no matter how large they are ( #4779 )
...
* feat: add an option to compact all overlapped files no matter how large they are
* chore: Apply suggestions from code review
* feat: always compact oerlapped files no matter how large they are
* chore: cleaup
2022-06-06 23:39:09 +00:00
04c685b3b7
chore(deps): Bump tokio-util from 0.7.2 to 0.7.3 ( #4784 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.2 to 0.7.3.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.2...tokio-util-0.7.3 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-06 14:46:27 +00:00
a1ea793e13
chore(deps): Bump tokio-stream from 0.1.8 to 0.1.9 ( #4785 )
...
Bumps [tokio-stream](https://github.com/tokio-rs/tokio ) from 0.1.8 to 0.1.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.8...tokio-stream-0.1.9 )
---
updated-dependencies:
- dependency-name: tokio-stream
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:21:54 +00:00
e03bf94420
chore(deps): Bump tokio from 1.18.2 to 1.19.1 ( #4783 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.2...tokio-1.19.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-06 14:15:12 +00:00
bd59c89103
Merge pull request #4767 from influxdata/cn/read-buffer-cache
...
feat: Actually hook up read buffer chunk cache into querier
2022-06-06 12:58:39 +00:00
412309e7b1
Merge branch 'main' into cn/read-buffer-cache
2022-06-06 12:52:48 +00:00
bc107aff1d
Merge pull request #4780 from influxdata/cn/chunk-updating
...
docs: Remove outdated references to the old MUB -> RUB -> OS chunk cycle
2022-06-06 12:52:34 +00:00
6070421245
Merge branch 'main' into cn/chunk-updating
2022-06-06 12:46:30 +00:00
55f5d0ae11
docs: Improving Readability r2 ( #4781 )
...
* docs: Improve TOML readability
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: Fix `jemalloc` link
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: Readability in `point_generation.rs`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-06-04 16:46:57 +00:00
bfd537c853
docs: Remove comments referencing number of test scenarios created
...
These comments aren't near the code that affects how many scenarios get
created, so they were incorrect and are likely to be incorrect in
different ways in the future.
2022-06-03 16:29:30 -04:00
5c6c086d26
docs: Improve description of ChunkStage in query test scenarios
...
Namely, that ChunkStage::Parquet probably doesn't correspond to
ParquetChunk; it means the data has been persisted to parquet and the
chunks are now managed by the querier.
2022-06-03 16:29:30 -04:00
c6cb594a6d
test: There are no more MUB chunk types, remove that from test helper fn
2022-06-03 16:29:29 -04:00
e1061ce623
docs: Don't attempt to list out chunk types exhaustively
2022-06-03 16:29:29 -04:00
63b59f6470
test: Document current possibly-incorrect behavior in the test
2022-06-03 14:33:04 -04:00
c73011d4c4
docs: update logging ( #4766 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 17:54:45 +00:00
7daf680e76
test: Add nonexistent column not equal; this currently fails
2022-06-03 12:51:12 -04:00
5af0cc6acf
fix: Handle read buffer column not existing for column_names in QueryChunk impl
2022-06-03 12:45:16 -04:00
7b080ca9e1
fix: Revert read_buffer table handling of column not existing
...
And change the test to a case that might be incorrect, but it at least
captures the current behavior
2022-06-03 11:41:33 -04:00
2e752157b9
docs: Clean up docs ( #4764 )
...
* docs: Remove outdated instructions for running OG
* docs: clarify docs / readme
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 14:21:14 +00:00
aa510ae4e6
fix: Remove test uses of parquet chunks and document as unused
...
The querier is now using read buffer chunks only, but we're leaving the
parquet chunk code around for the moment.
2022-06-03 09:16:04 -04:00
d3df9db1a6
test: Validate SQL referencing nonexistent column returns an error
2022-06-03 09:16:04 -04:00
6b882db671
fix: Don't return any column names if predicate references a column that doesn't exist
2022-06-03 09:16:04 -04:00
c7f52ce362
fix: Return a different error type to distinguish different situations
2022-06-03 09:16:04 -04:00
a4f51d99f6
feat: Use the read buffer chunk cache in the querier
2022-06-03 09:16:04 -04:00
40d3a09296
docs: Add some comments to InstrumentedAsyncOwnedSemaphorePermit ( #4775 )
2022-06-03 11:08:16 +00:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
9a21292db8
chore(deps): Bump async-trait from 0.1.53 to 0.1.56 ( #4774 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.53 to 0.1.56.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.53...0.1.56 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-06-03 09:10:40 +00:00
73a7e6f0a5
chore(deps): Bump syn from 1.0.95 to 1.0.96 ( #4773 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.95 to 1.0.96.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.95...1.0.96 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-03 08:02:56 +00:00
Andrew Lamb
Marco Neumann
kodiakhq[bot]
dependabot[bot]
Dom Dwyer
Nga Tran
Ryan Russell
Carol (Nichols || Goulding)