5af0cc6acf
fix: Handle read buffer column not existing for column_names in QueryChunk impl
2022-06-03 12:45:16 -04:00
7b080ca9e1
fix: Revert read_buffer table handling of column not existing
...
And change the test to a case that might be incorrect, but it at least
captures the current behavior
2022-06-03 11:41:33 -04:00
6b882db671
fix: Don't return any column names if predicate references a column that doesn't exist
2022-06-03 09:16:04 -04:00
c7f52ce362
fix: Return a different error type to distinguish different situations
2022-06-03 09:16:04 -04:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
3a33e806c7
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `14.0.0` ( #4619 )
...
* chore: Update datafusion deps
* chore: update arrow/parquet/arrow flight deps
* chore: Run cargo hakari tasks
* chore: Update location of utils
* chore: Update some more APIs
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-05-17 14:13:03 +00:00
e07bcd40c2
refactor: Remove unused dependencies
...
These were found by iterating over all of the dependencies of each
Cargo.toml, then grepping that crate for the dependency's name. If it
didn't show up, I attempted to remove it.
I left a few dependencies that this process flagged:
* generated_types
- `pbjson`,`serde`. Apparently used by the generated code.
* grpc-router-test-gen
- `prost`. Apparently used by the generated code.
* influxdb_iox
- `heappy`. Doesn't appear used, but is behind enough feature
flags that I don't care to reason about and it's already optional.
- `tikv_jemalloc_sys`. Appears to be setting a feature flag of an
indirect dependency.
* iox_gitops_adapter
- `k8s_openapi`. Appears to be setting a feature flag of an indirect
dependency.
2022-05-06 15:57:58 -04:00
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
bf1d08333d
fix: Move ChunkColumnSummary to read_buffer, the only place it's used
2022-05-06 14:45:39 -04:00
d2671355c3
fix: Move partition metadata types to data_types2
2022-05-06 14:45:37 -04:00
02893e598c
chore: Update datafusion and upgrade arrow/parquet/arrow-flight to 13 ( #4516 )
...
* chore: Tool for automating arrow version update
* chore: Update datafusion and arrow/parquet/arrow-flight
* fix: update for changes in Arrow API
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-05 00:21:02 +00:00
4c94e46642
chore(deps): Bump croaring from 0.5.2 to 0.6.0 ( #4408 )
...
* chore(deps): Bump croaring from 0.5.2 to 0.6.0
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.5.2 to 0.6.0.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.2...0.6.0 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: croaring 0.6.0 compat
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-04-25 16:41:08 +00:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
d7370de615
test: add property tests to RLE encoder
2022-04-14 11:40:33 +01:00
aa4ad56939
test: add test case
2022-04-14 11:35:19 +01:00
5ce11a37f8
test: add property tests for float byte trimmer
2022-04-14 11:35:19 +01:00
e548ba7b17
test: add property tests for byte trimmer
2022-04-14 11:35:19 +01:00
cf0d048037
refactor: rename helper
2022-04-14 11:35:19 +01:00
20bbb88dc5
refactor: remove table name from `TableSummary` ( #4170 )
...
This allows us to remove the table name from the low-level chunk
representations (like `ParquetFile`, RUB, ...) since table names are
already tracked by the higher-level data structures (e.g. catalog,
catalog chunk) that manage the low-level chunk representations.
This is similar to #4167 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:24:00 +00:00
2b76c31157
refactor: make statistics null counts optional ( #4160 )
...
Min/max values and distinct counts are already optional, so let's make
the null counts optional as well. This will be helpful for NG to deal w/
partial statistics (e.g. we only populate stats for the time column).
Note that the total count is still mandatory, but we normally have the
chunk/file-level row count at hand.
2022-03-29 17:47:57 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
204dd7c8e9
refactor: Fix some random clippy lints from the future ( #4118 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 09:21:29 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
33851be3a5
chore: upgrade Rust to 1.59 ( #3875 )
...
Mostly a few new clippy crates around `flat_map`, `and_then`, and
"underscore locks" (!!!):
https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_lock
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:14:19 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
37c65fc24f
fix: Use a string literal with unreachable ( #3768 )
...
This should have been taken care of with the update to Rust 2021
Edition, but rustc had a bug [1]. The bug will be fixed in Rust 1.59, which
means this code will stop compiling.
[1] https://github.com/rust-lang/rust/pull/93179
2022-02-16 16:44:48 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
04a85515d3
Merge branch 'main' into er/refactor/read_buffer_clippy
2022-02-01 15:11:00 +00:00
04a59f825f
refactor: fix clippy lints
2022-02-01 14:56:49 +00:00
fa546047fb
refactor: update dep with API change ( #3596 )
2022-02-01 14:53:10 +00:00
2f3cc046f6
chore: enable to_string restrictive lints
2022-02-01 14:47:30 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
87af662d3b
refactor: include comments in test cases ( #3550 )
2022-01-27 20:35:14 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
5befa7922b
refactor: move chunk metrics to module ( #3548 )
...
* refactor: move chunk metrics into module
* refactor: use Metrics as internal name
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:24:31 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
0a0b8b2150
feat: decouple read buffer row group size from Datafusion batch size ( #3538 )
...
* feat: add chunk builder
* test: test coverage for chunk builder
* refactor: apply suggestions from code review
* refactor: address PR feedback
2022-01-26 12:39:29 +00:00
d1816b662f
chore: update rust to 1.58 ( #3461 )
...
* chore: update rust to 1.58
* fix: clippy
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-01-13 21:12:46 +00:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
80c048528c
refactor: Clean up code by using `lit` and `lit_timestamp_nano` rather than `Expr::Literal(ScalarValue(Some(...)))` ( #3427 )
2022-01-07 17:23:14 +00:00
527885f7f8
chore: Update datafusion ( #3413 )
...
* chore: Update datafusion and update code to handle timezone aware timestamps
* fix: cargo hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-23 14:52:12 +00:00
a08246e3c7
Merge branch 'main' into er/refactor/debug_rub_query
2021-12-02 19:42:46 +00:00
083ba1daf4
refactor: apply PR feedback
2021-12-02 19:34:47 +00:00
0683f851e9
fix: Derive Default instead of doing an equivalent manual impl
...
Identified by clippy. The impl did have `#[cfg(test)]` on it, but the
entire struct does too, so this is fine.
https://rust-lang.github.io/rust-clippy/master/index.html#derivable_impls
2021-12-02 11:52:02 -05:00
1a899b939e
fix: Remove redundant closures identified by clippy
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
2021-12-02 11:52:02 -05:00
de25c8e1a4
refactor: add some trace timings
2021-12-02 12:23:44 +00:00
197634ed50
feat: reload chunk back into read buffer ( #3209 ) ( #3216 )
...
* feat: reload chunk back into read buffer (#3209 )
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:26:10 +00:00
Carol (Nichols || Goulding)
Andrew Lamb
Jake Goulding
dependabot[bot]
Edd Robinson
Marco Neumann
Carol (Nichols || Goulding)
Andrew Lamb
Raphael Taylor-Davies