72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
942a6100b5
fix: check schemas in `pretty_print_batches` ( #6309 )
...
* fix: check schemas in `pretty_print_batches`
I think most users of this function (and `assert_batches_eq`) assume
that all batches have the same schema. If not, `pretty_print_batches`
may either fail producing an actual table (some rows may have more or
less columns) or silently produce a table that looks "alright".
* fix: equalize schemas where it is required/desired
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-02 12:14:16 +00:00
14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )" ( #6296 )
...
This reverts commit b7e52c0d8d
2022-12-01 14:20:43 +00:00
b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )
...
This reverts commit 039a45ddd1
2022-12-01 11:38:42 +00:00
039a45ddd1
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )
...
* chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0`
* chore: Update thrift to 0.17
* fix: use workspace arrow-flight in ingester2
* chore: Update for API changes
* fix: test
* chore: Update hakari
* chore: Update hakari again
* chore: Update trace_exporters to latest thrift
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 14:12:30 +00:00
fc520e0c0f
refactor: Remove unecessary optimize_record_batch ( #6262 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 13:35:46 +00:00
f22b1e1a09
chore: Update datafusion (to get memory limiting code) ( #6246 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update to newer api
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 12:05:17 +00:00
e4c12fa6a5
fix: slice flight response batches ( #6205 )
...
* fix: slice flight response batches
Same as #6094 but for the Apache Flight interface.
Ref https://github.com/influxdata/idpe/issues/16073 .
* refactor: use `RecordBatch::slice`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 12:25:23 +00:00
a969754819
chore(deps): Bump chrono from 0.4.22 to 0.4.23 ( #6129 )
...
* chore(deps): Bump chrono from 0.4.22 to 0.4.23
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.22 to 0.4.23.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.22...v0.4.23 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* refactor: chrono future compat
Integer->timstamp conversions should not silently panic.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-14 13:34:09 +00:00
9b48437711
refactor: make influx column type mandatory ( #5978 )
...
We basically assume everywhere that a column falls into one of the three
known categories (time, tag, field), so lets encode this in our type
system instead of defining "unknown" as "undefined behavior, may or may
not crash".
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-26 11:20:29 +00:00
cd4087e00d
style: add no todo!() or dbg!() lints
...
Some crates had theme, some not - lets be consistent and have the
compiler spot dbg!() and todo!() macro calls - they should never be in
prod code!
2022-09-29 13:10:07 +02:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
c8108f01e7
chore: Upgrade to Rust 1.64 ( #5727 )
...
* chore: Upgrade to Rust 1.64
* fix: Use iter find instead of a for loop, thanks clippy
* fix: Remove some needless borrows, thanks clippy
* fix: Use then_some rather than then with a closure, thanks clippy
* fix: Use iter retain rather than filter collect, thanks clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-22 18:04:00 +00:00
9215a534d0
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0` ( #5229 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0`
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 08:10:47 +00:00
106d84c6e1
chore: readability improvements ( #4955 )
...
* chore(arrow_util): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* chore(tracker): readability improvements
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* chore(cache_system): improve readability
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(lru test): rename `test_panic_id_collision`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-28 14:28:14 +00:00
eca3b6b9a1
fix: reduce memory usage in ingester with less buffering prior to query engine ( #4830 )
...
* refactor: remove another buffer copy in ingester
* docs: Update arrow_util/src/util.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-06-14 18:22:55 +00:00
3592aa52d8
chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0` ( #4743 )
...
* chore: Update datafusion + `arrow`/`parquet`/`arrow-flight` to `15.0.0`
* chore: Update APIs
* chore: Run cargo hakari tasks
* feat: normalize parquet file metadata
* chore: update size tests
* chore: add docs on metadata stripping
* chore: TEMP UPDATE TO DF BRANCH
* chore: Update for new API
* fix: Update to latest DF
* fix: cargo hakari
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2022-06-03 10:32:26 +00:00
31d1b37d73
refactor: de-duplicate low-level arrow code ( #4697 )
...
It seems that during prototyping NG we've copied low level code (w/o
tests!) and never cleaned up. Let's not have this functionality twice.
2022-05-25 16:24:28 +00:00
41ee23463d
feat: is all set/unset BitSet query methods
...
Adds is_all_set() and is_all_unset() methods to the BitSet, to query the
state of all bits in the bitmap.
2022-05-18 13:33:24 +01:00
43bc473469
fix: bounds check BitSet access
...
Prior to this commit, it was possible to read/write to the allocated,
but unused storage bits outside of the "length" of the BitSet.
Bit access is now bounds checked.
2022-05-17 17:26:42 +01:00
7b436d37cd
fix: correctly set nulls in string columns ( #3939 )
...
* fix: correctly set nulls in string columns
* fix: test
* test: fix
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 20:15:44 +00:00
52866fe6a9
fix: merge record batches into one batch ( #3535 )
...
* fix: merge record batches into one batch
refactor: address review comments
* chore: update test output
2022-01-25 23:29:16 +00:00
9283432a0f
refactor: display as ns
2022-01-14 10:26:11 +00:00
6a842fc105
feat: add completed duration to system table
2022-01-14 10:26:11 +00:00
218042784f
feat: add `system.queries `system table ( #3328 )
...
* feat: Add query log and system table
* docs: Add docstrings for test normalization
2021-12-08 16:26:24 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
e444fa4cb2
feat: pbdata encode ( #2724 ) ( #3009 )
2021-11-02 18:31:53 +00:00
aafc618337
fix: PackedStringArray extend_from ( #2968 )
...
* fix: PackedStringArray extend_from
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-27 08:07:51 +00:00
f1fd40390a
feat: add MutableBatch::extend_from_ranges ( #2961 )
2021-10-25 14:41:25 +00:00
d5d885aac6
feat: add MutableBatch::extend_from_range ( #2917 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 11:58:11 +00:00
2aa1e24b89
feat: arrow_util extend_from_range ( #2904 )
...
* feat: arrow_util extend_from_range
* chore: review feedback
2021-10-20 13:11:43 +00:00
0b1e4a03a5
feat: add MutableBatch::extend_from ( #2900 )
...
* feat: add MutableBatch::extend_from
* chore: fix lint
* chore: improve panic message
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 12:31:06 +00:00
9974a5364c
chore(security): Replace prettytable with comfy-table ( #2905 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 10:44:36 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
708e862314
fix: truncate fuzz modulo zero ( #2883 )
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 18:53:57 +00:00
1518f30da3
feat: arrow_util truncate and bitset append ( #2879 )
...
* feat: arrow_util truncate and bitset append
* chore: check still mutable after truncate
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:58:17 +00:00
3c968ac092
feat: correctly account MUB sizes
...
Fixes #1565 .
2021-09-03 09:15:49 +02:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
01c79f1a1a
fix: Print all timestamps using RFC3339 format ( #2098 )
...
* fix: Use IOx pretty printer rather than arrow pretty printer
* chore: update tests in the query crate
* chore: update influxdb_iox tests
* chore: Update end to end tests
* chore: update query_tests
* chore: update mutable_buffer tests
* refactor: update parquet_file tests
* refactor: update db tests
* chore: update kafka integration test output
* fix: merge conflict
2021-07-22 19:04:52 +00:00
b3d6e3ed7b
fix: Implement pretty printer wity RFC3339 formatting for timestamps ( #2096 )
...
* fix: Implement pretty printer wity RFC3339 formatting for timestamps
* fix: doc strings + fmt
2021-07-22 17:08:27 +00:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
62d3305923
feat: optimize the dictionaries in the output of deduplicate node ( #1827 ) ( #1832 )
...
* feat: optimize dedup dictionaries (#1827 )
* fix: handle sliced null bitmasks
* chore: review feedback
2021-06-30 09:30:16 +00:00
f145c5d760
feat: add ability to optimize record batch dictionaries ( #1827 ) ( #1830 )
...
* feat: add ability to optimize record batch dictionaries (#1827 )
* chore: correct capacity
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
663a38862d
refactor: address PR feedback
2021-05-20 10:49:49 +01:00
0dc699252f
feat: add with_capacity constructor
2021-05-20 10:49:49 +01:00
8ab71be5eb
feat: use PackedStringArray in mutable buffer ( #1476 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-13 15:25:19 +00:00
b02105e47b
feat: construct StringDictionary from PackedStringArray ( #1475 )
...
* feat: construct StringDictionary from PackedStringArray
* chore: fix formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 20:16:25 +00:00
c85d1574eb
feat: move dictionary and bitset into arrow_utils ( #1459 )
...
* feat: move dictionary and bitset into arrow_utils
* chore: review feedback
* chore: remove redundant dictionary methods
* chore: consistent type parameter name in PackedStringArray
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 16:43:38 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
Carol (Nichols || Goulding)
Marco Neumann
Andrew Lamb
dependabot[bot]
Dom Dwyer
Andrew Lamb
Carol (Nichols || Goulding)
Ryan Russell
Nga Tran
Edd Robinson
Raphael Taylor-Davies
Carol (Nichols || Goulding)