be45889be1
chore: Upgrade datafusion ( #6407 )
...
* chore: Update datafusion
* chore: Update for API change
2022-12-15 06:51:35 +00:00
8729977851
chore: Upgrade datafusion / arrow to 29.0.0 to get flightsql client ( #6396 )
...
* chore: Update datafusion pin
* chore: Update for API change
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-13 20:16:09 +00:00
336ca761a3
chore: Update datafusion pin (to get sqlparser update) ( #6378 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-12 14:39:42 +00:00
e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) ( #6363 )
...
* chore: Update DataFusion pin to get median fix
* chore: Update for new Expr node
* test: add test for median
* test: add test for coercion of strings to timestamps
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 12:14:00 +00:00
061d582a9b
chore: patch object_store to get apache#3274 ( #6362 )
...
* chore: patch object_store to get apache#3274
* chore: Run cargo hakari tasks
* fix: add issue breadcrumb
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 14:52:25 +00:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
d0f1f6a4fd
chore: Upgrade datafusion to get memory limits ( #6297 )
...
* chore: Update datafusion
* fix: use correctly qualified column names
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 16:40:26 +00:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )" ( #6296 )
...
This reverts commit b7e52c0d8d
2022-12-01 14:20:43 +00:00
b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )
...
This reverts commit 039a45ddd1
2022-12-01 11:38:42 +00:00
039a45ddd1
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )
...
* chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0`
* chore: Update thrift to 0.17
* fix: use workspace arrow-flight in ingester2
* chore: Update for API changes
* fix: test
* chore: Update hakari
* chore: Update hakari again
* chore: Update trace_exporters to latest thrift
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-30 14:12:30 +00:00
f22b1e1a09
chore: Update datafusion (to get memory limiting code) ( #6246 )
...
* chore: Update datafusion
* chore: Run cargo hakari tasks
* fix: Update to newer api
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-29 12:05:17 +00:00
fc5697b8e7
chore: Update datafusion again (N of N) ( #6218 )
...
* chore: Update datafusion again (4 of N)
* fix: Update plans
* fix: Update for renamed API
* fix: Update more plans
* chore: Update to datafusion @ d355f69aae2cc951cfd021e5c0b690861ba0c4ac
* fix: update explain plan tests
* fix: update test after schema error
* chore: Update datafusion again
* fix: Add size() calculation to selectors
* chore: Run cargo hakari tasks
* fix: Update newly added test
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-28 17:09:40 +00:00
a66fc0b645
feat(ingester): ingester2 init
...
Adds an ingester2 crate to hold the MVP of the Kafkaless project.
This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.
This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.
This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
2022-11-24 15:34:02 +01:00
db63dc55a2
feat: Initial WAL implementation
2022-11-23 17:07:43 -05:00
71b29ae442
chore: Update DataFusion (3 of N) and arrow/parquet to 27.0.0 ( #6208 )
...
* chore: Update DataFusion and arrow/parquet to 27.0.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 14:21:47 +00:00
9fb1de0428
chore: Update datafusion (2 of N) right before arrow 27 upgrade ( #6207 )
...
* chore: Update datafusion (2 of N) right before arrow 27 upgrade
* fix: Update tests for better unsigned pushdown
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-23 11:04:14 +00:00
1a1ea74cb7
chore: Upgrade datafusion again ( #6160 )
...
* Revert "Revert "chore: Update datafusion again (#6108 )""
This reverts commit 766b3bbeb440618cfe332f6ee7d4f8a8217acc48.
* fix: Respect the partition sort key
* chore: update plans
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 19:28:26 +00:00
67712b595c
Revert "chore: Update datafusion again ( #6108 )" ( #6159 )
...
This reverts commit fbe9f27f10
2022-11-16 21:14:55 +00:00
9365d933f1
chore: router namespace api ( #6151 )
...
* chore: move ns api from querier to router
* chore: add explanatory comment in querier about moved namespace API
* fix: add namespace service to router
* fix: querier returns unimplemented error for ns retention, not panic
* chore: reuse namespace -> proto in router ns api
* chore: grpc namespace - consume ns to avoid clone
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 15:25:49 +00:00
fbe9f27f10
chore: Update datafusion again ( #6108 )
...
* chore: Update datafusion pin + api code
* chore: Run cargo hakari tasks
* refactor: combine_sort_key is more idomatic and add rationale comments
* refactor: satisfy borrow checker and updated comments
* fix: Add test case for combine_sort_key
* fix: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* fix: Add back test for deeply nested expression
* fix: Update output ordering
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 14:41:52 +00:00
cc17e5a54b
refactor: use a workspace dependency for hashbrown
2022-11-11 13:25:39 -05:00
034d9b371d
chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` ( #6061 )
...
* chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0`
* fix: Update query_functions
* fix: update for TimestampNanosecondArray API changes
* fix: update for TimestampNanosecondArray API changes
* chore: Update flatbuffers and remove rustsec warning
* chore: Update text
* fix: update more test
* fix: Lock ahash to exactly 0.8.0
* fix: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-07 11:01:58 +00:00
8c8e607dca
chore: Update datafusion pin ( #6054 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
* chore: Update expected error
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-11-03 19:41:31 +00:00
e7e9945207
chore: Update datafusion pin ( #6017 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-02 11:12:21 +00:00
00953460fb
chore: Update datafusion pin ( #6010 )
2022-10-31 17:14:56 +00:00
ace3c11f12
chore: Update datafusion ( #6004 )
...
* chore: Update datafusion
* chore: change path
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-31 16:16:28 +00:00
b27e2bd7d1
feat: Use workspace dep inheritance for the parquet crate
2022-10-26 10:37:51 -04:00
88c3a1f5e7
feat: Use workspace dep inheritance for the arrow-flight crate
2022-10-26 10:34:54 -04:00
3145e2c05b
feat: Use workspace dep inheritance for the arrow crate
2022-10-26 10:34:29 -04:00
44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate
2022-10-26 10:33:56 -04:00
99f36f8889
chore: unpatch clap V3 ( #5971 )
...
Upstream issue was solved by a new release.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-25 09:16:16 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
4ca869fcd9
fix: textwrap 0.15.1 was yanked ( #5953 )
...
Need to pull an unreleased clap v3 so we no longer depend on a yanked
version of textwrap. Also see https://github.com/clap-rs/clap/issues/4418
2022-10-24 07:19:51 +00:00
76d3fbbe53
refactor: remove old `packers` crate ( #5893 )
...
* refactor: remove old packers code
* refactor: Remove from workspace
2022-10-18 19:00:28 +00:00
9b7a7b5e40
refactor: remove read buffer ( #5891 )
2022-10-18 12:46:17 +00:00
81722dc19b
feat: AST traversal using Visitor pattern ( #5796 )
...
* feat: Partition implementation of Visitable for InfluxQL AST
* feat: Added consistent structures for each clause to simplify visitor
Continued to expand `accept` and `pre` / `post` visit implementations.
* feat: Added insta and tests using snapshots (thanks @crepererum)
The insta crate simplifies the process of validating the combination of
visitor and accept implementations are called and in the correct order.
* chore: Run cargo hakari tasks
* feat: Added remaining snapshot tests
Some tests are failing as some minor type changes must be added along
with the addition of related visitor functions.
* feat: Add types to represent each clause in numerous statements
These clauses permit distinct visit functions on the `Visitor` type.
* chore: Reformat `SELECT`
* chore: Explicitly specify access to export selected types only
This required completing all the missing documentation for the exported
types.
* chore: Update Cargo.lock
* chore: macro to implement common traits and hide 0th tuple element
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-10-13 22:37:49 +00:00
d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 ( #5792 )
...
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0
* fix: Update for coercion, fix explain plans for change in column name display
* chore: Update datafusion lock
* fix: Update for other API changes
* chore: Update to latest datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-10-12 16:19:14 +00:00
66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 ( #5694 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0
* chore: Update thrift / remove parquet_format
* fix: Update APIs
* chore: Update lock + Run cargo hakari tasks
* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-27 12:50:54 +00:00
65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol ( #5734 )
...
* feat: add `influxdb_iox debug parquet_to_lp` command
* chore: Run cargo hakari tasks
* fix: update command description
* fix: remove unecessary Result import
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2022-09-26 14:17:27 +00:00
f86d3e31da
chore: Update datafusion + object_store ( #5619 )
...
* chore: Update datafusion pin
* chore: update object_store to 0.5.0
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-13 12:34:54 +00:00
15369d4c52
feat: Implement gRPC binary logger ( #5473 )
...
* feat: Implement gRPC binary logger
* chore: Run cargo hakari tasks
* fix: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-09-01 10:57:02 +00:00
711ba77341
chore: update object_store to test IMDSv1 fallback ( #5509 )
...
* chore: update object_store to test IMDSv1 fallback
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-30 12:31:49 +00:00
66f0b59dbb
revert: remove Azure SDK / bump object_store
...
This reverts commit c2f8efa03a
2022-08-30 10:41:29 +02:00
c2f8efa03a
build: remove Azure SDK / bump object_store
...
Bumps the object_store pin to master to pick up:
https://github.com/apache/arrow-rs/pull/2509
This removes the Azure SDK.
2022-08-29 14:06:51 +02:00
b4e5895d7a
feat: Add influxdb_influxql_parser crate ( #5415 )
...
* feat: Add crate; parse quoted identifiers
* chore: Run cargo hakari tasks
* chore: satisfy linter
* chore: Use `test_helpers::Result`
* feat: Add all InfluxQL keywords
* chore: Update influxdb_influxql_parser/src/lib.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* chore: PR feedback
* chore: PR Feedback, remove Result<()>
* chore: Update Cargo.lock
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-08-18 23:09:45 +00:00
7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 ( #5419 )
...
* chore: update object_store
* chore: update hakari config
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-17 13:44:48 +00:00
c5f062bba0
feat: initial commit of schema merge bulk import tool ( #5344 )
...
* feat: initial commit of schema merge bulk import tool
* chore: use observability depds instead of tracing-*
* chore: removed debug printlns
* chore: fix feature decls for cloud providers for import crate
* chore: use println instead of info in import- no need for a simple CLI
* chore: tidy whitespace
* chore: remove unused dep in import
* chore: Run cargo hakari tasks
* chore: removed unimpld import job subcommand
* chore: clarifying comment about custom serialisation code
* chore: clarifying comment about schema merge code in import
* chore: fix wrong comment in import command
* chore: bump object store dep to get bugfix
* chore: rename import schema struct for clarity
* chore: run `cargo hakari generate`
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 09:07:38 +00:00
dadcc369b1
chore: update object_store to fix credentials client ( #5359 )
...
* chore: update object_store to fix credentials client
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-09 13:17:43 +00:00
Andrew Lamb
Raphael Taylor-Davies
kodiakhq[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Paul Dix
Luke Bond
Jake Goulding
Marco Neumann
Stuart Carnie
Andrew Lamb
Marko Mikulicic