Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
49,406 Commits
234 Branches
453 Tags
344 MiB
Commit Graph

573 Commits (praveen/use-heap-ringbuffer)

Author SHA1 Message Date
Andrew Lamb 27890b313f
chore: Update datafusion (#6997) 
* chore: Update datafusion

* chore: update the plans

* fix: update some plans

* chore: Update plans and port some explain plans to use insta snapshots

* fix: another plan

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-16 17:03:25 +00:00
Andrew Lamb 4d7aa1e48b
refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test (#6960) 
* refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test

* fix: Update compactor2/src/components/mod.rs

Co-authored-by: Marco Neumann <marco@crepererum.net>

---------

Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-13 12:06:42 +00:00
Andrew Lamb 143cb26e87
chore: Update datafusion again (#6958) 
* chore: Update datafusion pin

* chore: Update plans

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-13 10:28:44 +00:00
Andrew Lamb 501032c2fb
chore: Update datafusion again (#6929) 
* chore: Update DataFusion

* fix: disable repartitioned sorting

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-10 11:51:47 +00:00
Andrew Lamb 2f4d901fbe
chore: Update datafusion (#6893) 
* chore: Update datafusion

* chore: Run cargo hakari tasks

* fix: update for api change

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-08 18:27:18 +00:00
Andrew Lamb ae10b33f0c
chore: Update datafusion (#6868) 
* chore: Update datafusion

* chore: Run cargo hakari tasks

* fix: Update error message

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-02-06 14:22:45 +00:00
Raphael Taylor-Davies d3601a59f8
chore: update DataFusion, upgrade `arrow` `arrow-flight` and `parquet` to `32.0.0` (#6756) 
* chore: update DataFusion

* fix: test

* chore: format

* chore: clippy

* chore: update arrow

* chore: arrow upgrade fallout

* chore: Run cargo hakari tasks

* chore: remove failing warm compaction test

* fix: flight error propagation

* chore: update parquet size

* fix: Update error message

* chore: Update parquet metadata test

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-06 11:35:39 +00:00
Carol (Nichols || Goulding) b55e4ffa21
test: Port table_schema tests to end-to-end and get rid of query_tests (#6828) 
* test: Port table_schema query_tests to end-to-end

* fix: Remove now outdated query_tests crate

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-02-03 17:41:08 +00:00
Andrew Lamb 5b14caa780
chore: Update DataFusion (#6753) 
* chore: Update datafusion

* fix: Update for changes

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-30 14:48:52 +00:00
Andrew Lamb 0d32662eea
chore: Update datafusion again (#6722) 
* chore: Update datafusion

* fix: Update for API

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-27 18:59:27 +00:00
Andrew Lamb c3bc61f10e
refactor: Move `flightsql` code into its own module, add docs and tests (#6640) 
* refactor: Move `flightsql`  code into its own module

* fix: get schema from LogicalPlan

* refactor: use arrow_flight::sql::Any instead of prost_types::any

* fix: cleanup docs and avoid as_ref

* fix: Use Bytes

* fix: use Any::pack

* fix: doclink
 2023-01-24 18:24:32 +00:00
Andrew Lamb 9a61f36a53
chore: Update datafusion again (#6656) 
* chore: Update datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-23 12:13:07 +00:00
Andrew Lamb b09691dc6b
chore: Upgrade datafusion (again, I know) (#6639) 
* chore: Update datafusion

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-23 11:24:22 +00:00
kodiakhq[bot] 24ca1e6f8c
Merge branch 'main' into cn/test-old-with-new 2023-01-20 15:20:40 +00:00
Andrew Lamb d808c57cdc
chore: Remove `iox_arrow_flight` (#6621) 
* chore: Remove iox_arrow_flight

* fix: hack around tonic status errors

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-20 11:16:19 +00:00
Carol (Nichols || Goulding) 6afd782b3f
fix: Move query_tests2 into influxdb_iox/tests so that the code rebuilds 2023-01-19 16:44:31 -05:00
Andrew Lamb f639bf3e23
chore: refactor ingester to use upstream arrow-flight (#6622) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-19 15:16:13 +00:00
Andrew Lamb 6a4fa11dba
chore: Update to latest datafusion / arrow `31.0.0` (#6618) 
* chore: Update datafusion and arrow/parquet/arrow-flight `31.0.0`

* chore: Update for new API

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 20:21:10 +00:00
kodiakhq[bot] 33168b97f0
Merge branch 'main' into cn/query-tests-grpc 2023-01-18 19:03:51 +00:00
Marco Neumann e72173d58d
feat: very basic compactor2 skeleton (#6614) 
Sets up crate and wires up the main binary. No tests yet, no algorithm
framework, just the bare minimum.

Also I decided to not offer a gRPC server in `compactor2` at the moment
and hence did not implement any handle/delegate infrastructure. We add
this later if we need it. This also means compactor2 does NOT provide a
catalog service for now.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-18 16:36:40 +00:00
Andrew Lamb 8410998408
chore: Update datafusion to Jan 17, 2023 (2 / 2) and arrow/parquet `30.0.1` (#6604) 
* chore: Update datafusion to Jan 9, 2023 (2 / 2) and arrow/parquet `30.0.1`

* chore: Update for changes in arrow ipc

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-18 15:51:24 +00:00
Carol (Nichols || Goulding) f3b5dcaab7
feat: Reimagining query_tests 2023-01-18 10:24:17 -05:00
Andrew Lamb 57f08dbccd
chore: Update datafusion to Jan 9, 2023 (1 / 2) (#6603) 
* refactor: Update DataFusion pin to early Jan 2023

* fix: Update tests now that planning is async

* fix: Updates for API changes

* chore: Run cargo hakari tasks

* fix: Update comment

* refactor: nicer config setup

* fix: gapfill async

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-01-18 12:19:32 +00:00
dependabot[bot] cc6d8709c9
chore(deps): Bump hashbrown from 0.13.1 to 0.13.2 (#6574) 
Bumps [hashbrown](https://github.com/rust-lang/hashbrown) from 0.13.1 to 0.13.2.
- [Release notes](https://github.com/rust-lang/hashbrown/releases)
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.1...v0.13.2)

---
updated-dependencies:
- dependency-name: hashbrown
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-01-12 09:47:15 +00:00
Andrew Lamb dbe52f1ca1
chore: Upgrade datafusion (#6467) 
* chore: Update datafusion

* fix: Update for new apis

* chore: Update expected plan

* fix: Update for new config construction

* chore: update clippy

* fix: Fix error codes

* fix: update another test

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-01-03 15:29:11 +00:00
Andrew Lamb d0d5906476
chore: Update datafusion pin (#6442) 
* chore: Update datafusion pin

* refactor: Update iox_query for new apis

* chore: Update some more apis

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-12-19 20:02:42 +00:00
Andrew Lamb 94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight (#6427) 
* refactor: Extract common ArrowFlight client into iox_arrow_flight

* chore: Run cargo hakari tasks

* fix: clarify intent of iox_arrow_flight crate

* refactor: Apply suggestions from code review

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

* fix: loop --> while let

* fix: REmove make_tonic_error in favor of From impl

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-19 11:35:20 +00:00
Andrew Lamb be45889be1
chore: Upgrade datafusion (#6407) 
* chore: Update datafusion

* chore: Update for API change
 2022-12-15 06:51:35 +00:00
Andrew Lamb 8729977851
chore: Upgrade datafusion / arrow to 29.0.0 to get flightsql client (#6396) 
* chore: Update datafusion pin

* chore: Update for API change

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-12-13 20:16:09 +00:00
Andrew Lamb 336ca761a3
chore: Update datafusion pin (to get sqlparser update) (#6378) 
* chore: Update datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-12-12 14:39:42 +00:00
Andrew Lamb e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) (#6363) 
* chore: Update DataFusion pin to get median fix

* chore: Update for new Expr node

* test: add test for median

* test: add test for coercion of strings to timestamps

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-12 12:14:00 +00:00
Raphael Taylor-Davies 061d582a9b
chore: patch object_store to get apache#3274 (#6362) 
* chore: patch object_store to get apache#3274

* chore: Run cargo hakari tasks

* fix: add issue breadcrumb

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-09 14:52:25 +00:00
Andrew Lamb 9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers (#6349) 
* test: add tests for periods in measurement names

* chore: Update Datafusion

* chore: Update for changed APIs

* chore: Update expected plan output

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-08 11:27:13 +00:00
kodiakhq[bot] 9e3d0fcefb
Merge branch 'main' into cn/ingester2 2022-12-02 13:39:55 +00:00
Andrew Lamb d0f1f6a4fd
chore: Upgrade datafusion to get memory limits (#6297) 
* chore: Update datafusion

* fix: use correctly qualified column names

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-01 16:40:26 +00:00
Carol (Nichols || Goulding) 99698d0f46
feat: Create an ioxd_ingester2 crate 2022-12-01 11:37:21 -05:00
Andrew Lamb 14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` (#6279)" (#6294)" (#6296) 
This reverts commit b7e52c0d8d.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-12-01 14:20:43 +00:00
Andrew Lamb b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` (#6279)" (#6294) 
This reverts commit 039a45ddd1.
 2022-12-01 11:38:42 +00:00
Andrew Lamb 039a45ddd1
chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` (#6279) 
* chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0`

* chore: Update thrift to 0.17

* fix: use workspace arrow-flight in ingester2

* chore: Update for API changes

* fix: test

* chore: Update hakari

* chore: Update hakari again

* chore: Update trace_exporters to latest thrift

* fix: update test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-30 14:12:30 +00:00
Andrew Lamb f22b1e1a09
chore: Update datafusion (to get memory limiting code) (#6246) 
* chore: Update datafusion

* chore: Run cargo hakari tasks

* fix: Update to newer api

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-29 12:05:17 +00:00
Andrew Lamb fc5697b8e7
chore: Update datafusion again (N of N) (#6218) 
* chore: Update datafusion again (4 of N)

* fix: Update plans

* fix: Update for renamed API

* fix: Update more plans

* chore: Update to datafusion @ d355f69aae2cc951cfd021e5c0b690861ba0c4ac

* fix: update explain plan tests

* fix: update test after schema error

* chore: Update datafusion again

* fix: Add size() calculation to selectors

* chore: Run cargo hakari tasks

* fix: Update newly added test

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-28 17:09:40 +00:00
Dom Dwyer a66fc0b645
feat(ingester): ingester2 init 
Adds an ingester2 crate to hold the MVP of the Kafkaless project.

This was necessary due to the tight coupling of the ingester internals
with tests in external crates, and eases the parallel development of two
version of the ingester.

This commit contains various changes from the "ingester" crate, mostly
removing the concept/references to a "shard" or "ShardId" where
possible.

This commit does not copy over all of the "ingester" crate - only those
components that are definitely needed. I will drag across more as
functionality is implemented.
 2022-11-24 15:34:02 +01:00
Paul Dix db63dc55a2
feat: Initial WAL implementation 2022-11-23 17:07:43 -05:00
Andrew Lamb 71b29ae442
chore: Update DataFusion (3 of N) and arrow/parquet to 27.0.0 (#6208) 
* chore: Update DataFusion and arrow/parquet to 27.0.0

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-11-23 14:21:47 +00:00
Andrew Lamb 9fb1de0428
chore: Update datafusion (2 of N) right before arrow 27 upgrade (#6207) 
* chore: Update datafusion (2 of N) right before arrow 27 upgrade

* fix: Update tests for better unsigned pushdown

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-11-23 11:04:14 +00:00
Andrew Lamb 1a1ea74cb7
chore: Upgrade datafusion again (#6160) 
* Revert "Revert "chore: Update datafusion again (#6108)""

This reverts commit 766b3bbeb440618cfe332f6ee7d4f8a8217acc48.

* fix: Respect the partition sort key

* chore: update plans

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-22 19:28:26 +00:00
Andrew Lamb 67712b595c
Revert "chore: Update datafusion again (#6108)" (#6159) 
This reverts commit fbe9f27f10.
 2022-11-16 21:14:55 +00:00
Luke Bond 9365d933f1
chore: router namespace api (#6151) 
* chore: move ns api from querier to router

* chore: add explanatory comment in querier about moved namespace API

* fix: add namespace service to router

* fix: querier returns unimplemented error for ns retention, not panic

* chore: reuse namespace -> proto in router ns api

* chore: grpc namespace - consume ns to avoid clone

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-16 15:25:49 +00:00
Andrew Lamb fbe9f27f10
chore: Update datafusion again (#6108) 
* chore: Update datafusion pin + api code

* chore: Run cargo hakari tasks

* refactor: combine_sort_key is more idomatic and add rationale comments

* refactor: satisfy borrow checker and updated comments

* fix: Add test case for combine_sort_key

* fix:  Apply suggestions from code review

Co-authored-by: Marco Neumann <marco@crepererum.net>

* fix: Add back test for deeply nested expression

* fix: Update output ordering

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-16 14:41:52 +00:00
Jake Goulding cc17e5a54b refactor: use a workspace dependency for hashbrown 2022-11-11 13:25:39 -05:00
Andrew Lamb 034d9b371d
chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0` (#6061) 
* chore: Update datafusion and arrow/arrow-flight/parquet to `26.0.0`

* fix: Update query_functions

* fix: update for TimestampNanosecondArray API changes

* fix: update for TimestampNanosecondArray API changes

* chore: Update flatbuffers and remove rustsec warning

* chore: Update text

* fix: update more test

* fix: Lock ahash to exactly 0.8.0

* fix: Update datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-07 11:01:58 +00:00
Andrew Lamb 8c8e607dca
chore: Update datafusion pin (#6054) 
* chore: Update datafusion pin

* chore: Run cargo hakari tasks

* chore: Update expected error

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-11-03 19:41:31 +00:00
Andrew Lamb e7e9945207
chore: Update datafusion pin (#6017) 
* chore: Update datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-11-02 11:12:21 +00:00
Andrew Lamb 00953460fb
chore: Update datafusion pin (#6010) 2022-10-31 17:14:56 +00:00
Andrew Lamb ace3c11f12
chore: Update datafusion (#6004) 
* chore: Update datafusion

* chore: change path

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-31 16:16:28 +00:00
Carol (Nichols || Goulding) b27e2bd7d1
feat: Use workspace dep inheritance for the parquet crate 2022-10-26 10:37:51 -04:00
Carol (Nichols || Goulding) 88c3a1f5e7
feat: Use workspace dep inheritance for the arrow-flight crate 2022-10-26 10:34:54 -04:00
Carol (Nichols || Goulding) 3145e2c05b
feat: Use workspace dep inheritance for the arrow crate 2022-10-26 10:34:29 -04:00
Carol (Nichols || Goulding) 44936f661a
feat: Use workspace dep inheritance for datafusion instead of shim crate 2022-10-26 10:33:56 -04:00
Marco Neumann 99f36f8889
chore: unpatch clap V3 (#5971) 
Upstream issue was solved by a new release.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-25 09:16:16 +00:00
Carol (Nichols || Goulding) 2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition 2022-10-24 13:04:09 -04:00
Marco Neumann 4ca869fcd9
fix: textwrap 0.15.1 was yanked (#5953) 
Need to pull an unreleased clap v3 so we no longer depend on a yanked
version of textwrap. Also see https://github.com/clap-rs/clap/issues/4418
 2022-10-24 07:19:51 +00:00
Andrew Lamb 76d3fbbe53
refactor: remove old `packers` crate (#5893) 
* refactor: remove old packers code

* refactor: Remove from workspace
 2022-10-18 19:00:28 +00:00
Marco Neumann 9b7a7b5e40
refactor: remove read buffer (#5891) 2022-10-18 12:46:17 +00:00
Stuart Carnie 81722dc19b
feat: AST traversal using Visitor pattern (#5796) 
* feat: Partition implementation of Visitable for InfluxQL AST

* feat: Added consistent structures for each clause to simplify visitor

Continued to expand `accept` and `pre` / `post` visit implementations.

* feat: Added insta and tests using snapshots (thanks @crepererum)

The insta crate simplifies the process of validating the combination of
visitor and accept implementations are called and in the correct order.

* chore: Run cargo hakari tasks

* feat: Added remaining snapshot tests

Some tests are failing as some minor type changes must be added along
with the addition of related visitor functions.

* feat: Add types to represent each clause in numerous statements

These clauses permit distinct visit functions on the `Visitor` type.

* chore: Reformat `SELECT`

* chore: Explicitly specify access to export selected types only

This required completing all the missing documentation for the exported
types.

* chore: Update Cargo.lock

* chore: macro to implement common traits and hide 0th tuple element

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-10-13 22:37:49 +00:00
Andrew Lamb d57c99638c
chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0 (#5792) 
* chore: Update datafusion + `arrow`, `arrow-flight`, and `parquet` to 24.0.0.0

* fix: Update for coercion, fix explain plans for change in column name display

* chore: Update datafusion lock

* fix: Update for other API changes

* chore: Update to latest datafusion pin

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-10-12 16:19:14 +00:00
Andrew Lamb 66dbb9541f
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to 23.0.0, `thrift` to 0.16.0 (#5694) 
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight`  to 23.0.0

* chore: Update thrift / remove parquet_format

* fix: Update APIs

* chore: Update lock + Run cargo hakari tasks

* fix: use patched version of arrow-rs to work around https://github.com/apache/arrow-rs/issues/2779

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-09-27 12:50:54 +00:00
Andrew Lamb 65f1550126
feat: Implement `debug parquet_to_lp` command to convert parquet to line protocol (#5734) 
* feat: add `influxdb_iox debug parquet_to_lp` command

* chore: Run cargo hakari tasks

* fix: update command description

* fix: remove unecessary Result import

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2022-09-26 14:17:27 +00:00
Andrew Lamb f86d3e31da
chore: Update datafusion + object_store (#5619) 
* chore: Update datafusion pin

* chore: update object_store to 0.5.0

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-09-13 12:34:54 +00:00
Marko Mikulicic 15369d4c52
feat: Implement gRPC binary logger (#5473) 
* feat: Implement gRPC binary logger

* chore: Run cargo hakari tasks

* fix: Apply suggestions from code review

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-09-01 10:57:02 +00:00
Raphael Taylor-Davies 711ba77341
chore: update object_store to test IMDSv1 fallback (#5509) 
* chore: update object_store to test IMDSv1 fallback

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-08-30 12:31:49 +00:00
Dom Dwyer 66f0b59dbb revert: remove Azure SDK / bump object_store 
This reverts commit c2f8efa03a.
 2022-08-30 10:41:29 +02:00
Dom Dwyer c2f8efa03a build: remove Azure SDK / bump object_store 
Bumps the object_store pin to master to pick up:

	https://github.com/apache/arrow-rs/pull/2509

This removes the Azure SDK.
 2022-08-29 14:06:51 +02:00
Stuart Carnie b4e5895d7a
feat: Add influxdb_influxql_parser crate (#5415) 
* feat: Add crate; parse quoted identifiers

* chore: Run cargo hakari tasks

* chore: satisfy linter

* chore: Use `test_helpers::Result`

* feat: Add all InfluxQL keywords

* chore: Update influxdb_influxql_parser/src/lib.rs

Co-authored-by: Marco Neumann <marco@crepererum.net>

* chore: PR feedback

* chore: PR Feedback, remove Result<()>

* chore: Update Cargo.lock

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
 2022-08-18 23:09:45 +00:00
Andrew Lamb 7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 (#5419) 
* chore: update object_store

* chore: update hakari config

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-08-17 13:44:48 +00:00
Luke Bond c5f062bba0
feat: initial commit of schema merge bulk import tool (#5344) 
* feat: initial commit of schema merge bulk import tool

* chore: use observability depds instead of tracing-*

* chore: removed debug printlns

* chore: fix feature decls for cloud providers for import crate

* chore: use println instead of info in import- no need for a simple CLI

* chore: tidy whitespace

* chore: remove unused dep in import

* chore: Run cargo hakari tasks

* chore: removed unimpld import job subcommand

* chore: clarifying comment about custom serialisation code

* chore: clarifying comment about schema merge code in import

* chore: fix wrong comment in import command

* chore: bump object store dep to get bugfix

* chore: rename import schema struct for clarity

* chore: run `cargo hakari generate`

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-08-10 09:07:38 +00:00
Raphael Taylor-Davies dadcc369b1
chore: update object_store to fix credentials client (#5359) 
* chore: update object_store to fix credentials client

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-08-09 13:17:43 +00:00
Raphael Taylor-Davies ccb45d7bac
chore: update to rusoto-less object_store (#5342) 
* chore: update to rusoto-less object_store

* chore: Run cargo hakari tasks

* chore: further fixes

* chore: document workaround

* chore: review feedback

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-08-09 09:06:03 +00:00
Dom Dwyer 284a3069ce feat: Kafka client produce() instrumentation 
Adds a decorator over the underlying kafka client to capture the latency
distribution of the low-level kafka writes, independent of the
aggregation/DML batching framework that sits "above" this client.

The latency measurements include the serialisation overhead, protocol
overhead, and actual network I/O.
 2022-08-08 15:24:35 +02:00
Andrew Lamb e4dc8c2067
refactor: rename garbage collector crates for consistency (#5196) 
* refactor: rename garbage collector crates for consistency

* fix: cargo fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-07-25 12:44:37 +00:00
Jake Goulding f7a0fd43d2
feat: make object store garbage collector into a long-running service (#5135) 
* refactor: remove unused logging config

* chore: remove the object store garbage collector CLI tool

* refactor: accept an object store and catalog

* refactor: make Result type alias public like the error

* refactor: remove public modifier from modules

* refactor: allow shutting down the object store garbage collector

* feat: Introduce the object-store garbage collection server

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-07-18 21:27:38 +00:00
Marko Mikulicic 60df069a10 fix: Bump object_store crate for EKS support 2022-07-13 01:12:29 +02:00
Jake Goulding 428f41f747
feat: Walking skeleton of the object store GC tool 2022-07-07 09:48:04 -04:00
Marco Neumann 1962fcc229
chore: reduce dependencies and run `cargo update` (#4906) 
* chore: reduce proptest features

* chore: remove `grpc-router`

This crate is currently unused and we don't have immediate plans to use
it. And there's GIT, so it can always be restored.

* chore: `cargo update`
 2022-06-20 12:18:28 +00:00
Carol (Nichols || Goulding) 6417e7dc2a
feat: Extract sharder to its own crate 2022-06-15 10:01:45 -04:00
Luke Bond b76a0080d5
chore: remove unused iox_gitops_adapter (#4675) 
* chore: remove unused iox_gitops_adapter

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-05-24 10:28:43 +00:00
Marco Neumann 52346642a0
ci: fix cargo deny (#4629) 
* ci: fix cargo deny

* chore: downgrade `socket2`, version 0.4.5 was yanked

* chore: rename `query` to `iox_query`

`query` is already taken on crates.io and yanked and I am getting tired
of working around that.
 2022-05-18 09:38:35 +00:00
Raphael Taylor-Davies f2bb0fdf77
feat: update to crates.io object_store version (#4595) 
* feat: update to crates.io object_store version

* chore: Run cargo hakari tasks

* fix: tests

* chore: remove object store integration test plumbing

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-05-13 16:26:07 +00:00
Andrew Lamb 35d80fc512
refactor: rename `iox_catalog_service` to `service_grpc_catalog` for consistency (#4581) 
* refactor: rename `iox_catalog_service` to `service_grpc_catalog`

* fix: fmt

* fix: alphabetize crate

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-05-13 14:07:58 +00:00
kodiakhq[bot] 542ec97b66
Merge branch 'main' into cn/rename-no-ng 2022-05-13 13:47:48 +00:00
Marco Neumann cb0a4176fd
refactor: move `querier::cache_system` into its own crate (#4592) 2022-05-13 13:12:07 +00:00
Carol (Nichols || Goulding) 48e6e5713d
fix: Rename test_helpers_end_to_end_ng to test_helpers_end_to_end 
Connects to #4450.
 2022-05-12 16:09:07 -04:00
Andrew Lamb 7eff52d04e
refactor: rename `iox_object_store_service` to `service_grpc_object_store` for consisntency (#4572) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-05-12 14:22:18 +00:00
Andrew Lamb b8cb4c3f2b
feat: Interrogate schema from querier (as well as router) (#4557) 
* refactor: move SchemaService into `service_grpc_schema`

* feat: implement schema gRPC for querier

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-05-10 20:55:58 +00:00
Raphael Taylor-Davies 99b1a9b83f
refactor: split out ObjectStoreMetrics (#4547) 
* refactor: split out ObjectStoreMetrics

* chore: add workspace hack

* fix: compile
 2022-05-10 10:56:28 +00:00
Carol (Nichols || Goulding) fcd4815645
fix: Rename router2 to router 2022-05-06 14:51:52 -04:00
Carol (Nichols || Goulding) 0650a9bb77
fix: Rename ioxd_router2 to ioxd_router 2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding) 068096e7e1
fix: Rename data_types2 to data_types 2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding) e1bef1c218
fix: Remove OG data_types crate 2022-05-06 14:45:39 -04:00
Carol (Nichols || Goulding) ea46830954
fix: Remove iox_object_store crate; move ParquetFilePath to parquet_file 2022-05-06 14:45:36 -04:00
Carol (Nichols || Goulding) f8bdb022bc
fix: Remove job_registry crate 2022-05-06 11:35:11 -04:00
Carol (Nichols || Goulding) b88d071ce7
fix: Remove server 2022-05-06 11:30:36 -04:00
Carol (Nichols || Goulding) e0bc1801ac
fix: Remove router 2022-05-06 11:30:36 -04:00
Carol (Nichols || Goulding) ba8191c1eb
fix: Remove persistence_windows 2022-05-06 11:30:35 -04:00
Carol (Nichols || Goulding) cae32209da
fix: Remove parquet_catalog 2022-05-06 11:30:26 -04:00
Carol (Nichols || Goulding) 2d8656e2e1
fix: Remove mutable_buffer 2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding) e63f006398
fix: Remove lifecycle crate 2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding) e7de16732d
fix: Remove internal_types 2022-05-06 11:27:33 -04:00
Carol (Nichols || Goulding) 7286b4391a
fix: Remove db crate 2022-05-06 11:27:33 -04:00
Jake Goulding ece38417bc
feat: Remove the `run database` command 2022-05-06 09:48:30 -04:00
Jake Goulding b939d80db8
feat: Remove the `run router` command 2022-05-06 09:48:30 -04:00
Carol (Nichols || Goulding) e015d3bafb
feat: Remove the server_benchmarks crate (#4506) 2022-05-02 18:20:01 +00:00
Andrew Lamb 9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) (#4428) 
* refactor: Move all UDF implementations to query_function crate

* refactor: Move regex udf to query_functions

* refactor: Move functions out of query

* fix: lints, imports

* chore: Run cargo hakari tasks

* fix: clipy + benches

* fix: reduce borrowing and fix clippy

* fix: moar clippy

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-26 17:30:27 +00:00
二手掉包工程师 4b47d723b1
refactor: Rename time to iox_time (#4416) 
Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-26 00:19:59 +00:00
Paul Dix 5bf4550259
feat: add object store service to router (#4338) 
Add method to catalog to get parquet file by object store id.
Add gRPC service for object store to get a file from by its uuid.
Add the object store service to router2 with object store config.
 2022-04-16 17:58:31 +00:00
Paul Dix 99cbb28a89
feat: add initial catalog service to router (#4316) 
Create new crate for iox_catalog_service.
Add rpc to return parquet_file records by partition id.
Add CatalogService to router2.

The catalog service will be added to over time to provide access to the catalog over gRPC.
 2022-04-14 17:39:18 +00:00
Andrew Lamb edda409b19
refactor: Extract `ioxd_test`, `ioxd_compactor`, `ioxd_ingester`; remove `ioxd` (#4210) 
* refactor: Extract test, compactor, ingester, and test

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-03 10:42:22 +00:00
Andrew Lamb 833c10c083
feat: return write_token from HTTP writes to router2 (#4202) 
* feat: return write_token from HTTP writes to router2

* fix: Update router2/src/dml_handlers/instrumentation.rs

Co-authored-by: Dom <dom@itsallbroken.com>

* refactor: Use WriteSummary::default more vigorously

* fix: fix typo and add links to follow on issues

Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-04-02 10:34:51 +00:00
Andrew Lamb 532d227d11
refactor: extract router2 into ioxd_router2 (#4183) 
* refactor: extract router2 from ioxd

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-31 17:39:05 +00:00
Andrew Lamb 367e926d35
refactor: extract querier into ioxd_querier (#4182) 
* refactor: extract querier into ioxd_querier

* fix: dep
 2022-03-31 16:03:31 +00:00
Andrew Lamb f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` (#4162) 
* refactor: rename influxdb_ioxd to ioxd

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-30 13:31:27 +00:00
Andrew Lamb 4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd (#4149) 
* refactor: Extract common, OG database and router out of influxdb_ioxd

* chore: Run cargo hakari tasks

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-29 13:07:19 +00:00
Andrew Lamb fb75ef7b82
test: add end to end test for all in one mode, restructure fixture (#4114) 
* test: add end to end test for all in one mode, restructure fixture

* docs: fix typos and clarify schema requrements
 2022-03-24 12:53:25 +00:00
Andrew Lamb 29b89aaca7
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd (#4106) 
* refactor: extract grpc service implementations out of influxdb_ioxd

* chore: Run cargo hakari tasks

* refactor: rename server_common to service_common

* refactor: rename server_grpc_influxrpc to service_grpc_influxrpc

* refactor: rename server_grpc_flight to service_grpc_flight

* refactor: rename server_grpc_testing to service_grpc_testing

* fix: Cargo.toml

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2022-03-23 20:14:45 +00:00
Nga Tran f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates (#4013) 
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates

* chore: remove unused import
 2022-03-10 18:17:58 +00:00
Andrew Lamb d2c0acdd46
refactor: Remove serving readiness gate (#3986) 
* refactor: Remove serving_readiness

* fix: remove more

* fix: remove test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-09 12:17:44 +00:00
Andrew Lamb d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate (#3908) 
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox

split out serving readiness, get compiling

* fix: hakari

* fix: hakari again

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-03-03 16:48:30 +00:00
Carol (Nichols || Goulding) 8f3e44bf76
refactor: Extract a crate for shared data types in the new design 2022-03-02 12:16:15 -05:00
Raphael Taylor-Davies 43ada68f37
chore: reset release codegen-units to default (#3883) 2022-03-01 13:44:14 +00:00
Marco Neumann 9079e6ddb0
feat: backoff retries in ingester (#3841) 
* feat: add `backoff` crate

* feat: backoff retries in ingester

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-23 17:58:16 +00:00
Marco Neumann 459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` (#3819) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-23 09:10:43 +00:00
Luke Bond 0f012de70c
feat: adding compactor CLI command and crate 
Closes: #3777
 2022-02-21 12:24:09 +00:00
Luke Bond a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor (#3750) 
* chore: port sqlx-hotswap-pool over from conductor

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>

* chore: workspace hack fixes

* fix: unique schema per test db connection

* fix: adjust search path in catalog pg tests to see if it fixes test schema issue

* fix: actually fixed sqlx hotswap pool test

Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-15 16:18:36 +00:00
Luke Bond 54890525da
feat: gitops adapter ci build (#3692) 
* chore: remove references to perf_image in CI

* chore: adding gitops adapter image build in CI

* chore: gitops adapter bin now same as dir & package so docker build works

* fix: circle config package change after renaming gitops adapter package
 2022-02-10 11:23:06 +00:00
Carol (Nichols || Goulding) 73828323ac
feat: Ingester Flight gRPC API (#3623) 
* feat: Add a way to run ingester with an in-memory catalog from the CLI

If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.

Planning on using this in tests, so not documenting.

* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC

Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.

* feat: Add a flight API (handshake only) to ingester

* fix: Create partitions if using file-based write buffer

* fix: Change the server fixture to handle ingester server type

For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.

* feat: Start implementing ingester do_get, namely decoding the query

Skip serialization of the predicate for the moment.

* refactor: Rename ingest protos to ingester to match crate name

* refactor: Rename QueryResults to QueryData

* feat: Move ingester flight client to new querier crate

* fix: Off by one error, different starting indexes in sequencers

* fix: Create new CLI argument to pick the catalog type

* fix: Create a CLI option to set the number of topics to auto-create in the write buffer

* fix: Check the arrow flight service's health to tell that the ingester gRPC is up

* fix: Set postgres as the default catalog type

* fix: Return an error rather than panicking if CLI args aren't right
 2022-02-09 19:07:44 +00:00
Luke Bond de2a013786
feat: gitops adapter (#3656) 
* feat: scaffolding of gitops adapter bin crate

* chore: refactor gitops adapter; calls CLI now; status update fixes

* feat: gitops adapter now calls out to CLI once per topic; improved tests

* chore: add mock failure script for gitops adapter

* chore: update workspace-hack

* chore: refactor away unecessary to_string in gitops syncer

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-08 13:27:36 +00:00
Edd Robinson c1f5994660
refactor: move sql parsing -> RPC predicate into own crate (#3604) 
* chore: create crate

* refactor: move module to new crate
 2022-02-02 10:41:57 +00:00
NGA-TRAN ef336b4659 feat: add ingester crate and a few basic data structures for its data lifecycle 2022-01-17 15:38:03 -05:00
Paul Dix 4764e71c54 feat: Add initial iox_catalog skeleton 2022-01-17 09:56:20 -05:00
Dom a8cb8755de feat: new router2 crate 
This commit adds an almost-empty router2 crate containing enough of a
skeleton to plumb into the IOx CLI/server runner.
 2022-01-12 14:43:10 +00:00
Carol (Nichols || Goulding) f9174c483b
refactor: Extract server::db into its own crate (#3417) 
* refactor: Extract JobRegistry from the server crate

Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.

* refactor: Move db out of server into its own crate

Fixes #2821.
 2021-12-23 22:01:17 +00:00
Carol (Nichols || Goulding) 183de4519f
feat: Add incremental = true to the quick-release profile 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 255cf79104
docs: Explain tradeoffs and use cases of the different release profiles 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 758a17e199
feat: Create a build-time-optimized release build profile 
Fixes #3316.
 2021-12-08 09:53:19 -05:00
Carol (Nichols || Goulding) 994938b2f0
fix: Alphabetize/organize deps 2021-12-06 09:36:30 -05:00
Marco Neumann f2f090bde7 feat: test bench 2021-11-25 18:14:40 +01:00
Carol (Nichols || Goulding) f66cd681c2
feat: Results of using cargo hakari init workspace-hack 2021-11-19 09:20:35 -05:00
Carol (Nichols || Goulding) a2454b542d
fix: Small cleanups in Cargo.tomls (#3160) 
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles

* fix: Alphabetize dependencies

* fix: Add the data_types_conversions feature to get tests passing

* fix: Remove dev dependencies already listed under normal dependencies

* fix: Make sure the workspace is using the new resolver
 2021-11-18 22:26:33 +00:00
kodiakhq[bot] 49cfa046bb
Merge branch 'main' into crepererum/smaller_binaries 2021-11-12 11:17:09 +00:00
Raphael Taylor-Davies 6f268f8260
refactor: extract DML types (#2731) (#3084) 
* refactor: extract DML types (#2731)

* chore: fmt
 2021-11-11 12:34:07 +00:00
Marco Neumann 16fc610433 chore: faster and smaller IOx binaries 
Use `codegen-units = 1`, thin-LTO and debug section compression to make our binary smaller (which is good for deploy and
test times) and faster.

# Summary

The binary size of `influxdb_iox` after building with:

```console
$ cargo build --release --no-default-features --features="aws,gcp,azure,jemalloc_replacing_malloc"
```

The profile was:

```toml
[profile.release]
debug = true
```

The commit was:

```text
89ece8b493
```

The size results are:

| Method                                     | Size  |
| ------------------------------------------ | ----- |
| baseline                                   | 833MB |
| baseline + dbg compression                 | 222MB |
| baseline + strip                           |  49MB |
| codegen-units                              | 520MB |
| codegen-units + strip                      |  40MB |
| codegen-units + dbg compression            | 143MB |
| thin LTO                                   | 715MB |
| thin LTO + strip                           |  49MB |
| thin LTO + dbg compression                 | 199MB |
| codegen-units + thin LTO                   | 449MB |
| codegen-units + thin LTO + strip           |  40MB |
| codegen-units + thin LTO + dbg compression | 130MB |

For the methods that were successfully measured I couldn't really see any compile time differences on my laptop.

# Methods

## Strip
Remove debug symbols. We don't really want this, so this is just to get an idea of the size

```console
$ strip baseline
```

## Debug Sections compression
Debug sections make a large amount of our binary size (a stripped executable is 49MB instead of 833MB). Since we like to
have debug symbols we cannot just strip them. However these symbols are only used for:

- backtrace generation (something went wrong, not BAU)
- profiling
- debugging

So in normal operation and most test scenarios, we're just wasting memory. So we could compress them:

```console
$ objcopy --compress-debug-sections baseline baseline-dbg_compressed
```

There is also elfutils:

```console
$ eu-elfcompress test
```

Elfutils nearly ends up with the same size (220MB instead of 222MB that objcopy achieves), but takes more time and is
probably not worth it.

Note that compressed debug sections exist since many years. The Rust ecosystem supports reading them since over a year,
see:

- <https://github.com/gimli-rs/gimli/issues/195>
- <https://github.com/rust-lang/backtrace-rs/issues/342>

## Codegen Units
The rust compiler parallelizes codegen work. This split into units however means that optimizations are somewhat
limited. This can be change by:

```toml
[profile.release]
...
codegen-units = 1
```

As a nice side effect this should also make our code faster.

## Thin LTO
Get LLVM to run "thin" Link Time Optimization:

```toml
[profile.release]
...
lto = "thin"
```

As a nice side effect this should also make our code faster.

## Fat LTO
Get LLVM to run "fat" Link Time Optimization:

```toml
[profile.release]
...
lto = "fat"
```

There are no results for this because this took a massive amount of memory and CPU time and did not finish on my system.
 2021-11-10 15:49:19 +01:00
Raphael Taylor-Davies 60f0deaf1e
feat: remove flatbuffer entry (#3045) 2021-11-05 20:19:24 +00:00
Raphael Taylor-Davies 405ed6db1b
feat: add mutable_batch_tests (#2724) (#3011) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-03 18:15:35 +00:00
Marco Neumann a91e572b43 feat: implement basic router server type 
This only implements the deployment service and nothing else yet.
 2021-11-03 16:34:51 +01:00
Raphael Taylor-Davies 3010ff4176
feat: write entry to MutableBatch (#2972) 
* feat: write entry to MutableBatch

* feat: error on empty TableBatch

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-27 14:05:22 +00:00
Marco Neumann fc651ea193 chore: move `gprc-router-test-gen` to top level 2021-10-26 11:02:33 +02:00
Marco Neumann a0cdca7781 chore: move `influxdb_iox` into a proper workspace package 2021-10-26 11:02:33 +02:00
Marco Neumann 2833cefc12 test: remove Kafka from end2end tests 
Kafka is now sufficiently tested via the `write_buffer` crate. The
end2end tests can now use the in-memory mock implementation or -- if
servers can only be controlled via CLI -- the file-based implementation.
 2021-10-26 10:09:34 +02:00
Marco Neumann 81ccf85104 refactor: integrate print-cpu into main binary 
It's a less confusing structure and also allows us to have this
information available in our shipped binary.
 2021-10-25 18:29:32 +02:00
Marco Neumann 7a28982e2a fix: fix dependency features for `influxdb_iox` 2021-10-25 11:15:19 +02:00
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Raphael Taylor-Davies d2b41c5a13
feat: write pbdata to MutableBatch (#2724) (#2927) 
* feat: write pbdata to MutableBatch (#2724)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 14:32:35 +00:00
Raphael Taylor-Davies f64e749eb9
feat: write line protocol to MutableBatch (#2914) 
* feat: write line protocol to MutableBatch

* chore: fix lint

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-21 08:12:54 +00:00
Andrew Lamb 9974a5364c
chore(security): Replace prettytable with comfy-table (#2905) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-20 10:44:36 +00:00
Andrew Lamb a82dc6f5f0
chore: Update datafusion + arrow (#2903) 
* chore: Update datafusion to latest, arrow to 6.0.0

* fix: Update tests

* fix: bubble internal error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-19 17:14:08 +00:00
kodiakhq[bot] 0be89715e2
Merge branch 'main' into dependabot/cargo/rdkafka-0.27.0 2021-10-19 12:55:21 +00:00
Marco Neumann 7dafb0ff74 chore: remove unused `influxdb_iox` => `synchronized-writer` dep 2021-10-19 14:46:26 +02:00
dependabot[bot] 32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0 
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases)
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: rdkafka
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-19 11:49:34 +00:00
Marco Neumann 157b556d4c
feat: make flight responses streaming (#2876) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-18 15:01:50 +00:00
dependabot[bot] 70555ab33d
chore(deps): bump structopt from 0.3.23 to 0.3.25 (#2877) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.23 to 0.3.25.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.23...v0.3.25)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-10-18 14:49:15 +00:00
Raphael Taylor-Davies bdd6d67e7a
refactor: split out mutable_batch crate (#2841) 
* refactor: split out mutable_batch crate

* refactor: restore chunk module for better diffs

* chore: fmt

* chore: review feedback

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-14 16:46:54 +00:00
Marco Neumann 28195b9c0c chore: new `parquet_catalog` crate 2021-10-14 14:34:59 +02:00
Raphael Taylor-Davies 0554173684
feat: migrate write buffer to TimeProvider (#2722) (#2804) 
* feat: migrate write buffer to TimeProvider (#2722)

* chore: review feedback

Co-authored-by: Marco Neumann <marco@crepererum.net>

Co-authored-by: Marco Neumann <marco@crepererum.net>
 2021-10-12 10:32:34 +00:00
Raphael Taylor-Davies 7bd32a4916
feat: add time provider abstraction (#2722) (#2792) 
* feat: add time provider abstraction (#2722)

* chore: tests

* chore: more tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-11 15:00:29 +00:00
dependabot[bot] 61190a9d08
chore(deps): bump predicates from 2.0.2 to 2.0.3 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 14:32:08 +00:00
Raphael Taylor-Davies afe34751e7
refactor: split out schema crate (#2781) 
* refactor: split out schema crate

* chore: fix doc
 2021-10-11 09:45:08 +00:00
dependabot[bot] 1327735537
chore(deps): bump thiserror from 1.0.29 to 1.0.30 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.29 to 1.0.30.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.29...1.0.30)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 08:01:21 +00:00
dependabot[bot] 01a44899fd
chore(deps): bump assert_cmd from 2.0.1 to 2.0.2 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-11 01:45:35 +00:00
dependabot[bot] 7a6f381e69
chore(deps): bump arrow-flight from 5.4.0 to 5.5.0 
Bumps [arrow-flight](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow-flight
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 13:27:26 +00:00
kodiakhq[bot] 181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0 2021-10-04 13:10:42 +00:00
dependabot[bot] 6d9205d87c chore(deps): bump parquet from 5.4.0 to 5.5.0 
Bumps [parquet](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 12:15:11 +01:00
dependabot[bot] d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0 
Bumps [arrow](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 08:55:38 +00:00
Raphael Taylor-Davies 86cee568d5
feat: use upstream pbjson (#2650) 
* feat: use upstream pbjson

* chore: fmt
 2021-09-28 16:29:26 +00:00
Raphael Taylor-Davies daa2ec2f4c
revert: "fix: Revert "feat: remove routerify (#2586)" (#2623)" (#2649) 
This reverts commit 18941fcbff.
 2021-09-28 14:42:17 +00:00
Andrew Lamb 18941fcbff
fix: Revert "feat: remove routerify (#2586)" (#2623) 
This reverts commit 9fba7b0ba0.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-24 09:50:24 +00:00
Raphael Taylor-Davies 92e6173f49
feat: implement jaeger-agent protocol directly (#2607) 
* feat: implement jaeger-agent protocol directly

* chore: review feedback

* fix: remove jaeger feature flag
 2021-09-22 17:30:37 +00:00
Raphael Taylor-Davies 46088a7ff1
feat: disable rustyline dirs-next dependency (#2579) 2021-09-20 14:42:07 +00:00
Raphael Taylor-Davies 9fba7b0ba0
feat: remove routerify (#2586) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-20 08:57:19 +00:00
dependabot[bot] 1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0 
Bumps [dirs](https://github.com/soc/dirs-rs) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases)
- [Commits](https://github.com/soc/dirs-rs/commits)

---
updated-dependencies:
- dependency-name: dirs
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-20 08:28:04 +00:00
Raphael Taylor-Davies 3ffb16daa6
feat: remove parse_duration (#2574) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-19 11:27:24 +00:00
Raphael Taylor-Davies 1d55d9a1b5
feat: add pbjson support (#2468) 
* feat: add pbjson support

* chore: fix test
 2021-09-16 07:33:27 +00:00
Raphael Taylor-Davies c66095cad1
feat: remove metrics crate (#2552) 2021-09-15 19:43:33 +00:00
Raphael Taylor-Davies 6e7fa3e574
feat: migrate http ingest metrics (#2542) 
* feat: migrate http ingest metrics

* chore: review feedback

* refactor: RAII entry ingest recorder
 2021-09-15 19:01:10 +00:00
Marco Neumann bfaba78dc3 refactor: move `predicate` into its own crate 
Two reasons:

1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
   small follow-up PR).
2. `predicate` will have more and more features (like serialization)
   which justifies a new home
 2021-09-14 17:13:02 +02:00
Marco Neumann 3f2e46c397 feat: prune old transactions from preserved catalog 2021-09-14 12:08:17 +02:00
Raphael Taylor-Davies e07bc85670
feat: enable tracing for http API (#2273) (#2498) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-10 11:21:02 +00:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
kodiakhq[bot] 06311970fa
Merge branch 'main' into dependabot/cargo/parquet-5.3.0 2021-09-06 09:00:27 +00:00
dependabot[bot] ccfd6ad3da
chore(deps): bump parquet from 5.2.0 to 5.3.0 
Bumps [parquet](https://github.com/apache/arrow-rs) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.3.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.2.0...5.3.0)

---
updated-dependencies:
- dependency-name: parquet
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 08:52:30 +00:00
kodiakhq[bot] 0fd92c8b45
Merge branch 'main' into dependabot/cargo/structopt-0.3.23 2021-09-06 08:51:32 +00:00