833c10c083
feat: return write_token from HTTP writes to router2 ( #4202 )
...
* feat: return write_token from HTTP writes to router2
* fix: Update router2/src/dml_handlers/instrumentation.rs
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: Use WriteSummary::default more vigorously
* fix: fix typo and add links to follow on issues
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-02 10:34:51 +00:00
e8b0655ac8
chore(deps): Bump clap from 3.1.6 to 3.1.7 ( #4199 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.6 to 3.1.7.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.6...v3.1.7 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-01 09:49:30 +00:00
d37af1a7f5
fix: include git sha (again) in release build ( #4193 )
...
* fix: error if git-sha can not be found
* refactor: move main to influxdb_iox
* fix: fmt
2022-03-31 19:14:21 +00:00
532d227d11
refactor: extract router2 into ioxd_router2 ( #4183 )
...
* refactor: extract router2 from ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-31 17:39:05 +00:00
367e926d35
refactor: extract querier into ioxd_querier ( #4182 )
...
* refactor: extract querier into ioxd_querier
* fix: dep
2022-03-31 16:03:31 +00:00
a1df864283
feat: Support 'SHOW NAMESPACES' in sql repl ( #4164 )
...
* feat: Support `SHOW NAMESPACES` in sql repl
* feat: add basic support to clients
* fix: add get_namespaces service test
* fix: proper error handling
* test: end to end test for namespace client
* refactor: Use QuerierDatabase rather than Catalog
* refactor: remove unused function
2022-03-31 12:57:33 +00:00
fc4635a334
chore(deps): Bump lock_api from 0.4.6 to 0.4.7 ( #4186 )
...
Bumps [lock_api](https://github.com/Amanieu/parking_lot ) from 0.4.6 to 0.4.7.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/lock_api-0.4.6...lock_api-0.4.7 )
---
updated-dependencies:
- dependency-name: lock_api
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-31 07:45:03 +00:00
22b24bdab3
chore: Update datafusion again ( #4148 )
...
* chore: update datafusoon
* refactor: Update for DataFusion API changes
* chore: TEMP TEMP change df to local copy
* chore: Update to datafusion again
* fix: Update Cargo.lock
* fix: logical conflict
2022-03-30 16:51:48 +00:00
f2a3dd58b2
refactor: rename `influxdb_ioxd` to `ioxd` ( #4162 )
...
* refactor: rename influxdb_ioxd to ioxd
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 13:31:27 +00:00
9950bcee27
chore(deps): Bump indexmap from 1.8.0 to 1.8.1 ( #4166 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.8.0...1.8.1 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-30 08:49:07 +00:00
4ca52e5ae0
refactor: Extract common, OG database and router out of influxdb_ioxd ( #4149 )
...
* refactor: Extract common, OG database and router out of influxdb_ioxd
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-29 13:07:19 +00:00
decd018a6a
refactor: remove querier sync loop ( #4146 )
...
Namespaces are now created on demand and contain their full schema.
Tombstones/chunks are created on demand during the query.
Closes #4123 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 11:47:23 +00:00
e154d77bf5
chore(deps): Bump getrandom from 0.2.5 to 0.2.6 ( #4155 )
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.5 to 0.2.6.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.5...v0.2.6 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-29 08:53:34 +00:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
a83ad1ce2b
chore(deps): Bump syn from 1.0.89 to 1.0.90 ( #4150 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.89 to 1.0.90.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.89...1.0.90 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:28:30 +00:00
80b7e9cce1
feat: delete fully processed tombstones & integration tests for find_and_compact ( #4116 )
...
* feat: remove fully processed tombstones
* test: first few tests
* fix: delete SQL
* fix: test how IN (...) works in PG
* fix: test how IN (?) works in PG
* fix: test how IN (?) works in PG
* fix: dynamically add IN (?, ?, ...)
* fix: dynamically add IN (?, ?, ...) & its dynamic values
* fix: add argument directly in the SQL
* test: more tests for catalog read and update functions
* chore: move a subfunction to make it easier to read)
* test: first test for find_can_compact but disabled due to bug
* test: integration tests and a bug fix for find_and_compact
* chore: cleanup
* refactor: address review comments
* fix: put 2 delete processed tombstones and tombstones in a transaction
2022-03-28 18:35:54 +00:00
fb186c6733
refactor: `QueryDatabaseProvider::db` should be async ( #4143 )
...
This is required to fetch querier namespaces on demand.
Ref #4123 .
2022-03-28 11:18:20 +00:00
4f9515ffba
chore(deps): Bump async-trait from 0.1.52 to 0.1.53 ( #4141 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.52 to 0.1.53.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.52...0.1.53 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:55:24 +00:00
e5bbc74f7a
chore(deps): Bump paste from 1.0.6 to 1.0.7 ( #4140 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.6 to 1.0.7.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.6...1.0.7 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-28 08:44:01 +00:00
58c630d709
chore: Update datafusion ( #4133 )
...
* chore: Update datafusion
* fix: typo
* fix: Update explain plan output
* fix: update Cargo.locl
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-25 15:08:39 +00:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
8ca5c337b2
refactor: port more query tests to NG, some code clean up ( #4125 )
...
* refactor: inline function that is used once
* refactor: generalize multi-chunk creation for NG
* refactor: `TwoMeasurementsManyFieldsTwoChunks` is OG-specific
* refactor: generalize `OneMeasurementTwoChunksDifferentTagSet`
* refactor: port `OneMeasurementFourChunksWithDuplicates` to NG
* refactor: `TwoMeasurementsManyFieldsLifecycle` is OG-specific
* refactor: simplify NG chunk generation
* refactor: port `ThreeDeleteThreeChunks` to NG
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:07:09 +00:00
fb75ef7b82
test: add end to end test for all in one mode, restructure fixture ( #4114 )
...
* test: add end to end test for all in one mode, restructure fixture
* docs: fix typos and clarify schema requrements
2022-03-24 12:53:25 +00:00
29b89aaca7
refactor: extract influxrpc, flight and testing gRPC out of influxdb_ioxd ( #4106 )
...
* refactor: extract grpc service implementations out of influxdb_ioxd
* chore: Run cargo hakari tasks
* refactor: rename server_common to service_common
* refactor: rename server_grpc_influxrpc to service_grpc_influxrpc
* refactor: rename server_grpc_flight to service_grpc_flight
* refactor: rename server_grpc_testing to service_grpc_testing
* fix: Cargo.toml
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-23 20:14:45 +00:00
51da6dd7fa
feat: store sort key in NG metadata ( #4110 )
...
The sort key is optional and currently only produced by `iox_tests`.
Writing it within the ingester/compactor is tracked by #3968 . The sort
key is read by the querier (and this will be verified by the query tests
and is required to merge #4103 ).
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-23 18:24:46 +00:00
8ee9b793f5
chore(deps): Bump hyper from 0.14.17 to 0.14.18 ( #4109 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.17 to 0.14.18.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.17...v0.14.18 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-23 13:12:43 +00:00
36071e2d12
chore(deps): Bump log from 0.4.14 to 0.4.16 ( #4108 )
...
Bumps [log](https://github.com/rust-lang/log ) from 0.4.14 to 0.4.16.
- [Release notes](https://github.com/rust-lang/log/releases )
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/log/commits )
---
updated-dependencies:
- dependency-name: log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-23 13:01:30 +00:00
89206e013c
test: run SOME query tests for querier ( #4098 )
...
This includes some type changes to dispatch between OG and NG and allows
some tests to be run against the NG querier. This only contains parquet
files though, so it's somewhat a limited scope.
For #3934 .
2022-03-22 17:39:19 +00:00
b83b000590
chore: Update datafusion ( #4071 )
...
* chore: update to datafusion 5936edc2a94d5fb20702a41eab2b80695961b9dc
* chore: Update apis to match datafusion changes
2022-03-22 13:17:41 +00:00
55643945a1
refactor: `querier` w/o `db` ( #4063 )
...
* feat: `TombstoneRepo::list_by_table`
* feat: `ParquetFileRepo::list_by_table_not_to_delete`
* refactor: `querier` w/o `db`
Get the `querier` to work w/o relying on `db`. A few notes:
- Testing is kinda shallow, we really need to get `query_tests` working
w/ `querier` (see #3934 ).
- We still run a sync loop for namespaces, tables and schemas. This will
be a replaced by "update namespace incl. tables and schemas on demand".
Note however that we cannot fetch single tables and schemas on demand
at the moment, because DataFusion doesn't implement async schema
inspection (only `scan` / "give me all the chunks" is async). I think
that's OK for now and we can address this later.
- There is NO cache for parquet files and tombstones at the moment. For
correctness, they need to be fetched in a single transaction (or we
need a kinda tricky sequence number / logical clock tracking) and I am
not sure yet how this makes sense when we have the ingester data wired
up and predicates pushed down to the catalog (see next point). So
let's measure first and then decide on a caching strategy for this.
- Predicates are currently NOT pushed down to the catalog. I'll need to
figure out how to extract time range from generic DataFusion
expressions to make that work (it's easier for InfluxRPC queries, but
they are not tested at the moment, see first point).
Sorry that this commit is kinda huge. I initially planned to only
migrate the chunks away from `db` and leave the tables and schemas for a
follow-up PR, but the DataFusion trait structure (chunks are bound to
their tables) makes this kinda pointless.
Closes #3974 .
* docs: explain what we're doing
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* docs: mention tracking issues
* docs: explain what we're doing
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-03-21 16:58:00 +00:00
a23efce408
chore(deps): Bump kube-derive from 0.69.1 to 0.70.0 ( #4073 )
...
* chore(deps): Bump kube-derive from 0.69.1 to 0.70.0
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.69.1 to 0.70.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.1...0.70.0 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump kube-runtime from 0.69.1 to 0.70.0
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.69.1 to 0.70.0.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.1...0.70.0 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: upgrade kube to version 0.70
* chore: hakari
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-03-21 15:32:45 +00:00
933dc69ecf
feat: For each compacted data set, persist new parquet file to object store ( #4058 )
...
* feat: Rearrange skeleton functions for split/persist/catalog update
* feat: Persist compacted files to object storage
Fixes #3951 .
* docs: Add comment about batches' schemas
2022-03-21 14:16:03 +00:00
cd36229e27
chore(deps): Bump synchronized-writer from 1.1.10 to 1.1.11 ( #4075 )
...
Bumps [synchronized-writer](https://github.com/magiclen/synchronized-writer ) from 1.1.10 to 1.1.11.
- [Release notes](https://github.com/magiclen/synchronized-writer/releases )
- [Commits](https://github.com/magiclen/synchronized-writer/compare/v1.1.10...v1.1.11 )
---
updated-dependencies:
- dependency-name: synchronized-writer
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-21 10:53:11 +00:00
66cf34c2e2
chore(deps): Bump tokio-rustls from 0.23.2 to 0.23.3 ( #4074 )
...
Bumps [tokio-rustls](https://github.com/tokio-rs/tls ) from 0.23.2 to 0.23.3.
- [Release notes](https://github.com/tokio-rs/tls/releases )
- [Commits](https://github.com/tokio-rs/tls/commits )
---
updated-dependencies:
- dependency-name: tokio-rustls
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-21 10:43:07 +00:00
836aecc7ad
chore(deps): Bump libc from 0.2.120 to 0.2.121 ( #4076 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.120 to 0.2.121.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.120...0.2.121 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-21 09:28:39 +00:00
a122b1e2ca
refactor: dyn-typed DB for `query_tests` ( #4053 )
...
To test the `db::Db` as well as the `querier` with the same test
framework, they require a shared interface. Ideally this interface is
dynamically typed instead of static dispatched via generics because:
- `query_tests` already take ages to compile
- we often hold a list of scenarios and a single scenario should (in a
future PR) be able to represent both OG as well as NG
The vision here is that we basically keep the whole test setup but add
new scenarios which are NG-specific later on.
Now the issue w/ many query-related types is that they are NOT
object-safe because methods that don't take `&self` or they have
associated types that we cannot specify in general for OG and NG at the
same time.
So we need a bunch of wrappers that make dynamic dispatch possible. They
mostly call to an internal "interface" crate which is the actual `dyn`
part. The interface is currently only implemented for OG.
The scenarios currently also only contain OG databases. However,
creating a dynamic interface that can be used in all `query_tests` is
already a huge step.
Note that there are two places where we downcast the dynamic/abstract
database to `db::Db` again:
1. To create one scenario based on another and where we need to
manipulate `db::Db` with OG-specific semantics.
2. `server_benchmarks`. These contain OG databases only and there is no
point in benchmarking throw the dynamic dispatch interface because
prod (`influxdb_ioxd`) also uses static dispatch.
Ref #3934 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-18 10:11:17 +00:00
d9900f661b
refactor: ingester table & namespace count metrics
...
Record the number of tables / namespaces an ingester process has
observed.
2022-03-17 17:20:30 +00:00
58256c32ff
chore(deps): Bump syn from 1.0.88 to 1.0.89 ( #4050 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.88 to 1.0.89.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.88...1.0.89 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-17 11:30:42 +00:00
f4cd5c9ba8
Merge branch 'main' into dom/object-store-metrics
2022-03-15 14:27:19 +00:00
cc7f71126c
chore(deps): Bump syn from 1.0.86 to 1.0.88 ( #4037 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.86 to 1.0.88.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.86...1.0.88 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 12:22:18 +00:00
e2cf5c63e5
chore(deps): Bump nom from 7.1.0 to 7.1.1 ( #4035 )
...
Bumps [nom](https://github.com/Geal/nom ) from 7.1.0 to 7.1.1.
- [Release notes](https://github.com/Geal/nom/releases )
- [Changelog](https://github.com/Geal/nom/blob/main/CHANGELOG.md )
- [Commits](https://github.com/Geal/nom/compare/7.1.0...7.1.1 )
---
updated-dependencies:
- dependency-name: nom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-15 11:29:01 +00:00
eeb918ca2b
chore(deps): Bump libc from 0.2.119 to 0.2.120 ( #4034 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.119 to 0.2.120.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.119...0.2.120 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-15 10:56:52 +00:00
10831a0530
chore(deps): Bump reqwest from 0.11.9 to 0.11.10 ( #4032 )
...
Bumps [reqwest](https://github.com/seanmonstar/reqwest ) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases )
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10 )
---
updated-dependencies:
- dependency-name: reqwest
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-15 10:43:28 +00:00
8f1938a482
chore: Update datafusion ( #4022 )
...
* chore: Update datafusion
* chore: update for change in Expr
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-14 17:24:00 +00:00
27efb66237
test: add proptest for `AddressableHeap` ( #4025 )
...
* test: add proptest for `AddressableHeap`
For #3985 .
* refactor: simplify code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-03-14 12:38:30 +00:00
3f0f090c4e
chore(deps): Bump pretty_assertions from 1.1.0 to 1.2.0 ( #4024 )
...
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions ) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases )
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md )
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/commits )
---
updated-dependencies:
- dependency-name: pretty_assertions
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-14 10:33:27 +00:00
5a29d070ea
feat: Implement the compact function for NG Compactor ( #4001 )
...
* feat: initial implementation of compact a given list of overlapped parquet files
* feat: Add QueryableParquetChunk and some refactoring
* feat: build queryable parquet chunks for parquet files with tombstones
* feat: second half the implementation for Compactor's compact. Tests will be next
* fix: comments for trait funnctions fof QueryChunkMeta
* test: add tests for compactor's compact function
* fix: typos
* refactor: address Jake's review comments
* refactor: address Andrew's comments and add one more test for files in different order in the vector
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 20:25:19 +00:00
bfc473b8cd
feat: object store instrumentation
...
Implements a decorator of the ObjectStoreAPI trait, recording:
* Bytes uploaded / downloaded through the instrumented API
* Call latencies, broken down by operation & success / error state
All the current implementations that return a Stream from the get
operation actually return a "fake" stream containing all the data in one
go rather than streaming chunks from the upstream. I've instrumented the
Stream to future-proof any actual streaming impls in the future.
2022-03-11 13:38:48 +00:00
bd2824aa59
chore(deps): Bump pprof from 0.6.2 to 0.7.0 ( #4004 )
...
* chore(deps): Bump pprof from 0.6.2 to 0.7.0
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.6.2 to 0.7.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: `pprof` features for version 0.7
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-11 11:08:59 +00:00
f03ebd79ab
refactor: move querier's test utils to a new crate to get reused by tests in other crates ( #4013 )
...
* refactor: move querier's test utils to a new crate to be able resued by tests in other crates
* chore: remove unused import
2022-03-10 18:17:58 +00:00
67209fce7c
chore(deps): Bump clap from 3.1.5 to 3.1.6 ( #4010 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.5...v3.1.6 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:38:53 +00:00
e1344f7ca8
chore(deps): Bump sqlparser from 0.14.0 to 0.15.0 ( #4007 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.14.0...v0.15.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:28:48 +00:00
68e09264f2
chore(deps): Bump pbjson from 0.2.2 to 0.2.3 ( #4006 )
...
Bumps [pbjson](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:20:10 +00:00
f2834c4101
chore(deps): Bump handlebars from 4.2.1 to 4.2.2 ( #4009 )
...
Bumps [handlebars](https://github.com/sunng87/handlebars-rust ) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/sunng87/handlebars-rust/releases )
- [Changelog](https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md )
- [Commits](https://github.com/sunng87/handlebars-rust/compare/v4.2.1...v4.2.2 )
---
updated-dependencies:
- dependency-name: handlebars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:11:19 +00:00
c051e944f2
chore(deps): Bump tracing from 0.1.31 to 0.1.32 ( #4008 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.31 to 0.1.32.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.31...tracing-0.1.32 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 14:01:00 +00:00
cef4090bce
chore(deps): Bump pbjson-types from 0.2.2 to 0.2.3 ( #4005 )
...
Bumps [pbjson-types](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 13:52:08 +00:00
ed84b453d3
chore(deps): Bump tracing-core from 0.1.22 to 0.1.23 ( #4003 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.22...tracing-core-0.1.23 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 13:43:15 +00:00
0756642137
chore(deps): Bump pbjson-build from 0.2.2 to 0.2.3 ( #4002 )
...
Bumps [pbjson-build](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-10 13:34:00 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
944f628e29
fix: Remove data_types as a dependency of ng compactor ( #3993 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 17:03:02 +00:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files ( #3973 )
...
* feat: main function for finding and compacting parquet files
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename file and struct
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 16:34:43 +00:00
3278e35b41
build: bump regex
...
Bump regex version due to RUSTSEC-2022-0013, see:
https://rustsec.org/advisories/RUSTSEC-2022-0013
2022-03-08 15:37:23 +00:00
baae6910f2
build: remove openssl dependency
...
Switches a stray reqwest dependency to use rustls.
2022-03-07 13:47:39 +00:00
e47bfc40be
Merge branch 'main' into dom/router-grpc-write
2022-03-07 11:13:19 +00:00
d3321135eb
chore(deps): Bump clap from 3.1.2 to 3.1.5 ( #3957 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.2 to 3.1.5.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.2...v3.1.5 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-07 09:58:46 +00:00
3c58a08431
chore(deps): Bump siphasher from 0.3.9 to 0.3.10 ( #3956 )
...
Bumps [siphasher](https://github.com/jedisct1/rust-siphash ) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/jedisct1/rust-siphash/releases )
- [Commits](https://github.com/jedisct1/rust-siphash/compare/0.3.9...0.3.10 )
---
updated-dependencies:
- dependency-name: siphasher
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-07 09:10:09 +00:00
48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 ( #3955 )
...
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-07 08:54:20 +00:00
e09f39d6a0
chore: Update datafusion ( #3943 )
...
* chore: Update datafusion
* refactor: update for new datafusion
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-04 19:37:46 +00:00
b870b9340b
chore: remove uneeded dependencies ( #3929 )
...
* chore: remove unused deps in compactor
* chore: remove unused deps in influxdb_ioxd
* chore: remove unused deps in object_store
* chore: remove unused deps in server
* fix: object_store needs observability deps when compiled with aws
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 17:39:46 +00:00
26c43f0a2c
feat: grpc write handler
...
Implements the gRPC write handler endpoint.
2022-03-04 14:51:42 +00:00
8d00aaba90
feat: sync chunks in querier ( #3911 )
...
* feat: `ParquetFileRepo::list_by_namespace_not_to_delete`
* feat: `ChunkAddr: Clone`
* test: ensure that querier keeps same partition objects
* test: improve `create_parquet_file` flexibility
* feat: sync chunks in querier
* test: improve `test_parquet_file`
2022-03-04 08:53:39 +00:00
d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate ( #3908 )
...
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox
split out serving readiness, get compiling
* fix: hakari
* fix: hakari again
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
04a7a957fe
Merge branch 'main' into dom/rustls
2022-03-03 11:59:40 +00:00
da145ffbe4
feat: batch upsert of columns to catalog
...
Adds ColumnRepo::create_or_get_many() to upsert multiple columns in one
round trip to Postgres.
2022-03-03 11:17:30 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
4785a7d028
refactor: bump Azure SDK
...
Update to the latest Azure SDK to pick up rustls support.
2022-03-03 11:02:20 +00:00
02a2cfde54
feat: update rskafka ( #3805 ) ( #3910 )
2022-03-03 10:53:47 +00:00
3f2a58b47f
refactor: pub use data_types from data_types2
...
So it's clearer which parts of data_types the NG design is using, and
which types can be cleaned up eventually.
2022-03-02 13:55:31 -05:00
2a90841715
refactor: Move IngesterQueryRequest to data_types2
...
So that querier doesn't need to depend on ingester.
2022-03-02 13:52:13 -05:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
141a6087d0
feat: Querier able to send Flight queries to Ingester
...
Fixes #3773 .
2022-03-02 11:50:45 -05:00
44cfdc7aca
feat: update to latest rskafka logging ( #3805 ) ( #3896 )
2022-03-02 11:05:03 +00:00
af57664f53
feat: wire up flight into the querier ( #3889 )
2022-03-02 09:20:26 +00:00
936f51013d
test: querier shutdown and background task handling ( #3881 )
...
This is similar to what we already have for the ingester.
2022-03-02 08:59:50 +00:00
143311f63f
feat: additional write buffer logging ( #3805 ) ( #3887 )
...
* feat: additional write buffer logging (#3805 )
* fix: assertion direction
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-01 17:08:57 +00:00
0e0dc500f6
feat: prepare data to send to querier ( #3825 )
...
* feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes
* refactor: adjust the design after discussing with Paul
* feat: apply the coming tombstone on all data but persiting one
* chore: fmt
* fix: build on buffer tombstone
* test: delete & write tests for a parition and some cleanup
* feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied
* chore: cleanup
* feat: intitial implementation for preparing data to send back to the Querier
* feat: full implementation of prepare_data_to_querier
* fix: apply filters for the batches
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: cleanup
* fix: typos in comments
* fix: typos in comments
* fix: typos in comments
* test: create different scenarios and test them
* chore: fix typos
* test: add tests with deletes
* chore: make pub pub(crate)
* chore: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* refactor: address review comments
* fix: keep batches in their arrival order
* refactor: not assign unecessary values to enum
* refactor: use bitflags enum
* fix: use bitflags correctly
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: avoid using use at the end of the function
* chore: merge main to branch
* fix: fix downgrade versions
* refactor: address review comments
* chore: remove unnecessary comments
* refactor: Make the whole test_utils module test-only and bring paths into module scope
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-03-01 01:00:45 +00:00
6e2470bf5f
feat: create CatalogChunk in querier ( #3862 )
...
* feat: `NamespaceRepo::get_by_id`
* feat: create `CatalogChunk` in querier
2022-02-28 17:20:38 +00:00
b213796c98
feat: sync namespaces in querier ( #3865 )
...
* feat: `NamespaceRepo::list`
* feat: sync namespaces in querier
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:01:28 +00:00
77a0c74830
chore: `cargo update` + less rustsec exceptions ( #3873 )
...
It seems that dependabot doesn't catch with the flood of updates, so
this is a manual update run:
```console
$ cargo update
Updating crates.io index
Updating git repository `https://github.com/Azure/azure-sdk-for-rust.git `
Updating git repository `https://github.com/apache/arrow-datafusion.git `
Updating git repository `https://github.com/influxdata/rskafka.git `
Updating git repository `https://github.com/mkmik/heappy `
Updating anyhow v1.0.53 -> v1.0.55
Updating autocfg v1.0.1 -> v1.1.0
Updating clang-sys v1.3.0 -> v1.3.1
Updating clap_derive v3.1.0 -> v3.1.2
Updating core-foundation v0.9.2 -> v0.9.3
Updating crossbeam-epoch v0.9.6 -> v0.9.7
Updating crossbeam-queue v0.3.3 -> v0.3.4
Updating crossbeam-utils v0.8.6 -> v0.8.7
Updating debugid v0.7.2 -> v0.7.3
Updating fd-lock v3.0.3 -> v3.0.4
Updating inferno v0.10.10 -> v0.10.12
Updating io-lifetimes v0.4.4 -> v0.5.3
Updating kube-client v0.69.0 -> v0.69.1
Updating kube-core v0.69.0 -> v0.69.1
Updating libm v0.2.1 -> v0.2.2
Updating linux-raw-sys v0.0.37 -> v0.0.42
Removing memmap v0.7.0
Adding memmap2 v0.5.3
Updating ntapi v0.3.6 -> v0.3.7
Updating output_vt100 v0.1.2 -> v0.1.3
Updating rgb v0.8.31 -> v0.8.32
Updating rustix v0.32.1 -> v0.33.3
Updating rustls v0.20.2 -> v0.20.4
Updating security-framework v2.6.0 -> v2.6.1
Updating security-framework-sys v2.6.0 -> v2.6.1
Updating semver v1.0.4 -> v1.0.6
Updating symbolic-common v8.5.0 -> v8.6.1
Updating symbolic-demangle v8.5.0 -> v8.6.1
Updating tower-http v0.2.2 -> v0.2.3
Updating unicode-segmentation v1.8.0 -> v1.9.0
Updating zeroize v1.5.2 -> v1.5.3
```
This also allows us to remove a rustsec exception.
2022-02-28 11:24:32 +00:00
a9d52aaef3
chore(deps): Bump mockito from 0.30.0 to 0.31.0 ( #3871 )
...
* chore(deps): Bump mockito from 0.30.0 to 0.31.0
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.30.0 to 0.31.0.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.30.0...0.31.0 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: disallow `RUSTSEC-2020-0095`
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 10:05:24 +00:00
c289467239
chore(deps): Bump clap from 3.1.1 to 3.1.2 ( #3870 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.1...v3.1.2 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 09:53:25 +00:00
f966f4c7a4
feat: create `ParquetChunk` in querier ( #3857 )
...
Adds a small adapter that is able to produce `ParquetChunk`s for NG.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
49d1be30e7
feat: wire up `ParquetFilePath` for NG ( #3853 )
...
It's a bit of a duck-type hack, but if we wanna just `ParquetFileChunk`
in the new architecture, we somehow need it to accept new-gen paths.
Also path handling should be somewhat centralized since
ingester/compactor/querier all need to construct them. So having a
`ParquetFilePath` that supports both path styles seems to be a
not-to-bad solution. This should obviously be cleaned up in some
not-to-distant future.
2022-02-24 16:05:38 +00:00
4731913c44
feat: skeleton of querier CLI ( #3843 )
...
* feat: skeleton of querier CLI
* chore: wrap metrics in opt&arc in querier to satisfy new api
* chore: derive debug in querier handler
* chore: add join handles and their shutdown to nascent querier server
* chore: querier server http unimpl -> 404
* fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
9079e6ddb0
feat: backoff retries in ingester ( #3841 )
...
* feat: add `backoff` crate
* feat: backoff retries in ingester
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
3e69a5e1b4
Merge branch 'main' into dom/catalog-instrumentation
2022-02-23 16:14:40 +00:00
657ac249e9
feat: track ingester jobs ( #3836 )
2022-02-23 15:33:47 +00:00
a0b43323d6
feat: catalog instrumentation decorator
...
Adds a MetricDecorator type that wraps a Catalog impl, delegating calls
to the underlying impl and recording per-method call latency broken down
by success/error state.
2022-02-23 14:34:26 +00:00
b63f920d4c
chore(deps): Bump parquet from 9.0.2 to 9.1.0 ( #3828 )
...
* chore(deps): Bump parquet from 9.0.2 to 9.1.0
Bumps [parquet](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update chunk size test
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:25:15 +00:00
18c2ca1bb5
chore(deps): Bump tower from 0.4.11 to 0.4.12 ( #3817 )
...
* chore(deps): Bump tower from 0.4.11 to 0.4.12
Bumps [tower](https://github.com/tower-rs/tower ) from 0.4.11 to 0.4.12.
- [Release notes](https://github.com/tower-rs/tower/releases )
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.11...tower-0.4.12 )
---
updated-dependencies:
- dependency-name: tower
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:14:15 +00:00
5a79b3a68b
chore(deps): Bump arrow-flight from 9.0.2 to 9.1.0 ( #3829 )
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:03:22 +00:00
79144e227b
chore(deps): Bump integer-encoding from 3.0.2 to 3.0.3 ( #3827 )
...
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs ) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/dermesser/integer-encoding-rs/releases )
- [Commits](https://github.com/dermesser/integer-encoding-rs/commits )
---
updated-dependencies:
- dependency-name: integer-encoding
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:37:26 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
f7df54623e
chore(deps): Bump getrandom from 0.2.4 to 0.2.5 ( #3818 )
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.4...v0.2.5 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 08:59:05 +00:00
25c525f187
chore(deps): Bump tracing-subscriber from 0.3.8 to 0.3.9 ( #3815 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.8 to 0.3.9.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.8...tracing-subscriber-0.3.9 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 17:01:24 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
8574bb695c
chore(deps): Bump tikv-jemalloc-sys ( #3813 )
...
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator ) from 0.4.2+5.2.1-patched.2 to 0.4.3+5.2.1-patched.2.
- [Release notes](https://github.com/tikv/jemallocator/releases )
- [Changelog](https://github.com/tikv/jemallocator/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/jemallocator/commits )
---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:12:33 +00:00
ae5712fab0
chore(deps): Bump kube-derive from 0.69.0 to 0.69.1 ( #3812 )
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:49:14 +00:00
1dca869c3f
chore(deps): Bump sqlx from 0.5.10 to 0.5.11 ( #3799 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:14:36 +00:00
65ab5213e5
chore(deps): Bump clap from 3.0.14 to 3.1.1 ( #3809 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.14 to 3.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.14...v3.1.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 14:51:53 +00:00
c2a01c2407
chore(deps): Bump comfy-table from 5.0.0 to 5.0.1 ( #3800 )
...
Bumps [comfy-table](https://github.com/nukesor/comfy-table ) from 5.0.0 to 5.0.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases )
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nukesor/comfy-table/compare/v5.0.0...v5.0.1 )
---
updated-dependencies:
- dependency-name: comfy-table
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:36:52 +00:00
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
a16d4b80c4
chore(deps): Bump tracing from 0.1.30 to 0.1.31 ( #3798 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.30...tracing-0.1.31 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:40:09 +00:00
474a4099e1
chore(deps): Bump kube from 0.69.0 to 0.69.1 ( #3796 )
...
Bumps [kube](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:22:58 +00:00
271daa5eac
chore(deps): Bump sqlx-core from 0.5.10 to 0.5.11 ( #3792 )
...
Bumps [sqlx-core](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:05:31 +00:00
9ad10a5a5a
chore(deps): Bump libc from 0.2.117 to 0.2.119 ( #3795 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:56:17 +00:00
09e69a57b2
chore(deps): Bump cc from 1.0.72 to 1.0.73 ( #3793 )
...
Bumps [cc](https://github.com/alexcrichton/cc-rs ) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases )
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.72...1.0.73 )
---
updated-dependencies:
- dependency-name: cc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:47:47 +00:00
d6317a2ffd
chore(deps): Bump kube-runtime from 0.69.0 to 0.69.1 ( #3794 )
...
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 09:37:52 +00:00
7969ec682c
feat: catalog now supports dsn-file:// ( #3787 )
...
* feat: catalog now supports dsn-file://
* chore: rename fn in catalog PG hotswap code
2022-02-18 16:42:55 +00:00
f54ef92b77
fix: supervise and shutdown ingester background tasks ( #3769 )
...
* fix: supervise and shutdown ingester background tasks
Closes #3761 .
Closes #3762 .
* docs: improve wording
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* test: join/shutdown handling for ingester
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-02-18 09:35:29 +00:00
44e9eaf92b
test(iox_catalog): isolated catalog tests
...
This commit changes the iox_catalog test harness so that each test is
run in an independent, randomly generated schema to avoid concurrent
tests interfering with each other.
Each test creates a randomly-named schema, grants permissions to the new
schema, runs the full migration stack against the new schema, and then
executes the code under test.
2022-02-17 14:19:01 +00:00
b1215805a8
fix: upgrade rskafka for bugfixes and race-free start offsets ( #3775 )
...
This fixes the concerns that were brought up during the review of #3748 .
2022-02-17 11:22:25 +00:00
d0965bb0b2
Merge branch 'main' into dom/mb-partitioning
2022-02-16 11:30:42 +00:00
a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor ( #3750 )
...
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 16:18:36 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
92218ce8aa
feat: write partitioner
...
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
2022-02-15 11:29:32 +00:00
e99922d518
refactor: parametrise DML handler input type
...
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
2022-02-15 11:23:45 +00:00
4db27eec68
fix: defined behaviour when seeking to an unknown sequence number ( #3745 )
...
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-15 11:08:01 +00:00
1cd51b6071
chore(deps): Bump hyper from 0.14.16 to 0.14.17 ( #3744 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.16 to 0.14.17.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.16...v0.14.17 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 10:23:46 +00:00
2626e4891d
chore(deps): Bump tracing-subscriber from 0.3.7 to 0.3.8 ( #3741 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:51:21 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
85aa019f50
feat: Turn protobuf predicates into predicate::Predicate ( #3707 )
...
* feat: Turn protobuf predicates into predicate::Predicate
* fix: Take buf lint's suggestions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) ( #3736 )
...
* chore: update gitops adapter k8s deps (via dependabot)
* chore: workspace hack updates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:47:15 +00:00
b77d43fc22
chore: Update datafusion ( #3727 )
2022-02-11 21:41:36 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 ( #3706 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:19:19 +00:00
6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 ( #3698 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 21:21:24 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 ( #3696 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 19:40:03 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
7d11485c77
chore(deps): bump tracing from 0.1.29 to 0.1.30 ( #3677 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.29 to 0.1.30.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.29...tracing-0.1.30 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:44:16 +00:00
59b2141c0b
feat: Add lifecycle manager to ingester ( #3645 )
...
This adds the lifecycle manager to the ingester. It will trigger based on a threshold for max partition size or age or based on keeping total memory under a certain threshold.
It defines a new interface for a persister, which is stubbed out for IngesterData. I'm not sure yet how persistence errors should be handled. The assumption here is that the persister continues to retry persistence forever until it succeeds.
There is one scenario I can think of that may cause this lifecycle manager problems. If a single partition is very high throughput, it could cause things to back up as persistence is not parallelized within a single partition. Any given partition can currently only run one persistence operation at a time. We can address this later.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:23:40 +00:00
9369152096
chore(deps): bump tracing-core from 0.1.21 to 0.1.22 ( #3675 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.21 to 0.1.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.21...tracing-core-0.1.22 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:12:39 +00:00
ba87ae2918
chore(deps): bump crc32fast from 1.3.1 to 1.3.2 ( #3668 )
...
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast ) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases )
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.3.1...v1.3.2 )
---
updated-dependencies:
- dependency-name: crc32fast
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-08 14:48:31 +00:00
3c9a27b31b
chore(deps): bump futures-test from 0.3.19 to 0.3.21 ( #3667 )
...
Bumps [futures-test](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures-test
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-08 14:35:34 +00:00
9a874186d5
chore(deps): bump libc from 0.2.116 to 0.2.117 ( #3666 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.116 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.116...0.2.117 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:59:02 +00:00
5de4d6203f
refactor: catalog transaction ( #3660 )
...
* refactor: catalog Unit of Work (= transaction)
Setup an inteface to handle Units of Work within our catalog. Previously
both the Postgres and the in-mem backend used "mini-transactions on
demand". Now the caller has a clear way to establish boundaries and
gets read and write isolation. A single `Arc<dyn Catalog>` can create as
many `Box<dyn UnitOfWork>` as you like, but note that depending on the
backend you may not scale infinitely (postgres will likely impose
certain limits and the in-mem backend limits concurrency to 1 to keep
things simple).
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename Unit of Work to Transaction
* test: improve `test_txn_isolation`
* feat: clearify transaction drop semantics
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:38:33 +00:00
de2a013786
feat: gitops adapter ( #3656 )
...
* feat: scaffolding of gitops adapter bin crate
* chore: refactor gitops adapter; calls CLI now; status update fixes
* feat: gitops adapter now calls out to CLI once per topic; improved tests
* chore: add mock failure script for gitops adapter
* chore: update workspace-hack
* chore: refactor away unecessary to_string in gitops syncer
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:27:36 +00:00
0bd7941a18
fix(REPL): Don't buffer lines until a trailing semicolon is found and add history hinting ( #3630 )
...
* fix(REPL): Don't buffer lines until a trailing semicolon is found
The repl would silently buffer all lines until a trailing semicolon were found which
resulted in some very confusing error messages as I would input invalid commands followed
by a command I thought were valid, except I'd still get an error due to the previous command being buffered.
This uses rustyline's helper feature to detect incomplete input (no trailing semicolon) and makes
it accept multiline input until the input is completed.
I also included some of rustyline's default hint and highlighting while I was at it.
* chore: cargo clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 17:11:01 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
bc4b7f8a5b
test: ensure that rskafka and rdkakfa work together ( #3624 )
...
* chore: upgrade rskafka + enable snappy support
* test: ensure that rskafka and rdkakfa work together
Before removing rdkafka ensure that:
- rskafka can consume existing messages produced by rdkafka so we do not
need to drain existing topics
- rdkafka can consume new messages produced by rskafka so we can roll
back
I ran the whole `write_buffer` test suite (including the newly added
tests) using Apache Kafka as well as Redpanda.
* test: ensure we handle consumer offset in error case correctly
* docs: explain test setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:52:42 +00:00
ce46bbaada
feat: wire up the write buffer to the ingester process ( #3533 )
...
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.
I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.
There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
9079baf19a
Merge branch 'main' into dom/schema-validation
2022-02-02 16:27:35 +00:00
030a2cb4c1
chore: Update datafusion ( #3613 )
...
* chore: Update datafusion
* fix: update for latest DF API
* fix: another API change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 16:27:11 +00:00
4744c5804e
refactor: remove Dashmap
...
Swap Dashmap for a regular RwLock<HashMap<..,>> due to soundness issues:
https://rustsec.org/advisories/RUSTSEC-2022-0002
2022-02-02 14:04:53 +00:00
39d489d9e7
refactor: enable schema validation
...
Adds the SchemaValidator to the DML handler stack - this adds it into
the request path in router2.
2022-02-02 14:04:14 +00:00
6598023726
feat: cache NamespaceSchema in validator
...
Adds an in-memory cache of table schemas to the SchemaValidator DML
handler.
The cache pulls from the global catalog when observing a column for the
first time, and pushes the column type to set it for subsequent requests
if it does not exist (this pull & push is done by atomically by the
catalog in an "upsert" call).
The in-memory cache is sharded by namespace, with each shard guarded by
an individual lock to minimise contention between readers (the expected
average case) and writers (only when adding new columns/tables).
Relies on the catalog to serialise new column creation and validate
parallel creation requests.
2022-02-02 13:04:53 +00:00
c81f207298
feat: schema validation
...
Implements a write schema validation DML handler, denying requests that
conflict with the schema within the global catalog. Additive schema
changes are accepted, incrementally updating the global catalog schema.
Deletes are passed through unchanged and unvalidated.
2022-02-02 13:04:53 +00:00
5441682207
feat: add support for parsing predicate
2022-02-02 11:02:33 +00:00
c1f5994660
refactor: move sql parsing -> RPC predicate into own crate ( #3604 )
...
* chore: create crate
* refactor: move module to new crate
2022-02-02 10:41:57 +00:00
fa546047fb
refactor: update dep with API change ( #3596 )
2022-02-01 14:53:10 +00:00
443fd00c1b
Merge branch 'main' into er/feat/parse_sql_rpc
2022-02-01 13:55:08 +00:00
0a2cb36ddf
fix: workspace-hack ( #3593 )
2022-02-01 12:32:58 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
175732c3ca
feat: basic sqlparser -> RPCNode
2022-02-01 10:26:03 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
8bef2c105c
Merge branch 'main' into cn/persist
2022-01-31 18:50:45 +00:00
7b96a37165
chore: Update datafusion ( #3586 )
...
* chore: update DataFusion to f849968057ddddccc9aa19915ef3ea56bf14d80d
* fix: reduce overhead of creating physical expressions
* chore: use MemTrackingMetrics
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-31 18:15:28 +00:00
bf89162fa5
refactor: Move IoxMetadata to parquet_file
2022-01-31 10:36:33 -05:00
dd9620da0c
feat: Create a new proto definition for the new design's IoxMetadata
2022-01-31 10:36:32 -05:00
5e0e0d8aa7
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-31 10:36:32 -05:00
c633c9bc5c
feat: Wire object store into ingester persistence
2022-01-31 10:36:30 -05:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
8735ede74f
feat: IoxMetadata for parquet file ( #3547 )
...
* feat: IoxMetadata for parquet file
* fix: typos
* refactor: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 14:41:59 +00:00
1669acc0c2
build: update tokio ( #3566 )
...
Release notes:
https://github.com/tokio-rs/tokio/releases/tag/tokio-1.16.0
2022-01-28 10:36:12 +00:00
b486258dfb
chore: run cargo update ( #3559 )
...
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:05:31 +00:00
9201023ea4
feat: schema validation for MutableBuffer instances ( #3554 )
...
* refactor: Debug bounds on Catalog trait
* feat: validate MutableBatch schema
Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.
* refactor: Send bound on boxed errors
* refactor: clippy
Allow assert_eq!(bool, bool) for readability.
* refactor: no PartialEq<MB Column> for ColumnSchema
Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 20:55:18 +00:00
4a96e52290
feat: router2 sharder benchmarking ( #3558 )
...
* feat: benchmarking the router2 sharder
* chore: added throughput to sharder benchmarks; vary num buckets
2022-01-27 18:09:16 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
ce568ab447
build(iox_catalog): remove unused dependencies ( #3552 )
...
* build: don't pull in all of tokio
We already specify the tokio features we need so "full" (all features)
is not necessary.
* build: remove chrono dependency
Appears unused.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:14:42 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
7261571abf
fix: Revert "chore: temporarily hack around datafusion tempfiles" ( #3525 )
...
This reverts commit ae5763c1cb6bb4a98ffe0779a3a35f6daaf10971.
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 12:08:49 +00:00
bc44d33108
feat: Implement a snapshot method on DataBuffer ( #3518 )
...
* feat: Implement a snapshot method on DataBuffer
Fixes #3510 .
* test: Add a test snapshotting batches with different but compatible schemas
* fix: Simplify min/max sequencer number collection
The first batch should always have the min sequencer number. The last
batch should always have the max sequencer number. The min should always
be less than (or equal to, in case there's only one batch) the max.
2022-01-26 15:22:51 +00:00
2928254c0f
fix: test logging ( #3536 )
...
- Use a more standard way to setup the tracing subsystem (as described
in tracing-subscriber docs)
- Also capture content from `log` crate
- Play nice w/ Rust's libtest message capture
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-26 10:28:51 +00:00
b846ead320
feat(router2): shard writes/deletes into write buffer ( #3499 )
...
* feat: Sequencer wrapper
This type wraps an underlying WriteBufferWriter implementation, tagging
it with a sequencer ID it should use when enqueuing operations to the
buffer.
* feat: mock sharder
Implements a mock Sharder impl that returns pre-configured responses to
shard(), and captures the input to the call.
* feat: sharded write buffer
Implements sharding of ops into an underlying WriteBuffer.
Writes are sharded by some abstract Sharder impl, collated per shard to
maximise the size of each op (and therefore compression efficiency),
converted into a DML operation and then enqueued in parallel to the
underlying WriteBuffer implementation.
Deletes are modelled as being mapped to a single write buffer shard,
which is the case while we support sharding based on the table &
namespace only. Deletes will be extended to support (potentially)
multiple shards when column overrides are implemented.
* refactor: runtime write buffers
Switch from using static dispatch, to using a runtime specified
WriteBufferWriting implementation.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-25 15:19:48 +00:00
51a0f8a56a
chore: temporarily hack around datafusion tempfiles ( #3524 )
2022-01-25 12:30:29 +00:00
f9c1e80a7f
chore: update thread_local
...
chore: update thread_local
2022-01-24 13:37:52 -05:00
797ba459b9
chore: merge main to branch
2022-01-24 12:06:23 -05:00
5f98a07b7f
chore: add Corgo.lock
2022-01-24 12:03:02 -05:00
bb893510a0
feat: Add scaffolding for ingester server
...
* Adds a new ingester command to start an ingester server
* Moves previous ingester server over to handler
* Skeleton for gRPC and HTTP handlers
2022-01-21 18:02:19 -05:00
9615feacb3
fix(InfluxQL): Support RegEx with escape sequences not supported by Rust regex ( #3502 )
...
* fix(InfluxQL): Translate unsupported meta characters
* fix: remove debugging
* fix: clippy sacrifice
* docs: Add additional background and rationale for rewriting
* fix: doc link
2022-01-21 14:40:10 +00:00
5b22abf6a2
Merge branch 'main' into crepererum/wb_rskafka
2022-01-20 16:48:10 +00:00
Andrew Lamb
dependabot[bot]
Marco Neumann
Nga Tran
Carol (Nichols || Goulding)
Dom Dwyer
kodiakhq[bot]
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Luke Bond
Paul Dix
Markus Westerlind
Paul Dix
Andrew Lamb
Edd Robinson
Nga Tran
Luke Bond