ed84b453d3
chore(deps): Bump tracing-core from 0.1.22 to 0.1.23 ( #4003 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.22 to 0.1.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.22...tracing-core-0.1.23 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-10 13:43:15 +00:00
0756642137
chore(deps): Bump pbjson-build from 0.2.2 to 0.2.3 ( #4002 )
...
Bumps [pbjson-build](https://github.com/influxdata/pbjson ) from 0.2.2 to 0.2.3.
- [Release notes](https://github.com/influxdata/pbjson/releases )
- [Commits](https://github.com/influxdata/pbjson/compare/0.2.2...0.2.3 )
---
updated-dependencies:
- dependency-name: pbjson-build
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-10 13:34:00 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
944f628e29
fix: Remove data_types as a dependency of ng compactor ( #3993 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 17:03:02 +00:00
93b0cdbcc4
fix: Create the test database as part of ng server fixture startup
2022-03-09 09:55:43 -05:00
d2c0acdd46
refactor: Remove serving readiness gate ( #3986 )
...
* refactor: Remove serving_readiness
* fix: remove more
* fix: remove test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-09 12:17:44 +00:00
09fba1d2c0
feat: NG Compactor - main function for finding and compacting parquet files ( #3973 )
...
* feat: main function for finding and compacting parquet files
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename file and struct
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-08 16:34:43 +00:00
3278e35b41
build: bump regex
...
Bump regex version due to RUSTSEC-2022-0013, see:
https://rustsec.org/advisories/RUSTSEC-2022-0013
2022-03-08 15:37:23 +00:00
baae6910f2
build: remove openssl dependency
...
Switches a stray reqwest dependency to use rustls.
2022-03-07 13:47:39 +00:00
e47bfc40be
Merge branch 'main' into dom/router-grpc-write
2022-03-07 11:13:19 +00:00
d3321135eb
chore(deps): Bump clap from 3.1.2 to 3.1.5 ( #3957 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.2 to 3.1.5.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.2...v3.1.5 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-07 09:58:46 +00:00
3c58a08431
chore(deps): Bump siphasher from 0.3.9 to 0.3.10 ( #3956 )
...
Bumps [siphasher](https://github.com/jedisct1/rust-siphash ) from 0.3.9 to 0.3.10.
- [Release notes](https://github.com/jedisct1/rust-siphash/releases )
- [Commits](https://github.com/jedisct1/rust-siphash/compare/0.3.9...0.3.10 )
---
updated-dependencies:
- dependency-name: siphasher
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-07 09:10:09 +00:00
48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 ( #3955 )
...
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-07 08:54:20 +00:00
e09f39d6a0
chore: Update datafusion ( #3943 )
...
* chore: Update datafusion
* refactor: update for new datafusion
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-04 19:37:46 +00:00
b870b9340b
chore: remove uneeded dependencies ( #3929 )
...
* chore: remove unused deps in compactor
* chore: remove unused deps in influxdb_ioxd
* chore: remove unused deps in object_store
* chore: remove unused deps in server
* fix: object_store needs observability deps when compiled with aws
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 17:39:46 +00:00
26c43f0a2c
feat: grpc write handler
...
Implements the gRPC write handler endpoint.
2022-03-04 14:51:42 +00:00
8d00aaba90
feat: sync chunks in querier ( #3911 )
...
* feat: `ParquetFileRepo::list_by_namespace_not_to_delete`
* feat: `ChunkAddr: Clone`
* test: ensure that querier keeps same partition objects
* test: improve `create_parquet_file` flexibility
* feat: sync chunks in querier
* test: improve `test_parquet_file`
2022-03-04 08:53:39 +00:00
d6afda0227
refactor: split `influxdb_ioxd`, `clap_blocks` and `serving_readiness` from influxdb_iox crate ( #3908 )
...
* refactor: split influxdb_ioxd, clap_blocks, and serving_readiness out of influxdb_iox
split out serving readiness, get compiling
* fix: hakari
* fix: hakari again
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-03 16:48:30 +00:00
04a7a957fe
Merge branch 'main' into dom/rustls
2022-03-03 11:59:40 +00:00
da145ffbe4
feat: batch upsert of columns to catalog
...
Adds ColumnRepo::create_or_get_many() to upsert multiple columns in one
round trip to Postgres.
2022-03-03 11:17:30 +00:00
46bb107be4
refactor: use rustls
...
Removes openssl as a dependency, switching to rustls[1] as the TLS
implementation throughout.
It is important to note that this change brings with it a significant
behavioural difference - rustls does not currently support IP SANs in
certificates (instead only supporting fully-qualified names / DNS) and
this will manifest as a failure to connect to IP endpoints over TLS.
This might be a blocker that prevents us using rustls exclusively, but
there's noe asy way to know without trying it. Fortunately the rustls
project has received funding to work on IP SAN support[2].
[1]: https://github.com/rustls/rustls
[2]: https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/
2022-03-03 11:05:20 +00:00
4785a7d028
refactor: bump Azure SDK
...
Update to the latest Azure SDK to pick up rustls support.
2022-03-03 11:02:20 +00:00
02a2cfde54
feat: update rskafka ( #3805 ) ( #3910 )
2022-03-03 10:53:47 +00:00
3f2a58b47f
refactor: pub use data_types from data_types2
...
So it's clearer which parts of data_types the NG design is using, and
which types can be cleaned up eventually.
2022-03-02 13:55:31 -05:00
2a90841715
refactor: Move IngesterQueryRequest to data_types2
...
So that querier doesn't need to depend on ingester.
2022-03-02 13:52:13 -05:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
141a6087d0
feat: Querier able to send Flight queries to Ingester
...
Fixes #3773 .
2022-03-02 11:50:45 -05:00
44cfdc7aca
feat: update to latest rskafka logging ( #3805 ) ( #3896 )
2022-03-02 11:05:03 +00:00
af57664f53
feat: wire up flight into the querier ( #3889 )
2022-03-02 09:20:26 +00:00
936f51013d
test: querier shutdown and background task handling ( #3881 )
...
This is similar to what we already have for the ingester.
2022-03-02 08:59:50 +00:00
143311f63f
feat: additional write buffer logging ( #3805 ) ( #3887 )
...
* feat: additional write buffer logging (#3805 )
* fix: assertion direction
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-01 17:08:57 +00:00
0e0dc500f6
feat: prepare data to send to querier ( #3825 )
...
* feat: changes needed to apply tombstones correctly on the life-cycle ingest bacthes
* refactor: adjust the design after discussing with Paul
* feat: apply the coming tombstone on all data but persiting one
* chore: fmt
* fix: build on buffer tombstone
* test: delete & write tests for a parition and some cleanup
* feat: No need add processed tombstones for newly created parquet file in the ingester becasue all deletes before that parquet file is created were applied
* chore: cleanup
* feat: intitial implementation for preparing data to send back to the Querier
* feat: full implementation of prepare_data_to_querier
* fix: apply filters for the batches
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: cleanup
* fix: typos in comments
* fix: typos in comments
* fix: typos in comments
* test: create different scenarios and test them
* chore: fix typos
* test: add tests with deletes
* chore: make pub pub(crate)
* chore: Apply suggestions from code review
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
* refactor: address review comments
* fix: keep batches in their arrival order
* refactor: not assign unecessary values to enum
* refactor: use bitflags enum
* fix: use bitflags correctly
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: avoid using use at the end of the function
* chore: merge main to branch
* fix: fix downgrade versions
* refactor: address review comments
* chore: remove unnecessary comments
* refactor: Make the whole test_utils module test-only and bring paths into module scope
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-03-01 01:00:45 +00:00
6e2470bf5f
feat: create CatalogChunk in querier ( #3862 )
...
* feat: `NamespaceRepo::get_by_id`
* feat: create `CatalogChunk` in querier
2022-02-28 17:20:38 +00:00
b213796c98
feat: sync namespaces in querier ( #3865 )
...
* feat: `NamespaceRepo::list`
* feat: sync namespaces in querier
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:01:28 +00:00
77a0c74830
chore: `cargo update` + less rustsec exceptions ( #3873 )
...
It seems that dependabot doesn't catch with the flood of updates, so
this is a manual update run:
```console
$ cargo update
Updating crates.io index
Updating git repository `https://github.com/Azure/azure-sdk-for-rust.git `
Updating git repository `https://github.com/apache/arrow-datafusion.git `
Updating git repository `https://github.com/influxdata/rskafka.git `
Updating git repository `https://github.com/mkmik/heappy `
Updating anyhow v1.0.53 -> v1.0.55
Updating autocfg v1.0.1 -> v1.1.0
Updating clang-sys v1.3.0 -> v1.3.1
Updating clap_derive v3.1.0 -> v3.1.2
Updating core-foundation v0.9.2 -> v0.9.3
Updating crossbeam-epoch v0.9.6 -> v0.9.7
Updating crossbeam-queue v0.3.3 -> v0.3.4
Updating crossbeam-utils v0.8.6 -> v0.8.7
Updating debugid v0.7.2 -> v0.7.3
Updating fd-lock v3.0.3 -> v3.0.4
Updating inferno v0.10.10 -> v0.10.12
Updating io-lifetimes v0.4.4 -> v0.5.3
Updating kube-client v0.69.0 -> v0.69.1
Updating kube-core v0.69.0 -> v0.69.1
Updating libm v0.2.1 -> v0.2.2
Updating linux-raw-sys v0.0.37 -> v0.0.42
Removing memmap v0.7.0
Adding memmap2 v0.5.3
Updating ntapi v0.3.6 -> v0.3.7
Updating output_vt100 v0.1.2 -> v0.1.3
Updating rgb v0.8.31 -> v0.8.32
Updating rustix v0.32.1 -> v0.33.3
Updating rustls v0.20.2 -> v0.20.4
Updating security-framework v2.6.0 -> v2.6.1
Updating security-framework-sys v2.6.0 -> v2.6.1
Updating semver v1.0.4 -> v1.0.6
Updating symbolic-common v8.5.0 -> v8.6.1
Updating symbolic-demangle v8.5.0 -> v8.6.1
Updating tower-http v0.2.2 -> v0.2.3
Updating unicode-segmentation v1.8.0 -> v1.9.0
Updating zeroize v1.5.2 -> v1.5.3
```
This also allows us to remove a rustsec exception.
2022-02-28 11:24:32 +00:00
a9d52aaef3
chore(deps): Bump mockito from 0.30.0 to 0.31.0 ( #3871 )
...
* chore(deps): Bump mockito from 0.30.0 to 0.31.0
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.30.0 to 0.31.0.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.30.0...0.31.0 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: disallow `RUSTSEC-2020-0095`
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 10:05:24 +00:00
c289467239
chore(deps): Bump clap from 3.1.1 to 3.1.2 ( #3870 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.1...v3.1.2 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 09:53:25 +00:00
f966f4c7a4
feat: create `ParquetChunk` in querier ( #3857 )
...
Adds a small adapter that is able to produce `ParquetChunk`s for NG.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 08:54:16 +00:00
49d1be30e7
feat: wire up `ParquetFilePath` for NG ( #3853 )
...
It's a bit of a duck-type hack, but if we wanna just `ParquetFileChunk`
in the new architecture, we somehow need it to accept new-gen paths.
Also path handling should be somewhat centralized since
ingester/compactor/querier all need to construct them. So having a
`ParquetFilePath` that supports both path styles seems to be a
not-to-bad solution. This should obviously be cleaned up in some
not-to-distant future.
2022-02-24 16:05:38 +00:00
4731913c44
feat: skeleton of querier CLI ( #3843 )
...
* feat: skeleton of querier CLI
* chore: wrap metrics in opt&arc in querier to satisfy new api
* chore: derive debug in querier handler
* chore: add join handles and their shutdown to nascent querier server
* chore: querier server http unimpl -> 404
* fix: join/shutdown fix in querier; removed unused delegates
2022-02-24 15:42:56 +00:00
9079e6ddb0
feat: backoff retries in ingester ( #3841 )
...
* feat: add `backoff` crate
* feat: backoff retries in ingester
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 17:58:16 +00:00
3e69a5e1b4
Merge branch 'main' into dom/catalog-instrumentation
2022-02-23 16:14:40 +00:00
657ac249e9
feat: track ingester jobs ( #3836 )
2022-02-23 15:33:47 +00:00
a0b43323d6
feat: catalog instrumentation decorator
...
Adds a MetricDecorator type that wraps a Catalog impl, delegating calls
to the underlying impl and recording per-method call latency broken down
by success/error state.
2022-02-23 14:34:26 +00:00
b63f920d4c
chore(deps): Bump parquet from 9.0.2 to 9.1.0 ( #3828 )
...
* chore(deps): Bump parquet from 9.0.2 to 9.1.0
Bumps [parquet](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update chunk size test
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:25:15 +00:00
18c2ca1bb5
chore(deps): Bump tower from 0.4.11 to 0.4.12 ( #3817 )
...
* chore(deps): Bump tower from 0.4.11 to 0.4.12
Bumps [tower](https://github.com/tower-rs/tower ) from 0.4.11 to 0.4.12.
- [Release notes](https://github.com/tower-rs/tower/releases )
- [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.11...tower-0.4.12 )
---
updated-dependencies:
- dependency-name: tower
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:14:15 +00:00
5a79b3a68b
chore(deps): Bump arrow-flight from 9.0.2 to 9.1.0 ( #3829 )
...
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 11:03:22 +00:00
79144e227b
chore(deps): Bump integer-encoding from 3.0.2 to 3.0.3 ( #3827 )
...
Bumps [integer-encoding](https://github.com/dermesser/integer-encoding-rs ) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/dermesser/integer-encoding-rs/releases )
- [Commits](https://github.com/dermesser/integer-encoding-rs/commits )
---
updated-dependencies:
- dependency-name: integer-encoding
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:37:26 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
459974d1b8
refactor: move `DedicatedExecutor` from `query` to `executor` ( #3819 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 09:10:43 +00:00
f7df54623e
chore(deps): Bump getrandom from 0.2.4 to 0.2.5 ( #3818 )
...
Bumps [getrandom](https://github.com/rust-random/getrandom ) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/rust-random/getrandom/releases )
- [Changelog](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/getrandom/compare/v0.2.4...v0.2.5 )
---
updated-dependencies:
- dependency-name: getrandom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-23 08:59:05 +00:00
25c525f187
chore(deps): Bump tracing-subscriber from 0.3.8 to 0.3.9 ( #3815 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.8 to 0.3.9.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.8...tracing-subscriber-0.3.9 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 17:01:24 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
8574bb695c
chore(deps): Bump tikv-jemalloc-sys ( #3813 )
...
Bumps [tikv-jemalloc-sys](https://github.com/tikv/jemallocator ) from 0.4.2+5.2.1-patched.2 to 0.4.3+5.2.1-patched.2.
- [Release notes](https://github.com/tikv/jemallocator/releases )
- [Changelog](https://github.com/tikv/jemallocator/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/jemallocator/commits )
---
updated-dependencies:
- dependency-name: tikv-jemalloc-sys
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:12:33 +00:00
ae5712fab0
chore(deps): Bump kube-derive from 0.69.0 to 0.69.1 ( #3812 )
...
Bumps [kube-derive](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-derive
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:49:14 +00:00
1dca869c3f
chore(deps): Bump sqlx from 0.5.10 to 0.5.11 ( #3799 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:14:36 +00:00
65ab5213e5
chore(deps): Bump clap from 3.0.14 to 3.1.1 ( #3809 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.14 to 3.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.14...v3.1.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-22 14:51:53 +00:00
c2a01c2407
chore(deps): Bump comfy-table from 5.0.0 to 5.0.1 ( #3800 )
...
Bumps [comfy-table](https://github.com/nukesor/comfy-table ) from 5.0.0 to 5.0.1.
- [Release notes](https://github.com/nukesor/comfy-table/releases )
- [Changelog](https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md )
- [Commits](https://github.com/nukesor/comfy-table/compare/v5.0.0...v5.0.1 )
---
updated-dependencies:
- dependency-name: comfy-table
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 14:36:52 +00:00
0f012de70c
feat: adding compactor CLI command and crate
...
Closes : #3777
2022-02-21 12:24:09 +00:00
a16d4b80c4
chore(deps): Bump tracing from 0.1.30 to 0.1.31 ( #3798 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.30 to 0.1.31.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.30...tracing-0.1.31 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:40:09 +00:00
474a4099e1
chore(deps): Bump kube from 0.69.0 to 0.69.1 ( #3796 )
...
Bumps [kube](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:22:58 +00:00
271daa5eac
chore(deps): Bump sqlx-core from 0.5.10 to 0.5.11 ( #3792 )
...
Bumps [sqlx-core](https://github.com/launchbadge/sqlx ) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/compare/v0.5.10...v0.5.11 )
---
updated-dependencies:
- dependency-name: sqlx-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 10:05:31 +00:00
9ad10a5a5a
chore(deps): Bump libc from 0.2.117 to 0.2.119 ( #3795 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.117 to 0.2.119.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.117...0.2.119 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:56:17 +00:00
09e69a57b2
chore(deps): Bump cc from 1.0.72 to 1.0.73 ( #3793 )
...
Bumps [cc](https://github.com/alexcrichton/cc-rs ) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/alexcrichton/cc-rs/releases )
- [Commits](https://github.com/alexcrichton/cc-rs/compare/1.0.72...1.0.73 )
---
updated-dependencies:
- dependency-name: cc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-21 09:47:47 +00:00
d6317a2ffd
chore(deps): Bump kube-runtime from 0.69.0 to 0.69.1 ( #3794 )
...
Bumps [kube-runtime](https://github.com/kube-rs/kube-rs ) from 0.69.0 to 0.69.1.
- [Release notes](https://github.com/kube-rs/kube-rs/releases )
- [Changelog](https://github.com/kube-rs/kube-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/kube-rs/kube-rs/compare/0.69.0...0.69.1 )
---
updated-dependencies:
- dependency-name: kube-runtime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 09:37:52 +00:00
7969ec682c
feat: catalog now supports dsn-file:// ( #3787 )
...
* feat: catalog now supports dsn-file://
* chore: rename fn in catalog PG hotswap code
2022-02-18 16:42:55 +00:00
f54ef92b77
fix: supervise and shutdown ingester background tasks ( #3769 )
...
* fix: supervise and shutdown ingester background tasks
Closes #3761 .
Closes #3762 .
* docs: improve wording
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* test: join/shutdown handling for ingester
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2022-02-18 09:35:29 +00:00
44e9eaf92b
test(iox_catalog): isolated catalog tests
...
This commit changes the iox_catalog test harness so that each test is
run in an independent, randomly generated schema to avoid concurrent
tests interfering with each other.
Each test creates a randomly-named schema, grants permissions to the new
schema, runs the full migration stack against the new schema, and then
executes the code under test.
2022-02-17 14:19:01 +00:00
b1215805a8
fix: upgrade rskafka for bugfixes and race-free start offsets ( #3775 )
...
This fixes the concerns that were brought up during the review of #3748 .
2022-02-17 11:22:25 +00:00
d0965bb0b2
Merge branch 'main' into dom/mb-partitioning
2022-02-16 11:30:42 +00:00
a66e29e5b3
chore: port sqlx-hotswap-pool over from conductor ( #3750 )
...
* chore: port sqlx-hotswap-pool over from conductor
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: workspace hack fixes
* fix: unique schema per test db connection
* fix: adjust search path in catalog pg tests to see if it fixes test schema issue
* fix: actually fixed sqlx hotswap pool test
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 16:18:36 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
92218ce8aa
feat: write partitioner
...
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
2022-02-15 11:29:32 +00:00
e99922d518
refactor: parametrise DML handler input type
...
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
2022-02-15 11:23:45 +00:00
4db27eec68
fix: defined behaviour when seeking to an unknown sequence number ( #3745 )
...
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-15 11:08:01 +00:00
1cd51b6071
chore(deps): Bump hyper from 0.14.16 to 0.14.17 ( #3744 )
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.16 to 0.14.17.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.16...v0.14.17 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 10:23:46 +00:00
2626e4891d
chore(deps): Bump tracing-subscriber from 0.3.7 to 0.3.8 ( #3741 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:51:21 +00:00
60a7f87645
chore(deps): Bump serde_json from 1.0.78 to 1.0.79 ( #3739 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 20:42:54 +00:00
85aa019f50
feat: Turn protobuf predicates into predicate::Predicate ( #3707 )
...
* feat: Turn protobuf predicates into predicate::Predicate
* fix: Take buf lint's suggestions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:56:56 +00:00
d39cf0158b
chore: update gitops adapter k8s deps (via dependabot) ( #3736 )
...
* chore: update gitops adapter k8s deps (via dependabot)
* chore: workspace hack updates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-14 17:47:15 +00:00
b77d43fc22
chore: Update datafusion ( #3727 )
2022-02-11 21:41:36 +00:00
6c3b208438
chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0 ( #3665 )
...
* chore(deps): bump cache_loader_async from 0.1.2 to 0.2.0
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.2 to 0.2.0.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits/v0.2.0 )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: breaking changes from `cache_loader_async`
* fix: Update workspace hack: tokio full is no longer needed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-02-10 17:01:11 +00:00
54890525da
feat: gitops adapter ci build ( #3692 )
...
* chore: remove references to perf_image in CI
* chore: adding gitops adapter image build in CI
* chore: gitops adapter bin now same as dir & package so docker build works
* fix: circle config package change after renaming gitops adapter package
2022-02-10 11:23:06 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
f23574bc5f
chore(deps): bump futures from 0.3.19 to 0.3.21 ( #3706 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:19:19 +00:00
6a002f09e1
chore(deps): bump clap from 3.0.13 to 3.0.14 ( #3698 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.0.13 to 3.0.14.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.0.13...v3.0.14 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 21:21:24 +00:00
85d44552fa
chore(deps): bump cloud-storage from 0.10.3 to 0.11.0 ( #3697 )
...
Bumps [cloud-storage](https://github.com/ThouCheese/cloud-storage-rs ) from 0.10.3 to 0.11.0.
- [Release notes](https://github.com/ThouCheese/cloud-storage-rs/releases )
- [Changelog](https://github.com/ThouCheese/cloud-storage-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ThouCheese/cloud-storage-rs/commits )
---
updated-dependencies:
- dependency-name: cloud-storage
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 20:03:44 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 ( #3696 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-02-09 19:40:03 +00:00
73828323ac
feat: Ingester Flight gRPC API ( #3623 )
...
* feat: Add a way to run ingester with an in-memory catalog from the CLI
If you set the --catalog-dsn string to "mem", rather than using that as
a Postgres connection URL, create an in-memory catalog.
Planning on using this in tests, so not documenting.
* fix: Set default topic to the same value as SHARED_KAFKA_TOPIC
Namely, both should use an underscore. I don't think there's a way to
directly share these values between a constant and an annotation.
* feat: Add a flight API (handshake only) to ingester
* fix: Create partitions if using file-based write buffer
* fix: Change the server fixture to handle ingester server type
For now, the ingester doesn't implement the deployment API. Not sure if
it should or not.
* feat: Start implementing ingester do_get, namely decoding the query
Skip serialization of the predicate for the moment.
* refactor: Rename ingest protos to ingester to match crate name
* refactor: Rename QueryResults to QueryData
* feat: Move ingester flight client to new querier crate
* fix: Off by one error, different starting indexes in sequencers
* fix: Create new CLI argument to pick the catalog type
* fix: Create a CLI option to set the number of topics to auto-create in the write buffer
* fix: Check the arrow flight service's health to tell that the ingester gRPC is up
* fix: Set postgres as the default catalog type
* fix: Return an error rather than panicking if CLI args aren't right
2022-02-09 19:07:44 +00:00
7d11485c77
chore(deps): bump tracing from 0.1.29 to 0.1.30 ( #3677 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.29 to 0.1.30.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.29...tracing-0.1.30 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 16:44:16 +00:00
59b2141c0b
feat: Add lifecycle manager to ingester ( #3645 )
...
This adds the lifecycle manager to the ingester. It will trigger based on a threshold for max partition size or age or based on keeping total memory under a certain threshold.
It defines a new interface for a persister, which is stubbed out for IngesterData. I'm not sure yet how persistence errors should be handled. The assumption here is that the persister continues to retry persistence forever until it succeeds.
There is one scenario I can think of that may cause this lifecycle manager problems. If a single partition is very high throughput, it could cause things to back up as persistence is not parallelized within a single partition. Any given partition can currently only run one persistence operation at a time. We can address this later.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:23:40 +00:00
9369152096
chore(deps): bump tracing-core from 0.1.21 to 0.1.22 ( #3675 )
...
Bumps [tracing-core](https://github.com/tokio-rs/tracing ) from 0.1.21 to 0.1.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.21...tracing-core-0.1.22 )
---
updated-dependencies:
- dependency-name: tracing-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 15:12:39 +00:00
ba87ae2918
chore(deps): bump crc32fast from 1.3.1 to 1.3.2 ( #3668 )
...
Bumps [crc32fast](https://github.com/srijs/rust-crc32fast ) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/srijs/rust-crc32fast/releases )
- [Commits](https://github.com/srijs/rust-crc32fast/compare/v1.3.1...v1.3.2 )
---
updated-dependencies:
- dependency-name: crc32fast
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-08 14:48:31 +00:00
3c9a27b31b
chore(deps): bump futures-test from 0.3.19 to 0.3.21 ( #3667 )
...
Bumps [futures-test](https://github.com/rust-lang/futures-rs ) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21 )
---
updated-dependencies:
- dependency-name: futures-test
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-08 14:35:34 +00:00
9a874186d5
chore(deps): bump libc from 0.2.116 to 0.2.117 ( #3666 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.116 to 0.2.117.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.116...0.2.117 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:59:02 +00:00
5de4d6203f
refactor: catalog transaction ( #3660 )
...
* refactor: catalog Unit of Work (= transaction)
Setup an inteface to handle Units of Work within our catalog. Previously
both the Postgres and the in-mem backend used "mini-transactions on
demand". Now the caller has a clear way to establish boundaries and
gets read and write isolation. A single `Arc<dyn Catalog>` can create as
many `Box<dyn UnitOfWork>` as you like, but note that depending on the
backend you may not scale infinitely (postgres will likely impose
certain limits and the in-mem backend limits concurrency to 1 to keep
things simple).
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename Unit of Work to Transaction
* test: improve `test_txn_isolation`
* feat: clearify transaction drop semantics
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:38:33 +00:00
de2a013786
feat: gitops adapter ( #3656 )
...
* feat: scaffolding of gitops adapter bin crate
* chore: refactor gitops adapter; calls CLI now; status update fixes
* feat: gitops adapter now calls out to CLI once per topic; improved tests
* chore: add mock failure script for gitops adapter
* chore: update workspace-hack
* chore: refactor away unecessary to_string in gitops syncer
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-08 13:27:36 +00:00
0bd7941a18
fix(REPL): Don't buffer lines until a trailing semicolon is found and add history hinting ( #3630 )
...
* fix(REPL): Don't buffer lines until a trailing semicolon is found
The repl would silently buffer all lines until a trailing semicolon were found which
resulted in some very confusing error messages as I would input invalid commands followed
by a command I thought were valid, except I'd still get an error due to the previous command being buffered.
This uses rustyline's helper feature to detect incomplete input (no trailing semicolon) and makes
it accept multiline input until the input is completed.
I also included some of rustyline's default hint and highlighting while I was at it.
* chore: cargo clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 17:11:01 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
bc4b7f8a5b
test: ensure that rskafka and rdkakfa work together ( #3624 )
...
* chore: upgrade rskafka + enable snappy support
* test: ensure that rskafka and rdkakfa work together
Before removing rdkafka ensure that:
- rskafka can consume existing messages produced by rdkafka so we do not
need to drain existing topics
- rdkafka can consume new messages produced by rskafka so we can roll
back
I ran the whole `write_buffer` test suite (including the newly added
tests) using Apache Kafka as well as Redpanda.
* test: ensure we handle consumer offset in error case correctly
* docs: explain test setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:52:42 +00:00
ce46bbaada
feat: wire up the write buffer to the ingester process ( #3533 )
...
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.
I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.
There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:47:28 +00:00
9079baf19a
Merge branch 'main' into dom/schema-validation
2022-02-02 16:27:35 +00:00
030a2cb4c1
chore: Update datafusion ( #3613 )
...
* chore: Update datafusion
* fix: update for latest DF API
* fix: another API change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 16:27:11 +00:00
4744c5804e
refactor: remove Dashmap
...
Swap Dashmap for a regular RwLock<HashMap<..,>> due to soundness issues:
https://rustsec.org/advisories/RUSTSEC-2022-0002
2022-02-02 14:04:53 +00:00
39d489d9e7
refactor: enable schema validation
...
Adds the SchemaValidator to the DML handler stack - this adds it into
the request path in router2.
2022-02-02 14:04:14 +00:00
6598023726
feat: cache NamespaceSchema in validator
...
Adds an in-memory cache of table schemas to the SchemaValidator DML
handler.
The cache pulls from the global catalog when observing a column for the
first time, and pushes the column type to set it for subsequent requests
if it does not exist (this pull & push is done by atomically by the
catalog in an "upsert" call).
The in-memory cache is sharded by namespace, with each shard guarded by
an individual lock to minimise contention between readers (the expected
average case) and writers (only when adding new columns/tables).
Relies on the catalog to serialise new column creation and validate
parallel creation requests.
2022-02-02 13:04:53 +00:00
c81f207298
feat: schema validation
...
Implements a write schema validation DML handler, denying requests that
conflict with the schema within the global catalog. Additive schema
changes are accepted, incrementally updating the global catalog schema.
Deletes are passed through unchanged and unvalidated.
2022-02-02 13:04:53 +00:00
5441682207
feat: add support for parsing predicate
2022-02-02 11:02:33 +00:00
c1f5994660
refactor: move sql parsing -> RPC predicate into own crate ( #3604 )
...
* chore: create crate
* refactor: move module to new crate
2022-02-02 10:41:57 +00:00
fa546047fb
refactor: update dep with API change ( #3596 )
2022-02-01 14:53:10 +00:00
443fd00c1b
Merge branch 'main' into er/feat/parse_sql_rpc
2022-02-01 13:55:08 +00:00
0a2cb36ddf
fix: workspace-hack ( #3593 )
2022-02-01 12:32:58 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
175732c3ca
feat: basic sqlparser -> RPCNode
2022-02-01 10:26:03 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
8bef2c105c
Merge branch 'main' into cn/persist
2022-01-31 18:50:45 +00:00
7b96a37165
chore: Update datafusion ( #3586 )
...
* chore: update DataFusion to f849968057ddddccc9aa19915ef3ea56bf14d80d
* fix: reduce overhead of creating physical expressions
* chore: use MemTrackingMetrics
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-31 18:15:28 +00:00
bf89162fa5
refactor: Move IoxMetadata to parquet_file
2022-01-31 10:36:33 -05:00
dd9620da0c
feat: Create a new proto definition for the new design's IoxMetadata
2022-01-31 10:36:32 -05:00
5e0e0d8aa7
feat: Write parquet to object storage in a similar way as parquet_file::Storage
2022-01-31 10:36:32 -05:00
c633c9bc5c
feat: Wire object store into ingester persistence
2022-01-31 10:36:30 -05:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
8735ede74f
feat: IoxMetadata for parquet file ( #3547 )
...
* feat: IoxMetadata for parquet file
* fix: typos
* refactor: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 14:41:59 +00:00
1669acc0c2
build: update tokio ( #3566 )
...
Release notes:
https://github.com/tokio-rs/tokio/releases/tag/tokio-1.16.0
2022-01-28 10:36:12 +00:00
b486258dfb
chore: run cargo update ( #3559 )
...
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:05:31 +00:00
9201023ea4
feat: schema validation for MutableBuffer instances ( #3554 )
...
* refactor: Debug bounds on Catalog trait
* feat: validate MutableBatch schema
Changes the schema validation code to validate MutableBatch instances
(coming from a pre-parsed LP write, and non-LP-based writes) instead of
parsed LP lines.
* refactor: Send bound on boxed errors
* refactor: clippy
Allow assert_eq!(bool, bool) for readability.
* refactor: no PartialEq<MB Column> for ColumnSchema
Remove the PartialEq<mutable_buffer::Column> for ColumnSchema - it's
definitely more readable as a method call.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 20:55:18 +00:00
4a96e52290
feat: router2 sharder benchmarking ( #3558 )
...
* feat: benchmarking the router2 sharder
* chore: added throughput to sharder benchmarks; vary num buckets
2022-01-27 18:09:16 +00:00
2062267d0f
chore: Update hashbrown ( #3551 )
...
* chore: Update hashbrown
* fix: hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:34:10 +00:00
ce568ab447
build(iox_catalog): remove unused dependencies ( #3552 )
...
* build: don't pull in all of tokio
We already specify the tokio features we need so "full" (all features)
is not necessary.
* build: remove chrono dependency
Appears unused.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 15:14:42 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
7261571abf
fix: Revert "chore: temporarily hack around datafusion tempfiles" ( #3525 )
...
This reverts commit ae5763c1cb6bb4a98ffe0779a3a35f6daaf10971.
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 12:08:49 +00:00
bc44d33108
feat: Implement a snapshot method on DataBuffer ( #3518 )
...
* feat: Implement a snapshot method on DataBuffer
Fixes #3510 .
* test: Add a test snapshotting batches with different but compatible schemas
* fix: Simplify min/max sequencer number collection
The first batch should always have the min sequencer number. The last
batch should always have the max sequencer number. The min should always
be less than (or equal to, in case there's only one batch) the max.
2022-01-26 15:22:51 +00:00
2928254c0f
fix: test logging ( #3536 )
...
- Use a more standard way to setup the tracing subsystem (as described
in tracing-subscriber docs)
- Also capture content from `log` crate
- Play nice w/ Rust's libtest message capture
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-26 10:28:51 +00:00
b846ead320
feat(router2): shard writes/deletes into write buffer ( #3499 )
...
* feat: Sequencer wrapper
This type wraps an underlying WriteBufferWriter implementation, tagging
it with a sequencer ID it should use when enqueuing operations to the
buffer.
* feat: mock sharder
Implements a mock Sharder impl that returns pre-configured responses to
shard(), and captures the input to the call.
* feat: sharded write buffer
Implements sharding of ops into an underlying WriteBuffer.
Writes are sharded by some abstract Sharder impl, collated per shard to
maximise the size of each op (and therefore compression efficiency),
converted into a DML operation and then enqueued in parallel to the
underlying WriteBuffer implementation.
Deletes are modelled as being mapped to a single write buffer shard,
which is the case while we support sharding based on the table &
namespace only. Deletes will be extended to support (potentially)
multiple shards when column overrides are implemented.
* refactor: runtime write buffers
Switch from using static dispatch, to using a runtime specified
WriteBufferWriting implementation.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-25 15:19:48 +00:00
51a0f8a56a
chore: temporarily hack around datafusion tempfiles ( #3524 )
2022-01-25 12:30:29 +00:00
f9c1e80a7f
chore: update thread_local
...
chore: update thread_local
2022-01-24 13:37:52 -05:00
797ba459b9
chore: merge main to branch
2022-01-24 12:06:23 -05:00
5f98a07b7f
chore: add Corgo.lock
2022-01-24 12:03:02 -05:00
bb893510a0
feat: Add scaffolding for ingester server
...
* Adds a new ingester command to start an ingester server
* Moves previous ingester server over to handler
* Skeleton for gRPC and HTTP handlers
2022-01-21 18:02:19 -05:00
9615feacb3
fix(InfluxQL): Support RegEx with escape sequences not supported by Rust regex ( #3502 )
...
* fix(InfluxQL): Translate unsupported meta characters
* fix: remove debugging
* fix: clippy sacrifice
* docs: Add additional background and rationale for rewriting
* fix: doc link
2022-01-21 14:40:10 +00:00
5b22abf6a2
Merge branch 'main' into crepererum/wb_rskafka
2022-01-20 16:48:10 +00:00
9b6e626626
chore: Update datafusion (and get fix for influxql test failure) ( #3484 )
...
* test: add tests for comparing dictionary arrays
* chore: update datafusion deps
* refactor: Update code for DataFusion API changes
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 14:01:47 +00:00
76dd62a6c2
feat: RSKafka-driven write buffer
2022-01-20 12:36:10 +01:00
36d50d083f
feat: sharder trait & impl
...
This commit defines the Sharder trait that should allow us to implement
multiple sharding strategies over a defined set of input types (such as
a MutableBatch for writes, DeletePredicate for deletes, etc).
This commit also includes a jump hash implementation that consistently
shards (table name, namespace) tuples to a given shard for all input
types.
2022-01-20 11:10:37 +00:00
d825dab8e2
fix: hakari workspace hack
2022-01-19 14:48:00 -05:00
41038721e1
feat: Add parquet file records to iox_catalog
...
* Adds ParquetFile and scaffolding to IOx catalog
* Changed the file_location in parquet_file to object_store_id which is a uuid
2022-01-19 14:14:54 -05:00
edb97f51cf
refactor: add persisting struct
2022-01-19 12:36:18 -05:00
dependabot[bot]
Andrew Lamb
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Nga Tran
Dom Dwyer
kodiakhq[bot]
Marco Neumann
Raphael Taylor-Davies
Luke Bond
Paul Dix
Markus Westerlind
Paul Dix
Andrew Lamb
Edd Robinson
Nga Tran
Luke Bond