12937ee724
feat: add SOCKS5 support to Kafka write buffer ( #4623 )
2022-05-17 15:21:35 +00:00
259d2486c1
chore(deps): Bump tokio-util from 0.7.1 to 0.7.2 ( #4605 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.1...tokio-util-0.7.2 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-05-16 11:42:31 +00:00
068096e7e1
fix: Rename data_types2 to data_types
2022-05-06 14:45:39 -04:00
0541c6e40f
fix: Remove data_types crate where it's no longer used
2022-05-06 14:45:39 -04:00
3ab0788a94
fix: Move DeletePredicate types to data_types2
2022-05-06 14:45:37 -04:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
694ffd2238
chore(deps): Bump httparse from 1.6.0 to 1.7.0 ( #4277 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.6.0...v1.7.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-12 09:37:08 +00:00
17af5fcbd1
chore(deps): Bump tokio-util from 0.7.0 to 0.7.1 ( #4154 )
...
* chore(deps): Bump tokio-util from 0.7.0 to 0.7.1
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.7.0...tokio-util-0.7.1 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 08:39:02 +00:00
02a2cfde54
feat: update rskafka ( #3805 ) ( #3910 )
2022-03-03 10:53:47 +00:00
44cfdc7aca
feat: update to latest rskafka logging ( #3805 ) ( #3896 )
2022-03-02 11:05:03 +00:00
143311f63f
feat: additional write buffer logging ( #3805 ) ( #3887 )
...
* feat: additional write buffer logging (#3805 )
* fix: assertion direction
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-01 17:08:57 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
b1215805a8
fix: upgrade rskafka for bugfixes and race-free start offsets ( #3775 )
...
This fixes the concerns that were brought up during the review of #3748 .
2022-02-17 11:22:25 +00:00
89105ccfab
chore(deps): Bump tokio-util from 0.6.9 to 0.7.0 ( #3743 )
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.9 to 0.7.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/commits )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-15 11:33:41 +00:00
4db27eec68
fix: defined behaviour when seeking to an unknown sequence number ( #3745 )
...
* chore: upgrade rskafka
* refactor: less cloning
* fix: defined behaviour when seeking to an unknown sequence number
The new, defined behavior is: "return an error once and then end the
stream".
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-15 11:08:01 +00:00
ad60dc6949
chore(deps): bump httparse from 1.5.1 to 1.6.0 ( #3708 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.5.1 to 1.6.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.5.1...v1.6.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 09:54:42 +00:00
50cff27b01
chore: remove rdkafka dependency ( #3625 )
...
All features are now covered by rskafka. This also removes the need to
specify a server ID for write buffer consumers. This was only used for
rdkafka since there we needed to specify a consumer group, even though
we did not use any transactions.
2022-02-03 13:33:56 +00:00
bc4b7f8a5b
test: ensure that rskafka and rdkakfa work together ( #3624 )
...
* chore: upgrade rskafka + enable snappy support
* test: ensure that rskafka and rdkakfa work together
Before removing rdkafka ensure that:
- rskafka can consume existing messages produced by rdkafka so we do not
need to drain existing topics
- rdkafka can consume new messages produced by rskafka so we can roll
back
I ran the whole `write_buffer` test suite (including the newly added
tests) using Apache Kafka as well as Redpanda.
* test: ensure we handle consumer offset in error case correctly
* docs: explain test setup
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 12:52:42 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
b326b62b44
feat: buffer writes when writing to RSKafka ( #3520 )
2022-02-01 10:07:52 +00:00
4101d16f71
chore: feature flag consistency ( #3574 )
...
* chore: feature flag consistency
* chore: add aarch64-apple-darwin to hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-28 16:38:59 +00:00
2928254c0f
fix: test logging ( #3536 )
...
- Use a more standard way to setup the tracing subsystem (as described
in tracing-subscriber docs)
- Also capture content from `log` crate
- Play nice w/ Rust's libtest message capture
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-26 10:28:51 +00:00
76dd62a6c2
feat: RSKafka-driven write buffer
2022-01-20 12:36:10 +01:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
403dcae93c
feat: Put kafka write_buffer code behind a feature flag
...
Which is off by default. This makes rdkafka optional to minimize
build-time dependencies for users that don't plan on using a Kafka write
buffer.
2021-12-09 08:49:34 -05:00
7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs
...
Connects to #3117 .
2021-12-06 09:37:31 -05:00
16d8ae5e04
fix: Match tokio features to what's actually in use in each crate
...
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
2021-12-06 09:37:16 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
c729fc6a25
chore(deps): bump rdkafka from 0.27.0 to 0.28.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.27.0...v0.28.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-29 08:18:02 +00:00
edf7becd20
fix: address review comments
2021-11-24 12:09:52 +01:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
60f0deaf1e
feat: remove flatbuffer entry ( #3045 )
2021-11-05 20:19:24 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
08fcd87337
feat: use protobuf encoding in write buffer ( #2724 ) ( #3018 )
2021-11-03 15:19:05 +00:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
6ec0bd5bab
feat: file-based write write_buffer
...
Closes #2849 .
2021-10-19 15:26:43 +02:00
32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-19 11:49:34 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
07a72a1daa
chore: tell cargo-udeps that `dotenv` is needed for `write_buffer`
2021-09-22 11:09:39 +02:00
801cf08be7
feat: auto-creation of sequencers by write buffer
...
For Kafka, that basically means that we create a topic if it doesn't
exist yet.
Closed #2455 .
Fixes #2189 .
2021-09-07 18:24:57 +02:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
Marco Neumann
dependabot[bot]
Carol (Nichols || Goulding)
二手掉包工程师
Andrew Lamb
Raphael Taylor-Davies
Andrew Lamb
Carol (Nichols || Goulding)