d6c6e9a6c7
fix: Default kafka timeout to be shorter than gRPC timeout (60 sec --> 10 sec) ( #3131 )
...
* fix: Default kafka timeout to be shorter than gRPC timeout
* docs: fix link style
2021-11-17 12:19:53 +00:00
79929c8cf4
feat: add more Kafka metrics
2021-11-16 17:18:41 +01:00
9ee004946e
fix: do not overload rdkafka w/ statistics
2021-11-16 17:18:41 +01:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
a6d83a3026
feat: WriteBufferReader use DmlOperation ( #2731 ) ( #3096 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
11f4a1dee8
feat: add connection management for router
2021-11-08 11:11:52 +01:00
60f0deaf1e
feat: remove flatbuffer entry ( #3045 )
2021-11-05 20:19:24 +00:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
08fcd87337
feat: use protobuf encoding in write buffer ( #2724 ) ( #3018 )
2021-11-03 15:19:05 +00:00
51c6348e54
refactor: extract codec module for write buffer ( #2724 ) ( #3017 )
2021-11-03 14:07:33 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
6ceab054ab
refactor: move DbWrite to mutable_batch ( #2986 )
...
* refactor: move DbWrite to mutable_batch
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
3af1504ed2
fix: race condition in file-based write buffer
2021-10-26 10:09:34 +02:00
93f6519c34
fix: address review comments
2021-10-26 10:09:34 +02:00
d527775aec
feat: allow gaps in file-based WB + improve error handling
2021-10-26 10:09:34 +02:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
a5f15e6e76
refactor: improve directory names
2021-10-19 16:15:22 +02:00
6ec0bd5bab
feat: file-based write write_buffer
...
Closes #2849 .
2021-10-19 15:26:43 +02:00
b2698cca44
feat: add `format_jaeger_trace_context`
2021-10-19 15:26:05 +02:00
32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-19 11:49:34 +00:00
45c2c26168
Merge branch 'main' into crepererum/write_buffer_optional_span_ctx
2021-10-15 07:25:21 +00:00
85be39de40
test: make `headers_case_handling` test easier to understand
2021-10-15 09:20:41 +02:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
d6da68b762
fix: do not panic when writing to an unknown sequencer
2021-10-14 17:27:19 +02:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
90d2f1d60d
refactor: use Kafka headers similar to HTTP
2021-10-12 16:31:06 +02:00
b955ecc6a7
feat: add format header to Kafka messages
...
This allows us to transition to new formats in the future.
2021-10-12 16:31:06 +02:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
896ce03415
refactor: use sets and maps for write buffer sequencers
...
Use the type to reflect that entries are unique and sorted.
2021-10-12 11:13:09 +02:00
2e8af77e41
feat: allow write buffer producers to read possible sequencer IDs
2021-10-12 10:37:32 +02:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
07a72a1daa
chore: tell cargo-udeps that `dotenv` is needed for `write_buffer`
2021-09-22 11:09:39 +02:00
c33e5c22e6
feat: pull WriteBuffer consumer out of Db and onto Database ( #2243 ) ( #2525 )
...
* feat: pull WriteBuffer consumer out of Db and onto Database (#2243 )
* chore: restore WritingOnlyAllowedThroughWriteBuffer error
* refactor: remove WriteBufferConfig
* chore: fix docs
* chore: move WriteBufferConsumer tests out of db.rs
* chore: document WriteBufferFactory member functions
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-14 16:04:58 +00:00
7e434d16d2
fix: MockBufferForReading waker registration ( #2520 )
...
* fix: MockBufferForReading waker registration
* chore: remove unnecessary sleep
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-13 21:21:21 +00:00
bbb8898d36
refactor: make writer buffer auto-creation types nicer to read
2021-09-08 11:13:48 +02:00
2cc1297c96
fix: typos
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-09-07 18:24:59 +02:00
c0cc239781
fix: improve Kafka error handling
2021-09-07 18:24:59 +02:00
801cf08be7
feat: auto-creation of sequencers by write buffer
...
For Kafka, that basically means that we create a topic if it doesn't
exist yet.
Closed #2455 .
Fixes #2189 .
2021-09-07 18:24:57 +02:00
924e460bf7
feat: sequencer auto-creation for mocked write buffer
2021-09-07 18:18:20 +02:00
ecbd0165fc
feat: prepare auto-creation of sequencers for mocked write buffer
2021-09-07 18:18:20 +02:00
82f9750ba7
test: prepare write buffer test suite for failable reader and writer creation
...
This is required to work w/ sequencer auto-creation.
2021-09-07 18:18:20 +02:00
Andrew Lamb
Marco Neumann
Raphael Taylor-Davies
dependabot[bot]
kodiakhq[bot]