a5f15e6e76
refactor: improve directory names
2021-10-19 16:15:22 +02:00
6ec0bd5bab
feat: file-based write write_buffer
...
Closes #2849 .
2021-10-19 15:26:43 +02:00
b2698cca44
feat: add `format_jaeger_trace_context`
2021-10-19 15:26:05 +02:00
32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0
...
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka ) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases )
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md )
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0 )
---
updated-dependencies:
- dependency-name: rdkafka
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-19 11:49:34 +00:00
45c2c26168
Merge branch 'main' into crepererum/write_buffer_optional_span_ctx
2021-10-15 07:25:21 +00:00
85be39de40
test: make `headers_case_handling` test easier to understand
2021-10-15 09:20:41 +02:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
d6da68b762
fix: do not panic when writing to an unknown sequencer
2021-10-14 17:27:19 +02:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
90d2f1d60d
refactor: use Kafka headers similar to HTTP
2021-10-12 16:31:06 +02:00
b955ecc6a7
feat: add format header to Kafka messages
...
This allows us to transition to new formats in the future.
2021-10-12 16:31:06 +02:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
896ce03415
refactor: use sets and maps for write buffer sequencers
...
Use the type to reflect that entries are unique and sorted.
2021-10-12 11:13:09 +02:00
2e8af77e41
feat: allow write buffer producers to read possible sequencer IDs
2021-10-12 10:37:32 +02:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
07a72a1daa
chore: tell cargo-udeps that `dotenv` is needed for `write_buffer`
2021-09-22 11:09:39 +02:00
c33e5c22e6
feat: pull WriteBuffer consumer out of Db and onto Database ( #2243 ) ( #2525 )
...
* feat: pull WriteBuffer consumer out of Db and onto Database (#2243 )
* chore: restore WritingOnlyAllowedThroughWriteBuffer error
* refactor: remove WriteBufferConfig
* chore: fix docs
* chore: move WriteBufferConsumer tests out of db.rs
* chore: document WriteBufferFactory member functions
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-14 16:04:58 +00:00
7e434d16d2
fix: MockBufferForReading waker registration ( #2520 )
...
* fix: MockBufferForReading waker registration
* chore: remove unnecessary sleep
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-13 21:21:21 +00:00
bbb8898d36
refactor: make writer buffer auto-creation types nicer to read
2021-09-08 11:13:48 +02:00
2cc1297c96
fix: typos
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-09-07 18:24:59 +02:00
c0cc239781
fix: improve Kafka error handling
2021-09-07 18:24:59 +02:00
801cf08be7
feat: auto-creation of sequencers by write buffer
...
For Kafka, that basically means that we create a topic if it doesn't
exist yet.
Closed #2455 .
Fixes #2189 .
2021-09-07 18:24:57 +02:00
924e460bf7
feat: sequencer auto-creation for mocked write buffer
2021-09-07 18:18:20 +02:00
ecbd0165fc
feat: prepare auto-creation of sequencers for mocked write buffer
2021-09-07 18:18:20 +02:00
82f9750ba7
test: prepare write buffer test suite for failable reader and writer creation
...
This is required to work w/ sequencer auto-creation.
2021-09-07 18:18:20 +02:00
f1864813f4
docs: document write buffer test suite
2021-09-07 18:18:20 +02:00
2ea3b600d0
test: harden `assert_reader_content` a bit
...
- entries should be sorted by the stream, there is no need to sort the
results
- ensure that there are no leftover entries in the stream by asserting
that it is "pending"
2021-09-07 18:18:20 +02:00
d5662328b0
refactor: `n_sequencers` should be non-zero
2021-09-07 18:18:20 +02:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2
...
Bumps [parking_lot](https://github.com/Amanieu/parking_lot ) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases )
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2 )
---
updated-dependencies:
- dependency-name: parking_lot
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 01:18:24 +00:00
a63eb53ac5
feat: forward connection config to Kafka write buffer
2021-09-02 16:53:31 +02:00
ecf1f99ddb
refactor: more flexible writer buffer config
...
This allows:
- different types (instead of guessing through the connection URL)
- sequencer counts (not used yet but will be by #2455 )
- extensible configs (e.g. to configure Kafka in a more granular way,
not wired up yet)
- future extensions (since we use a message now instead of a single
string)
**BREAKING: This requires changes for deployed systems / existing DBs!**
2021-09-02 16:41:35 +02:00
f81885c172
fix: limit kafka consumer queue to 10MB
...
I think that while this value limits the maximum memory consumption, it
is too optimistic.
2021-08-20 09:55:49 +02:00
6c0cefded3
fix: limit kafka consumer queue to 100MB
...
By default the queue will be filled to up to 1GB which makes finding
memory leaks quite hard.
2021-08-19 11:36:15 +02:00
817ed4b46b
feat: enable rdkafka statistics ( #2312 ) ( #2314 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-17 09:28:02 +00:00
4a3fe01743
test: don't overdo it
2021-08-16 18:31:45 +02:00
5caa2ad8ec
fix: typo
2021-08-16 18:31:45 +02:00
1a7293015b
test: allow write buffer mocks that always fail
2021-08-16 18:27:09 +02:00
e3c263a006
test: add tooling to purge kafka topic data
2021-08-16 13:47:07 +02:00
21ebdee5a1
feat: make kafka topic creation code reusable
2021-08-16 13:47:07 +02:00
a72bacae67
test: use proper ignore instead of commenting out
2021-08-12 11:38:02 +02:00
a5c74f2798
feat: ability to inject mocked write buffers into server/database
2021-08-12 10:46:16 +02:00
55f68f0401
feat: add `type_name` to write buffer interface
2021-08-12 10:46:16 +02:00
32a5c8ca3a
feat: add ability to clear messages from mocked write buffers
2021-08-12 10:46:16 +02:00
3de6b44e23
build: use new rustdoc lint name ( #2261 )
...
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
2021-08-11 19:48:51 +00:00
99b358d481
feat(iox): Enable snappy compression in the producer
2021-07-29 16:37:11 +02:00
eb310ecada
fix: ensure that the Kafka producer TS we report is also truncated
2021-07-28 15:48:02 +02:00
Marco Neumann
dependabot[bot]
kodiakhq[bot]
Raphael Taylor-Davies
Dom
Marko Mikulicic