Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,142 Commits
188 Branches
453 Tags
378 MiB
Commit Graph

80 Commits (b0d8b679bed0c9064a2a3f24340a20ef0c3fccaa)

Author SHA1 Message Date
Marco Neumann 07a72a1daa chore: tell cargo-udeps that `dotenv` is needed for `write_buffer` 2021-09-22 11:09:39 +02:00
Raphael Taylor-Davies c33e5c22e6
feat: pull WriteBuffer consumer out of Db and onto Database (#2243) (#2525) 
* feat: pull WriteBuffer consumer out of Db and onto Database (#2243)

* chore: restore WritingOnlyAllowedThroughWriteBuffer error

* refactor: remove WriteBufferConfig

* chore: fix docs

* chore: move WriteBufferConsumer tests out of db.rs

* chore: document WriteBufferFactory member functions

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-14 16:04:58 +00:00
Raphael Taylor-Davies 7e434d16d2
fix: MockBufferForReading waker registration (#2520) 
* fix: MockBufferForReading waker registration

* chore: remove unnecessary sleep

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-13 21:21:21 +00:00
Marco Neumann bbb8898d36 refactor: make writer buffer auto-creation types nicer to read 2021-09-08 11:13:48 +02:00
Marco Neumann 2cc1297c96 fix: typos 
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2021-09-07 18:24:59 +02:00
Marco Neumann c0cc239781 fix: improve Kafka error handling 2021-09-07 18:24:59 +02:00
Marco Neumann 801cf08be7 feat: auto-creation of sequencers by write buffer 
For Kafka, that basically means that we create a topic if it doesn't
exist yet.

Closed #2455.
Fixes #2189.
 2021-09-07 18:24:57 +02:00
Marco Neumann 924e460bf7 feat: sequencer auto-creation for mocked write buffer 2021-09-07 18:18:20 +02:00
Marco Neumann ecbd0165fc feat: prepare auto-creation of sequencers for mocked write buffer 2021-09-07 18:18:20 +02:00
Marco Neumann 82f9750ba7 test: prepare write buffer test suite for failable reader and writer creation 
This is required to work w/ sequencer auto-creation.
 2021-09-07 18:18:20 +02:00
Marco Neumann f1864813f4 docs: document write buffer test suite 2021-09-07 18:18:20 +02:00
Marco Neumann 2ea3b600d0 test: harden `assert_reader_content` a bit 
- entries should be sorted by the stream, there is no need to sort the
  results
- ensure that there are no leftover entries in the stream by asserting
  that it is "pending"
 2021-09-07 18:18:20 +02:00
Marco Neumann d5662328b0 refactor: `n_sequencers` should be non-zero 2021-09-07 18:18:20 +02:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
Marco Neumann a63eb53ac5 feat: forward connection config to Kafka write buffer 2021-09-02 16:53:31 +02:00
Marco Neumann ecf1f99ddb refactor: more flexible writer buffer config 
This allows:

- different types (instead of guessing through the connection URL)
- sequencer counts (not used yet but will be by #2455)
- extensible configs (e.g. to configure Kafka in a more granular way,
  not wired up yet)
- future extensions (since we use a message now instead of a single
  string)

**BREAKING: This requires changes for deployed systems / existing DBs!**
 2021-09-02 16:41:35 +02:00
Marco Neumann f81885c172 fix: limit kafka consumer queue to 10MB 
I think that while this value limits the maximum memory consumption, it
is too optimistic.
 2021-08-20 09:55:49 +02:00
Marco Neumann 6c0cefded3 fix: limit kafka consumer queue to 100MB 
By default the queue will be filled to up to 1GB which makes finding
memory leaks quite hard.
 2021-08-19 11:36:15 +02:00
Raphael Taylor-Davies 817ed4b46b
feat: enable rdkafka statistics (#2312) (#2314) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-17 09:28:02 +00:00
Marco Neumann 4a3fe01743 test: don't overdo it 2021-08-16 18:31:45 +02:00
Marco Neumann 5caa2ad8ec fix: typo 2021-08-16 18:31:45 +02:00
Marco Neumann 1a7293015b test: allow write buffer mocks that always fail 2021-08-16 18:27:09 +02:00
Marco Neumann e3c263a006 test: add tooling to purge kafka topic data 2021-08-16 13:47:07 +02:00
Marco Neumann 21ebdee5a1 feat: make kafka topic creation code reusable 2021-08-16 13:47:07 +02:00
Marco Neumann a72bacae67 test: use proper ignore instead of commenting out 2021-08-12 11:38:02 +02:00
Marco Neumann a5c74f2798 feat: ability to inject mocked write buffers into server/database 2021-08-12 10:46:16 +02:00
Marco Neumann 55f68f0401 feat: add `type_name` to write buffer interface 2021-08-12 10:46:16 +02:00
Marco Neumann 32a5c8ca3a feat: add ability to clear messages from mocked write buffers 2021-08-12 10:46:16 +02:00
Dom 3de6b44e23
build: use new rustdoc lint name (#2261) 
* fix: nocache feature code rot

The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.

* build: use updated broken_intra_doc_links name

The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links

https://doc.rust-lang.org/rustdoc/lints.html
 2021-08-11 19:48:51 +00:00
Marko Mikulicic 99b358d481
feat(iox): Enable snappy compression in the producer 2021-07-29 16:37:11 +02:00
Marco Neumann eb310ecada fix: ensure that the Kafka producer TS we report is also truncated 2021-07-28 15:48:02 +02:00
Marco Neumann 0fcec6b742 refactor: move ingest timestamp from sequence to sequended entry 2021-07-28 15:40:35 +02:00
Marco Neumann d940d4f6d3 docs: explain how test timestamps are measured 2021-07-28 14:46:33 +02:00
Marco Neumann e736bc6953 feat: add ingest timestamp to `Sequence` 
This allows us to track wall-clock ingest time for entries that we
receive via write buffer (e.g. Kafka).
 2021-07-28 14:41:18 +02:00
Marco Neumann 55490c279a fix: Kafka watermark error for new partitions 2021-07-21 15:21:52 +02:00
Marco Neumann 5df88c70aa feat: add ability to fetch watermarks from write buffer 2021-07-21 11:59:52 +02:00
kodiakhq[bot] 58dd7e9532
Merge branch 'main' into crepererum/writer_buffer_seek 2021-07-20 12:29:18 +00:00
Marko Mikulicic c01cfbc34c
fix: Increase kafka message size 2021-07-20 14:17:37 +02:00
Marco Neumann ec7ebdff29 refactor: use lifetimes to ensure single stream / no seek while streaming 2021-07-20 13:52:33 +02:00
Marco Neumann b0663a0337 feat: disallow multiple write buffer streams and seeking while streams 
Multiple streams will mess up ordering. Seeking while streaming is
likely a bug and should not work.
 2021-07-20 12:35:20 +02:00
Marco Neumann 38f4eec20e feat: implement `seek` for write buffer 
This is required to control replay ranges.
 2021-07-20 10:25:56 +02:00
Marco Neumann 592424c896 refactor: use one stream per sequencer/partition 
Advantages are:

- for large DBs w/ many partitions we can ingest data in-parallel
- on top of this change we can implement per-sequencer seeking, which is
  required for replay
 2021-07-19 12:26:58 +02:00
Edd Robinson 1676c33113
refactor: update write_buffer/src/kafka.rs 2021-07-16 16:56:03 +01:00
Marko Mikulicic de0dce79f6
docs: Update write_buffer/src/kafka.rs 
Co-authored-by: Edd Robinson <me@edd.io>
 2021-07-16 17:53:02 +02:00
Marko Mikulicic 92623b5f32
fix: Set acks=all for kafka writes 2021-07-16 17:49:04 +02:00
Marko Mikulicic cbadd65cfe
fix: Update write_buffer/src/kafka.rs 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-07-15 23:00:45 +02:00
Marko Mikulicic 06399e88e0
chore: Add some debug logs to write buffer 2021-07-15 22:18:03 +02:00
Carol (Nichols || Goulding) 7301268b4f fix: Increase the internal librdkafka producer queue size 
Given that we've increased the max message size by a factor of 10, also
increase the internal producer queue max size by a factor of 10 to
reduce the number of retries needed to successfully enqueue messages to
Kafka.

Connects to #2007.
 2021-07-15 11:35:55 -04:00
Carol (Nichols || Goulding) fa3a2db0d3 fix: Retry adding Kafka messages to queue forever 
By using [producer.send][] rather than [producer.send_result][] and
specifying Timeout::never.

Connects to #2007.

[producer.send]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send
[producer.send_result]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send_result
 2021-07-15 11:34:23 -04:00