Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,764 Commits
234 Branches
453 Tags
344 MiB
Commit Graph

25 Commits (1d326bd4045251b72c3681bab8371d937355b0fb)

Author SHA1 Message Date
Raphael Taylor-Davies 60f0deaf1e
feat: remove flatbuffer entry (#3045) 2021-11-05 20:19:24 +00:00
Raphael Taylor-Davies 07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata (#2743) (#3021) 
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743)

* fix: Kafka message content type header

* chore: fix doc
 2021-11-04 10:20:40 +00:00
Raphael Taylor-Davies 08fcd87337
feat: use protobuf encoding in write buffer (#2724) (#3018) 2021-11-03 15:19:05 +00:00
Raphael Taylor-Davies f1a6468e7b
feat: migrate write buffer consumer to use DbWrite (#2724) (#3003) 
* feat: migrate write buffer consumer to use DbWrite (#2724)

* fix: doc

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-01 16:38:48 +00:00
dependabot[bot] c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 11:21:59 +00:00
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Marco Neumann 6ec0bd5bab feat: file-based write write_buffer 
Closes #2849.
 2021-10-19 15:26:43 +02:00
dependabot[bot] 32e18b6436
chore(deps): bump rdkafka from 0.26.0 to 0.27.0 
Bumps [rdkafka](https://github.com/fede1024/rust-rdkafka) from 0.26.0 to 0.27.0.
- [Release notes](https://github.com/fede1024/rust-rdkafka/releases)
- [Changelog](https://github.com/fede1024/rust-rdkafka/blob/master/changelog.md)
- [Commits](https://github.com/fede1024/rust-rdkafka/compare/v0.26.0...v0.27.0)

---
updated-dependencies:
- dependency-name: rdkafka
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-19 11:49:34 +00:00
Marco Neumann 5e06519afb feat: propagate trace information through write buffer 2021-10-14 11:07:41 +02:00
Raphael Taylor-Davies 8a82f92c5d
refactor: add TimeProvider abstraction (#2722) (#2815) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-12 21:19:03 +00:00
Raphael Taylor-Davies 0554173684
feat: migrate write buffer to TimeProvider (#2722) (#2804) 
* feat: migrate write buffer to TimeProvider (#2722)

* chore: review feedback

Co-authored-by: Marco Neumann <marco@crepererum.net>

Co-authored-by: Marco Neumann <marco@crepererum.net>
 2021-10-12 10:32:34 +00:00
Marco Neumann 07a72a1daa chore: tell cargo-udeps that `dotenv` is needed for `write_buffer` 2021-09-22 11:09:39 +02:00
Marco Neumann 801cf08be7 feat: auto-creation of sequencers by write buffer 
For Kafka, that basically means that we create a topic if it doesn't
exist yet.

Closed #2455.
Fixes #2189.
 2021-09-07 18:24:57 +02:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
Marco Neumann e736bc6953 feat: add ingest timestamp to `Sequence` 
This allows us to track wall-clock ingest time for entries that we
receive via write buffer (e.g. Kafka).
 2021-07-28 14:41:18 +02:00
Marco Neumann 592424c896 refactor: use one stream per sequencer/partition 
Advantages are:

- for large DBs w/ many partitions we can ingest data in-parallel
- on top of this change we can implement per-sequencer seeking, which is
  required for replay
 2021-07-19 12:26:58 +02:00
Marko Mikulicic 06399e88e0
chore: Add some debug logs to write buffer 2021-07-15 22:18:03 +02:00
Marco Neumann b5428e53a5 refactor: write buffer testing + better mocking 
This refactors the write buffer a bit for:

- **Testing:** Add generic tests for the Kafka and the mocking
  implementation. The same interface can be used easily add new
  implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
  specify the partition/sequencer ID. The implicit partitioning of the
  Kafka writer would have lead to broken data since we must never spill
  entries w/ the same primary key over multiple partitions. At the
  moment we will only use partition 0 but we can easily implement
  better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
  that feels more real. Especially the handling around multiple streams
  and "write while read" has been improved. This will be helpful for
  testing and for new features like seeking (during replay). A solid
  realistic mock also helps us to ensure that the tests using the mock
  do not rely on unrealistic behavior too much.
 2021-07-15 17:20:45 +02:00
Marco Neumann 9cb9ae0874 chore: move write buffer into its own crate 2021-07-14 14:09:18 +02:00
Carol (Nichols || Goulding) f4a9a5ae56 fix: Remove write buffer 2021-06-04 14:40:17 -04:00
Carol (Nichols || Goulding) f0b93c5c8c refactor: Rename the wal crate to write_buffer 2021-04-21 17:43:03 +00:00
Andrew Lamb 48c43b136c
refactor: rename write_buffer --> mutable_buffer (#595) 
* refactor: git mv write_buffer mutable_buffer

* refactor: update crate name references

* refactor: update some more references
 2020-12-22 10:49:53 -05:00
Andrew Lamb cdb26e60e4
refactor: rename `storage` crate to `query` to better reflect what it is (#475) 
* refactor: rename storage --> query

* refactor: update a few more referenes
 2020-11-24 14:19:29 -05:00
Andrew Lamb a52e0001c5
refactor: rename all crates that start with`delorean_` in preparation for rename (#415) 
* refactor: rename delorean_cluster --> cluster

* refactor: rebane delorean_generated_types --> generated_types

* refactor: rename delorean_write_buffer --> write_buffer

* refactor: rename delorean_ingest --> ingest

* refactor: rename delorean_storage --> storage

* refactor: rename delorean_tsm --> tsm

* refactor: rename delorean_test_helpers --> test_helpers

* refactor: rename delorean_arrow --> arrow_deps

* refactor: rename delorean_line_parser --> influxdb_line_protocol
 2020-11-05 13:44:36 -05:00