Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
6,263 Commits
205 Branches
453 Tags
382 MiB
Commit Graph

137 Commits (d289ba6d03a33a59c6918add4497228fc06cad9c)

Author SHA1 Message Date
Raphael Taylor-Davies 8a82f92c5d
refactor: add TimeProvider abstraction (#2722) (#2815) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-12 21:19:03 +00:00
Marco Neumann 90d2f1d60d refactor: use Kafka headers similar to HTTP 2021-10-12 16:31:06 +02:00
Marco Neumann b955ecc6a7 feat: add format header to Kafka messages 
This allows us to transition to new formats in the future.
 2021-10-12 16:31:06 +02:00
Raphael Taylor-Davies 0554173684
feat: migrate write buffer to TimeProvider (#2722) (#2804) 
* feat: migrate write buffer to TimeProvider (#2722)

* chore: review feedback

Co-authored-by: Marco Neumann <marco@crepererum.net>

Co-authored-by: Marco Neumann <marco@crepererum.net>
 2021-10-12 10:32:34 +00:00
Marco Neumann 896ce03415 refactor: use sets and maps for write buffer sequencers 
Use the type to reflect that entries are unique and sorted.
 2021-10-12 11:13:09 +02:00
Marco Neumann 2e8af77e41 feat: allow write buffer producers to read possible sequencer IDs 2021-10-12 10:37:32 +02:00
Raphael Taylor-Davies f35a49edd0
refactor: move Sequence to data_types (#2780) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-11 09:23:00 +00:00
Marco Neumann 07a72a1daa chore: tell cargo-udeps that `dotenv` is needed for `write_buffer` 2021-09-22 11:09:39 +02:00
Raphael Taylor-Davies c33e5c22e6
feat: pull WriteBuffer consumer out of Db and onto Database (#2243) (#2525) 
* feat: pull WriteBuffer consumer out of Db and onto Database (#2243)

* chore: restore WritingOnlyAllowedThroughWriteBuffer error

* refactor: remove WriteBufferConfig

* chore: fix docs

* chore: move WriteBufferConsumer tests out of db.rs

* chore: document WriteBufferFactory member functions

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-14 16:04:58 +00:00
Raphael Taylor-Davies 7e434d16d2
fix: MockBufferForReading waker registration (#2520) 
* fix: MockBufferForReading waker registration

* chore: remove unnecessary sleep

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-13 21:21:21 +00:00
Marco Neumann bbb8898d36 refactor: make writer buffer auto-creation types nicer to read 2021-09-08 11:13:48 +02:00
Marco Neumann 2cc1297c96 fix: typos 
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
 2021-09-07 18:24:59 +02:00
Marco Neumann c0cc239781 fix: improve Kafka error handling 2021-09-07 18:24:59 +02:00
Marco Neumann 801cf08be7 feat: auto-creation of sequencers by write buffer 
For Kafka, that basically means that we create a topic if it doesn't
exist yet.

Closed #2455.
Fixes #2189.
 2021-09-07 18:24:57 +02:00
Marco Neumann 924e460bf7 feat: sequencer auto-creation for mocked write buffer 2021-09-07 18:18:20 +02:00
Marco Neumann ecbd0165fc feat: prepare auto-creation of sequencers for mocked write buffer 2021-09-07 18:18:20 +02:00
Marco Neumann 82f9750ba7 test: prepare write buffer test suite for failable reader and writer creation 
This is required to work w/ sequencer auto-creation.
 2021-09-07 18:18:20 +02:00
Marco Neumann f1864813f4 docs: document write buffer test suite 2021-09-07 18:18:20 +02:00
Marco Neumann 2ea3b600d0 test: harden `assert_reader_content` a bit 
- entries should be sorted by the stream, there is no need to sort the
  results
- ensure that there are no leftover entries in the stream by asserting
  that it is "pending"
 2021-09-07 18:18:20 +02:00
Marco Neumann d5662328b0 refactor: `n_sequencers` should be non-zero 2021-09-07 18:18:20 +02:00
dependabot[bot] b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 09:11:38 +00:00
dependabot[bot] b1bb390893
chore(deps): bump parking_lot from 0.11.1 to 0.11.2 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.1 to 0.11.2.
- [Release notes](https://github.com/Amanieu/parking_lot/releases)
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.11.1...0.11.2)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-06 01:18:24 +00:00
Marco Neumann a63eb53ac5 feat: forward connection config to Kafka write buffer 2021-09-02 16:53:31 +02:00
Marco Neumann ecf1f99ddb refactor: more flexible writer buffer config 
This allows:

- different types (instead of guessing through the connection URL)
- sequencer counts (not used yet but will be by #2455)
- extensible configs (e.g. to configure Kafka in a more granular way,
  not wired up yet)
- future extensions (since we use a message now instead of a single
  string)

**BREAKING: This requires changes for deployed systems / existing DBs!**
 2021-09-02 16:41:35 +02:00
Marco Neumann f81885c172 fix: limit kafka consumer queue to 10MB 
I think that while this value limits the maximum memory consumption, it
is too optimistic.
 2021-08-20 09:55:49 +02:00
Marco Neumann 6c0cefded3 fix: limit kafka consumer queue to 100MB 
By default the queue will be filled to up to 1GB which makes finding
memory leaks quite hard.
 2021-08-19 11:36:15 +02:00
Raphael Taylor-Davies 817ed4b46b
feat: enable rdkafka statistics (#2312) (#2314) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-08-17 09:28:02 +00:00
Marco Neumann 4a3fe01743 test: don't overdo it 2021-08-16 18:31:45 +02:00
Marco Neumann 5caa2ad8ec fix: typo 2021-08-16 18:31:45 +02:00
Marco Neumann 1a7293015b test: allow write buffer mocks that always fail 2021-08-16 18:27:09 +02:00
Marco Neumann e3c263a006 test: add tooling to purge kafka topic data 2021-08-16 13:47:07 +02:00
Marco Neumann 21ebdee5a1 feat: make kafka topic creation code reusable 2021-08-16 13:47:07 +02:00
Marco Neumann a72bacae67 test: use proper ignore instead of commenting out 2021-08-12 11:38:02 +02:00
Marco Neumann a5c74f2798 feat: ability to inject mocked write buffers into server/database 2021-08-12 10:46:16 +02:00
Marco Neumann 55f68f0401 feat: add `type_name` to write buffer interface 2021-08-12 10:46:16 +02:00
Marco Neumann 32a5c8ca3a feat: add ability to clear messages from mocked write buffers 2021-08-12 10:46:16 +02:00
Dom 3de6b44e23
build: use new rustdoc lint name (#2261) 
* fix: nocache feature code rot

The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.

* build: use updated broken_intra_doc_links name

The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links

https://doc.rust-lang.org/rustdoc/lints.html
 2021-08-11 19:48:51 +00:00
Marko Mikulicic 99b358d481
feat(iox): Enable snappy compression in the producer 2021-07-29 16:37:11 +02:00
Marco Neumann eb310ecada fix: ensure that the Kafka producer TS we report is also truncated 2021-07-28 15:48:02 +02:00
Marco Neumann 0fcec6b742 refactor: move ingest timestamp from sequence to sequended entry 2021-07-28 15:40:35 +02:00
Marco Neumann d940d4f6d3 docs: explain how test timestamps are measured 2021-07-28 14:46:33 +02:00
Marco Neumann e736bc6953 feat: add ingest timestamp to `Sequence` 
This allows us to track wall-clock ingest time for entries that we
receive via write buffer (e.g. Kafka).
 2021-07-28 14:41:18 +02:00
Marco Neumann 55490c279a fix: Kafka watermark error for new partitions 2021-07-21 15:21:52 +02:00
Marco Neumann 5df88c70aa feat: add ability to fetch watermarks from write buffer 2021-07-21 11:59:52 +02:00
kodiakhq[bot] 58dd7e9532
Merge branch 'main' into crepererum/writer_buffer_seek 2021-07-20 12:29:18 +00:00
Marko Mikulicic c01cfbc34c
fix: Increase kafka message size 2021-07-20 14:17:37 +02:00
Marco Neumann ec7ebdff29 refactor: use lifetimes to ensure single stream / no seek while streaming 2021-07-20 13:52:33 +02:00
Marco Neumann b0663a0337 feat: disallow multiple write buffer streams and seeking while streams 
Multiple streams will mess up ordering. Seeking while streaming is
likely a bug and should not work.
 2021-07-20 12:35:20 +02:00
Marco Neumann 38f4eec20e feat: implement `seek` for write buffer 
This is required to control replay ranges.
 2021-07-20 10:25:56 +02:00
Marco Neumann 592424c896 refactor: use one stream per sequencer/partition 
Advantages are:

- for large DBs w/ many partitions we can ingest data in-parallel
- on top of this change we can implement per-sequencer seeking, which is
  required for replay
 2021-07-19 12:26:58 +02:00
Edd Robinson 1676c33113
refactor: update write_buffer/src/kafka.rs 2021-07-16 16:56:03 +01:00
Marko Mikulicic de0dce79f6
docs: Update write_buffer/src/kafka.rs 
Co-authored-by: Edd Robinson <me@edd.io>
 2021-07-16 17:53:02 +02:00
Marko Mikulicic 92623b5f32
fix: Set acks=all for kafka writes 2021-07-16 17:49:04 +02:00
Marko Mikulicic cbadd65cfe
fix: Update write_buffer/src/kafka.rs 
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2021-07-15 23:00:45 +02:00
Marko Mikulicic 06399e88e0
chore: Add some debug logs to write buffer 2021-07-15 22:18:03 +02:00
Carol (Nichols || Goulding) 7301268b4f fix: Increase the internal librdkafka producer queue size 
Given that we've increased the max message size by a factor of 10, also
increase the internal producer queue max size by a factor of 10 to
reduce the number of retries needed to successfully enqueue messages to
Kafka.

Connects to #2007.
 2021-07-15 11:35:55 -04:00
Carol (Nichols || Goulding) fa3a2db0d3 fix: Retry adding Kafka messages to queue forever 
By using [producer.send][] rather than [producer.send_result][] and
specifying Timeout::never.

Connects to #2007.

[producer.send]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send
[producer.send_result]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send_result
 2021-07-15 11:34:23 -04:00
Marco Neumann a064820a70 fix: code comment should match the code 
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2021-07-15 17:20:46 +02:00
Marco Neumann b5428e53a5 refactor: write buffer testing + better mocking 
This refactors the write buffer a bit for:

- **Testing:** Add generic tests for the Kafka and the mocking
  implementation. The same interface can be used easily add new
  implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
  specify the partition/sequencer ID. The implicit partitioning of the
  Kafka writer would have lead to broken data since we must never spill
  entries w/ the same primary key over multiple partitions. At the
  moment we will only use partition 0 but we can easily implement
  better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
  that feels more real. Especially the handling around multiple streams
  and "write while read" has been improved. This will be helpful for
  testing and for new features like seeking (during replay). A solid
  realistic mock also helps us to ensure that the tests using the mock
  do not rely on unrealistic behavior too much.
 2021-07-15 17:20:45 +02:00
Marko Mikulicic 8d23dd6d6d
fix: Set kafka max message size in client 2021-07-14 14:46:49 +02:00
Marco Neumann 9cb9ae0874 chore: move write buffer into its own crate 2021-07-14 14:09:18 +02:00
Carol (Nichols || Goulding) f4a9a5ae56 fix: Remove write buffer 2021-06-04 14:40:17 -04:00
Marco Neumann eddc9319ff docs: deny broken intradoc links 2021-04-27 13:22:28 +02:00
Carol (Nichols || Goulding) 938acb98e8 refactor: Rename WAL within the types, variables, and comments in the write_buffer crate 2021-04-21 17:43:03 +00:00
Carol (Nichols || Goulding) f0b93c5c8c refactor: Rename the wal crate to write_buffer 2021-04-21 17:43:03 +00:00
Andrew Lamb 48c43b136c
refactor: rename write_buffer --> mutable_buffer (#595) 
* refactor: git mv write_buffer mutable_buffer

* refactor: update crate name references

* refactor: update some more references
 2020-12-22 10:49:53 -05:00
Andrew Lamb 263af1eeac
feat: implement read_group in the write_buffer (#583) 
* feat: implement read_group in the write_buffer

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: rustfmt

* fix: adjust tests for min/max

* fix: Update write_buffer/src/table.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-12-22 09:03:20 -05:00
Andrew Lamb 28eac06d8f
refactor: Organize window_bounds the same as selector functions (#594) 
* refactor: Organize window_bounds the same as selector functions

* fix: add missing file
 2020-12-21 12:51:36 -05:00
Andrew Lamb bb96142564
chore: Update arrow dependencies, remove custom min/max implementation (#585) 
* chore: Update arrow dependency

* fix: Update code for changes in datafusion

* fix: use arrow version of min_boolean
 2020-12-21 12:31:39 -05:00
Andrew Lamb 699504a022
feat: implement read_group for Sum, Count and Mean aggregates (#557) 
* feat: Implement read_group for "normal" aggregates

* test: add tests
 2020-12-15 09:35:00 -05:00
Andrew Lamb a6d2c13888
chore: Update arrow + other depenencies (#540) 
* chore: Update arrow + other depenencies

* chore: Update write_buffer and query crate
 2020-12-15 08:46:27 -05:00
Dom 6f473984d0 style: wrap comments 
Runs rustfmt with the new config.
 2020-12-11 18:22:26 +00:00
Andrew Lamb ea6b2f6bc8
refactor: remove minor code duplication (#555) 2020-12-11 11:18:00 -05:00
Paul Dix fa3ecbd4ed
feat: Implement write buffer to Parquet snapshotting (#526) 
* feat: Implement write buffer to Parquet snapshotting

This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
 2020-12-08 14:20:43 -05:00
Andrew Lamb 4ec75a4f22
fix: Fix gRPC panic` when multiple field selections are provided (#523) 
* fix: do not assert when multiple fields are selected

* fix: clippy

* fix: write unit test, fix bug

* fix: tweak comments
 2020-12-03 12:31:02 -05:00
Andrew Lamb ecc4eee8e1
refactor: Move SQL functions into is own trait (#511) 
* refactor: remove uneeded function table_to_arrow from Trait

* refactor: Move SQL functions into is own trait
 2020-12-02 08:23:37 -05:00
Andrew Lamb 5ef499bb63
refactor: rename Database --> TSDatabase to better reflect its purpose (#510) 
* refactor: rename Database --> TSDatabase to better reflect its purpose

* refactor: rename field_columns to field_column_names

* fix: clippy?
 2020-12-01 12:37:11 -05:00
Andrew Lamb 1646397891
refactor: consolidate GroupedSeriesSet and SeriesSet (#502) 2020-11-30 14:23:58 -05:00
Andrew Lamb 20f421e9c6
fix: Do not send GroupFrames in response to read_window_aggregate (#497) 
* fix: Do not send GroupFrames in response to read_window_aggregate

* fix: clippy and test
 2020-11-30 05:59:05 -05:00
Andrew Lamb 8c8af66d4d
refactor: use datafusion expression building code in write_buffer (#484) 2020-11-25 14:41:39 -05:00
Andrew Lamb 0eaa90e89d
feat: Hook up read_window_aggregate into the write_buffer, end-to-end tests (#483) 
* feat: read_window_aggregate_plans

* fix: clippy sacrifice

* fix: clippy

* fix: clippy
 2020-11-25 10:20:49 -05:00
Andrew Lamb 9f6427c94f
refactor: query/src/groupby.rs -> query/src/group_by.rs (#477) 
* refactor: query/src/groupby.rs -> query/src/group_by.rs

* refactor: update references
 2020-11-25 06:43:11 -05:00
Andrew Lamb cdb26e60e4
refactor: rename `storage` crate to `query` to better reflect what it is (#475) 
* refactor: rename storage --> query

* refactor: update a few more referenes
 2020-11-24 14:19:29 -05:00
Andrew Lamb 85921fe401
feat: Implement gRPC and storage interface for read_group_aggregate (#474) 
* feat: Implement gRPC and storage interface for read_group_aggregate

* fix: clippy

* docs: Tweak comments

* fix: moar clippy

* fix: fmt

* docs: Apply doc improvement suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: improve error creation

* refactor: use match instead of if

* refactor: clearer match

* refactor: clean up storage aggregate matching

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
 2020-11-24 13:41:11 -05:00
Andrew Lamb 597933622d
fix: improve error messages with more context (#455) 2020-11-16 16:40:29 -05:00
Andrew Lamb 2fa0e03162
fix: Use datafusion optimizer in IOx query plans (#439) 
* chore: update arrow dep to 8e4d9ebef3

* fix: checkin Cargo.lock

* fix: Enable datafusion optimizer, use display_indent_schema
 2020-11-11 18:06:21 -05:00
Andrew Lamb a52e0001c5
refactor: rename all crates that start with`delorean_` in preparation for rename (#415) 
* refactor: rename delorean_cluster --> cluster

* refactor: rebane delorean_generated_types --> generated_types

* refactor: rename delorean_write_buffer --> write_buffer

* refactor: rename delorean_ingest --> ingest

* refactor: rename delorean_storage --> storage

* refactor: rename delorean_tsm --> tsm

* refactor: rename delorean_test_helpers --> test_helpers

* refactor: rename delorean_arrow --> arrow_deps

* refactor: rename delorean_line_parser --> influxdb_line_protocol
 2020-11-05 13:44:36 -05:00