8276511bd3
feat: allow to construct partition checkpoint from partition
2021-07-16 11:45:34 +02:00
a9ea8e9ced
docs: add docstring to some `Partition` methods
2021-07-16 11:45:34 +02:00
81c90868be
docs: note that `MinMaxSequence` are inclusive start/end
2021-07-16 11:45:34 +02:00
71b5030fc0
refactor: remove unused `LockableChunk::write_to_object_store`
2021-07-16 11:45:34 +02:00
00b89cd751
fix: freeze chunks in write path ( #2021 ) ( #2022 )
...
* fix: freeze chunks in write path (#2021 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-16 08:51:37 +00:00
3613b18e32
Merge pull request #2026 from influxdata/ntran/refactor_use_sort_key
...
refactor: address review comments and add useful log info to catch resort
2021-07-15 21:25:00 +00:00
50aa1f857d
Merge branch 'main' into ntran/refactor_use_sort_key
2021-07-15 21:17:26 +00:00
12121ad1c4
Merge pull request #2028 from influxdata/debugkafka
...
chore: Add some debug logs to write buffer
2021-07-15 21:14:57 +00:00
76d9b8f7cc
Merge branch 'main' into debugkafka
2021-07-15 21:07:35 +00:00
6218957bd8
fix: flaky lifecycle test ( #1994 ) ( #2020 )
...
* fix: flaky lifecycle test (#1994 )
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 21:07:09 +00:00
cbadd65cfe
fix: Update write_buffer/src/kafka.rs
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-15 23:00:45 +02:00
d5dcb40438
refactor: track future execution ( #2014 )
...
* refactor: track future execution
* refactor: update server/src/db/lifecycle/compact.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 20:58:52 +00:00
06399e88e0
chore: Add some debug logs to write buffer
2021-07-15 22:18:03 +02:00
f3b44d81f2
refactor: remove empty lines
2021-07-15 15:43:45 -04:00
cfe0bfa88b
refactor: address review comments and add useful log info to catch resort
2021-07-15 15:39:12 -04:00
74b8bb76e6
chore: Update to correct pre-release version of DF ( #2023 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2021-07-15 18:13:42 +00:00
5027131bb2
Merge pull request #2019 from influxdata/cn/kafka-queue-settings
...
fix: Kafka queueing
2021-07-15 17:47:42 +00:00
0db38a84db
Merge branch 'main' into cn/kafka-queue-settings
2021-07-15 17:40:18 +00:00
3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks ( #2017 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
7301268b4f
fix: Increase the internal librdkafka producer queue size
...
Given that we've increased the max message size by a factor of 10, also
increase the internal producer queue max size by a factor of 10 to
reduce the number of retries needed to successfully enqueue messages to
Kafka.
Connects to #2007 .
2021-07-15 11:35:55 -04:00
fa3a2db0d3
fix: Retry adding Kafka messages to queue forever
...
By using [producer.send][] rather than [producer.send_result][] and
specifying Timeout::never.
Connects to #2007 .
[producer.send]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send
[producer.send_result]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send_result
2021-07-15 11:34:23 -04:00
5a9c73a4dd
Merge pull request #2012 from influxdata/crepererum/write_buffer_testing
...
refactor: write buffer testing + better mocking
2021-07-15 15:29:49 +00:00
a064820a70
fix: code comment should match the code
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-07-15 17:20:46 +02:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
3bb32594ba
refactor: rename end-to-end.rs to end_to_end.rs ( #2015 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:50:32 +00:00
a79c0b4e75
feat: add mub row count threshold to lifecycle rules ( #1876 ) ( #2016 )
...
* feat: add mub row count threshold to lifecycle rules (#1876 )
* chore: update docstring
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:42:17 +00:00
0442991e45
docs: Add links to latest tech talks ( #2013 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:10:49 +00:00
6a4c08ec28
refactor: extract DatabaseBuilder for end-to-end test cases ( #2004 )
...
* refactor: extract DatabaseBuilder for end-to-end test cases
* chore: fix kafka tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 12:25:21 +00:00
d71f38f27c
feat: compute PartitionCheckpoint from PersistenceWindows ( #2011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 12:17:23 +00:00
0c86d1dccf
feat: Record parquet bytes size in catalog / parquet_file ( #2006 )
...
* feat: Store object store size in parquet_file
* fix: update TRANSACTION_VERSION to 8
* refactor: rename os_bytes --> file_size_bytes
2021-07-15 12:07:11 +00:00
86f677cba2
Merge pull request #1959 from influxdata/crepererum/persist_drop
...
feat: persist "drop chunk"
2021-07-15 10:15:11 +00:00
4741483f72
docs: explain why we update memory metrics when lifecycle action is cleared
2021-07-15 12:07:56 +02:00
924b0db542
fix: account for memory size in drop lifecycle action
2021-07-15 12:07:56 +02:00
cccdd8a43f
fix: correct code comment
2021-07-15 12:07:56 +02:00
77a9191a11
fix: chunk dropping over lifecycle policy should also respect the preserved catalog
2021-07-15 12:07:56 +02:00
71cb15f017
refactor: use lifecycle action to drop chunks
...
This avoids holding partition locks while the preserved catalog IO is
done.
2021-07-15 12:07:56 +02:00
956086fa6d
feat: add "drop chunk" job type
2021-07-15 12:07:56 +02:00
e570c66697
feat: add "dropping" chunk lifecycle action
2021-07-15 12:07:56 +02:00
68e20779a2
test: add test for clearing lifecycle actions from chunks
2021-07-15 12:07:56 +02:00
d89fca00be
feat: persist "drop chunk"
2021-07-15 12:07:56 +02:00
40047a76bc
refactor: `remove_parquet` cannot fail
2021-07-15 12:07:56 +02:00
3e0d1eb560
refactor: introduce PartitionAddr ( #2010 )
2021-07-15 10:01:33 +00:00
6182c0974b
Merge pull request #1970 from influxdata/ntran/use_sort_key
...
feat: Using sort_key to avoid resorts
2021-07-14 21:43:41 +00:00
0b1f2b1fd0
chore: merge main to branch
2021-07-14 16:17:14 -04:00
ef271d1e1c
test: make the tests clearer
2021-07-14 15:42:30 -04:00
b4d86dcb7d
fix: make the order of sort key deterministic
2021-07-14 14:50:19 -04:00
9ffaf863fa
refactor: cleanup
2021-07-14 14:30:04 -04:00
552e3fb691
fix: Padd stats compute deterministic order of sort key and update tests that got changed by the use of sort key
2021-07-14 14:06:41 -04:00
d156998b46
fix: remove unused parameter `mutable_linger_seconds` from dbrules ( #2003 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 18:06:20 +00:00
2bcf6d280b
Merge pull request #1999 from influxdata/cn/exploration
...
fix: Change RUB chunk table_summaries to table_summary
2021-07-14 17:38:22 +00:00
Marco Neumann
Raphael Taylor-Davies
kodiakhq[bot]
Marko Mikulicic
Edd Robinson
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)