76d9b8f7cc
Merge branch 'main' into debugkafka
2021-07-15 21:07:35 +00:00
6218957bd8
fix: flaky lifecycle test ( #1994 ) ( #2020 )
...
* fix: flaky lifecycle test (#1994 )
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 21:07:09 +00:00
cbadd65cfe
fix: Update write_buffer/src/kafka.rs
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-15 23:00:45 +02:00
d5dcb40438
refactor: track future execution ( #2014 )
...
* refactor: track future execution
* refactor: update server/src/db/lifecycle/compact.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 20:58:52 +00:00
06399e88e0
chore: Add some debug logs to write buffer
2021-07-15 22:18:03 +02:00
74b8bb76e6
chore: Update to correct pre-release version of DF ( #2023 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2021-07-15 18:13:42 +00:00
5027131bb2
Merge pull request #2019 from influxdata/cn/kafka-queue-settings
...
fix: Kafka queueing
2021-07-15 17:47:42 +00:00
0db38a84db
Merge branch 'main' into cn/kafka-queue-settings
2021-07-15 17:40:18 +00:00
3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks ( #2017 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
7301268b4f
fix: Increase the internal librdkafka producer queue size
...
Given that we've increased the max message size by a factor of 10, also
increase the internal producer queue max size by a factor of 10 to
reduce the number of retries needed to successfully enqueue messages to
Kafka.
Connects to #2007 .
2021-07-15 11:35:55 -04:00
fa3a2db0d3
fix: Retry adding Kafka messages to queue forever
...
By using [producer.send][] rather than [producer.send_result][] and
specifying Timeout::never.
Connects to #2007 .
[producer.send]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send
[producer.send_result]: https://docs.rs/rdkafka/0.26.0/rdkafka/producer/future_producer/struct.FutureProducer.html#method.send_result
2021-07-15 11:34:23 -04:00
5a9c73a4dd
Merge pull request #2012 from influxdata/crepererum/write_buffer_testing
...
refactor: write buffer testing + better mocking
2021-07-15 15:29:49 +00:00
a064820a70
fix: code comment should match the code
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-07-15 17:20:46 +02:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
3bb32594ba
refactor: rename end-to-end.rs to end_to_end.rs ( #2015 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:50:32 +00:00
a79c0b4e75
feat: add mub row count threshold to lifecycle rules ( #1876 ) ( #2016 )
...
* feat: add mub row count threshold to lifecycle rules (#1876 )
* chore: update docstring
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:42:17 +00:00
0442991e45
docs: Add links to latest tech talks ( #2013 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:10:49 +00:00
6a4c08ec28
refactor: extract DatabaseBuilder for end-to-end test cases ( #2004 )
...
* refactor: extract DatabaseBuilder for end-to-end test cases
* chore: fix kafka tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 12:25:21 +00:00
d71f38f27c
feat: compute PartitionCheckpoint from PersistenceWindows ( #2011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 12:17:23 +00:00
0c86d1dccf
feat: Record parquet bytes size in catalog / parquet_file ( #2006 )
...
* feat: Store object store size in parquet_file
* fix: update TRANSACTION_VERSION to 8
* refactor: rename os_bytes --> file_size_bytes
2021-07-15 12:07:11 +00:00
86f677cba2
Merge pull request #1959 from influxdata/crepererum/persist_drop
...
feat: persist "drop chunk"
2021-07-15 10:15:11 +00:00
4741483f72
docs: explain why we update memory metrics when lifecycle action is cleared
2021-07-15 12:07:56 +02:00
924b0db542
fix: account for memory size in drop lifecycle action
2021-07-15 12:07:56 +02:00
cccdd8a43f
fix: correct code comment
2021-07-15 12:07:56 +02:00
77a9191a11
fix: chunk dropping over lifecycle policy should also respect the preserved catalog
2021-07-15 12:07:56 +02:00
71cb15f017
refactor: use lifecycle action to drop chunks
...
This avoids holding partition locks while the preserved catalog IO is
done.
2021-07-15 12:07:56 +02:00
956086fa6d
feat: add "drop chunk" job type
2021-07-15 12:07:56 +02:00
e570c66697
feat: add "dropping" chunk lifecycle action
2021-07-15 12:07:56 +02:00
68e20779a2
test: add test for clearing lifecycle actions from chunks
2021-07-15 12:07:56 +02:00
d89fca00be
feat: persist "drop chunk"
2021-07-15 12:07:56 +02:00
40047a76bc
refactor: `remove_parquet` cannot fail
2021-07-15 12:07:56 +02:00
3e0d1eb560
refactor: introduce PartitionAddr ( #2010 )
2021-07-15 10:01:33 +00:00
6182c0974b
Merge pull request #1970 from influxdata/ntran/use_sort_key
...
feat: Using sort_key to avoid resorts
2021-07-14 21:43:41 +00:00
0b1f2b1fd0
chore: merge main to branch
2021-07-14 16:17:14 -04:00
ef271d1e1c
test: make the tests clearer
2021-07-14 15:42:30 -04:00
b4d86dcb7d
fix: make the order of sort key deterministic
2021-07-14 14:50:19 -04:00
9ffaf863fa
refactor: cleanup
2021-07-14 14:30:04 -04:00
552e3fb691
fix: Padd stats compute deterministic order of sort key and update tests that got changed by the use of sort key
2021-07-14 14:06:41 -04:00
d156998b46
fix: remove unused parameter `mutable_linger_seconds` from dbrules ( #2003 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 18:06:20 +00:00
2bcf6d280b
Merge pull request #1999 from influxdata/cn/exploration
...
fix: Change RUB chunk table_summaries to table_summary
2021-07-14 17:38:22 +00:00
833debd5b5
Merge branch 'main' into cn/exploration
2021-07-14 17:30:55 +00:00
cbeeb97cff
feat: flush open window on persist ( #1985 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 16:58:20 +00:00
83f72c465d
Merge pull request #2002 from influxdata/er/refactor/batch
...
refactor: increase compaction batch size
2021-07-14 16:39:13 +00:00
2ef6671eca
Merge branch 'main' into er/refactor/batch
2021-07-14 17:31:42 +01:00
1d00fa2fd8
refactor: track memory metrics in catalog ( #1995 )
...
* refactor: track memory metrics in catalog
* chore: update comment
2021-07-14 16:23:00 +00:00
46ac15a77e
refactor: increase compaction batch size
2021-07-14 17:19:11 +01:00
8070065e2f
fix: Change RUB chunk table_summaries to table_summary
...
Because chunks now have only one table.
Connects to #1718 , #1613 , #1295
2021-07-14 11:18:02 -04:00
cedd6269c7
Merge pull request #1997 from influxdata/cn/alt-table-summary
...
feat: Make a TableSummaryAndTimes type for incremental replacement of TableSummary
2021-07-14 14:56:54 +00:00
9bc7ea2ace
Merge branch 'main' into cn/alt-table-summary
2021-07-14 14:49:59 +00:00
89b15fd3d2
Merge pull request #1998 from influxdata/crepererum/fix_entry_clone
...
fix: fix `Entry::clone`
2021-07-14 14:49:37 +00:00
kodiakhq[bot]
Raphael Taylor-Davies
Marko Mikulicic
Edd Robinson
Andrew Lamb
Carol (Nichols || Goulding)
Marco Neumann
Nga Tran