63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
ce5b24e65d
refactor: use DateTime<Utc> in PersistenceWindows ( #2722 ) ( #2743 )
...
* refactor: use DateTime<Utc> in PersistenceWindows (#2722 )
* chore: fix benchmark
* chore: fmt
* chore: review feedback
2021-10-06 09:39:32 +00:00
e8eab2cc97
feat: allow compaction and persistence to retun no chunk ( #2664 ) ( #2700 )
...
* feat: allow compaction and persistence to retun no chunk (#2664 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-04 10:54:47 +00:00
b402423e9e
feat: remove move lifecycle action ( #2674 )
...
* feat: remove move_chunk lifecycle action
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-30 16:58:05 +00:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
cef5aeee52
refactor: introduce `ChunkId` type
2021-09-20 13:10:41 +02:00
bfaba78dc3
refactor: move `predicate` into its own crate
...
Two reasons:
1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
small follow-up PR).
2. `predicate` will have more and more features (like serialization)
which justifies a new home
2021-09-14 17:13:02 +02:00
31cbb646b9
feat: skip individual rows during replay based on timestamp
2021-09-07 11:44:52 +02:00
0946ffe916
refactor: reuse IOxExecutionContext ( #2373 )
...
* refactor: reuse IOxExecutionContext
* fix: orphaned comment
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-23 15:47:15 +00:00
98627944e7
refactor: make packers a dev-dependency of read buffer ( #2345 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-19 11:09:34 +00:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
4689b5e4e5
refactor: Remove first/last write times from MUB chunks
2021-07-28 11:02:57 -04:00
0b88deea43
refactor: don't pass sequence to MUB ( #2107 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-26 14:40:39 +00:00
446af5eb15
fix: consistent write timestamps ( #2104 )
...
* fix: consistent write timestamps
* chore: fix benchmarks
2021-07-23 18:04:15 +00:00
1ef2bc1887
refactor: `Db::{write_chunk_to_object_store => Db::persist_partition}`
...
The previous method allowed to persist any chunk -- even ones that
should not be persisted yet and w/o any order of peristence. That will
break our persistence windows. So instead offer a sane higher-level
interface that can trigger persistence of a partition within the
boundaries of the lifecycle rules. This needs some adjustments for our
test suite.
2021-07-16 12:07:58 +02:00
cbeeb97cff
feat: flush open window on persist ( #1985 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 16:58:20 +00:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
7602bde850
chore: Update datafusion deps ( #1799 )
...
* chore: Update datafusion deps + rework code
* refactor: remove workaround as it has been contributed upstream
* fix: Update query/src/exec/split.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 10:58:32 +00:00
5ca9760c94
test: make partioning in DB tests consistent w/ DB rules
2021-07-08 09:18:09 +02:00
9e1723620c
refactor: rename load_chunk_to_read_buffer to move_chunk_to_read_buffer ( #1857 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 16:53:18 +00:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
93881da016
feat: Make Write Buffer store_entry async
...
In preparation for the Kafka write buffer implementation needing to call
async functions.
2021-06-23 10:48:18 -04:00
ea04ce40dc
feat: transactional lifecycle API ( #1753 )
...
* feat: transactional lifecycle API
* chore: remove redundant upgrade
* feat: lifecycle error propagation
* chore: add usage doctest
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-21 13:09:53 +00:00
a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk ( #1711 )
...
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk
* fix: fmt
2021-06-14 13:35:20 +00:00
856751deec
feat: Lifecycle manager unloads, rather than drop, chunks when soft limit is hit ( #1701 )
...
* feat: unload chunks from memory rather than dropping them
* docs: Update server/src/db/lifecycle.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* docs: Update comment wording
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:14:39 +00:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
88ec1ef0cf
test: enable checkpointing in catalog benchmark
...
This now creates a checkpoint every 10 transactions. To make it a bit
more fair increase the chunk count to 109, so we have some transactions
after the last checkpoint. With that we improve performance from 10.5s
to 1.2s (or even 0.3s if we would keep the chunk count at 100).
2021-06-14 10:08:32 +02:00
2e6f51cbfb
fix: fix `server_benchmarks::benches::catalog_persistence`
2021-06-14 10:08:32 +02:00
876f642860
test: add benchmark for catalog loading
2021-06-10 15:42:21 +02:00
a614fef5bc
chore: remove more unused dependencies ( #1658 )
...
* chore: remove more unused deps
* refactor: move benchmarks into server_benchmarks crate
2021-06-09 10:17:20 +00:00
42f26b609b
refactor: Move `query_tests` and `server_benchmarks` into their own crate --> smaller `server` ( #1628 )
...
* refactor: Separate query_tests into its own crate
* fix: references
* refactor: break out server benchmarks
* fix: Update query_tests/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-06-04 17:31:19 +00:00
Marco Neumann
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Andrew Lamb