a4704dd165
chore: update parquet_cache_limit to u64 and 0 for default
2021-07-20 15:41:06 -04:00
297e059085
feat: add parquet cache size setting to database rules
2021-07-20 15:41:06 -04:00
a852dce450
refactor: default to num_cpu
2021-07-19 14:00:10 +01:00
4ca19ad13f
refactor: add max_active_compaction config option
2021-07-19 14:00:10 +01:00
5fc98c7c56
feat: add failure reporting to TaskTracker ( #2031 )
...
* feat: add failure reporting to TaskTracker
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-19 09:17:20 +00:00
f57ba6afdb
fix: use fixed-size timestamps for parquet metadata ( #2032 )
...
This fixes flaky tests that rely on predictable files sizes.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-16 13:14:02 +00:00
3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks ( #2017 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
a79c0b4e75
feat: add mub row count threshold to lifecycle rules ( #1876 ) ( #2016 )
...
* feat: add mub row count threshold to lifecycle rules (#1876 )
* chore: update docstring
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 13:42:17 +00:00
0c86d1dccf
feat: Record parquet bytes size in catalog / parquet_file ( #2006 )
...
* feat: Store object store size in parquet_file
* fix: update TRANSACTION_VERSION to 8
* refactor: rename os_bytes --> file_size_bytes
2021-07-15 12:07:11 +00:00
956086fa6d
feat: add "drop chunk" job type
2021-07-15 12:07:56 +02:00
e570c66697
feat: add "dropping" chunk lifecycle action
2021-07-15 12:07:56 +02:00
d156998b46
fix: remove unused parameter `mutable_linger_seconds` from dbrules ( #2003 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 18:06:20 +00:00
4800b36949
chore: Update IOx to a pre-release version of arrow and datafusion to test out performance improvement
2021-07-13 15:44:57 -04:00
157a0cc98c
chore: update flatbuffers to 2.0
2021-07-13 15:44:45 +02:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
3d008f4d27
feat: add API+CLI to unload chunks
...
Closes #1919 .
2021-07-12 14:06:01 +02:00
6f2d20cb19
chore: reword comment on late_arrive_window_seconds for clarity
2021-07-11 08:25:31 -04:00
2854b54420
refactor: leave deprecated mutable_linger_seconds in proto
2021-07-10 12:48:50 -04:00
0c8c81a321
refactor: remove mutable_linger_seconds from lifecycle
...
The interplay between mutable_linger_seconds, late_arrive_window and persist_age_threshold_seconds can be tricky to reason about. I realized that the lifecycle rules can be simplified by removing mutable_linger_seconds and instead using late_arrive_window_seconds for the same purpose. Semantically, they basically mean the same thing. We want to give data around this amount of time to arrive before the system persists it, which gives it more of an opportunity to persist non-overlapping data.
When a partition goes cold for writes, after we've waiting past this window, we should compact and persist that partition. This removes one unnecessary knob from the lifecycle configuration and also removes the potential for conflicting configuration options.
2021-07-10 08:04:33 -04:00
9534220035
feat: Add any lifecycle_action to system.chunks and API ( #1947 )
2021-07-09 17:38:29 +00:00
e41fd2a821
refactor: remove unused mutable_minimum_age_seconds lifecycle setting
...
Closes #1878
2021-07-08 18:34:01 -04:00
e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
7059f16b9e
refactor: Turn mutable_linger_seconds into a non-optional ( #1917 )
2021-07-08 11:25:57 +02:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
090b0aba11
refactor: remove unused `mutable_size_threshold` lifecycle setting ( #1909 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 17:03:15 +00:00
3d644b63a1
feat: add `Replay` state to DB init
2021-07-06 14:24:39 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
b817ea88dd
Merge branch 'main' into crepererum/parquet_metadata_protobuf
2021-07-01 07:52:39 +00:00
cc038010cd
feat: add persist_age_threshold to LifecycleRules ( #1853 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 21:27:06 +00:00
4204127b05
refactor: use protobuf for in-parquet metadata
2021-06-30 16:51:37 +02:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
8e69202270
feat: catalog wiping gRPC
2021-06-21 09:31:23 +02:00
d17b5710a8
feat: add server functionality to wipe preserved catalogs
2021-06-21 09:31:23 +02:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
2eb2aca091
fix: fix discrepancy of ckpting config over CLI and protobuf
2021-06-14 10:27:47 +02:00
eae73591f3
feat: add `catalog_checkpoint_interval` lifecycle config
2021-06-14 10:04:50 +02:00
edbaaedfc3
docs: clarify behavior of unspecified transaction encoding
2021-06-10 15:42:21 +02:00
be9b3a4853
fix: protobuf lint fixes
2021-06-10 15:42:21 +02:00
33e364ed78
feat: add encoding info to transaction protobuf
...
This should help with #1381 .
2021-06-10 15:42:21 +02:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
4c12fd9b81
fix: Switch tabs for spaces
...
Not sure why my editor did that...
2021-06-07 13:06:50 -04:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2418e91001
feat: Add a DatabaseRule field for an optional Kafka write buffer connection string
2021-06-07 09:56:23 -04:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
f38a6378a6
fix: make `getServerStatus` gRPC-compliant
2021-06-04 12:00:49 +02:00
Paul Dix
Edd Robinson
Raphael Taylor-Davies
Marco Neumann
Andrew Lamb
Andrew Lamb
Carol (Nichols || Goulding)
Marko Mikulicic
Jacob Marble
kodiakhq[bot]