54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
cc038010cd
feat: add persist_age_threshold to LifecycleRules ( #1853 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 21:27:06 +00:00
3ae8ac2467
chore: improve wait_for_chunk failure output ( #1835 )
2021-06-29 11:54:32 +00:00
5287f6a577
feat: print operations on wait_for_chunk failure ( #1809 ) ( #1833 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 11:09:11 +00:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
c66f9e5aeb
feat: Write entries to Kafka when configured as the write buffer
2021-06-23 10:48:18 -04:00
5cd911c74a
fix: correct row count for object store chunks ( #1789 )
2021-06-23 12:06:49 +00:00
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
5362c7c924
feat: enable query deduplication ( #1762 )
2021-06-21 18:49:04 +00:00
31ad5c85f9
fix: Consistently refer to docker-compose
2021-06-21 09:41:37 -04:00
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
a153f841d8
feat: add `--force` flag to CLI wipe command
2021-06-21 09:31:23 +02:00
c0766f1c26
feat: catalog wiping CLI
2021-06-21 09:31:23 +02:00
8e69202270
feat: catalog wiping gRPC
2021-06-21 09:31:23 +02:00
14ba02ec87
feat: expose server and DB init errors over gRPC
...
Closes #1624 .
2021-06-14 17:34:57 +02:00
856751deec
feat: Lifecycle manager unloads, rather than drop, chunks when soft limit is hit ( #1701 )
...
* feat: unload chunks from memory rather than dropping them
* docs: Update server/src/db/lifecycle.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* docs: Update comment wording
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:14:39 +00:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
2eb2aca091
fix: fix discrepancy of ckpting config over CLI and protobuf
2021-06-14 10:27:47 +02:00
4224b693d9
refactor: combine preservation.rs and persistence.rs ( #1692 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-11 11:33:14 +00:00
0cbe74dbde
fix: persistence to parquet by swapping order of arguments ( #1687 )
...
* fix: fix order of arguments
* test: for persistence
2021-06-11 10:55:40 +00:00
11b25b3aaf
refactor: swap order of partition and table in in-memory catalog ( #1678 )
...
* refactor: swap order of partition and table in in-memory catalog
* chore: review feedback
* chore: validate panic message
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 16:40:30 +00:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2418e91001
feat: Add a DatabaseRule field for an optional Kafka write buffer connection string
2021-06-07 09:56:23 -04:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
afe88eeb7c
chore: fix flaky test ( #1643 )
2021-06-07 12:52:11 +00:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
e06d65bb2a
refactor: migrate "DBs initialized" RPC to "server status"
2021-06-04 11:33:41 +02:00
b30d7e2821
feat: move DB loading into background worker
...
Before this change we loaded databases eagerly when a serverID was
passed on startup BEFORE starting up the gRPC server. Since loading
(esp. at its current state without checkpoints and with too many small
parquet files) can take very long, K8s thinks IOx is unhealthy. With
this change we are now loading databases in the server background worker
once a serverID is available. Until then we block all DB-related
interactions including adding new databases (since without inspecting
the object store there is now way we can check if the DB already
exists).
Furthermore we now load database no matter if the serverID was passed on
startup (via CLI or environment variable) or was set later via gRPC
call. Before this change the latter case was somewhat forgotten.
2021-06-04 11:33:41 +02:00
bbd73e59be
feat: jitter background clean-up job + wait on first job
2021-06-03 11:23:29 +02:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
9765c53bb4
feat: Implement service readiness API
2021-05-24 17:57:45 +02:00
5c5064bdac
fix: Set default line timestamp and default partition time to same value ( #1512 )
...
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
e28c031f76
refactor: make `ServerFixture::restart_server` consume the fixture
...
This also avoids an additional channel lock.
2021-05-20 14:30:30 +02:00
93251f22c7
feat: read perserved catalog during DB startup
...
Closes #1382 .
2021-05-20 10:28:31 +02:00
beee3115f4
feat: expose regex =\~ and to gRPC API
2021-05-07 13:44:51 +01:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
57616c3d16
feat: implement --format and `SET FORMAT <pretty,csv,json>` in SQL repl ( #1404 )
...
* feat: add `set format` and --format flag to sql REPL
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 17:02:10 +00:00
f36f1efeeb
feat: add row counts to SQL repl output ( #1405 )
...
* feat: add row counts to repl
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 10:57:16 +00:00
3b7c5ac350
fix(storage rpc): do not send back tags with empty values ( #1403 )
2021-05-04 10:35:24 +00:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
da440d8d27
feat: add a id() method to `management::Operations` ( #1385 )
...
* feat: add Operation::id()
* feat: add a id() method to Operations management client
* fix: broken doc link
2021-04-30 17:09:28 +00:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
eb8d91cf1c
refactor: remove additional uses of RecordBatch::try_new ( #1378 )
...
* refactor: remove additional uses of RecordBatch::try_new
* fix: fix accidental change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 10:24:47 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
c64f25e777
feat: Add `OBSERVER` mode to SQL repl along with SQL primer / cookbook ( #1349 )
...
* feat: add OBSERVER command to repl
* fix: fixup test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 11:47:25 +00:00
Marco Neumann
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Marko Mikulicic
Carol (Nichols || Goulding)
Edd Robinson