54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
cc038010cd
feat: add persist_age_threshold to LifecycleRules ( #1853 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 21:27:06 +00:00
f6731c60d7
fix: change timeout to have all tests passed on slow laptop
2021-06-30 16:04:02 -04:00
89757d7232
fix: do not print test output to logs except on failure ( #1840 )
...
* fix: do not print test output to logs except on failure
* docs: update CONTRIBUTING.md
2021-06-30 13:20:11 +00:00
eac9261507
chore: print end-to-end output ( #1838 )
...
* chore: print end-to-end output
* chore: clippy
* chore: update CONTRIBUTING.md
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 15:09:43 +00:00
3ae8ac2467
chore: improve wait_for_chunk failure output ( #1835 )
2021-06-29 11:54:32 +00:00
5287f6a577
feat: print operations on wait_for_chunk failure ( #1809 ) ( #1833 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 11:09:11 +00:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
c66f9e5aeb
feat: Write entries to Kafka when configured as the write buffer
2021-06-23 10:48:18 -04:00
5cd911c74a
fix: correct row count for object store chunks ( #1789 )
2021-06-23 12:06:49 +00:00
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
5362c7c924
feat: enable query deduplication ( #1762 )
2021-06-21 18:49:04 +00:00
31ad5c85f9
fix: Consistently refer to docker-compose
2021-06-21 09:41:37 -04:00
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
a153f841d8
feat: add `--force` flag to CLI wipe command
2021-06-21 09:31:23 +02:00
c0766f1c26
feat: catalog wiping CLI
2021-06-21 09:31:23 +02:00
8e69202270
feat: catalog wiping gRPC
2021-06-21 09:31:23 +02:00
51f27de2ee
docs: fix typo
...
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-06-14 17:34:57 +02:00
14ba02ec87
feat: expose server and DB init errors over gRPC
...
Closes #1624 .
2021-06-14 17:34:57 +02:00
a449d5ef74
test: make some `server_fixture` functionality public
...
This is useful when you want to test a server boot-up with custom
configs.
2021-06-14 17:34:57 +02:00
856751deec
feat: Lifecycle manager unloads, rather than drop, chunks when soft limit is hit ( #1701 )
...
* feat: unload chunks from memory rather than dropping them
* docs: Update server/src/db/lifecycle.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* docs: Update comment wording
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:14:39 +00:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
2eb2aca091
fix: fix discrepancy of ckpting config over CLI and protobuf
2021-06-14 10:27:47 +02:00
4224b693d9
refactor: combine preservation.rs and persistence.rs ( #1692 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-11 11:33:14 +00:00
0cbe74dbde
fix: persistence to parquet by swapping order of arguments ( #1687 )
...
* fix: fix order of arguments
* test: for persistence
2021-06-11 10:55:40 +00:00
11b25b3aaf
refactor: swap order of partition and table in in-memory catalog ( #1678 )
...
* refactor: swap order of partition and table in in-memory catalog
* chore: review feedback
* chore: validate panic message
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 16:40:30 +00:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2418e91001
feat: Add a DatabaseRule field for an optional Kafka write buffer connection string
2021-06-07 09:56:23 -04:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
5749a2c119
chore: cleanup legacy TSM -> parquet code ( #1639 )
...
* chore: cleanup legacy parquet code
* chore: remove tests of removed functionality
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-07 12:59:33 +00:00
afe88eeb7c
chore: fix flaky test ( #1643 )
2021-06-07 12:52:11 +00:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
c4a2a7243f
fix: formatting
2021-06-04 12:58:25 +02:00
34939e37c7
fix: style
...
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
2021-06-04 12:46:28 +02:00
e06d65bb2a
refactor: migrate "DBs initialized" RPC to "server status"
2021-06-04 11:33:41 +02:00
b30d7e2821
feat: move DB loading into background worker
...
Before this change we loaded databases eagerly when a serverID was
passed on startup BEFORE starting up the gRPC server. Since loading
(esp. at its current state without checkpoints and with too many small
parquet files) can take very long, K8s thinks IOx is unhealthy. With
this change we are now loading databases in the server background worker
once a serverID is available. Until then we block all DB-related
interactions including adding new databases (since without inspecting
the object store there is now way we can check if the DB already
exists).
Furthermore we now load database no matter if the serverID was passed on
startup (via CLI or environment variable) or was set later via gRPC
call. Before this change the latter case was somewhat forgotten.
2021-06-04 11:33:41 +02:00
bbd73e59be
feat: jitter background clean-up job + wait on first job
2021-06-03 11:23:29 +02:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
9765c53bb4
feat: Implement service readiness API
2021-05-24 17:57:45 +02:00
5c5064bdac
fix: Set default line timestamp and default partition time to same value ( #1512 )
...
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
e28c031f76
refactor: make `ServerFixture::restart_server` consume the fixture
...
This also avoids an additional channel lock.
2021-05-20 14:30:30 +02:00
93251f22c7
feat: read perserved catalog during DB startup
...
Closes #1382 .
2021-05-20 10:28:31 +02:00
25ec0ab4ca
test: make end2end server fixture restartable
2021-05-20 10:22:20 +02:00
beee3115f4
feat: expose regex =\~ and to gRPC API
2021-05-07 13:44:51 +01:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
Marco Neumann
Raphael Taylor-Davies
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)
kodiakhq[bot]
Marko Mikulicic
Carol (Nichols || Goulding)
Edd Robinson