bf54ab51f2
refactor: split lifecycle into separate crate ( #1730 )
2021-06-15 15:57:47 +00:00
f96e05d26a
refactor: traitify lifecycle policy ( #1729 )
...
* refactor: traitify lifecycle policy
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 14:00:06 +00:00
b756e09904
refactor: Rename parquet_file::Chunk --> ParquetChunk ( #1722 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 11:21:49 +00:00
09f2ae1616
Merge branch 'main' into crepererum/issue1595
2021-06-15 11:12:01 +00:00
adc3a059ca
refactor: improve server background task logging
...
- rename `name` to `db_name`
- add `table_name` to error-detection logs
- use `Display` instead of `Debug` fmt for errors, which results in
nicer outputs and follows the rest of the stack
This is for #1725 .
2021-06-15 10:28:12 +02:00
dcfaa81969
feat: info-log server ID during init
...
Add a info log when the server ID is set. Because this is done where the
server ID is also stored, this automatically affects all ways to set it
(via CLI, via environment variable, via gRPC call).
Closes #1595 .
2021-06-15 10:09:53 +02:00
19f684ee14
Merge branch 'main' into crepererum/issue1506
2021-06-15 07:36:49 +00:00
55fc5e564b
refactor: remove serverID and DB name args from catalog state
...
They are no longer required.
2021-06-15 09:35:41 +02:00
057c99d431
fix: tighten memory ordering
2021-06-14 17:34:57 +02:00
2ea24b6467
feat: allow to fail initializing a single DB
...
- keep errors encountered during DB init
- treat failed DB inits as existing DBs
- effectively poison failed DBs (there is no way to recover except by
restarting the server, yet)
2021-06-14 17:34:57 +02:00
0b5552f131
refactor: ensure that DBs are reserved before doing expensive IO
2021-06-14 17:34:57 +02:00
233235365a
refactor: de-couple DB rules commit from name reservation
...
This allows us to put DBs in a controlled error state when we try to
load rules from a file but the rules are somewhat broken.
2021-06-14 17:34:57 +02:00
318af9b801
feat: keep error that occurred during server init
2021-06-14 17:34:57 +02:00
bf0ba6ba6c
test: rename some server init tests to better reflect their nature
2021-06-14 17:34:57 +02:00
250ccdcdcd
refactor: use `IOxMetadata` instead of path parsing for parquet chunks
2021-06-14 16:24:50 +02:00
d51e7a127c
feat: include table name, partition key, and chunk ID in `IoxMetadata`
2021-06-14 16:24:50 +02:00
a14e9ab27c
refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk ( #1711 )
...
* refactor: rename mutable_buffer::Chunk --> mutable_buffer::MBChunk
* fix: fmt
2021-06-14 13:35:20 +00:00
856751deec
feat: Lifecycle manager unloads, rather than drop, chunks when soft limit is hit ( #1701 )
...
* feat: unload chunks from memory rather than dropping them
* docs: Update server/src/db/lifecycle.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* docs: Update comment wording
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:14:39 +00:00
fc1b5ea165
Merge branch 'main' into crepererum/parquet_metadata_wrapper
2021-06-14 11:20:39 +00:00
9d1ca95a52
refactor: Rename catalog::Chunk --> catalog::CatalogChunk ( #1702 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 11:20:14 +00:00
518f7c6f15
refactor: wrap upstream parquet MD into struct + clean up interface
...
This prevents users from `parquet_file::metadata` to also depend on
`parquet` directly. Furthermore they don't need to important dozend of
functions and can instead just use `IoxParquetMetaData` directly.
2021-06-14 13:17:01 +02:00
665919786e
test: fix test
2021-06-14 10:52:23 +02:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
898c638630
feat: wire up catalog checkpointing
...
Closes #1381 .
2021-06-14 10:08:32 +02:00
df866f72e0
refactor: store parquet metadata in chunk
...
This will be useful for #1381 .
At the moment we parse schema and stats eagerly and store them alongside
the parquet metadata in memory. Technically this is not required since
this is basically duplicate data. In the future we might trade-off some
of this memory against CPU consumption by parsing schema and stats on
demand.
2021-06-14 10:08:31 +02:00
ff19beb0ad
refactor: export rb chunk as RBChunk
2021-06-11 18:33:10 +01:00
71e2a8fbaa
Merge branch 'main' into crepererum/inline_parquet_table_struct
2021-06-11 11:22:48 +00:00
0cbe74dbde
fix: persistence to parquet by swapping order of arguments ( #1687 )
...
* fix: fix order of arguments
* test: for persistence
2021-06-11 10:55:40 +00:00
f8a518bbed
refactor: inline `Table` into `parquet_file::chunk::Chunk`
...
Note that the resulting size estimations are different because we were
double-counting `Table`. `mem::size_of::<Self>()` is recursive for
non-boxed types since the child will be part of the parent structure.
Issue: #1295 .
2021-06-11 11:54:31 +02:00
11b25b3aaf
refactor: swap order of partition and table in in-memory catalog ( #1678 )
...
* refactor: swap order of partition and table in in-memory catalog
* chore: review feedback
* chore: validate panic message
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 16:40:30 +00:00
13bb290a7c
chore: enforce `clippy::future_not_send` for `server` + top-level crate ( #1679 )
...
* chore: enforce `clippy::future_not_send` for `server`
* chore: enforce `clippy::future_not_send` for top-level crate
2021-06-10 15:01:12 +00:00
294c304491
feat: impl catalog checkpointing infrastructure
...
This implements a way to add checkpoints to the preserved catalog and
speed up replay.
Note: This leaves the "hook it up into the actual DB" for a future PR.
Issue: #1381 .
2021-06-10 15:42:21 +02:00
3ba27bdbd9
Merge branch 'main' into crepererum/clippy_future_not_send_part1
2021-06-10 07:19:31 +00:00
5f863a59fd
Merge branch 'main' into crepererum/extract_server_init
2021-06-10 07:14:57 +00:00
44d8fb9472
Merge branch 'main' into crepererum/clippy_future_not_send_part1
2021-06-10 07:10:11 +00:00
eed73a30c5
Merge branch 'main' into ntran/dedup_within_chunk
2021-06-09 18:19:17 +00:00
c1c58018fc
refactor: address review comments
2021-06-09 14:17:47 -04:00
4fe2d7af9c
chore: enforce `clippy::future_not_send` for `parquet_file`
2021-06-09 18:18:27 +02:00
d9c38dfe88
refactor: extract server init code
...
This prepares for #1624 , so the end results looks a bit cleaner.
2021-06-09 16:53:11 +02:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
07c4277ca7
refactor: schema merge to give more control over field merging ( #1653 )
...
* refactor: schema merge to give more control over field merging
* chore: review feedback
2021-06-09 06:30:45 +00:00
3e10351538
test: add tests for the sort plan
2021-06-08 21:40:46 -04:00
68e3a2121f
feat: add SortExec
2021-06-08 15:04:31 -04:00
fd8a87484e
feat: Hook up chunk grouping into provider
2021-06-08 14:42:37 -04:00
edbf1b7d5e
Merge branch 'main' into ntran/dedup_within_chunk
2021-06-08 13:18:40 -04:00
40cb4f741f
feat: initial implementaton
2021-06-08 13:17:36 -04:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2bb2c4ba47
docs: Add some doc comments about the WriteBuffer trait
2021-06-07 11:22:33 -04:00
a8a4a5f29d
fix: Return the Sequence type from the write buffer, not vague WriteMetadata
2021-06-07 11:15:46 -04:00
a63c12acfb
fix: Remove references to Kafka from db tests
2021-06-07 10:58:34 -04:00
Raphael Taylor-Davies
Andrew Lamb
kodiakhq[bot]
Marco Neumann
Edd Robinson
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)