979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
262bf446ab
feat: change background worker to use backoff instead of poll ( #1339 ) ( #1347 )
...
* feat: change background worker to use backoff instead (#1339 )
* chore: fix comment
* chore: fix tests
* chore: review comments
* chore: review feedback
2021-04-29 11:10:51 +00:00
1f3bf6b5a6
Merge branch 'main' into sharding_refa
2021-04-28 21:58:21 +00:00
b1924d10b4
fix: Only show current database's system.operations table ( #1310 )
...
* test: add test for operations.system_tables
* fix: only show operations for current database
* fix: update test
* fix: improve test
* refactor: filter in Schema provider rather than in job tracker
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
91ee005ada
feat: Add multiple matchers to shard config
2021-04-28 17:02:58 +02:00
b6054f9df9
feat: Make background task period configurable
2021-04-28 15:47:31 +02:00
9b57214344
feat: move closing chunks as soon as possible ( #1329 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
84c863aeb0
refactor: remove unused PersistSegment job ( #1245 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:41:59 +00:00
934bc75855
fix: Remove PartialEq<u32> impls for ServerId, not really necessary
2021-04-26 18:44:32 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
bdb772b243
refactor: Alphabetize module declarations
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
9c74074095
feat: List background operations as system.operations table
2021-04-26 16:59:36 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
ed6025b400
refactor: Rename Wal metadata endpoint types to WriteBuffer
2021-04-21 17:43:03 +00:00
5f024401a4
refactor: Rename data_types::wal module to data_types::write_buffer
2021-04-21 17:43:03 +00:00
cdb9574beb
fix: Rename wal/meta as wb/meta
2021-04-21 17:43:03 +00:00
0677c544bb
refactor: Rename WalBufferRollover to WriteBufferRollover
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
8f0ed5e51c
refactor: satisfy new clippy lints
2021-04-20 17:30:50 +00:00
757933afc4
fix: use Self when possible
2021-04-19 08:48:11 -04:00
e2954b918e
fix: set defaults for `persist` and `drop_non_persisted` to `true`
...
This should prevent writer deaths for default configs.
Closes #1225 .
2021-04-19 13:03:04 +02:00
8bfcc1782d
feat: Implement sharding logic
2021-04-16 15:14:57 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
e516f65cbe
feat: add WriteChunk job
...
For #1083 .
2021-04-14 12:37:43 +02:00
440cef59df
docs: fix docs of `LifecycleRules::buffer_size_soft`
2021-04-14 12:37:43 +02:00
ef7de0aa05
feat: add persist flag to LifecycleRules
...
For #1083 .
2021-04-14 12:37:43 +02:00
bd13c09bad
refactor: make sharder optional when generating entry
2021-04-13 12:52:14 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
7f77a01e61
chore: merged main to branch and resolved conflicts
2021-04-12 12:09:04 -04:00
453aeaf1a0
feat: Add tests for writing RB chunks to Object Store
2021-04-09 17:39:23 -04:00
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
f501a74aea
refactor: Address review comments
2021-04-07 21:28:03 -04:00
be6e1e48e4
feat: add writer_id and object_store in Db
2021-04-07 18:36:07 -04:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
a7659addbe
chore: cleanup entry.rs based on PR feedback and add better errors
2021-04-06 17:56:58 -04:00
2a134a3ac1
feat: implment line protocol to flatbuffers
...
This implements a builder for converting line protocol to the Entry flatbuffers. It also contains wrapper structs to make working with the flatbuffers a little easier.
The flatbuffer needed an addition to keep track of how many bits in the null mask should be used.
2021-04-06 17:56:56 -04:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
18ca5c5292
refactor: Implement `as_str()` directly rather than using a crate
2021-04-01 21:22:04 +00:00
Paul Dix
Raphael Taylor-Davies
Andrew Lamb
kodiakhq[bot]
Marko Mikulicic
Marco Neumann
Carol (Nichols || Goulding)
Jake Goulding
Andrew Lamb
Edd Robinson
Nga Tran
Jacob Marble