4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
febc1538ff
chore: Update Rust version ( #1445 )
...
* chore: Update Rust version
* refactor: Make struct constructor field orderings consistent
Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)
More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor
* refactor: Use flatten where appropriate
One instance is a false positive with a clippy bug.
More info:
- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
* refactor: Use Option map instead of match
More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
7d5c988fba
feat: Actually route SequencedEntry to the Write Buffer, if present
...
Connects to #1157 .
Rearrange some code and comments to be consistent with the design. Make
some more places not care whether they're getting an owned or borrowed
SequencedEntry.
2021-05-05 10:55:11 -04:00
979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
9aefcd216f
fix: Validate that ClockValue is never 0
2021-04-28 13:54:55 -04:00
2f4d7189ff
fix: Validate ServerId when creating structs from flatbuffers
...
When we get the flatbuffers, we won't have the server ID in addition to
the flatbuffers-- it's in the flatbuffers. But we want to validate the
`ServerId` once when the `SequencedEntry` is created so that its
`server_id` method can assume it has a valid `ServerId`.
2021-04-28 13:06:12 -04:00
6bdc153361
feat: sort RUB (read buffer) chunks ( #1308 )
...
* feat: sort chunks before upserting to read buffer (#1216 )
* chore: review feedback
* chore: fix merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 16:50:48 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4db3a4b3b5
test: enable writer to split large batches
2021-04-14 09:36:39 +00:00
bd13c09bad
refactor: make sharder optional when generating entry
2021-04-13 12:52:14 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
5893c17905
refactor: PR feedback and change ClockValue to actual type.
2021-04-12 18:43:14 +00:00
31115742ec
feat: Add writing of Entry structures to MB Chunk
...
This adds writing of Entry of a vec of TableWriteBatch to the Mutable Buffer Chunk. This is additional to the previous method of writing via ReplicatedWrite. The next step is to remove the old ReplicatedWrite bits.
Test helpers for parsing line protocol into Entry and writing line protocol directly to Chunks have also been added.
2021-04-12 18:43:14 +00:00
0a3386f24a
refactor: Make PartitionWrite, Table, and Column return keys/names
2021-04-12 18:43:14 +00:00
3f928ed374
refactor: Add ClockValue and WriterId to Entry
2021-04-12 18:43:14 +00:00
dad8d6bafd
chore: Update Entry with test helpers
2021-04-12 18:43:14 +00:00
3bb283fe20
chore: make SequencedEntryRaw the method of choice for SequencedEntry
2021-04-08 12:57:11 -04:00
81926279fc
chore: add benchmark for SequencedEntryRaw::new_from_entry_bytes
2021-04-08 12:57:11 -04:00
c002d83e9a
feat: add SequencedEntryRaw for raw entry bytes
2021-04-08 12:57:11 -04:00
0546968e13
chore: add SequencedEntry::new_from_entry benchmark
2021-04-08 12:57:11 -04:00
0c082e2347
feat: add sequenced entry builder
2021-04-08 12:57:11 -04:00
531ffe7ab0
chore: fix fmt and add comment to entry.fbs
2021-04-06 18:53:22 -04:00
a7659addbe
chore: cleanup entry.rs based on PR feedback and add better errors
2021-04-06 17:56:58 -04:00
2a134a3ac1
feat: implment line protocol to flatbuffers
...
This implements a builder for converting line protocol to the Entry flatbuffers. It also contains wrapper structs to make working with the flatbuffers a little easier.
The flatbuffer needed an addition to keep track of how many bits in the null mask should be used.
2021-04-06 17:56:56 -04:00
d4c090cb84
refactor: extract OnceNonZeroU32 ( #1134 )
2021-04-06 21:38:04 +00:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
c47a252c15
feat: Check in generated WAL flatbuffers code
2021-03-22 09:38:59 -04:00
52e415ae1b
fix: Verify flatbuffers when restoring from a file, and not after that
2021-03-22 09:38:59 -04:00
7c6f543d35
fix: wb::ColumnValue is now a set of associated consts, not a real Rust enum
2021-03-22 09:38:58 -04:00
24af745b23
chore: Upgrade to flatbuffers 0.8
2021-03-22 09:38:58 -04:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
Raphael Taylor-Davies
Andrew Lamb
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Paul Dix
Marco Neumann
Jake Goulding
Marko Mikulicic
Edd Robinson
Jacob Marble