20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
35bc09ef7c
refactor: use `FromIter` rather than array builders to create system table arrays ( #1246 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:47:11 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
e33af0c084
fix: remove wait_for_complete
2021-04-26 16:59:36 +00:00
fcf5c304a6
fix: fixup merge conflicts
2021-04-26 16:59:36 +00:00
348f38c2c9
feat: add status column
2021-04-26 16:59:36 +00:00
9c74074095
feat: List background operations as system.operations table
2021-04-26 16:59:36 +00:00
faec98eab9
refactor: remove time column from row group
2021-04-26 09:51:06 +00:00
dd2bee3343
fix: don't over count new chunks
2021-04-23 20:25:37 +01:00
552409ea70
feat: add metrics for chunk lifecycle
2021-04-23 17:13:43 +00:00
0e3b7bff49
refactor: plumb metrics registry to catalog
2021-04-23 17:13:43 +00:00
eacfe9af6a
refactor: address PR feedback
2021-04-23 15:58:48 +00:00
d3218802ab
refactor: more API metric coverage
2021-04-23 15:58:48 +00:00
97b2369140
refactor: swap existing metrics for THE NEW WAY
2021-04-23 15:58:48 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
74c25f541d
feat: fast MUB dictionary arrow conversion ( #1273 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
5f024401a4
refactor: Rename data_types::wal module to data_types::write_buffer
2021-04-21 17:43:03 +00:00
46f9360c58
fix: Rename WAL_DIR to WRITE_BUFFER_DIR and redefine it as wb, not wal
2021-04-21 17:43:03 +00:00
8baeaab8ca
refactor: Rename Db.wal_buffer to write_buffer
2021-04-21 17:43:03 +00:00
4bfcb68536
refactor: Remove a WalError that doesn't appear to be used
2021-04-21 17:43:03 +00:00
0677c544bb
refactor: Rename WalBufferRollover to WriteBufferRollover
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
1cf3ab3978
feat: enable disabling MUB snapshot caching ( #1203 ) ( #1264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-21 08:21:31 +00:00
508aa8be8c
refactor: satisfy new clippy lints
2021-04-20 17:30:50 +00:00
c5d99ba9af
feat: wire up hard buffer limit
...
Fixes #1255 .
2021-04-20 13:25:31 +02:00
d05585f736
refactor: log chunk migration to rb at info
2021-04-19 16:08:49 +00:00
82c1d94ce1
refactor: Use Option.map where possible
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
f092294da3
fix: Use MAX (window end) for timestamps in read group ( #1228 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4001ec73c9
Merge branch 'main' into ntran/query_parquet_2
2021-04-14 20:33:03 +00:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
025f760479
refactor: change sync::RwLock to parking_lot
2021-04-14 19:18:03 +00:00
59ca090aef
feat: Use single db-wide executor for running queries ( #1198 )
...
* refactor: plumb executor into all Db instances
* refactor: Route all query executions through worker pool
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 16:46:02 +00:00
5bb34e9a97
refactor: use read_buffer column range for time range
2021-04-14 16:10:24 +00:00
04f594bf94
perf: use column_values in read buffer
2021-04-14 15:13:35 +00:00
f5f768d750
feat: Add a dedicated threadpool for running queries ( #1191 )
...
* feat: use a dedicated tokio threadpool for running queries
* feat: plumb number of executor threads through to command line
thread through command line
* fix: Logical merge conflict
* fix: another logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
f430013141
refactor: rename `Db::load_chunk_to_object_store` to `Db::write_chunk_to_object_store
...
For #1083 .
2021-04-14 12:37:43 +02:00
1c6690ae3b
fix: fix constant use in `server::db::tests::read_from_read_buffer`
2021-04-14 12:37:43 +02:00
26ce520cf1
feat: hook RB->OS into lifecycle
...
Fixes #1083 .
2021-04-14 12:37:43 +02:00
7b0b975433
feat: add Db::load_chunk_to_object_store_in_background
...
For #1083 .
2021-04-14 12:37:43 +02:00
84b2ce4571
docs: extend docs of `Db::load_chunk_to_read_buffer_in_background`
2021-04-14 12:37:43 +02:00
Raphael Taylor-Davies
Andrew Lamb
Carol (Nichols || Goulding)
Jake Goulding
Andrew Lamb
Edd Robinson
Marco Neumann
Marko Mikulicic
Nga Tran
kodiakhq[bot]