4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
4204127b05
refactor: use protobuf for in-parquet metadata
2021-06-30 16:51:37 +02:00
0a625b50e6
feat: store transaction timestamp in preserved catalog
2021-06-02 09:41:19 +02:00
d3711a5591
refactor: Use ParquetExec from DataFusion to read parquet files ( #1580 )
...
* refactor: use ParquetExec to read parquet files
* fix: test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-01 14:44:07 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
19a2733d30
feat: preserve transaction metadata in parquets
2021-05-25 09:56:12 +02:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
86771ea629
chore: update arrow/datafusion deps ( #1433 )
...
* chore: update datafusion deps
* chore: update arrow deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-05 22:37:31 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
1f42eb89cd
feat: implement parquet metadata handling
...
Closes #1379 and contributes to #1380 .
2021-05-05 13:29:16 +02:00
136c35cb88
feat: implement transaction handling for catalog
...
Closes #1253 .
2021-05-03 10:04:35 +02:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
4a6d6bd7ad
feat: initial work for querying data from parquet file in object store
2021-04-13 13:57:46 -04:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
c2355aca6d
feat: add basic memory tracking ( #1125 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
6e01fbc382
feat: ause TableSummary as metadata for parquet chunk's tables and read buffer's read_filter ot get data
2021-04-05 15:37:34 -04:00
4bdf8963e6
feat: continue buidling foundation for writing RB chunks to parquet files
2021-04-02 16:06:25 -04:00
19a453a483
feat: finally have some framework with clear todos for writing a chunk into parquet files
2021-03-31 16:21:53 -04:00
cd409b471f
feat: continue the implementation
2021-03-30 21:31:51 -04:00
0bcd52d5c9
feat: Add more changes
2021-03-30 18:31:09 -04:00
Marco Neumann
Andrew Lamb
Raphael Taylor-Davies
Nga Tran