d70d7a63a2
Merge branch 'main' into crepererum/remove_invalid_chunk_state
2021-05-28 10:20:05 +00:00
c6f42cf304
refactor: Remove unnecessary code ( #1573 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 10:12:47 +00:00
5cfede51f2
refactor: remove `ChunkState::Invalid`
...
This seems to only exist to fight the borrow checker and we can actually
live without it.
2021-05-28 11:16:06 +02:00
3ae44a0375
refactor: Chunks can have at most one object store path ( #1574 )
...
* refactor: Chunk can have at most one path
* fix: update tests
2021-05-27 19:52:09 +00:00
f3bec93ef1
feat: Cache TableSummary in Catalog rather than computing it on demand ( #1569 )
...
* feat: Cache `TableSummary` in catalog Chunks
* refactor: use consistent table summary
2021-05-27 16:03:05 +00:00
5d342d7779
feat: associate tracker with lifecycle action ( #1099 ) ( #1556 )
...
* feat: associate tracker with lifecycle action (#1099 )
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:47:35 +00:00
792bff07d1
feat: only store ChunkSnapshot in Closed state ( #1560 )
...
* feat: only store ChunkSnapshot in Closed state
* chore: review feedback
* feat: record MUB size as closed size
* chore: document column ordering assumption
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:36:47 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
efe077da8f
Merge branch 'main' into crepererum/issue1313
2021-05-26 14:46:18 +00:00
24ec1a472e
fix: do NOT delete parquet files that are reachable by time travel
2021-05-26 12:38:54 +02:00
c03b8a3963
refactor: remove tables from ChunkSnapshot ( #1295 ) ( #1558 )
2021-05-26 10:37:40 +00:00
1fb6af2364
refactor: split DB background loop into lifecycle and cleanup
...
This should prevent one from blocking / stalling the other.
2021-05-26 11:09:30 +02:00
5983336366
refactor: rename `parquet_file::{utils => test_utils}`
2021-05-26 11:09:29 +02:00
dd6bbeec42
feat: add background task to clean up OS
...
Closes #1313 .
2021-05-26 11:04:56 +02:00
cc78b5317d
feat: add method to get all parquet files from catalog state
2021-05-26 11:02:40 +02:00
166851d952
Merge branch 'main' into crepererum/in_file_metadata
2021-05-26 07:39:53 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
19a2733d30
feat: preserve transaction metadata in parquets
2021-05-25 09:56:12 +02:00
fe8e6301fe
refactor: move `read_schema_from_parquet_metadata` back to `parquet_file::metadata`
...
Let us pool all metadata handling in a single module, which makes it
easier to review.
2021-05-25 09:37:53 +02:00
a4215f0a56
fix: Fix 'acive' jemalloc stat misreporting
2021-05-25 02:55:27 +02:00
018e1e0246
chore: add a comment to trick github to check semantic
2021-05-24 17:25:14 -04:00
40a5d7d4ba
chore: Merge branch 'main' into tran/pushdown_parquet
2021-05-24 16:31:06 -04:00
e72ae81a8e
feat: support predicate pushdown for parquet files
2021-05-24 16:22:52 -04:00
db96286ed7
Merge branch 'main' into er/refactor/scalar_comp
2021-05-24 17:02:14 +00:00
c464ffadad
refactor: remove special case timestamp_range in parquet chunk ( #1543 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-24 16:19:44 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
abe64c6edc
test: uncomment tests to fix
2021-05-24 16:18:53 +01:00
5c5064bdac
fix: Set default line timestamp and default partition time to same value ( #1512 )
...
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
27e5b8fabf
refactor: Remove multiple table support from Parquet Chunk ( #1541 )
2021-05-24 08:40:31 -04:00
1f70d1f9c8
chore: remove a couple more comments
2021-05-21 17:06:53 -04:00
f113abacb5
feat: more unit & e2e tests plus cleanup and addressing review comments of Andrew and Edd
2021-05-21 16:48:43 -04:00
1093542578
fix: now all tests pass. Next step is cleaning up and addressing review comments
2021-05-21 13:29:20 -04:00
784ef88fcd
chore: merge main to branch and add more tests that expose a wrong result bug on unsigned int
2021-05-21 12:38:06 -04:00
93afc9c213
chore: more tests
2021-05-21 11:39:12 -04:00
5b619733d9
refactor: split lifecycle tracking from chunk state ( #1361 ) ( #1099 ) ( #1397 )
...
* refactor: split lifecycle tracking from chunk state (#1361 ) (#1099 )
* chore: namespace internal errors
* chore: fix logical conflict
* chore: don't remove moving chunk size metric
2021-05-21 09:27:44 +00:00
e44a3a87db
feat: fnow predicate is actuallu pushed down to RUB but there are bugs and not working yet
2021-05-20 16:56:15 -04:00
f028a356f4
Merge branch 'main' into crepererum/issue1382-c
2021-05-20 15:51:47 +00:00
aac00d4fa6
Merge branch 'main' into crepererum/remove_snapshotting
2021-05-20 14:14:58 +00:00
0e37d500eb
feat: remove snapshot feature
...
The parquet files produced by this code path are only semi-specified and
will miss many important metadata aspects that we will require for data
lineage.
2021-05-20 14:59:04 +02:00
462a5590c6
fix: fmt
2021-05-20 14:58:50 +02:00
c908cf0f98
fix: review suggestion
...
Co-authored-by: Edd Robinson <me@edd.io>
2021-05-20 14:40:02 +02:00
aa90329c1f
feat: Add remote_template for simpler remote configuration
2021-05-20 12:45:08 +02:00
7e55544eef
fix: correctly track chunk ID counter during catalog replay
2021-05-20 10:32:40 +02:00
93251f22c7
feat: read perserved catalog during DB startup
...
Closes #1382 .
2021-05-20 10:28:31 +02:00
91d7189e6d
feat: Log cached connections
2021-05-20 10:27:20 +02:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
00dacb5394
feat: add tests to verify the correctness as well as the explain of the plan
2021-05-19 17:31:16 -04:00
11561111d5
chore: merge main to branch
2021-05-19 15:11:15 -04:00
087d61f229
feat: Part 1 of predicate push down - Send predicates to MUB, RUB, and Parquet File. Note that MUB has not handled predicates yet
2021-05-19 13:59:51 -04:00
ce2f8351be
fix: Cache outbound gRPC connections
2021-05-19 18:28:45 +02:00
kodiakhq[bot]
Andrew Lamb
Marco Neumann
Raphael Taylor-Davies
Marko Mikulicic
Nga Tran
Edd Robinson
Carol (Nichols || Goulding)