4a961694ec
refactor: make caller sync mem<>OS view during catalog transactions
...
This is for #1740 . Greatly simplifies the integration of the persisted
catalog into the DB.
2021-06-23 10:25:05 +02:00
d1db0dfaeb
refactor: remove type parameter from preserved catalog
...
For #1740 .
2021-06-22 10:53:10 +02:00
ff60627500
refactor: make preserved catalog NOT own the in-mem catalog
...
Works towards #1740 .
2021-06-21 18:39:43 +02:00
881729bd23
refactor: make caller responsible to create checkpoint data
...
This decouples the in-mem and preserved catalog a bit and works
towards #1740 .
2021-06-21 18:33:23 +02:00
aba973a6e1
refactor: make catalog `wipe` a freestanding function
...
It does not interact with the `CatalogState` so users can call this
function without that type.
2021-06-21 09:31:23 +02:00
258a6b1956
chore: remove more dead code ( #1760 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-18 21:28:22 +00:00
de67bd3efe
refactor: Remove PartitionChunk::table_schema ( #1756 )
...
* refactor: Remove PartitionChunk::table_schema
* docs: update comments
2021-06-18 16:13:16 +00:00
f6dbc8d6f2
refactor: add ChunkAddr to describe location of chunk in catalog ( #1745 )
...
* refactor: add ChunkPath to describe location of chunk in catalog
* refactor: rename ChunkPath to ChunkAddr
* chore: further renames
* chore: even more renames
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-17 12:04:37 +00:00
e056d97cf6
test: always test transaction aborts
2021-06-16 11:01:14 +02:00
caaf95c6ec
refactor: remove lock from `TestCatalogState`
2021-06-16 10:51:15 +02:00
c8c412f6fe
refactor: rework catalog state interface
...
This now allows not only for copy-based transaction handling but also
for eager exec and rollbacks. This will be useful to properly implement
transaction aborts for the "real" catalog.
2021-06-16 10:51:15 +02:00
e064a6bbba
test: add test suite for `CatalogState` impls
...
This makes it easier to check if `CatalogState` correctly implement all
features, including transaction aborting.
2021-06-16 10:50:47 +02:00
b756e09904
refactor: Rename parquet_file::Chunk --> ParquetChunk ( #1722 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 11:21:49 +00:00
64c815dd50
fix: bump catalog version ( #1726 )
...
This should have been done in #1714 . Also add a note so that future devs
might hopefully not forget. In any case though the code also works w/o
this bump, it's just that the error message is a bit less nice ("cannot
parse IOxMetadata" instead of "unsupported catalog version").
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 10:26:30 +00:00
55fc5e564b
refactor: remove serverID and DB name args from catalog state
...
They are no longer required.
2021-06-15 09:35:41 +02:00
776b6c011c
feat: remove path parsing functionality
...
Paths to parquet files are an implementation detail and should not be
parsed.
Closes #1506 .
2021-06-14 16:24:50 +02:00
250ccdcdcd
refactor: use `IOxMetadata` instead of path parsing for parquet chunks
2021-06-14 16:24:50 +02:00
d51e7a127c
feat: include table name, partition key, and chunk ID in `IoxMetadata`
2021-06-14 16:24:50 +02:00
b57f397057
Merge branch 'main' into crepererum/checkpoint_during_restore
2021-06-14 13:54:03 +00:00
0a7dcc3779
test: adjust read-write parquet test to newest test data
2021-06-14 14:24:24 +02:00
d6f6ddfdaa
fix: fix NULL handling in parquet stats
2021-06-14 14:24:09 +02:00
eae56630fb
test: add test for all-NULL float column metadata
2021-06-14 13:48:34 +02:00
3f9bcf7cd9
fix: fix NaN handling in parquet stats
2021-06-14 13:44:52 +02:00
ea96210e98
test: enable unblocked test
2021-06-14 13:44:52 +02:00
518f7c6f15
refactor: wrap upstream parquet MD into struct + clean up interface
...
This prevents users from `parquet_file::metadata` to also depend on
`parquet` directly. Furthermore they don't need to important dozend of
functions and can instead just use `IoxParquetMetaData` directly.
2021-06-14 13:17:01 +02:00
030d0d2b9a
feat: create checkpoint during catalog rebuild
2021-06-14 10:55:56 +02:00
df866f72e0
refactor: store parquet metadata in chunk
...
This will be useful for #1381 .
At the moment we parse schema and stats eagerly and store them alongside
the parquet metadata in memory. Technically this is not required since
this is basically duplicate data. In the future we might trade-off some
of this memory against CPU consumption by parsing schema and stats on
demand.
2021-06-14 10:08:31 +02:00
e6699ff15a
test: ensure that `find_last_transaction_timestamp` considers checkpoints
2021-06-14 10:04:50 +02:00
f8a518bbed
refactor: inline `Table` into `parquet_file::chunk::Chunk`
...
Note that the resulting size estimations are different because we were
double-counting `Table`. `mem::size_of::<Self>()` is recursive for
non-boxed types since the child will be part of the parent structure.
Issue: #1295 .
2021-06-11 11:54:31 +02:00
28d1dc4da1
chore: bump preserved catalog version
2021-06-10 16:01:13 +02:00
80ee36cd1a
refactor: slightly streamline path parsing code in pres. catalog
2021-06-10 15:59:28 +02:00
7e7332c9ce
refactor: make comparison a bit less confusing
2021-06-10 15:42:21 +02:00
fd581e2ec9
docs: fix confusion wording in `CatalogState::files`
2021-06-10 15:42:21 +02:00
be9b3a4853
fix: protobuf lint fixes
2021-06-10 15:42:21 +02:00
294c304491
feat: impl catalog checkpointing infrastructure
...
This implements a way to add checkpoints to the preserved catalog and
speed up replay.
Note: This leaves the "hook it up into the actual DB" for a future PR.
Issue: #1381 .
2021-06-10 15:42:21 +02:00
188cacec54
refactor: use `Arc` to pass `ParquetFileMetaData`
...
This will be handy when the catalog state must be able to return
metadata objects so that we can create checkpoints, esp. when we use
multi-chunk parquet files in some midterm future.
2021-06-10 15:42:21 +02:00
c7412740e4
refactor: prepare to read and write multiple file types for catalog
...
Prepares #1381 .
2021-06-10 15:42:21 +02:00
33e364ed78
feat: add encoding info to transaction protobuf
...
This should help with #1381 .
2021-06-10 15:42:21 +02:00
4fe2d7af9c
chore: enforce `clippy::future_not_send` for `parquet_file`
2021-06-09 18:18:27 +02:00
ab0aed0f2e
refactor: Remove a layer of channels in parquet read stream ( #1648 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-07 16:47:04 +00:00
1e7ef193a6
refactor: use field metadata to store influx types ( #1642 )
...
* refactor: use field metadata to store influx types
make SchemaBuilder non-consuming
* chore: remove unused variants
* chore: fix lints
2021-06-07 13:26:39 +00:00
c830542464
feat: add info log when cleanup limit is reached
2021-06-04 11:12:29 +02:00
91df8a30e7
feat: limit number of files during storage cleanup
...
Since the number of parquet files can potentially be unbound (aka very
very large) and we do not want to hold the transaction lock for too
long and also want to limit memory consumption of the cleanup routine,
let's limit the number of files that we collect for cleanup.
2021-06-03 17:43:11 +02:00
85139abbbb
fix: use structured logging for cleanup logs
2021-06-03 11:23:29 +02:00
32c6ed1f34
refactor: More cleanup related to multi-table chunks ( #1604 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-02 17:00:23 +00:00
e5b65e10ac
test: ensure that `find_last_transaction_timestamp` indeed returns the last timestamp
2021-06-02 10:15:06 +02:00
98e413d5a9
fix: do not unwrap broken timestamps in serialized catalog
2021-06-02 10:15:06 +02:00
fc0a74920f
fix: use clearer error text
2021-06-02 09:41:19 +02:00
2a0b2698c6
fix: use structured logging
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-06-02 09:41:19 +02:00
64bf8c5182
docs: add code comment explaining why we parse transaction timestamps
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-06-02 09:41:19 +02:00
77aeb5ca5d
refactor: use protobuf-native Timestamp instead of string
2021-06-02 09:41:19 +02:00
9b9400803b
refactor!: bump transaction version to 2
2021-06-02 09:41:19 +02:00
5f77b7b92b
feat: add `parquet_file::catalog::find_last_transaction_timestamp`
2021-06-02 09:41:19 +02:00
9aee961e2a
test: test loading catalogs from broken protobufs
2021-06-02 09:41:19 +02:00
0a625b50e6
feat: store transaction timestamp in preserved catalog
2021-06-02 09:41:19 +02:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
d3711a5591
refactor: Use ParquetExec from DataFusion to read parquet files ( #1580 )
...
* refactor: use ParquetExec to read parquet files
* fix: test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-01 14:44:07 +00:00
64328dcf1c
feat: cache schema on catalog chunks too ( #1575 )
2021-06-01 12:42:46 +00:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
db432de137
feat: add distinct count to StatValues ( #1568 )
2021-05-28 17:41:34 +00:00
6098c7cd00
Merge branch 'main' into crepererum/issue1376
2021-05-28 07:13:15 +00:00
f3bec93ef1
feat: Cache TableSummary in Catalog rather than computing it on demand ( #1569 )
...
* feat: Cache `TableSummary` in catalog Chunks
* refactor: use consistent table summary
2021-05-27 16:03:05 +00:00
dd2a976907
feat: add a flag to ignore metadata errors during catalog rebuild
2021-05-27 13:10:14 +02:00
bc7389dc38
fix: fix typo
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-05-27 12:51:01 +02:00
48307e4ab2
docs: adjust error description to reflect internal errors
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-05-27 12:51:01 +02:00
d6f0dc7059
feat: implement catalog rebuilding from files
...
Closes #1376 .
2021-05-27 12:51:01 +02:00
024323912a
docs: explain what `PreservedCatalog::wipe` offers
2021-05-27 12:48:41 +02:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
9f451423d5
feat: log files that are deleted
2021-05-26 12:49:44 +02:00
24ec1a472e
fix: do NOT delete parquet files that are reachable by time travel
2021-05-26 12:38:54 +02:00
5983336366
refactor: rename `parquet_file::{utils => test_utils}`
2021-05-26 11:09:29 +02:00
d7e3bc569e
refactor: shorten time we hold the transaction lock during clean-up
2021-05-26 11:04:57 +02:00
18f5dd9ae1
test: ensure transaction lock exists during cleanup planning
2021-05-26 11:04:57 +02:00
b55eae98da
fix: do not delete non-parquet files during catalog-driven cleanup
2021-05-26 11:04:57 +02:00
5ed16ff294
refactor: improve error message in `parquet_file::cleanup`
2021-05-26 11:04:57 +02:00
14fdf3b7c7
feat: implement object store cleanup core routine
2021-05-26 11:02:40 +02:00
cc78b5317d
feat: add method to get all parquet files from catalog state
2021-05-26 11:02:40 +02:00
953114af2e
feat: add method to abort catalog transaction
2021-05-26 11:02:40 +02:00
92fcd7e940
feat: add a way to get OS, server ID and DB name from catalog
2021-05-26 11:02:40 +02:00
9daa4d00d6
test: re-organize `parquet_file` test utils a bit
2021-05-26 11:02:39 +02:00
38183928c8
refactor: extract path generator for data location
2021-05-26 10:59:40 +02:00
19a2733d30
feat: preserve transaction metadata in parquets
2021-05-25 09:56:12 +02:00
fe8e6301fe
refactor: move `read_schema_from_parquet_metadata` back to `parquet_file::metadata`
...
Let us pool all metadata handling in a single module, which makes it
easier to review.
2021-05-25 09:37:53 +02:00
ac83d99f66
feat: add a way to get current revision and UUID from transaction handle
2021-05-25 09:37:53 +02:00
fdc553b257
refactor: replace unwrap with expect
2021-05-25 09:37:53 +02:00
c464ffadad
refactor: remove special case timestamp_range in parquet chunk ( #1543 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-24 16:19:44 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
27e5b8fabf
refactor: Remove multiple table support from Parquet Chunk ( #1541 )
2021-05-24 08:40:31 -04:00
8bdddfd475
docs: mention that catalog wiping does not delete parquet files
2021-05-20 10:22:20 +02:00
b1a06246d6
feat: implement function to wipe a preserved catalog
2021-05-20 10:22:20 +02:00
6c405aa6f9
feat: check if preserved catalog exists when creating an empty one
2021-05-20 10:22:20 +02:00
c6a6005f65
feat: add `PreservedCatalog.exists`
2021-05-20 10:22:20 +02:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
8db26485a4
refactor: empty transaction during catalog creation
...
That involves some refactoring which we are going to need anyway for
hooking up the "read" path of the catalog into the DB startup, namely:
- make `Db::new` require a preserved catalog
- introduce a helper function that can provide that
- as a consequence, all test-creations of a Db are now async
This prepares for #1382 .
2021-05-18 17:42:07 +02:00
cdf0ada6a6
test: test preserved catalog <-> Db write wiring
2021-05-17 13:57:31 +02:00
68729dd5ee
refactor: avoid string allocation
2021-05-17 12:32:34 +02:00
adcd8132e7
docs: more comments regarding catalog transaction handling
2021-05-17 12:05:08 +02:00
a99d53e771
docs: document `OpenTransaction::handle_action*`
2021-05-17 11:48:51 +02:00
4fb800c7a6
refactor: make PreservedCatalog easier to integrate
2021-05-17 11:33:22 +02:00
f4d7154746
fix: table summaries must include timestamp as well
2021-05-17 11:33:22 +02:00
Marco Neumann
Andrew Lamb
Raphael Taylor-Davies
kodiakhq[bot]