8bdddfd475
docs: mention that catalog wiping does not delete parquet files
2021-05-20 10:22:20 +02:00
b1a06246d6
feat: implement function to wipe a preserved catalog
2021-05-20 10:22:20 +02:00
6c405aa6f9
feat: check if preserved catalog exists when creating an empty one
2021-05-20 10:22:20 +02:00
c6a6005f65
feat: add `PreservedCatalog.exists`
2021-05-20 10:22:20 +02:00
37880ee89a
refactor: store chunk IDs only in catalog ( #1521 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-20 04:07:14 +00:00
8db26485a4
refactor: empty transaction during catalog creation
...
That involves some refactoring which we are going to need anyway for
hooking up the "read" path of the catalog into the DB startup, namely:
- make `Db::new` require a preserved catalog
- introduce a helper function that can provide that
- as a consequence, all test-creations of a Db are now async
This prepares for #1382 .
2021-05-18 17:42:07 +02:00
cdf0ada6a6
test: test preserved catalog <-> Db write wiring
2021-05-17 13:57:31 +02:00
68729dd5ee
refactor: avoid string allocation
2021-05-17 12:32:34 +02:00
adcd8132e7
docs: more comments regarding catalog transaction handling
2021-05-17 12:05:08 +02:00
a99d53e771
docs: document `OpenTransaction::handle_action*`
2021-05-17 11:48:51 +02:00
4fb800c7a6
refactor: make PreservedCatalog easier to integrate
2021-05-17 11:33:22 +02:00
f4d7154746
fix: table summaries must include timestamp as well
2021-05-17 11:33:22 +02:00
7cced3242f
feat: add a way to parse infos from parquet paths
2021-05-17 11:33:22 +02:00
5969caccb0
feat: return parquet metadata from `write_to_object_store`
2021-05-17 11:33:22 +02:00
f9178dbb5f
feat: push metrics into catalog ( #1488 )
...
* feat: push metrics into catalog
* chore: minor cleanup
* fix: include db labels in chunk metric domains
* chore: fmt
* fix: don't allow dropping moving chunks
* chore: further tweaks
* chore: review feedback
* feat: use new_unregistered() for metric instruments instead of default
* chore: use &[KeyValue] instead of &Vec<KeyValue>
* refactor: make GauageValue non default constructible
2021-05-14 17:37:39 +00:00
9583636748
feat: we now can read parquet files form all kind of object stores
2021-05-12 18:05:34 -04:00
795f5bfcb7
refactor: make `StatValues::{min,max}` optional + handle NaNs
...
This will allow us to:
- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
anymore)
2021-05-10 17:12:25 +02:00
c6b933eb63
chore: merge main to branch
2021-05-07 18:40:17 -04:00
f2c19ec080
refactor: further address Carol's comment
2021-05-07 17:40:40 -04:00
971500681f
refactor: address Andrew's and Carol's comment
2021-05-07 17:33:19 -04:00
e2cc4634bf
fix: Use PathBuf rather than debug formatting and back to String
...
This is the same fix I made in 54c5f98
2021-05-07 15:58:11 -04:00
31d49db0ed
chore: a litlle more cleanup
2021-05-07 09:38:41 -04:00
ba015ee4df
refactor: clean up and add comments
2021-05-07 09:31:41 -04:00
1a998d4116
feat: preserve parquet metadata in catalog
...
Closes #1380 .
2021-05-07 09:51:44 +02:00
c3d523fc4f
refactor: add col prefixes to make_chunk & Co
2021-05-07 09:51:44 +02:00
5db504300d
refactor: use parsed paths instead of raw strings for catalog paths
2021-05-07 09:51:44 +02:00
55bf848bd2
feat: Now we can query directly from files in object store
2021-05-06 18:02:17 -04:00
884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns ( #1415 )
...
* feat: add column_type and influxdb_column_type, remove row_count from system.columns
* fix: update tests
* fix: more test update
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
86771ea629
chore: update arrow/datafusion deps ( #1433 )
...
* chore: update datafusion deps
* chore: update arrow deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-05 22:37:31 +00:00
a5c92fae8a
chore: merge main to branch
2021-05-05 13:48:42 -04:00
3bdb451529
chore: merge main to branch
2021-05-05 13:18:39 -04:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
2b46f51e5b
chore: address Dom's comment
2021-05-05 12:55:41 -04:00
a1f3413c89
refactor: move private test helpers to utils module to be used by many modules
2021-05-05 11:41:46 -04:00
fcb37a0b1d
feat: more testing scenarios for quering parquet files
2021-05-05 10:57:02 -04:00
1f42eb89cd
feat: implement parquet metadata handling
...
Closes #1379 and contributes to #1380 .
2021-05-05 13:29:16 +02:00
056c29aaa2
feat: add a way to retrieve timestamp range from parquet chunk
2021-05-05 13:29:16 +02:00
c54109113e
feat: add a way to retrieve storage path from parquet chunks
2021-05-05 13:29:16 +02:00
136c35cb88
feat: implement transaction handling for catalog
...
Closes #1253 .
2021-05-03 10:04:35 +02:00
34a3388a49
feat: unload chunks from read buffer but keep them in object store
2021-04-30 16:12:02 -04:00
e87973babe
refactor: address review comments
2021-04-29 13:15:43 -04:00
402d9c748c
chore: cargo fmt
2021-04-28 16:52:52 -04:00
2a2760bd18
feat: complete tests where data in both RUB and OS
2021-04-28 16:14:07 -04:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
657bfa1b20
refactor: address Andrew's comments
2021-04-16 17:44:46 -04:00
b3e110a241
refactor: address Jake's comment
2021-04-16 17:27:40 -04:00
Marco Neumann
Raphael Taylor-Davies
Nga Tran
Carol (Nichols || Goulding)
Andrew Lamb
Jake Goulding