31d49db0ed
chore: a litlle more cleanup
2021-05-07 09:38:41 -04:00
ba015ee4df
refactor: clean up and add comments
2021-05-07 09:31:41 -04:00
1a998d4116
feat: preserve parquet metadata in catalog
...
Closes #1380 .
2021-05-07 09:51:44 +02:00
c3d523fc4f
refactor: add col prefixes to make_chunk & Co
2021-05-07 09:51:44 +02:00
5db504300d
refactor: use parsed paths instead of raw strings for catalog paths
2021-05-07 09:51:44 +02:00
55bf848bd2
feat: Now we can query directly from files in object store
2021-05-06 18:02:17 -04:00
884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns ( #1415 )
...
* feat: add column_type and influxdb_column_type, remove row_count from system.columns
* fix: update tests
* fix: more test update
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
86771ea629
chore: update arrow/datafusion deps ( #1433 )
...
* chore: update datafusion deps
* chore: update arrow deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-05 22:37:31 +00:00
a5c92fae8a
chore: merge main to branch
2021-05-05 13:48:42 -04:00
3bdb451529
chore: merge main to branch
2021-05-05 13:18:39 -04:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
2b46f51e5b
chore: address Dom's comment
2021-05-05 12:55:41 -04:00
a1f3413c89
refactor: move private test helpers to utils module to be used by many modules
2021-05-05 11:41:46 -04:00
fcb37a0b1d
feat: more testing scenarios for quering parquet files
2021-05-05 10:57:02 -04:00
1f42eb89cd
feat: implement parquet metadata handling
...
Closes #1379 and contributes to #1380 .
2021-05-05 13:29:16 +02:00
056c29aaa2
feat: add a way to retrieve timestamp range from parquet chunk
2021-05-05 13:29:16 +02:00
c54109113e
feat: add a way to retrieve storage path from parquet chunks
2021-05-05 13:29:16 +02:00
136c35cb88
feat: implement transaction handling for catalog
...
Closes #1253 .
2021-05-03 10:04:35 +02:00
34a3388a49
feat: unload chunks from read buffer but keep them in object store
2021-04-30 16:12:02 -04:00
e87973babe
refactor: address review comments
2021-04-29 13:15:43 -04:00
402d9c748c
chore: cargo fmt
2021-04-28 16:52:52 -04:00
2a2760bd18
feat: complete tests where data in both RUB and OS
2021-04-28 16:14:07 -04:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
657bfa1b20
refactor: address Andrew's comments
2021-04-16 17:44:46 -04:00
b3e110a241
refactor: address Jake's comment
2021-04-16 17:27:40 -04:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
231ebb54d4
chore: fix a format
2021-04-14 16:32:25 -04:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
05bf28ce85
feat: Add 2 main functions table_schema and table_names for Parquet Chunk ato pay a foundation for querying it
2021-04-13 18:23:55 -04:00
4a6d6bd7ad
feat: initial work for querying data from parquet file in object store
2021-04-13 13:57:46 -04:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
453aeaf1a0
feat: Add tests for writing RB chunks to Object Store
2021-04-09 17:39:23 -04:00
f501a74aea
refactor: Address review comments
2021-04-07 21:28:03 -04:00
be6e1e48e4
feat: add writer_id and object_store in Db
2021-04-07 18:36:07 -04:00
c2355aca6d
feat: add basic memory tracking ( #1125 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
6e01fbc382
feat: ause TableSummary as metadata for parquet chunk's tables and read buffer's read_filter ot get data
2021-04-05 15:37:34 -04:00
4bdf8963e6
feat: continue buidling foundation for writing RB chunks to parquet files
2021-04-02 16:06:25 -04:00
49267114d3
chore: merge main into branch and resolve conflicts
2021-04-01 13:22:49 -04:00
1463c6645f
feat: Add ChunkState::ObjectStore and rename ParquetChunk to Chunk
2021-04-01 11:53:03 -04:00
19a453a483
feat: finally have some framework with clear todos for writing a chunk into parquet files
2021-03-31 16:21:53 -04:00
cd409b471f
feat: continue the implementation
2021-03-30 21:31:51 -04:00
0bcd52d5c9
feat: Add more changes
2021-03-30 18:31:09 -04:00
Nga Tran
Marco Neumann
Andrew Lamb
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Jake Goulding