2e30483f1f
refactor: Remove predicate module from predicate crate ( #3648 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 14:54:07 +00:00
17fbeaaade
feat: insert the persisted info into the catalog in one transaction ( #3636 )
...
* feat: add ProcessedTombstoneRepo
* feat: add function add_parquet_file_with_tombstones
* fix: remove unecessary use
* feat: handling transaction when adding parquet file and its processed tombstones
* feat: tests update catalog for parquet file and processed tombstones
* fix: make add parquet file & its processed tombstones fully transactional
* chore: cleanup
* test: add integration tests for new catalog update functions
* chore: remove catalog_update.rs
* chore: cleanup
* fix: assert the right values
* fix: create unique namespace
* fix: support non transaction create_many
* test: remove tests that do not work in a transaction
* fix: one more case with unique namespace
* chore: more verification around for better understanding why certain tests fail
* fix: compare difference rather than absolute becasue the DB already has data
* fix: fix the argument provided to SQL
* fix: return non-empty processed tombstones
* fix: insert the right parquet file
* chore: remove unsed file
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 14:44:15 +00:00
62a2ad289b
feat: Implement deserializing IoxMetadata from protobuf ( #3589 )
...
Fixes #3587 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 16:05:21 +00:00
22778a3a80
chore: upgrade rskafka and parking_lot ( #3592 )
2022-02-01 11:50:42 +00:00
093d5acfd4
fix: Unify temporary multiple definitions of IoxMetadata
2022-01-31 10:48:29 -05:00
8f81ce5501
refactor: Share parquet_file::storage code between new and old metadata
2022-01-31 10:36:33 -05:00
bf89162fa5
refactor: Move IoxMetadata to parquet_file
2022-01-31 10:36:33 -05:00
0f72a881ef
refactor: Rename Rust struct parquet_file::IoxMetadata to be IoxMetadataOld
2022-01-31 10:36:33 -05:00
1b298bb5bd
refactor: Alias the old proto definitions to make clearer the new ones coming in
2022-01-31 10:36:33 -05:00
32d7c4cbfe
refactor: remove InfluxColumnType::IOx ( #3565 )
...
* refactor: remove InfluxColumnType::IOx
Remove unused column variant - see #3554 for context.
* refactor: reserve SEMANTIC_TYPE_IOX name in proto
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 21:15:36 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
cdf5c21cd4
fix: Fix max timestamp value comparison in chunk metadata ( #3453 )
...
* fix: Fix max timestamp value comparison in chunk metadata
* refactor: rename contains to overlaps
Co-authored-by: Edd Robinson <me@edd.io>
2022-01-13 16:58:30 +00:00
c5cf03511c
fix: parquet column count statistics ( #2124 ) ( #3444 )
...
* fix: parquet metadata total_count (#2124 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-11 21:56:24 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
37bb7f2120
chore: `cargo update`
...
dependabot currently doesn't work due to
https://github.com/dependabot/dependabot-core/issues/4574
Excluded `quote` due to
https://github.com/dtolnay/quote/issues/204
2022-01-11 14:57:51 +01:00
ec8644a39a
refactor: return clearer error message
2021-12-07 12:24:28 -05:00
561c5ed8e7
refactor: make checking no data happen during reading inout stream
2021-12-07 12:03:41 -05:00
c992c82582
chore: Merge branch 'main' into ntran/compact_os_tests
2021-12-07 11:08:12 -05:00
5fdaa5b4ab
chore: don't panic with invalid parquet ( #3309 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-06 21:15:35 +00:00
7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs
...
Connects to #3117 .
2021-12-06 09:37:31 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
0b24b3c227
fix: Use a consistent version specifier when depending on the futures crate
2021-12-06 09:37:12 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
11067bfe3f
feat: simplify parquet reader ( #3282 ) ( #3291 )
...
* feat: simplify parquet reader (#3282 )
* chore: add back log line
2021-12-03 23:21:58 +00:00
86f9fe0bcb
refactor: no longer need to create and test no-row-groups parquet files
2021-12-03 15:14:04 -05:00
152281e428
fix: Capture the right 'no data' while parquet has no data
2021-12-03 12:19:48 -05:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
88aedc556e
feat: add FromStr implementation
2021-12-02 12:59:52 +00:00
bf74608dc8
docs: not persist of the input stream is empty
2021-12-01 17:53:19 -05:00
f085af034e
refactor: not persist empty chunk resulting from deleting & deduplicating ( #3274 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 20:57:30 +00:00
f53cdca010
feat: handling empty compacted stream
2021-11-30 18:13:36 -05:00
197634ed50
feat: reload chunk back into read buffer ( #3209 ) ( #3216 )
...
* feat: reload chunk back into read buffer (#3209 )
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
d16a7759ca
Merge branch 'main' into cn/workspace-hack
2021-11-22 17:05:31 +00:00
73d60539ad
refactor: use ChunkGenerator in parquet_catalog ( #2209 ) ( #3167 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:29:33 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
ca4e0ad13b
refactor: add parquet chunk generator ( #2209 ) ( #3163 )
...
* refactor: add parquet chunk generator (#2209 )
* fix: tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 12:35:18 +00:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
1902c4f8a9
chore: Update DataFusion ( #3012 )
...
* chore: Update DataFusion
* fix: restore Cargo.log
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:06:21 +00:00
4c9570b519
refactor: move `catalog` protobuf to `preserved_catalog`
...
This makes it clearer what's going since the contained messages are
only for the preserved part, not the in-mem catalog and its management.
2021-11-01 18:07:25 +01:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
d8f35d8ee9
chore: remove unused `parquet_file` => `chrono` dep
2021-10-19 14:45:56 +02:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
0568452a0c
chore: Update datafusion ( #2838 )
...
* chore: update datafusion version
* refactor: Update to use new datafusion apis
* fix: do not upgrade other packages
2021-10-13 20:51:19 +00:00
Carol (Nichols || Goulding)
Nga Tran
Marco Neumann
Carol (Nichols || Goulding)
Dom
Andrew Lamb
Raphael Taylor-Davies
Nga Tran
kodiakhq[bot]
Edd Robinson
dependabot[bot]
Carol (Nichols || Goulding)