ec8644a39a
refactor: return clearer error message
2021-12-07 12:24:28 -05:00
561c5ed8e7
refactor: make checking no data happen during reading inout stream
2021-12-07 12:03:41 -05:00
c992c82582
chore: Merge branch 'main' into ntran/compact_os_tests
2021-12-07 11:08:12 -05:00
5fdaa5b4ab
chore: don't panic with invalid parquet ( #3309 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-06 21:15:35 +00:00
7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs
...
Connects to #3117 .
2021-12-06 09:37:31 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
0b24b3c227
fix: Use a consistent version specifier when depending on the futures crate
2021-12-06 09:37:12 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
11067bfe3f
feat: simplify parquet reader ( #3282 ) ( #3291 )
...
* feat: simplify parquet reader (#3282 )
* chore: add back log line
2021-12-03 23:21:58 +00:00
86f9fe0bcb
refactor: no longer need to create and test no-row-groups parquet files
2021-12-03 15:14:04 -05:00
152281e428
fix: Capture the right 'no data' while parquet has no data
2021-12-03 12:19:48 -05:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
5d0fd1c603
fix: Allow dead code on fields that are now detected as never read
2021-12-02 11:52:01 -05:00
88aedc556e
feat: add FromStr implementation
2021-12-02 12:59:52 +00:00
bf74608dc8
docs: not persist of the input stream is empty
2021-12-01 17:53:19 -05:00
f085af034e
refactor: not persist empty chunk resulting from deleting & deduplicating ( #3274 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-01 20:57:30 +00:00
f53cdca010
feat: handling empty compacted stream
2021-11-30 18:13:36 -05:00
197634ed50
feat: reload chunk back into read buffer ( #3209 ) ( #3216 )
...
* feat: reload chunk back into read buffer (#3209 )
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
d16a7759ca
Merge branch 'main' into cn/workspace-hack
2021-11-22 17:05:31 +00:00
73d60539ad
refactor: use ChunkGenerator in parquet_catalog ( #2209 ) ( #3167 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:29:33 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
ca4e0ad13b
refactor: add parquet chunk generator ( #2209 ) ( #3163 )
...
* refactor: add parquet chunk generator (#2209 )
* fix: tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 12:35:18 +00:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
1902c4f8a9
chore: Update DataFusion ( #3012 )
...
* chore: Update DataFusion
* fix: restore Cargo.log
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-02 18:06:21 +00:00
4c9570b519
refactor: move `catalog` protobuf to `preserved_catalog`
...
This makes it clearer what's going since the contained messages are
only for the preserved part, not the in-mem catalog and its management.
2021-11-01 18:07:25 +01:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
d8f35d8ee9
chore: remove unused `parquet_file` => `chrono` dep
2021-10-19 14:45:56 +02:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
0568452a0c
chore: Update datafusion ( #2838 )
...
* chore: update datafusion version
* refactor: Update to use new datafusion apis
* fix: do not upgrade other packages
2021-10-13 20:51:19 +00:00
1523e0edcd
refactor: clean up preserved catalog interface
...
1. Remove `new_empty` logic. It's a leftover from the time when the
`PreservedCatalog` owned the in-memory catalog.
2. Make `db_name` a part of the `PreservedCatalogConfig`.
2021-10-13 13:58:11 +02:00
8414e6edbb
feat: migrate preserved catalog to TimeProvider ( #2722 ) ( #2808 )
...
* feat: migrate preserved catalog to TimeProvider (#2722 )
* fix: deterministic catalog prune tests
* fix: failing test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 14:43:05 +00:00
3dfe400e6b
feat: migrate write path to TimeProvider ( #2722 ) ( #2807 )
2021-10-12 12:09:08 +00:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
06c2c23322
refactor: create PreservedCatalogConfig struct ( #2793 )
...
* refactor: create PreservedCatalogConfig struct
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:43:05 +00:00
5da2f7b1b0
Merge branch 'main' into cn/less-database-name
2021-10-11 10:35:42 -04:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
8407735e00
fix: Pass the database name into PreservedCatalog
2021-10-08 15:25:10 -04:00
276aef69c9
refactor: Move PreservedCatalog test helper functions to test helpers and use them more
2021-10-08 15:25:10 -04:00
3aff4fcb07
refactor: Extract test helper functions for common catalog operations
...
This will make the next change easier, and I think it makes the tests
easier to read.
2021-10-08 15:25:10 -04:00
559a7e0221
Merge branch 'main' into cn/chunk-addr-smaller
2021-10-08 17:26:20 +00:00
fbe76935f4
fix: Remove some calls to iox_object_store.database_name
2021-10-08 09:50:14 -04:00
64bda1fc08
feat: improve `Debug`/`Display` for test `ChunkId`s
2021-10-08 13:55:56 +02:00
d3de6bb6e4
refactor: `max_persisted_timestamp` => `flush_timestamp`
...
There might be data left before this timestamp that wasn't persisted
(e.g. incoming data while the persistence was running).
2021-10-08 12:36:23 +02:00
63a932fa37
refactor: "min unpersisted ts" => "max persisted ts"
...
Store the "maximum persisted timestamp" instead of the "minimum
unpersisted timestamp". This avoids the need to calculate the next
timestamp from the current one (which was done via "max TS + 1ns").
The old calculation was prone to overflow panics. Since the
timestamps in this calculation originate from user-provided data (and
not the wall clock), this was an easy DoS vector that could be triggered
via the following line protocol:
```text
table_1 foo=1 <i64::MAX>
```
which is
```text
table_1 foo=1 9223372036854775807
```
Bonus points: the timestamp persisted in the partition
checkpoints is now the very same that was used by the split query during
persistence. Consistence FTW!
Fixes #2225 .
2021-10-08 11:52:49 +02:00
7d6be3f500
Merge branch 'main' into crepererum/issue2748
2021-10-07 09:04:18 +00:00
Nga Tran
Raphael Taylor-Davies
Carol (Nichols || Goulding)
kodiakhq[bot]
Edd Robinson
Andrew Lamb
Marco Neumann
dependabot[bot]
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)