acf698c366
fix: delete predicate sorting
2021-09-20 10:48:32 +02:00
c900704e58
chore: upgrade datafusion
2021-09-20 10:48:32 +02:00
0c5ba3786b
refactor: rename closure to make syntax a bit clearer
2021-09-20 10:48:32 +02:00
4c4fd59724
docs: extend comment about (not) cleanup up delete predicates
2021-09-20 10:48:32 +02:00
492d991f49
feat: delete catalog pres. catalog <=> in-mem catalog API
...
First step towards #2518 . Creates the Rust API to communicate delete
predicates between the preserved catalog and the in-memory catalog and
adds tests ensuring that the in-mem catalog produces the wanted errors
as well as correct checkpoints (similar to how this is done for the
parquet file tracking already).
**This does NOT contain the actual preservation!**
2021-09-20 10:48:32 +02:00
1752bf5563
Merge pull request #2584 from influxdata/dependabot/cargo/rand_distr-0.4.2
...
chore(deps): bump rand_distr from 0.4.1 to 0.4.2
2021-09-20 08:47:57 +00:00
876bb10cf8
chore(deps): bump rand_distr from 0.4.1 to 0.4.2
...
Bumps [rand_distr](https://github.com/rust-random/rand ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/rust-random/rand/releases )
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/rand/compare/rand_distr-0.4.1...rand_distr-0.4.2 )
---
updated-dependencies:
- dependency-name: rand_distr
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:39:39 +00:00
598179160a
Merge pull request #2583 from influxdata/dependabot/cargo/dirs-4.0.0
...
chore(deps): bump dirs from 3.0.2 to 4.0.0
2021-09-20 08:37:40 +00:00
1112597c44
chore(deps): bump dirs from 3.0.2 to 4.0.0
...
Bumps [dirs](https://github.com/soc/dirs-rs ) from 3.0.2 to 4.0.0.
- [Release notes](https://github.com/soc/dirs-rs/releases )
- [Commits](https://github.com/soc/dirs-rs/commits )
---
updated-dependencies:
- dependency-name: dirs
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:28:04 +00:00
5182ae92b0
Merge pull request #2582 from influxdata/dependabot/cargo/tracing-0.1.28
...
chore(deps): bump tracing from 0.1.27 to 0.1.28
2021-09-20 08:26:03 +00:00
c926994491
chore(deps): bump tracing from 0.1.27 to 0.1.28
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.27 to 0.1.28.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.27...tracing-0.1.28 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:17:24 +00:00
6f72028189
Merge pull request #2581 from influxdata/dependabot/cargo/hyper-0.14.13
...
chore(deps): bump hyper from 0.14.12 to 0.14.13
2021-09-20 08:13:33 +00:00
82c3fb35e7
chore(deps): bump hyper from 0.14.12 to 0.14.13
...
Bumps [hyper](https://github.com/hyperium/hyper ) from 0.14.12 to 0.14.13.
- [Release notes](https://github.com/hyperium/hyper/releases )
- [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md )
- [Commits](https://github.com/hyperium/hyper/compare/v0.14.12...v0.14.13 )
---
updated-dependencies:
- dependency-name: hyper
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 07:53:18 +00:00
fa5c884aab
Merge pull request #2528 from influxdata/crepererum/compress_parquet_metadata
...
feat: compress encoded parquet metadata
2021-09-20 07:51:24 +00:00
831e55d79e
refactor: make error messages more precise
2021-09-20 09:42:55 +02:00
0f5198c88d
test: fix tests dealing w/ parquet metadata sizes
...
Sizes now depend on the actual content and therefore we need
deterministic timestamps.
2021-09-20 09:42:53 +02:00
e15631002e
test: allow test code to specify exact parquet creation timestamp
...
This is required for deterministic sizes since different timestamp lead
to different compression ratios.
2021-09-20 09:42:52 +02:00
9c80d32af5
refactor: use normal google timestamps in parquet metadata again
...
We changed from Google timestamp (which use variable-sized integers) to
our own fixed-sized integer timestamps so that the size of the parquet
metadata does not depend on the timestamp. However with the introduction
of compression this is the case anyways (since slightly different
timestamps lead to different compression results) and we need now
derministic timestamps for tests. So there is now point in using our own
timestamp type. Switching back to the variable-sized type also shrinks
the post-compression results a bit.
2021-09-20 09:34:03 +02:00
afc507ae14
feat: compress encoded parquet metadata
...
Depending on the number of columns, this should safe between 60% and
75%.
2021-09-20 09:33:18 +02:00
7e804db0a3
fix: use btree map for some protobuf messages for deterministic outputs
2021-09-20 09:33:18 +02:00
5298543321
feat: don't use opentelemetry async trace exporter ( #2576 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 13:20:13 +00:00
4382bc0b71
feat: disable Snafu futures feature ( #2573 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:36:24 +00:00
3ffb16daa6
feat: remove parse_duration ( #2574 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-19 11:27:24 +00:00
62f302b31a
feat: remove human format ( #2577 )
2021-09-19 11:14:26 +00:00
0ff2f9bb60
feat: alter http query endpoint ( #2559 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-18 09:24:27 +00:00
63a1ed147f
Merge pull request #2566 from influxdata/cn/detailed-db-list
...
feat: Detailed database list command
2021-09-17 20:23:10 +00:00
364d245eae
feat: apply negated delete predicates during scan
2021-09-17 16:20:42 -04:00
ac9c25d33c
fix: Use prettytable for detailed database output
2021-09-17 15:59:36 -04:00
6520985b5d
refactor: Make the test regexes more readable
2021-09-17 15:27:23 -04:00
51a40b31bf
feat: Add a --detailed option to the database list CLI
...
That will list both active and deleted databases with their generations.
Closes #2462 .
2021-09-17 15:27:23 -04:00
423a976744
refactor: Use iterator adaptors rather than for loops
2021-09-17 15:27:23 -04:00
44a89cdf75
refactor: Change DeletedDatabase to DetailedDatabase
...
So this info can be reused for active databases in detailed database
lists.
2021-09-17 15:27:22 -04:00
cbf166c035
Merge pull request #2464 from influxdata/cn/restore
...
feat: Add an API for restoring a database that was marked deleted
2021-09-17 18:01:45 +00:00
23cc980d9e
Merge branch 'main' into cn/restore
2021-09-17 17:52:56 +00:00
60a866ddcb
refactor: merge delete predicates into select predicate
2021-09-17 07:52:33 -04:00
0168c043ad
Merge pull request #2565 from influxdata/ntran/delete_tests
...
test: tests for delete
2021-09-16 22:05:37 +00:00
243cc1f88c
fix: compile error after merge from main
2021-09-16 17:56:33 -04:00
0444d1b4fd
chore: merge main to branch
2021-09-16 17:28:37 -04:00
6cfeeb352b
refactor: address review comments
2021-09-16 17:21:06 -04:00
37b615f301
feat: migrate operations CLI to use pbjson ( #2562 )
...
* feat: migrate operations CLI to use pbjson
* fix: reserve removed field
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 19:15:24 +00:00
5878d3977c
chore: turn test on
2021-09-16 15:09:57 -04:00
cf4fd500b9
refactor: remove tests moved to query_tests
2021-09-16 15:05:48 -04:00
472e8a9e49
fix: fix compile error
2021-09-16 15:02:18 -04:00
2bae14df60
test: delete tests
2021-09-16 14:51:26 -04:00
4ae702549f
Merge pull request #2461 from influxdata/jpg/datagen-file-perf
...
perf: Don't open the output file each time we write.
2021-09-16 17:33:45 +00:00
997a96cf30
Merge branch 'main' into jpg/datagen-file-perf
2021-09-16 17:24:37 +00:00
9b3a13f984
feat: remove legacy http list partitions endpoint ( #2557 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 16:56:20 +00:00
f34eab70b3
feat: add pbjson bytes support ( #2560 )
...
* feat: add pbjson bytes support
* chore: fix lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 16:46:12 +00:00
315cbb8105
Merge pull request #2561 from influxdata/crepererum/arc_delete_predicates
...
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
2021-09-16 15:26:20 +00:00
ec943081c7
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
...
The motivations are:
1. The API uses a SINGLE predicate and adds that to many chunks. With
`Arc<Vec<...>>` you gain nothing, with `Vec<Arc<...>>` the predicate
is only stored once (in many vectors)
2. While we currently add predicates blindly to all chunks, we can be way
smarter in the future and prune out tables, partitions or even single
chunks (based on statistics). With that, it will be rare that many
chunks share the exact same set of predicates.
3. It would be nice if we could de-duplicate predicates when writing them
to the preserved catalog without needing to repeat the pruning
discussed in point 2. This is way easier to implement whan chunks
exists in `Arc`s.
4. As a side-note: the `Arc<Vec<...>>` wasn't really cloned around but
instead was created many time. So the new version should be more
memory efficient out of the box.
2021-09-16 17:16:09 +02:00
Marco Neumann
kodiakhq[bot]
dependabot[bot]
Raphael Taylor-Davies
Nga Tran
Carol (Nichols || Goulding)