1805f7fbe3
refactor: SequenceNumberSet::as_bytes -> to_bytes
...
The conversion is not (always) a cheap cast/conversion, and therefore
should be prefixed with "to_" and not "as_".
2023-02-15 11:03:15 +01:00
ac1502aeba
Merge pull request #6984 from influxdata/dom/soft-deleted-namespaces
...
feat(router): soft-delete namespace RPC handler
2023-02-15 09:57:41 +00:00
5ca165b76e
docs: fix two typos
...
Comments in test code.
2023-02-15 10:49:39 +01:00
dbe9219a8b
test: assert correct retention period
...
Use the correct constant in the retention period assert.
2023-02-15 10:49:38 +01:00
61fb92b85c
feat(router): soft-delete RPC handler
...
This implements the RPC "delete_namespace" handler, allowing a namespace
to be soft-deleted.
Adds unit coverage for all handlers & e2e test coverage for the new
handler (the rest were already covered).
The tests also highlight the caching issue documented here:
https://github.com/influxdata/influxdb_iox/issues/6175
2023-02-15 10:49:38 +01:00
4a30df0e07
chore(deps): Bump once_cell from 1.17.0 to 1.17.1 ( #6991 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.17.0 to 1.17.1.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.17.0...v1.17.1 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-15 09:37:50 +00:00
f499022511
feat: add compaction level to commit metrics ( #6985 )
...
* feat: add compaction level to commit metrics
* test: more realism
2023-02-15 09:28:19 +00:00
7fb052208f
feat: allow gap filling to produce multiple batches ( #6986 )
...
* feat: allow gap filling to produce multiple batches
* chore: code review feedback
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-14 22:48:29 +00:00
4db9df7197
feat: Add logging of commands sent via the `influxdb storage` command ( #6987 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-14 22:41:54 +00:00
0ffb211c54
test: more compactor layout tests ( #6988 )
...
* test: more compactor layout tests
* chore: address review comments
2023-02-14 22:14:06 +00:00
04bd47e64a
feat(compactor): Add more tests, improve sizes to simulator run display more ( #6981 )
...
* refactor: Split layout tests into their own module
* feat: Add more tests, improve sizes to simulator run display more
* fix: Apply suggestions from code review
Co-authored-by: Nga Tran <nga-tran@live.com>
* fix: fix comment wording
* fix: reporting order of skipped compactions
* chore: Run cargo hakari tasks
* fix: revert changes to Cargo.lock
* fix: revert workspace hack change
---------
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-02-14 19:34:19 +00:00
ed007cb71f
refactor: replace IF-statement w/ optimizer rule ( #6982 )
...
* refactor: replace IF-statement w/ optimizer rule
This replaces a single IF-statement within the physical plan
construction with a physical optimizer rule. While on its own this seems
kinda pointless, it sets the foundation for #6098 . W/o the optimizer
some EXPLAIN query tests would fail.
* test: use insta snapshots
* fix: update test snapshots
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-14 17:10:41 +00:00
5c506058da
feat: skip partitions of wide tables ( #6978 )
...
* feat: skip partitions of wide tables
* test: one more test
* refactor: address review comments
2023-02-14 16:42:13 +00:00
3a1b16e890
chore: ignore `RUSTSEC-2022-0090` ( #6983 )
2023-02-14 15:12:11 +00:00
363afd7271
Merge pull request #6979 from influxdata/cn/ie2e
...
test: Add an e2e test of just the ingester's API
2023-02-14 11:40:58 +00:00
2934e15f2f
Merge branch 'main' into cn/ie2e
2023-02-14 11:34:32 +00:00
a3b662f30b
docs: explain new physical plan construction ( #6970 )
...
* docs: explain new physical plan construction
There is no code yet, but this is the rough plan.
Ref #6098 .
* docs: clarify wording
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* docs: typos
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* docs: add links
* docs: extend reasoning
---------
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2023-02-14 11:13:22 +00:00
123c36e5b4
Merge branch 'main' into cn/ie2e
2023-02-14 11:12:38 +00:00
ce31ad7607
chore: `cargo update` ( #6980 )
...
dependabot doesn't seem to keep up at the moment, so let's do a global
manual update:
```console
❯ cargo update
Updating crates.io index
Updating git repository `https://github.com/apache/arrow-datafusion.git `
Updating git repository `https://github.com/influxdata/rskafka.git `
Updating git repository `https://github.com/mkmik/heappy `
Updating anyhow v1.0.68 -> v1.0.69
Updating axum v0.6.2 -> v0.6.6
Updating axum-core v0.3.1 -> v0.3.2
Removing bstr v0.2.17
Removing bstr v1.1.0
Adding bstr v1.2.0
Updating bytemuck v1.12.3 -> v1.13.0
Updating crc v3.0.0 -> v3.0.1
Updating csv v1.1.6 -> v1.2.0
Updating cxx v1.0.86 -> v1.0.90
Updating cxx-build v1.0.86 -> v1.0.90
Updating cxxbridge-flags v1.0.86 -> v1.0.90
Updating cxxbridge-macro v1.0.86 -> v1.0.90
Updating encoding_rs v0.8.31 -> v0.8.32
Updating fastrand v1.8.0 -> v1.9.0
Updating fd-lock v3.0.8 -> v3.0.10
Updating gimli v0.27.0 -> v0.27.1
Updating hermit-abi v0.3.0 -> v0.3.1
Updating inferno v0.11.13 -> v0.11.15
Removing itoa v0.4.8
Updating js-sys v0.3.60 -> v0.3.61
Updating num-complex v0.4.2 -> v0.4.3
Updating object v0.30.2 -> v0.30.3
Updating parking_lot_core v0.9.6 -> v0.9.7
Updating pest v2.5.3 -> v2.5.5
Updating pest_derive v2.5.3 -> v2.5.5
Updating pest_generator v2.5.3 -> v2.5.5
Updating pest_meta v2.5.3 -> v2.5.5
Updating petgraph v0.6.2 -> v0.6.3
Updating rayon-core v1.10.1 -> v1.10.2
Updating rgb v0.8.34 -> v0.8.35
Updating signal-hook-registry v1.4.0 -> v1.4.1
Updating spin v0.9.4 -> v0.9.5
Updating sync_wrapper v0.1.1 -> v0.1.2
Updating thread_local v1.1.4 -> v1.1.7
Updating tinyvec_macros v0.1.0 -> v0.1.1
Updating unicode-bidi v0.3.8 -> v0.3.10
Updating unicode-segmentation v1.10.0 -> v1.10.1
Updating wasm-bindgen v0.2.83 -> v0.2.84
Updating wasm-bindgen-backend v0.2.83 -> v0.2.84
Updating wasm-bindgen-futures v0.4.33 -> v0.4.34
Updating wasm-bindgen-macro v0.2.83 -> v0.2.84
Updating wasm-bindgen-macro-support v0.2.83 -> v0.2.84
Updating wasm-bindgen-shared v0.2.83 -> v0.2.84
Updating web-sys v0.3.60 -> v0.3.61
Updating which v4.3.0 -> v4.4.0
❯ cargo hakari generate
info: contents updated
```
2023-02-14 10:34:48 +00:00
969319dfd3
fix: Allow all valid characters following a keyword ( #6959 )
...
* fix: Allow all valid characters following a keyword
Closes #6382
* chore: Identified additional test cases
2023-02-13 22:21:11 +00:00
263d8fe21f
chore: Layout tests with `TargetLevel` algorithm + update display ( #6977 )
...
* refactor: move ParquetFileSimulator to compactor2_test_utils
* chore: Test with new algorithm + update display
* chore: Updates
* chore: Update setting to match prod
2023-02-13 22:12:55 +00:00
9e27709c84
test: Add an e2e test of just the ingester's API
2023-02-13 16:36:59 -05:00
adddd71c4d
chore: Add some comments about fatal panic rationale ( #6955 )
...
* chore: Add some comments and log messages about fatal panics
* fix: Update panic_logging/src/lib.rs
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 18:54:57 +00:00
1424d4a0b7
chore: remove dead code ( #6969 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 17:14:02 +00:00
13ce6da3df
refactor: extract `FileClassifer` component ( #6946 )
...
* refactor: extract `FileClassifer` component
Make the driver slightly smaller. Also makes the "all-in-one" mode
easier to understand.
* docs: add some
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 16:25:05 +00:00
9c18a3e3ef
Merge pull request #6971 from influxdata/dom/router-e2e-restart
...
test(router): router restarts + caching effects
2023-02-13 16:12:33 +00:00
4a0149a418
test(router): retention change after cache reset
...
Assert that the new retention period value is used once a router is
restarted and the cache converged.
2023-02-13 14:40:14 +01:00
ce12daa96e
refactor(e2e): simulate router restart
...
Allow a router to be "restarted" within an e2e test.
2023-02-13 14:40:11 +01:00
0e10561ce0
fix: Port router unit tests to the RPC write path ( #6956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:28:06 +00:00
2277cd80f5
Merge pull request #6952 from influxdata/dom/namespace-soft-delete-catalog
...
feat: namespace soft-delete support
2023-02-13 12:21:24 +00:00
4f330b1935
Merge branch 'main' into dom/namespace-soft-delete-catalog
2023-02-13 12:14:55 +00:00
2751c77de6
chore: add extra debugging to help track down jdbc_client failures ( #6968 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:14:39 +00:00
9d20f26006
Merge branch 'main' into dom/namespace-soft-delete-catalog
2023-02-13 12:07:03 +00:00
4d7aa1e48b
refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test ( #6960 )
...
* refactor: extract compactor2 test utils into `compactor2_test_utils` and integration test
* fix: Update compactor2/src/components/mod.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
---------
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 12:06:42 +00:00
2d46a364dc
feat: namespace soft-delete support
...
This commit adds initial support for "soft" namespace deletion, where
the actual records & data remain, but are no longer queryable /
writeable.
Soft deletion is eventually consistent - users can expect to continue
writing to and reading from a bucket after issuing a soft delete call,
until the various components either restart, or have their caches
flushed.
The components treat soft-deleted namespaces differently:
* router: ignore soft deleted namespaces
* ingester: accept soft deleted namespaces
* compactor: accept soft deleted namespaces
* querier: ignore soft deleted namespaces
* various gRPC services: ignore soft deleted namespaces
This ensures that the ingester & compactor do not see rows "vanishing"
from the database, and continue to make forward progress.
Writes for the deleted namespace that are buffered in the ingester will
be persisted as normal, allowing us to support "un-delete" operations
where the system is restored to a the state at which the delete was
issued (rather than loosing the buffered data).
Follow-on work is required to ensure GC drops the orphaned parquet files
after the configured GC time, and optimisations such as not compacting
parquet from soft-deleted namespaces seems like a trivial win.
2023-02-13 12:01:35 +01:00
143cb26e87
chore: Update datafusion again ( #6958 )
...
* chore: Update datafusion pin
* chore: Update plans
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-13 10:28:44 +00:00
0cbd9f6a82
chore(deps): Bump tokio-util from 0.7.5 to 0.7.7 ( #6964 )
...
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-13 10:10:53 +00:00
083f9bc0a7
chore(deps): Bump zstd-safe from 6.0.3+zstd.1.5.2 to 6.0.4+zstd.1.5.4 ( #6962 )
...
Bumps [zstd-safe](https://github.com/gyscos/zstd-rs ) from 6.0.3+zstd.1.5.2 to 6.0.4+zstd.1.5.4.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits/zstd-safe-6.0.4 )
---
updated-dependencies:
- dependency-name: zstd-safe
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-13 09:58:17 +00:00
c805a7527b
chore(deps): Bump zstd-sys from 2.0.6+zstd.1.5.2 to 2.0.7+zstd.1.5.4 ( #6963 )
...
Bumps [zstd-sys](https://github.com/gyscos/zstd-rs ) from 2.0.6+zstd.1.5.2 to 2.0.7+zstd.1.5.4.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits/zstd-sys-2.0.7 )
---
updated-dependencies:
- dependency-name: zstd-sys
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-13 09:23:39 +00:00
06d50e8611
chore(deps): Bump croaring from 0.7.0 to 0.8.0 ( #6965 )
...
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.7.0...0.8.0 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-13 09:09:32 +00:00
c15de72249
fix: Remove the use of write buffer config from import schema command ( #6957 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-11 15:02:32 +00:00
0db37f564c
chore: Update display, and add initial scenario tests ( #6954 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 20:20:19 +00:00
a2510c8343
feat: partial implementation of gap filling operator ( #6911 )
...
* feat: partial implementation of gap filling operator
* chore: code review feedback
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 19:03:34 +00:00
779fb93ce7
refactor: move test builders out of compactor2 code ( #6953 )
...
* refactor: move test builders out of compactor2 code
* fix: docs
2023-02-10 18:28:09 +00:00
de765a99b7
Merge pull request #6936 from influxdata/cn/multi-ingester-test
...
test: Test with multiple ingesters, like in production
2023-02-10 15:11:48 +00:00
8824342763
Merge branch 'main' into cn/multi-ingester-test
2023-02-10 15:05:39 +00:00
d790406085
feat(compactor): Implement ParquetFileSimulator and use it to show layout testing ( #6932 )
...
* feat(compactor): implement ParquetFileSimulator, show it working in tests
* fix: Update compactor2/src/components/parquet_files_sink/simulator.rs
Co-authored-by: Nga Tran <nga-tran@live.com>
* feat: Improve display of plan_ir
* refactor: return CompactResult, avoid `mut TestSetup`
---------
Co-authored-by: Nga Tran <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 14:51:17 +00:00
92bdb450e0
Merge pull request #6947 from influxdata/dom/expose-soft-delete-column
...
refactor(catalog): expose deleted_at on Namespace
2023-02-10 14:38:45 +00:00
a85dcd745b
refactor(catalog): expose deleted_at on Namespace
...
Add the new catalog column to the Namespace representation/model.
2023-02-10 14:15:01 +01:00
2283b3d401
fix: `clap_blocks` clippy warning ( #6945 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-10 12:29:45 +00:00
Dom Dwyer
dependabot[bot]
Marco Neumann
Christopher M. Wolff
Andrew Lamb
Nga Tran
Stuart Carnie
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]