d75df2b610
chore: `cargo update` ( #5426 )
...
```
bumpalo v3.10.0 -> v3.11.0
either v1.7.0 -> v1.8.0
iana-time-zone v0.1.45 -> v0.1.46
rustix v0.35.8 -> v0.35.9
```
`rustix` is important because `0.35.8` was yanked.
2022-08-18 08:53:00 +00:00
8eb3a79d7f
Merge pull request #5348 from influxdata/cn/upgrade-l0-metrics
...
feat: Add metrics on the size of files created by ingestion and used for compaction
2022-08-17 16:08:59 +00:00
2b3ca54168
Merge branch 'main' into cn/upgrade-l0-metrics
2022-08-17 16:01:42 +00:00
f4443f0b3a
feat: import schema override ( #5420 )
...
* chore: struct for overrides of import schema conflicts
* chore: import schema override shouldn't support tags
* feat: import schema merge can take an override schema
* fix: schema override in test had superfluous tag
* chore: test for batch schema merge with override in import schema
* feat: import schema merge now takes override schema
2022-08-17 14:59:50 +00:00
3dca9c1b43
feat: async sleep with `TimeProvider` ( #5417 )
...
* feat: async sleep with `TimeProvider`
This is helpful to mock "ticked" loops or to control certain async time
periods.
Will be used to test the refresh policy developed in #5318 .
* refactor: use a single `TimeProvider::sleep` impl
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 14:50:35 +00:00
f34f99c5ed
refactor: port LRU cache backend to policy framework ( #5406 )
...
* refactor: port LRU cache backend to policy framework
Closes #5320 .
* test: extend `test_oversized_entries`
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 14:43:24 +00:00
7f0ae53d6f
chore: Update to (almost) released object_store 0.4.0 ( #5419 )
...
* chore: update object_store
* chore: update hakari config
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-08-17 13:44:48 +00:00
bd4b708055
chore: Update datafusion pin + other dependencies ( #5418 )
...
* chore: Update datafusion pin
* chore: Update other depdecies
* fix: Update for changes in API
2022-08-17 10:42:37 +00:00
2e638fe19c
chore(deps): Bump libc from 0.2.131 to 0.2.132 ( #5414 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.131 to 0.2.132.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.131...0.2.132 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-17 08:40:57 +00:00
78665d3092
chore(deps): Bump once_cell from 1.13.0 to 1.13.1 ( #5413 )
...
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.13.0 to 1.13.1.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.13.0...v1.13.1 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 08:31:46 +00:00
7e1ad40522
refactor: lift `Send` requirement from `ChangeRequest` ( #5404 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-17 08:24:44 +00:00
51141f2c78
docs: Edit catalog API docs ( #5409 )
...
* docs: Edit Catalog docs string
* docs: Edit top-level catalog module doc
* docs: Mark `sealed` trait w/ `doc(hidden)`
* docs: Edit catalog transaction docs
* docs: Edit Catolog trait docs
* docs: Edit `RepoCollection` docs
Clarify concept of repository.
Add links.
* docs: Add link to `Transaction`
Co-authored-by: pierwill <pierwill@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-16 21:36:29 +00:00
a6f28d0709
Merge pull request #5411 from influxdata/dom/table-scoped-errs
...
feat: table name in schema validation errors
2022-08-16 17:07:31 +00:00
180ff9f681
feat: table name in schema validation errors
...
Scopes all schema validation errors to include the table name in the
error output.
2022-08-16 19:00:44 +02:00
7e97620b37
chore: `cargo-update` ( #5405 )
...
```
anyhow v1.0.60 -> v1.0.61
bytemuck v1.11.0 -> v1.12.0
hdrhistogram v7.5.0 -> v7.5.1
iana-time-zone v0.1.44 -> v0.1.45
memmap2 v0.5.5 -> v0.5.7
os_str_bytes v6.2.0 -> v6.3.0
```
`iana-time-zone` is important because `0.1.44` was yanked.
2022-08-16 12:51:46 +00:00
10fee5535a
feat: import schema updates iox catalog ( #5385 )
...
* feat: import schema updates iox catalog
- renamed import/schema module to aggregate_tsm_schema to not conflic
with schema crate
- fetch schema from iox catalog, and validate/merge/create as needed
chore: add catalog dsn config to import schema command
chore: import schema command connects to catalog
chore: import schema merge validation errors return non-zero code
chore: simplified and tidies import update catalog code
chore: tests and refactoring of import schema catalog update
* chore: require retention on ns creation in import
* chore: fixed bad test in import schema validation
* chore: friendlier errors & more tests in import schema catalog update
2022-08-16 11:05:27 +00:00
b60e0beee2
fix: read_buffer benchmarks array type creation ( #5401 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-16 10:28:27 +00:00
49ab568ca8
refactor: convert `remove_if` feature to policy framework ( #5398 )
...
* refactor: allow `ChangeRequest` to carry a lifetime
Let's not restrict our change functions to `'static` because this would
require us to clone loads of data to achieve predicate-based
`remove_if`.
* refactor: convert `remove_if` feature to policy framework
Decided to drop the "shared" functionality. We only use the small
`remove_if` bit which is way easier to reason about.
For #5320 .
* refactor: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-16 08:23:27 +00:00
0a7e6919f2
chore: do not build benchmark binary for lib targets ( #5400 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-16 08:11:23 +00:00
724759862f
chore(deps): Bump pin-project from 1.0.11 to 1.0.12 ( #5402 )
...
Bumps [pin-project](https://github.com/taiki-e/pin-project ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/taiki-e/pin-project/releases )
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md )
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.11...v1.0.12 )
---
updated-dependencies:
- dependency-name: pin-project
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-16 08:04:25 +00:00
0ccefa0d0c
refactor: port TTL backend to policy framework ( #5396 )
...
* refactor: port TTL backend to policy framework
Note that this is "just" a port, it does NOT change how TTL works. This
will be done in #5318 .
Helps with #5320 .
* fix: ensure inner backend is empty
* test: add some smoke test
2022-08-15 16:48:16 +00:00
ef716a5b90
fix: Remove compaction level attribute from the compaction_input_file_bytes metric
2022-08-15 10:50:04 -04:00
a9ed32df89
fix: Remove compaction_counter as it's now redundant with the compaction_input_file_bytes histogram
2022-08-15 10:23:29 -04:00
af95ce7ca6
feat: Add a histogram tracking sizes of files used as inputs to compaction
...
Fixes #5348 .
2022-08-15 10:13:54 -04:00
cd6c809fe0
fix: Change metric tracking sizes of files selected for compaction to a histogram
...
Connects to #5348 .
2022-08-15 10:13:54 -04:00
ed44817ed1
feat: Add a histogram of ingested (new L0) Parquet file sizes
...
Connects to #5348 .
2022-08-15 10:13:54 -04:00
b2caf54b3a
docs: clarify clamping behavior in `TimestampRange::new` ( #5399 )
...
Closes #5248 .
2022-08-15 12:43:01 +00:00
0a44a42999
chore(deps): Bump futures from 0.3.21 to 0.3.23 ( #5394 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 12:19:17 +00:00
69ccfafd14
chore(deps): Bump futures-task from 0.3.21 to 0.3.23 ( #5393 )
...
Bumps [futures-task](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-task
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 10:27:22 +00:00
2c934addf5
chore(deps): Bump futures-channel from 0.3.21 to 0.3.23 ( #5388 )
...
Bumps [futures-channel](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-channel
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 09:42:53 +00:00
a14e161109
chore(deps): Bump futures-core from 0.3.21 to 0.3.23 ( #5387 )
...
Bumps [futures-core](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 08:59:53 +00:00
6a4425ae08
chore(deps): Bump chrono from 0.4.21 to 0.4.22 ( #5392 )
...
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.21 to 0.4.22.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.21...v0.4.22 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:31:23 +00:00
8a5cd1cddd
refactor: abstract change requests for cache policies ( #5382 )
...
* refactor: abstract change requests for cache policies
Tl;Dr; Lets replace the hard-coded enum of change requests with a more
flexible and easier-to-understand function-based approach. It also
unifies the interface for "initial requests" and "reactions".
Story time:
I just wanted to port over the "shared" backend with its "remove if"
functionality to the policy framework. Turns out that this backend does
not just use simple atomic operations but is basically a
"compare&exchange"-like operation (technically: GET+REMOVE) that relies
on a single mutable access lifetime to make sure that this behavior is
sane and nobody messes with the value between the GET and REMOVE. I call
this a "compound request".
The new policy framework did not support this kind of requests so I
thought how I could shoehorn it in. Basically I wanted to extend
`ChangeRequest` to support these kind of operations. But listing
combinations explicitly kinda seemed like some hack. So my brain slowly
came up with an abstract approach how to encode these operations and
after a while I was close to design some kind of mini-VM. I also
considered some ugly workarounds or to not express the "remove if"
functionality" as a policy. My nice vision of a policy framework started
to crack...
This is when sanity kicked in and I realized: every change request is
technically just a function on a (virtual) cache backend. Sure,
all-powerful Rust functions allow you to do ugly stuff that will result
in deadlocks, but with good docs and helpers for the common operations
the risk is very low. Furthermore this is already close to the system I
had for the "initial requests" and which I called a "callback backend".
So I unified both systems and made the change request abstract and all
tests pass and I think it is the better design.
Helps with #5320 .
* docs: typos
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:24:25 +00:00
38277cf0fb
chore(deps): Bump nix from 0.24.2 to 0.25.0 ( #5391 )
...
* chore(deps): Bump nix from 0.24.2 to 0.25.0
Bumps [nix](https://github.com/nix-rust/nix ) from 0.24.2 to 0.25.0.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.2...v0.25.0 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:10:09 +00:00
822ec18fcd
chore(deps): Bump clap from 3.2.16 to 3.2.17 ( #5390 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.16 to 3.2.17.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/v3.2.17/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.16...v3.2.17 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 07:56:11 +00:00
44d72b2e94
Merge pull request #5384 from influxdata/dom/parallel-init
...
refactor: parallelise Kafka partition client init
2022-08-12 14:01:22 +01:00
bd88ac6149
refactor: parallelise Kafka partition client init
...
Changes the Kafka write buffer impl to parallelise initialisation of the
PartitionClient instances.
Now that the PartitionClient constructor also performs leader discovery
(using cached metadata, influxdata/rskafka#164 ) and establishes a broker
connection (influxdata/rskafka#166 ) executing them in parallel will
cause a proportional decrease in the time taken to bring IOx up.
2022-08-12 14:45:23 +02:00
b2e50f0c1a
Merge pull request #5383 from influxdata/dom/deploy
...
docs: router writes to partitions
2022-08-12 13:03:55 +01:00
d3dfc4221a
docs: router writes to partitions
...
Though it can write to different topics, it currently writes to
different partitions.
2022-08-12 13:48:04 +02:00
d79f63b7bc
Merge pull request #5380 from influxdata/dom/bump-rskafka
...
build: bump rskafka
2022-08-12 09:27:39 +01:00
dbe6b4947c
Merge branch 'main' into dom/bump-rskafka
2022-08-12 09:20:37 +01:00
7118334774
build: bump rskafka
...
Bump rskafka to pick up connection pre-warming:
https://github.com/influxdata/rskafka/pull/166
2022-08-12 10:13:25 +02:00
f0ae8c093b
chore(deps): Bump libc from 0.2.129 to 0.2.131 ( #5379 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.129 to 0.2.131.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/commits )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-12 07:53:06 +00:00
b820f69b05
Merge pull request #5378 from influxdata/cn/update-rust
...
chore: Upgrade to Rust 1.63
2022-08-11 19:54:16 +00:00
30ac1c7725
fix: Macros need to be linked from their place in the crate root
...
I'm not sure why this just started failing now, but sure.
2022-08-11 15:28:18 -04:00
d4a472f775
fix: Remove let bindings that are immediately returned
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#let_and_return
2022-08-11 15:21:02 -04:00
549a267e3c
fix: Use Self instead of unnecessary structure name repetition
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#use_self
2022-08-11 15:21:02 -04:00
ee91a91700
fix: Remove needless returns
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#needless_return
2022-08-11 15:21:01 -04:00
d8d29bc4b4
fix: Reduce type complexity
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity
2022-08-11 15:04:15 -04:00
3a501a4a10
fix: Remove an immediate ref to a deref
...
Caught by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#borrow_deref_ref
2022-08-11 15:04:14 -04:00
Marco Neumann
kodiakhq[bot]
Luke Bond
Andrew Lamb
dependabot[bot]
pierwill
Dom Dwyer
Carol (Nichols || Goulding)