724759862f
chore(deps): Bump pin-project from 1.0.11 to 1.0.12 ( #5402 )
...
Bumps [pin-project](https://github.com/taiki-e/pin-project ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/taiki-e/pin-project/releases )
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md )
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.11...v1.0.12 )
---
updated-dependencies:
- dependency-name: pin-project
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-16 08:04:25 +00:00
0ccefa0d0c
refactor: port TTL backend to policy framework ( #5396 )
...
* refactor: port TTL backend to policy framework
Note that this is "just" a port, it does NOT change how TTL works. This
will be done in #5318 .
Helps with #5320 .
* fix: ensure inner backend is empty
* test: add some smoke test
2022-08-15 16:48:16 +00:00
b2caf54b3a
docs: clarify clamping behavior in `TimestampRange::new` ( #5399 )
...
Closes #5248 .
2022-08-15 12:43:01 +00:00
0a44a42999
chore(deps): Bump futures from 0.3.21 to 0.3.23 ( #5394 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 12:19:17 +00:00
69ccfafd14
chore(deps): Bump futures-task from 0.3.21 to 0.3.23 ( #5393 )
...
Bumps [futures-task](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-task
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 10:27:22 +00:00
2c934addf5
chore(deps): Bump futures-channel from 0.3.21 to 0.3.23 ( #5388 )
...
Bumps [futures-channel](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-channel
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 09:42:53 +00:00
a14e161109
chore(deps): Bump futures-core from 0.3.21 to 0.3.23 ( #5387 )
...
Bumps [futures-core](https://github.com/rust-lang/futures-rs ) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23 )
---
updated-dependencies:
- dependency-name: futures-core
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 08:59:53 +00:00
6a4425ae08
chore(deps): Bump chrono from 0.4.21 to 0.4.22 ( #5392 )
...
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.21 to 0.4.22.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.21...v0.4.22 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:31:23 +00:00
8a5cd1cddd
refactor: abstract change requests for cache policies ( #5382 )
...
* refactor: abstract change requests for cache policies
Tl;Dr; Lets replace the hard-coded enum of change requests with a more
flexible and easier-to-understand function-based approach. It also
unifies the interface for "initial requests" and "reactions".
Story time:
I just wanted to port over the "shared" backend with its "remove if"
functionality to the policy framework. Turns out that this backend does
not just use simple atomic operations but is basically a
"compare&exchange"-like operation (technically: GET+REMOVE) that relies
on a single mutable access lifetime to make sure that this behavior is
sane and nobody messes with the value between the GET and REMOVE. I call
this a "compound request".
The new policy framework did not support this kind of requests so I
thought how I could shoehorn it in. Basically I wanted to extend
`ChangeRequest` to support these kind of operations. But listing
combinations explicitly kinda seemed like some hack. So my brain slowly
came up with an abstract approach how to encode these operations and
after a while I was close to design some kind of mini-VM. I also
considered some ugly workarounds or to not express the "remove if"
functionality" as a policy. My nice vision of a policy framework started
to crack...
This is when sanity kicked in and I realized: every change request is
technically just a function on a (virtual) cache backend. Sure,
all-powerful Rust functions allow you to do ugly stuff that will result
in deadlocks, but with good docs and helpers for the common operations
the risk is very low. Furthermore this is already close to the system I
had for the "initial requests" and which I called a "callback backend".
So I unified both systems and made the change request abstract and all
tests pass and I think it is the better design.
Helps with #5320 .
* docs: typos
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:24:25 +00:00
38277cf0fb
chore(deps): Bump nix from 0.24.2 to 0.25.0 ( #5391 )
...
* chore(deps): Bump nix from 0.24.2 to 0.25.0
Bumps [nix](https://github.com/nix-rust/nix ) from 0.24.2 to 0.25.0.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.2...v0.25.0 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-15 08:10:09 +00:00
822ec18fcd
chore(deps): Bump clap from 3.2.16 to 3.2.17 ( #5390 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.16 to 3.2.17.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/v3.2.17/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.16...v3.2.17 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-15 07:56:11 +00:00
44d72b2e94
Merge pull request #5384 from influxdata/dom/parallel-init
...
refactor: parallelise Kafka partition client init
2022-08-12 14:01:22 +01:00
bd88ac6149
refactor: parallelise Kafka partition client init
...
Changes the Kafka write buffer impl to parallelise initialisation of the
PartitionClient instances.
Now that the PartitionClient constructor also performs leader discovery
(using cached metadata, influxdata/rskafka#164 ) and establishes a broker
connection (influxdata/rskafka#166 ) executing them in parallel will
cause a proportional decrease in the time taken to bring IOx up.
2022-08-12 14:45:23 +02:00
b2e50f0c1a
Merge pull request #5383 from influxdata/dom/deploy
...
docs: router writes to partitions
2022-08-12 13:03:55 +01:00
d3dfc4221a
docs: router writes to partitions
...
Though it can write to different topics, it currently writes to
different partitions.
2022-08-12 13:48:04 +02:00
d79f63b7bc
Merge pull request #5380 from influxdata/dom/bump-rskafka
...
build: bump rskafka
2022-08-12 09:27:39 +01:00
dbe6b4947c
Merge branch 'main' into dom/bump-rskafka
2022-08-12 09:20:37 +01:00
7118334774
build: bump rskafka
...
Bump rskafka to pick up connection pre-warming:
https://github.com/influxdata/rskafka/pull/166
2022-08-12 10:13:25 +02:00
f0ae8c093b
chore(deps): Bump libc from 0.2.129 to 0.2.131 ( #5379 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.129 to 0.2.131.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/commits )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-12 07:53:06 +00:00
b820f69b05
Merge pull request #5378 from influxdata/cn/update-rust
...
chore: Upgrade to Rust 1.63
2022-08-11 19:54:16 +00:00
30ac1c7725
fix: Macros need to be linked from their place in the crate root
...
I'm not sure why this just started failing now, but sure.
2022-08-11 15:28:18 -04:00
d4a472f775
fix: Remove let bindings that are immediately returned
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#let_and_return
2022-08-11 15:21:02 -04:00
549a267e3c
fix: Use Self instead of unnecessary structure name repetition
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#use_self
2022-08-11 15:21:02 -04:00
ee91a91700
fix: Remove needless returns
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#needless_return
2022-08-11 15:21:01 -04:00
d8d29bc4b4
fix: Reduce type complexity
...
As now caught by clippy. https://rust-lang.github.io/rust-clippy/master/index.html#type_complexity
2022-08-11 15:04:15 -04:00
3a501a4a10
fix: Remove an immediate ref to a deref
...
Caught by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#borrow_deref_ref
2022-08-11 15:04:14 -04:00
b982bdaf2f
fix: Derive Eq when we derive PartialEq and members can derive Eq
...
Allow this in generated code that we don't control, though.
Recommended by clippy now. https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq
2022-08-11 15:04:06 -04:00
545a1a5228
chore: Upgrade to Rust 1.63
2022-08-11 15:04:00 -04:00
6b8afef2ea
Merge pull request #5377 from influxdata/dom/bump-rskafka
...
build: bump rskafka
2022-08-11 17:05:32 +01:00
faa1db9a24
build: bump rskafka
...
Bump rskafka & fix minor breakage in order to pick up client
pre-warming:
https://github.com/influxdata/rskafka/pull/165
2022-08-11 17:26:06 +02:00
26f3849191
feat: improve cache policies and their testing ( #5375 )
...
1. add GET support for the subscribers (this is needed so that
TTL/refreshers and LRU systems "know" when a key was used)
2. improve multi-threading test to not rely on a wait loop but use a 2nd
barrier instead
3. ensure that panics in the testing background thread are propagated
and make the test fail
4. implement defaults for `Subscriber` methods to reduce boilerplate for
implementators
Helps with #5320 .
2022-08-11 13:53:22 +00:00
90fec1365f
feat: intern schemas during query planning ( #5215 )
...
* feat: intern schemas during query planning
Helps with #5202 .
* refactor: `SchemaMerger::build` shall return an `Arc`
* feat: `SchemaMerger::with_interner`
* refactor: hash-based schema interning
2022-08-11 12:28:51 +00:00
4867c6b682
Merge pull request #5376 from influxdata/dom/rskafka-bump
...
build: bump rskafka
2022-08-11 12:20:25 +00:00
7174f38f3f
build: bump rskafka
...
Bumps rskafka to HEAD to pick up:
https://github.com/influxdata/rskafka/pull/164
2022-08-11 14:00:21 +02:00
ae6ac27960
chore(deps): Bump console-subscriber from 0.1.6 to 0.1.7 ( #5374 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.6 to 0.1.7.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.6...tokio-console-v0.1.7 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-11 11:08:10 +00:00
13f2a2ebc7
feat: new policy system for caches ( #5370 )
...
* feat: new policy system for caches
This is the framework part for the policy system outlined in #5320 . It
does NOT port any existing policies (likely TTL, LRU and shared) over to
the new system. This will be a follow-up.
* docs: improve example
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-08-11 09:41:17 +00:00
692ec97c2f
chore(deps): Bump ahash from 0.7.6 to 0.8.0 ( #5373 )
...
* chore(deps): Bump ahash from 0.7.6 to 0.8.0
Bumps [ahash](https://github.com/tkaitchuck/ahash ) from 0.7.6 to 0.8.0.
- [Release notes](https://github.com/tkaitchuck/ahash/releases )
- [Commits](https://github.com/tkaitchuck/ahash/compare/v0.7.6...v0.8.0 )
---
updated-dependencies:
- dependency-name: ahash
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
* fix: `ahash` features
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-11 08:37:31 +00:00
6b8b922fe7
fix: do not loose data when Kafka reports that offset is above watermark ( #5322 )
...
* fix: do not loose data when Kafka reports that offset is above watermark
This can happen in certain cluster rebalance settings.
This is also linked to https://github.com/influxdata/rskafka/issues/147
but for the upstream issue I currently have no idea how to fix it, so
let's at least harden IOx against it.
Fixes #5128 .
* refactor: panic for `SequenceNumberAfterWatermark`
2022-08-11 07:32:04 +00:00
3a945dbcb2
chore: return a struct with named and documented fields from `compact_persisting_batch` ( #5346 )
...
* chore: return a struct with named and documented fields from `compact_persisting_batch`
* docs: Remove extra 'the' and fix a typo
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-08-10 20:22:29 +00:00
b834bc630c
chore: more readability improvements to sort keys ( #5366 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-10 17:59:25 +00:00
07729bf153
Merge pull request #5334 from influxdata/cn+jg/compactor-e2e-tests
...
feat: Compactor e2e tests and a "run once" compactor command
2022-08-10 16:19:06 +00:00
5a672dc3da
Merge branch 'main' into cn+jg/compactor-e2e-tests
2022-08-10 16:11:50 +00:00
061444a23c
chore: added helper script for a docker catalog ( #5372 )
2022-08-10 15:49:40 +00:00
1b77abdda7
fix: Explain the purpose of this macro, make arg name better
2022-08-10 11:33:43 -04:00
96acf3c54b
fix: Don't rustfmt this module because of a rustfmt bug
2022-08-10 11:30:39 -04:00
75bdd470a2
docs: Rewrap doc comments to 100 cols now that they're indented more
2022-08-10 11:30:22 -04:00
9321c96aaf
test: Add a step for running compaction in e2e tests
2022-08-10 11:30:22 -04:00
8ec9117836
test: Log a command run by e2e tests in a convenient way
...
So that you can copy-paste it if you want to run it again! 🎉
2022-08-10 11:30:22 -04:00
9981d1636b
refactor: Move dump_log_to_stdout so it can be used in more than server fixtures
2022-08-10 11:30:22 -04:00
3915841a53
feat: Introduce a separate config for the compactor command
2022-08-10 11:30:21 -04:00
dependabot[bot]
Marco Neumann
Dom
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Luke Bond
Jake Goulding