48466bfa89
feat(metrics): bytes/frames sent/received & peers
...
Emit metrics tracking the number of bytes sent / received, and number of
frames sent / received by the local node.
Track the number of discovered peers to record peer discovery rate and
current number of known peers per node.
2023-07-10 12:11:15 +02:00
93789d7abb
feat: refuse oversized gossip payloads
...
Calculate the available byte size for a user payload sent via gossip,
and pro-actively check this limit earlier, when the caller is attempting
to send the frame, rather than later in the reactor where there's no
feedback to the caller.
DRY frame serialisation to simplify enforcement, and validate/refuse
oversized frames in the reactor so that frames are unlikely to be
truncated by receivers.
2023-07-10 12:11:14 +02:00
bc9ebc9c66
feat: gossip primitive
...
Adds a simple "gossip" implementation (more accurately described as a
pub/sub primitive currently) that supports broadcasting
application-level messages to the set of active peers.
This implementation uses UDP as a transport for best-effort delivery,
and enables zero-copy use of the payload using the Bytes crate.
Only peers explicitly provided as "seeds" when initialising will be
known to a gossip node - there's currently no peer exchange mechanism.
This implementation tolerates seeds changing their DNS entries when
restarting to point at new socket addresses (such as within Kubernetes
when pods move around).
2023-07-10 12:11:14 +02:00
48aa4a5e33
feat(gossip): frame proto definitions
...
Adds a proto definition and configures prost to build the rust types
from it.
The gossip framing is intended to be flexible and decoupled - the gossip
library will batch together one or more opaque application messages
and/or control frames, and uniquely identify each peer with a
per-instance UUID to detect crashes/restarts and track peers.
2023-07-10 12:11:13 +02:00
5c191ce6cf
ci: enable standard lint set
...
Adds the somewhat "standard" lint set we use to the gossip lib.
2023-07-10 12:11:12 +02:00
69ab70ce99
feat: init gossip package
...
Adds a new empty "gossip" package to the workspace.
2023-07-10 12:11:12 +02:00
5027c9a88c
chore: sort workspace members
...
Sort the package names in the workspace member declaration.
2023-07-10 12:11:03 +02:00
3ce11d8d66
chore: Update DataFusion ( #8190 )
...
* chore: Update DataFusion
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: use display format
* chore: Update explain plan output
* fix: update plans
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-10 09:54:50 +00:00
1f1b17f712
Merge pull request #8188 from influxdata/dom/partition-query-concurrency
...
test(bench): concurrent ingester partition queries
2023-07-10 09:41:03 +00:00
19b59d9de5
Merge branch 'main' into dom/partition-query-concurrency
2023-07-10 09:35:22 +00:00
1f82f6b059
chore(deps): Bump snafu from 0.7.4 to 0.7.5 ( #8193 )
...
Bumps [snafu](https://github.com/shepmaster/snafu ) from 0.7.4 to 0.7.5.
- [Changelog](https://github.com/shepmaster/snafu/blob/main/CHANGELOG.md )
- [Commits](https://github.com/shepmaster/snafu/compare/0.7.4...0.7.5 )
---
updated-dependencies:
- dependency-name: snafu
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-10 09:35:10 +00:00
d395a4907a
Merge branch 'main' into dom/partition-query-concurrency
2023-07-10 10:34:43 +01:00
f0789b74ce
chore(deps): Bump regex from 1.9.0 to 1.9.1 ( #8195 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.9.0 to 1.9.1.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.9.0...1.9.1 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-10 09:29:12 +00:00
341dcf2124
Merge branch 'main' into dom/partition-query-concurrency
2023-07-10 10:24:09 +01:00
ab16180f15
chore(deps): Bump serde from 1.0.167 to 1.0.168 ( #8194 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.167 to 1.0.168.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.167...v1.0.168 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-10 09:24:01 +00:00
d87f69a76c
Merge pull request #8197 from influxdata/dependabot/cargo/async-channel-1.9.0
...
chore(deps): Bump async-channel from 1.8.0 to 1.9.0
2023-07-10 10:17:46 +01:00
12317fee23
chore(deps): Bump async-channel from 1.8.0 to 1.9.0
...
Bumps [async-channel](https://github.com/smol-rs/async-channel ) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/smol-rs/async-channel/releases )
- [Changelog](https://github.com/smol-rs/async-channel/blob/master/CHANGELOG.md )
- [Commits](https://github.com/smol-rs/async-channel/compare/v1.8.0...v1.9.0 )
---
updated-dependencies:
- dependency-name: async-channel
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-07-10 01:42:26 +00:00
c2442c31f3
chore: create partition table index for created_at
2023-07-07 16:27:05 -05:00
b5faa37152
fix: Plumb tracing header name env/flag to client ( #8189 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-07 21:07:29 +00:00
048fc32bd5
feat: add `influxdb_iox debug build-catalog` command ( #8067 )
...
* feat: add `influxdb_iox debug build-catalog` command
* fix: tests
* fix: Use info! logs instead of println for status
* fix: Set partition_hash_id as well
* fix: remove leftover code
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-07 18:32:27 +00:00
ea38e93511
test(bench): concurrent partition queries
...
Benchmark the performance of concurrent queries against a single
partition, varying the number of concurrent queries and size of buffered
data in the partition.
2023-07-07 16:27:44 +02:00
1736333e32
Merge pull request #8173 from influxdata/jrb_58_throttle_partition_processing
...
feat: throttle partition evaluation in the compactor
2023-07-07 08:52:30 -05:00
ec6a609f63
Merge branch 'main' into jrb_58_throttle_partition_processing
2023-07-07 08:46:22 -05:00
5239540b1d
refactor: replace `println!` w/ `info!` ( #8185 )
...
`println!` is:
- confusing: like why is everything colored but this few lines are
special?!
- annoying: you cannot filter them which esp. for debugging is a pain
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-07 13:05:39 +00:00
115eda07d6
Merge pull request #8070 from influxdata/savage/remove-op-level-sequence-number-for-writes
...
refactor(wal): Remove op-level `sequence_number`, use per table map
2023-07-07 10:17:13 +00:00
e06b6987f0
Merge branch 'main' into savage/remove-op-level-sequence-number-for-writes
2023-07-07 10:12:04 +00:00
18d5b51a78
chore(deps): Bump sysinfo from 0.29.2 to 0.29.4 ( #8183 )
...
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) from 0.29.2 to 0.29.4.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits )
---
updated-dependencies:
- dependency-name: sysinfo
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:43:37 +00:00
8b000862e1
chore(deps): Bump pretty_assertions from 1.3.0 to 1.4.0 ( #8182 )
...
Bumps [pretty_assertions](https://github.com/rust-pretty-assertions/rust-pretty-assertions ) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/rust-pretty-assertions/rust-pretty-assertions/releases )
- [Changelog](https://github.com/rust-pretty-assertions/rust-pretty-assertions/blob/main/CHANGELOG.md )
- [Commits](https://github.com/rust-pretty-assertions/rust-pretty-assertions/compare/v1.3.0...v1.4.0 )
---
updated-dependencies:
- dependency-name: pretty_assertions
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:35:18 +00:00
057ee40cb9
chore(deps): Bump thiserror from 1.0.41 to 1.0.43 ( #8181 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.41 to 1.0.43.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.41...1.0.43 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:25:12 +00:00
9df1186472
chore(deps): Bump is-terminal from 0.4.8 to 0.4.9 ( #8180 )
...
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.8 to 0.4.9.
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.8...v0.4.9 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-07 09:16:43 +00:00
778299112b
chore(deps): Bump serde from 1.0.166 to 1.0.167 ( #8179 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.166 to 1.0.167.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.166...v1.0.167 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-07 09:05:08 +00:00
51a70f0ae9
Merge pull request #8184 from influxdata/sgc/issue/8168_integer_division
...
fix: InfluxQL planner promotes integer division to floats
2023-07-07 04:40:51 +00:00
1ca547b313
fix: Teach planner to rewrite binary expressions for div operator
...
Specifically when the operands are integers, to match InfluxQL OG
2023-07-07 11:22:03 +10:00
05dacf7fe7
fix: Teach type evaluator about division when operands are integers
2023-07-07 11:20:54 +10:00
65f928f227
chore: unwrap option in log ( #8178 )
2023-07-06 23:01:39 +00:00
afb899b204
chore: add logging when compactor fetches partitions ( #8176 )
2023-07-06 21:51:04 +00:00
eb162c775b
feat: log query and namespace on querier erorr ( #8174 )
2023-07-06 19:05:30 +00:00
9f522bfd30
Revert "feat(idpe-17789): scheduler job_status() ( #8121 )" ( #8175 )
...
This reverts commit 5d19fa3635
2023-07-06 18:52:25 +00:00
28eb3dcd92
feat: throttle partition evaluation in the compactor
2023-07-06 11:54:18 -05:00
197078d6c7
feat(ingester): Use WAL reference actor for file deletion during graceful shutdown
...
This change integrates the WAL reference actor with the graceful
shutdown buffer drain & persist behaviour, relying on its knowledge of
partial persistence for deletion and shutdown timing.
2023-07-06 17:18:12 +01:00
5d19fa3635
feat(idpe-17789): scheduler job_status() ( #8121 )
...
This block of work moves into the scheduler some of the specific downstream actions affiliated with compaction outcomes. Which responsibilities stay in the compactor, versus moved to the scheduler, roughly followed the heuristic of whether the action (a) had an impact on global catalog state (a.k.a. commits and partition skipping), (b) whether it's logging affiliated with compactor health (e.g. ParitionDoneSink logging outcomes) versus system health (e.g. logging commits), and (c) reporting to the scheduler on any errors encountered during compaction. This boundary is subject to change as we move forward.
Also, a noted caveat (TODO) on this commit. We have a CompactionJob which is used to track work handed off to each compactor. Currently it still uses the partition_id for tracking, but the followup PR will start moving the compactor to have more CompactionJob uuid awareness.
2023-07-06 09:15:59 -07:00
ecc5072197
Merge pull request #8006 from influxdata/7899/wal-disk-metrics
...
feat(7899): wal disk metric
2023-07-06 15:04:06 +01:00
a005f344d8
Merge branch 'main' into 7899/wal-disk-metrics
2023-07-06 14:44:11 +01:00
d46a5f0c51
refactor: update metrics out of phase
...
Emit the metrics out-of-phase with the metric scraper.
2023-07-06 15:36:30 +02:00
d979739576
perf: read disks once, resolve mount point once
...
Instead of refreshing every metric in the System every 10 seconds,
refresh only the disk statistics for the disk we're interested in.
Additionally resolve the parent disk for the directory path once,
instead of each loop.
2023-07-06 15:33:35 +02:00
b2c1532d3a
chore(deps): Bump regex from 1.8.4 to 1.9.0 ( #8160 )
...
* chore(deps): Bump regex from 1.8.4 to 1.9.0
Bumps [regex](https://github.com/rust-lang/regex ) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.8.4...1.9.0 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-06 12:48:27 +00:00
bbcf3f0556
chore(deps): Bump toml from 0.7.5 to 0.7.6 ( #8161 )
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.7.5 to 0.7.6.
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.7.5...toml-v0.7.6 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-06 12:35:47 +00:00
c55367366f
fix: return error rather than panic on unsupported predicate ( #8158 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-06 11:29:22 +00:00
bea9dbf7ab
test(ingester): Integration test dropping of persisted WAL segments
...
This adds an integration test that writes some data to the ingester,
waits for the WAL to be rotated and then ensures that the segment file
has been dropped.
2023-07-06 12:12:58 +01:00
a6df3a4b36
chore(deps): Bump clap from 4.3.10 to 4.3.11 ( #8162 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.3.10 to 4.3.11.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.3.10...v4.3.11 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-06 10:15:25 +00:00
Dom Dwyer
Andrew Lamb
kodiakhq[bot]
dependabot[bot]
Joe-Blount
Marko Mikulicic
Joe-Blount
Marco Neumann
Stuart Carnie
Fraser Savage
wiedld