7c6ccdb6d7
fix: Use keys and values functions. Thanks clippy!
2022-12-21 14:32:35 -05:00
bfc74db94c
fix: Use into_values function. Thanks clippy!
2022-12-21 14:32:35 -05:00
a082ae85f2
fix: Use clamp function. Thanks clippy!
2022-12-21 14:32:34 -05:00
39acfc4f0d
fix: Remove needless casts. Thanks clippy!
2022-12-21 14:32:34 -05:00
72aab99951
fix: Remove needless borrow. Thanks clippy!
2022-12-21 14:32:34 -05:00
a672fea405
chore: Update to Rust 1.66
2022-12-21 13:31:31 -05:00
b0cf20eb93
Merge pull request #6454 from influxdata/dom/persist-saturation-metric
...
feat(ingester2): persist saturation metric
2022-12-21 17:21:05 +00:00
6df3c1d4ca
Merge branch 'main' into dom/persist-saturation-metric
2022-12-21 17:07:42 +00:00
ff9a05d157
Merge pull request #6456 from influxdata/dom/fix-sort-key-retry
...
fix(persist): invalidate cached sort key
2022-12-21 17:07:23 +00:00
c85c25e963
Merge branch 'main' into dom/persist-saturation-metric
2022-12-21 17:05:29 +00:00
23dc2c4e06
refactor: consistent metric naming
...
Removes _ns (and incorrect _ms) suffix.
2022-12-21 18:04:20 +01:00
7f01911c81
Merge branch 'main' into dom/fix-sort-key-retry
2022-12-21 17:00:22 +00:00
38d7102710
Merge pull request #6455 from influxdata/cn/why-querier-why
...
fix: Allow partitions from ingesters to overlap in RPC write mode
2022-12-21 16:56:57 +00:00
9cdf512ed1
Merge branch 'main' into dom/fix-sort-key-retry
2022-12-21 16:50:14 +00:00
125228eb8e
Merge branch 'main' into cn/why-querier-why
2022-12-21 16:50:11 +00:00
23b781f274
fix(persist): invalidate cached sort key
...
The sort-key conflict path invalidated the cached sort key in the
PartitionData, but not the cached sort key in the persist's Context. Now
both are invalidated.
2022-12-21 17:45:48 +01:00
56ba3b17de
fix: Allow partitions from ingesters to overlap in RPC write mode
...
This was added in c82d0d8ca6dc02dcdd40a4c656a1ee51f3f9bfee with the
comment:
> Right now this would clearly indicate a bug and before I am trying to
> understand some prod issues, I wanna rule that one out.
In the RPC write path, this isn't a bug, it's quite expected.
2022-12-21 11:32:58 -05:00
ae296886c2
chore(deps): Bump prost-types from 0.11.2 to 0.11.5 ( #6452 )
...
* chore(deps): Bump prost-types from 0.11.2 to 0.11.5
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.11.2 to 0.11.5.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.2...v0.11.5 )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 16:31:35 +00:00
257c155d1e
chore: Line wrapping at 100 cols
2022-12-21 11:18:47 -05:00
679c6a7896
feat(ingester2): persist saturation metric
...
Expose a metric ("ingester_persist_saturated_duration_ns") that records
the cumulative duration of time the persist system has spent in the
"saturated" state.
2022-12-21 17:01:22 +01:00
ed17069087
chore(deps): Bump num_cpus from 1.14.0 to 1.15.0 ( #6451 )
...
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus ) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases )
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md )
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0 )
---
updated-dependencies:
- dependency-name: num_cpus
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-21 15:17:55 +00:00
7abef9b7bd
Merge pull request #6453 from influxdata/dom/persist-worker
...
refactor(persist): explicit worker module
2022-12-21 14:18:11 +00:00
15cff11b08
refactor(persist): explicit worker module
...
Separate out persist worker types & routines into a separate worker
module rather than commingling them with the persist handle, and rename
the unimaginative "inner" to reflect the actual usage.
2022-12-21 14:28:30 +01:00
e1059a9009
feat: FlightSQL Milestone 2 basic FlightSQL client and FlightSQL server implementation and plumbing ( #6398 )
...
* feat: Add basic Flight and FlightSQL client into IOx codebase
Basic flight end to end test
* fix: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-20 17:34:00 +00:00
d1845057cf
Merge pull request #6448 from influxdata/dom/retry-cas-sort-key-failure
...
fix(ingester2): handle concurrent sort key updates
2022-12-20 16:20:49 +00:00
7b133f85a1
docs: rust doclink failure
...
Rustdoc is so picky about indented text.
2022-12-20 17:13:24 +01:00
b3363639f5
chore: nudge CI
2022-12-20 17:05:03 +01:00
5f4acf186d
docs: fix bad doc link
...
Rust hates indented URLs.
2022-12-20 15:25:34 +01:00
e083f3276c
feat(persist): accept concurrent matching updates
...
As an optimisation, allow a persist task to progress if it observes a
concurrent catalog sort key update that exactly matches the sort key it
was committing.
2022-12-20 15:15:39 +01:00
f64ffbe035
fix(ingester2): handle concurrent sort key updates
...
Allow an ingester2 instance to tolerate concurrent partition sort key
updates in the catalog.
A persist job is optimistically executed with the locally cached sort
key. If an ingester2 instance observes a concurrent update, it aborts
both the sort key update, and the overall persist operation (before
making the parquet file visible) and retries the operation with the
newly observed sort key. Concurrent sort key updates are theorised to be
relatively rare overall.
Any orphaned parquet files uploaded as part of a persist job that aborts
due to a concurrent sort key update are eventually removed by the
(external) object store GC task.
See https://github.com/influxdata/influxdb_iox/issues/6439
2022-12-20 15:15:39 +01:00
925685f3a3
Merge pull request #6447 from influxdata/dom/cas-sort-key-catalog
...
feat(catalog): linearise sort key updates
2022-12-20 12:11:02 +00:00
2f88fc71ce
docs: fix incomplete comment
2022-12-20 12:31:01 +01:00
adc6fcfb04
feat(catalog): linearise sort key updates
...
Updating the sort key is not commutative and MUST be serialised. The
correctness of the current catalog interface relies on the caller
serialising updates globally, something it cannot reasonably assert in a
distributed system.
This change of the catalog interface pushes this responsibility to the
catalog itself where it can be effectively enforced, and allows a caller
to detect parallel updates to the sort key.
2022-12-20 12:31:00 +01:00
3e2362ea9b
fix: Do not panic when bad arguments are passed to selectors ( #6441 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-20 10:37:56 +00:00
96fe89bd3e
Merge pull request #6445 from influxdata/dependabot/cargo/snafu-0.7.4
...
chore(deps): Bump snafu from 0.7.3 to 0.7.4
2022-12-20 10:25:46 +00:00
9d4b2b56ae
chore(deps): Bump snafu from 0.7.3 to 0.7.4
...
Bumps [snafu](https://github.com/shepmaster/snafu ) from 0.7.3 to 0.7.4.
- [Release notes](https://github.com/shepmaster/snafu/releases )
- [Changelog](https://github.com/shepmaster/snafu/blob/main/CHANGELOG.md )
- [Commits](https://github.com/shepmaster/snafu/compare/0.7.3...0.7.4 )
---
updated-dependencies:
- dependency-name: snafu
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-20 01:04:43 +00:00
d0d5906476
chore: Update datafusion pin ( #6442 )
...
* chore: Update datafusion pin
* refactor: Update iox_query for new apis
* chore: Update some more apis
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-19 20:02:42 +00:00
200f4fe9bd
fix: Disable parquet file filtering in the querier based on max seq num in RPC write mode ( #6443 )
...
Connects to #6421 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 18:01:21 +00:00
9b22ede3f0
refactor: Make arrow flight client return `futures::Streams` ( #6438 )
...
* refactor: Make arrow flight client use futures::Streams
* refactor: concision
2022-12-19 17:09:26 +00:00
dfa70269cb
fix: Make multiple ingester addresses in the router work ( #6440 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 16:27:57 +00:00
f469854a0d
Merge pull request #6436 from influxdata/dom/query-types-refactor
...
refactor(ingester2): query types
2022-12-19 15:10:53 +00:00
dbbe43f241
Merge branch 'main' into dom/query-types-refactor
2022-12-19 15:03:16 +00:00
17c85cda25
Merge pull request #6408 from influxdata/cn/compactor2
...
feat: Add a compactor2 command that doesn't limit what it tries to compact by shard
2022-12-19 14:31:31 +00:00
a2166f8d8c
Merge branch 'main' into cn/compactor2
2022-12-19 14:24:37 +00:00
fb50d2ad31
Merge pull request #6437 from influxdata/dom/fix-comment
...
docs: fix incomplete comment
2022-12-19 14:24:24 +00:00
c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
84e29791e5
docs: fix incomplete comment
...
Finishes the incomplete sentence that
2022-12-19 12:46:21 +01:00
94c2f94ea1
refactor: Extract common ArrowFlight client into iox_arrow_flight ( #6427 )
...
* refactor: Extract common ArrowFlight client into iox_arrow_flight
* chore: Run cargo hakari tasks
* fix: clarify intent of iox_arrow_flight crate
* refactor: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: loop --> while let
* fix: REmove make_tonic_error in favor of From impl
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 11:35:20 +00:00
2ba564eaed
Merge pull request #6431 from influxdata/dependabot/cargo/thiserror-1.0.38
...
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
2022-12-19 11:09:55 +00:00
299f0e99f9
chore(deps): Bump thiserror from 1.0.37 to 1.0.38
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.37 to 1.0.38.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.37...1.0.38 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 10:33:32 +00:00
Carol (Nichols || Goulding)
Dom
kodiakhq[bot]
dependabot[bot]
Andrew Lamb
Carol (Nichols || Goulding)