149de84313
chore(deps): Bump serde from 1.0.150 to 1.0.151 ( #6432 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.150 to 1.0.151.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.150...v1.0.151 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:24:11 +00:00
ecd6cfa3a8
chore(deps): Bump syn from 1.0.105 to 1.0.107 ( #6434 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.105 to 1.0.107.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/commits )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:17:11 +00:00
c72734473c
chore(deps): Bump async-trait from 0.1.59 to 0.1.60 ( #6433 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.59 to 0.1.60.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.59...0.1.60 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-19 10:09:23 +00:00
7f2aa8b10c
chore(deps): Bump serde_json from 1.0.89 to 1.0.91
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.89 to 1.0.91.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.89...v1.0.91 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 01:44:18 +00:00
84698b3532
feat: add size_data to mutable batch ( #6425 )
...
This method will be used in the new ingestion pipeline to approximate how much memory a butable batch will take to convert to arrow and persist. It is meant only as a very rough estimate to trigger persistence for hot partitions.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-16 17:20:16 +00:00
cfb7c16bb1
Merge branch 'main' into dom/optimal-persist-parallelism
2022-12-16 09:12:22 +00:00
6a841bdf5a
chore(deps): Bump cc from 1.0.77 to 1.0.78
...
Bumps [cc](https://github.com/rust-lang/cc-rs ) from 1.0.77 to 1.0.78.
- [Release notes](https://github.com/rust-lang/cc-rs/releases )
- [Commits](https://github.com/rust-lang/cc-rs/compare/1.0.77...1.0.78 )
---
updated-dependencies:
- dependency-name: cc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-16 01:08:24 +00:00
aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
...
And standardize on ingester2 and router2 for consistency.
Connects to #6402 .
2022-12-15 14:06:59 -05:00
933ab1f8c7
feat(ingester2): optimal persist parallelism
...
This commit changes the behaviour of the persist system to enable
optimal parallelism of persist operations, and improve the accuracy of
the outstanding job bound / back-pressure.
Previously all persist operations for a given partition were
consistently hashed to a single worker task. This serialised persistence
per partition, ensuring all updates to the partition sort key were
serialised. However, this also unnecessarily serialises persist
operations that do not need to update the sort key, reducing the
potential throughput of the system; in the worst case of a single
partition receiving all the writes, only one worker would be persisting,
and the other N-1 workers would be idle.
After this change, the sort key is inspected when enqueuing the persist
operation and if it can be determined that no sort key update is
necessary (the typical case), then the persist task is placed into a
global work queue from which all workers consume. This allows for
maximal parallelisation of these jobs, and the removes the per-worker
head-of-line blocking.
In the case that the sort key does need updating, these jobs continue to
be consistently hashed to a single worker, ensuring serialised sort key
updates only where necessary.
To support these changes, the back-pressure system has been changed to
account for all outstanding persist jobs in the system, regardless of
type or assigned worker - a logical, bounded queue is composed together
of a semaphore limiting the number of persist tasks overall, and a
series of physical, unbounded queues - one to each worker & the global
queue. The overall system remains bounded by the
INFLUXDB_IOX_PERSIST_QUEUE_DEPTH value, and is now simpler to reason
about (it is independent of the number of workers, etc).
2022-12-15 18:30:51 +01:00
6324707110
chore(deps): Bump backtrace from 0.3.66 to 0.3.67 ( #6410 )
...
Bumps [backtrace](https://github.com/rust-lang/backtrace-rs ) from 0.3.66 to 0.3.67.
- [Release notes](https://github.com/rust-lang/backtrace-rs/releases )
- [Commits](https://github.com/rust-lang/backtrace-rs/compare/0.3.66...0.3.67 )
---
updated-dependencies:
- dependency-name: backtrace
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-15 09:23:12 +00:00
f82f516c0d
chore(deps): Bump toml from 0.5.9 to 0.5.10 ( #6409 )
...
Bumps [toml](https://github.com/toml-rs/toml ) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/toml-rs/toml/releases )
- [Commits](https://github.com/toml-rs/toml/commits/toml-v0.5.10 )
---
updated-dependencies:
- dependency-name: toml
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-15 08:48:53 +00:00
be45889be1
chore: Upgrade datafusion ( #6407 )
...
* chore: Update datafusion
* chore: Update for API change
2022-12-15 06:51:35 +00:00
d9c72bb93f
feat: optimize wal with batching ( #6399 )
...
* feat: optimize wal with batching
Simplified the wal writer so that it batches up write operations. Currently it waits 10ms between fsync calls. We can pull this out to a config variable later if we want, but I think this is good enough for now.
Also updated the reader to be a more simple blocking reader without the extra tasks and channels as that wasn't really getting us anything that I know of.
* chore: cleanup wal code for PR feedback
2022-12-14 16:07:20 +00:00
8729977851
chore: Upgrade datafusion / arrow to 29.0.0 to get flightsql client ( #6396 )
...
* chore: Update datafusion pin
* chore: Update for API change
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-13 20:16:09 +00:00
9e8ae1485f
Merge branch 'main' into dom/wal-bench
2022-12-13 15:19:32 +00:00
e108a8b6c9
chore(deps): Bump paste from 1.0.9 to 1.0.10 ( #6384 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.9 to 1.0.10.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.9...1.0.10 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-13 06:03:05 +00:00
727efcbdee
Merge branch 'main' into cn/ingester2-uuid
2022-12-12 16:21:15 +00:00
336ca761a3
chore: Update datafusion pin (to get sqlparser update) ( #6378 )
...
* chore: Update datafusion pin
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-12-12 14:39:42 +00:00
7c28a30d1b
test(ingester2): WAL replay benchmark
...
This adds a simple WAL replay benchmark to ingester2 that executes a
replay of a single line of LP.
Unfortunately each file in the benches directory is compiled as it's own
binary/crate, and as such is restricted to importing only "pub" types.
This sucks, as it requires you to either benchmark at a high level
(macro, not microbenchmarks - i.e. benchmarking the ingester startup,
not just the WAL replay) or you are forced to mark the reliant types &
functions as "pub", as well as all the other types/traits they reference
in their signatures. Because the performance sensitive code is usually
towards the lower end of the call stack, this can quickly lead to an
explosion of "pub" types causing a large amount of internal code to be
exported.
Instead this commit uses a middle-ground; benchmarked types & fns are
conditionally marked as "pub" iff the "benches" feature is enabled. This
prevents them from being visible by default, but allows the benchmark
function to call them.
The benchmark itself is also restricted to only run when this feature is
enabled.
2022-12-12 15:02:36 +01:00
e0ecacf6cc
chore: Update DataFusion (get median fix and automatic string to timestamp coercion) ( #6363 )
...
* chore: Update DataFusion pin to get median fix
* chore: Update for new Expr node
* test: add test for median
* test: add test for coercion of strings to timestamps
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 12:14:00 +00:00
95969ad24f
chore(deps): Bump base64 from 0.13.1 to 0.20.0 ( #6371 )
...
* chore(deps): Bump base64 from 0.13.1 to 0.20.0
Bumps [base64](https://github.com/marshallpierce/rust-base64 ) from 0.13.1 to 0.20.0.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases )
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/marshallpierce/rust-base64/compare/v0.13.1...v0.20.0 )
---
updated-dependencies:
- dependency-name: base64
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 07:07:19 +00:00
9305e4c566
chore(deps): Bump insta from 1.22.0 to 1.23.0 ( #6370 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.22.0...1.23.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-12 06:48:35 +00:00
a66895ecdc
chore(deps): Bump serde from 1.0.149 to 1.0.150 ( #6369 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.149 to 1.0.150.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.149...v1.0.150 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-12 06:35:16 +00:00
061d582a9b
chore: patch object_store to get apache#3274 ( #6362 )
...
* chore: patch object_store to get apache#3274
* chore: Run cargo hakari tasks
* fix: add issue breadcrumb
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 14:52:25 +00:00
0221820123
feat: rate-limit Jaeger UDP messages ( #6354 )
...
* feat: rate-limit Jaeger UDP messages
The Jaeger UDP protocol provides no way to signal backpressure /
overload. In certain situations, we are emitting that many tracing spans
in a short period of time that the OS, the network, or Jaeger drop them.
While a rate limit is not a perfect solution, it for sure helps a lot
(tested locally).
Note that the limiter does NOT lead to unlimited buffering because we
already have a limited outbox queue in place (see
`trace_exporters::export::CHANNEL_SIZE`).
Fixes #5446 .
* fix: only warn ones when the tracing channel is full
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-09 09:46:07 +00:00
62db312a8f
feat: Switch to escargot to get more control over running Cargo bins
2022-12-08 15:29:44 -05:00
9175f4a0b5
chore: Upgrade datafusion to get correct support for multi-part identifiers ( #6349 )
...
* test: add tests for periods in measurement names
* chore: Update Datafusion
* chore: Update for changed APIs
* chore: Update expected plan output
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 11:27:13 +00:00
8c5c6aee0b
chore(deps): Bump filetime from 0.2.18 to 0.2.19 ( #6350 )
...
Bumps [filetime](https://github.com/alexcrichton/filetime ) from 0.2.18 to 0.2.19.
- [Release notes](https://github.com/alexcrichton/filetime/releases )
- [Commits](https://github.com/alexcrichton/filetime/compare/0.2.18...0.2.19 )
---
updated-dependencies:
- dependency-name: filetime
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-08 09:26:52 +00:00
c97b025751
chore(deps): Bump zstd from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2 ( #6344 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.0+zstd.1.5.2 to 0.12.1+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/compare/v0.12.0...v0.12.1 )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-07 09:19:35 +00:00
abe60ee0dc
chore(deps): Bump serde from 1.0.148 to 1.0.149 ( #6341 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.148 to 1.0.149.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.148...v1.0.149 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 17:10:26 +00:00
687493e8e6
chore(deps): Bump sqlparser from 0.27.0 to 0.28.0 ( #6342 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.27.0 to 0.28.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.27.0...v0.28.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 12:02:01 +00:00
04e9253ade
chore(deps): Bump rustix from 0.36.4 to 0.36.5 ( #6340 )
...
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.4 to 0.36.5.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.4...v0.36.5 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-06 09:47:00 +00:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
2c771ea9f2
chore(deps): Bump mockito from 0.31.0 to 0.31.1 ( #6338 )
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.31.0 to 0.31.1.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.31.0...0.31.1 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:44:07 +00:00
c3a2ac99aa
chore(deps): Bump insta from 1.21.2 to 1.22.0 ( #6332 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.21.2...1.22.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 10:18:59 +00:00
05cc735849
chore(deps): Bump tower-http from 0.3.4 to 0.3.5 ( #6331 )
...
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.4 to 0.3.5.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.4...tower-http-0.3.5 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:18:12 +00:00
fc9431f6ae
chore(deps): Bump assert_cmd from 2.0.6 to 2.0.7 ( #6330 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.6...v2.0.7 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:05:56 +00:00
3740110cb5
chore(deps): Bump libc from 0.2.137 to 0.2.138 ( #6329 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.137 to 0.2.138.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.137...0.2.138 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 08:49:09 +00:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
f524687602
feat(ingester2): parallel partition persistence
...
Implements actor-based, parallel persistence in ingester2 with
controllable fan-out parallelism and queue depths.
This implementation encapsulates the complexity of persistence, queuing
and parallelism - the caller simply uses the handle to persist a
partition, while the actor handles fan-out to a set of persistence
workers, compaction in a separate thread-pool, and optional completion
notifications.
By consistently hashing persist jobs onto workers, parallelism is
achieved across partitions, but serialisation of partition persists is
enforced so that the sort key update is correctly serialised.
2022-12-02 16:34:03 +01:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
ec2e72d223
test: simplify test executors ( #6312 )
...
Have a single global test executor w/ reasonable defaults. Also don't
require tests to join/await executor shutdowns (most tests forget this
anyways and will get a runtime warning).
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-02 11:38:18 +00:00
8742ed6c67
chore(deps): Bump syn from 1.0.104 to 1.0.105 ( #6308 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.104 to 1.0.105.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.104...1.0.105 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-02 09:01:19 +00:00
d0f1f6a4fd
chore: Upgrade datafusion to get memory limits ( #6297 )
...
* chore: Update datafusion
* fix: use correctly qualified column names
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 16:40:26 +00:00
b9e424582f
refactor: Extract a clap block for the ingester2 RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
0133479122
feat: Create an ingester2 server
2022-12-01 11:39:29 -05:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
14a9bc92e9
Revert "Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )" ( #6296 )
...
This reverts commit b7e52c0d8d
2022-12-01 14:20:43 +00:00
b7e52c0d8d
Revert "chore: Update Datafusion and arrow/arrow-flight/parquet to `28.0.0` ( #6279 )" ( #6294 )
...
This reverts commit 039a45ddd1
2022-12-01 11:38:42 +00:00
c008219692
feat: Add a feature flag to switch to the router RPC write path ( #6247 )
...
* feat: Add a feature flag to switch to the router RPC write path
Fixes #6242 .
* refactor: Remove a weird arc clone/rename that's not needed
I'm sure this was needed at some point, but it doesn't make much sense.
I wasn't going to change this, but I'm now trying to minimize the
differences between this function and the write path init function, so
make this one better too.
* fix: Add the namespace autocreation to the RPC write path too
The topic/query pool don't really apply to this case, but use them
anyway to be able to use the existing catalog methods.
Also add a bunch of comments pointing out where the RPC write path
initializer and the old router's initializer are the same and where
they're different, so that perhaps it'll be easier to keep them in sync
while they both exist.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 11:05:39 +00:00
dependabot[bot]
Paul Dix
kodiakhq[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Andrew Lamb
Raphael Taylor-Davies
Marco Neumann
Carol (Nichols || Goulding)