c0f2ba09ee
Merge branch 'main' into cn/compactor2
2022-12-19 14:22:56 +00:00
7f2aa8b10c
chore(deps): Bump serde_json from 1.0.89 to 1.0.91
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.89 to 1.0.91.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.89...v1.0.91 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-12-19 01:44:18 +00:00
d7e75d43ea
fix: Make shard ID optional for compactor queries in RPC write mode
2022-12-16 17:28:53 -05:00
2406cdb24b
feat: Create a compactor2 cli
2022-12-16 17:22:06 -05:00
c830a83105
feat(ingester2): hot partition persistence
...
This PR uses the MutableBatch persist cost estimation added in #6425 to
selectively mark "hot" partitions for persistence.
This uses a (composable!) "post-write" observer that is invoked after
each buffer call - this allows the HotPartitionPersister in this commit
to inspect the cost of the partition after applying the write, and if it
exceeds the configurable cost threshold, enqueue it for persistence
(rotating the buffer within the partition in the process).
Unlike ingester(1), this implementation prevents overrun - the
application of the write that exceeds the cost limit, and enqueueing the
partition for persistence is atomic.
2022-12-16 19:33:34 +01:00
f419e2c378
feat: warm compaction ( #6192 )
...
* feat: warm compaction
chore: add missing warm compaction config
chore: tests for warm compaction
chore: modify count usage in warm compaction sql
chore: catalog test for warm compaction; sql fixes
feat: settable target level for compact w/ budget
chore: tests for warm compaction
chore: clarifying comments in warm compaction test
chore: fixed erroneous comment in catalog test
chore: improve warm compactor test by checking file exists
chore: tests for warm compaction
chore: warm compactor test tidy-ups
* chore: improve test for warm compaction
* chore: fix erroneous comment in warm compaction code
2022-12-16 15:59:45 +00:00
cfb7c16bb1
Merge branch 'main' into dom/optimal-persist-parallelism
2022-12-16 09:12:22 +00:00
22d6b78899
docs: Fix outdated comment on querier mode switching behavior
2022-12-15 14:16:14 -05:00
2a1e540ee3
fix: Rename INFLUXDB_IOX_MODE to INFLUXDB_IOX_RPC_MODE
2022-12-15 14:13:01 -05:00
7d216ba1fd
feat: Error if you run the wrong command with the wrong env var set
...
Connects to #6402 .
2022-12-15 14:06:59 -05:00
aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
...
And standardize on ingester2 and router2 for consistency.
Connects to #6402 .
2022-12-15 14:06:59 -05:00
933ab1f8c7
feat(ingester2): optimal persist parallelism
...
This commit changes the behaviour of the persist system to enable
optimal parallelism of persist operations, and improve the accuracy of
the outstanding job bound / back-pressure.
Previously all persist operations for a given partition were
consistently hashed to a single worker task. This serialised persistence
per partition, ensuring all updates to the partition sort key were
serialised. However, this also unnecessarily serialises persist
operations that do not need to update the sort key, reducing the
potential throughput of the system; in the worst case of a single
partition receiving all the writes, only one worker would be persisting,
and the other N-1 workers would be idle.
After this change, the sort key is inspected when enqueuing the persist
operation and if it can be determined that no sort key update is
necessary (the typical case), then the persist task is placed into a
global work queue from which all workers consume. This allows for
maximal parallelisation of these jobs, and the removes the per-worker
head-of-line blocking.
In the case that the sort key does need updating, these jobs continue to
be consistently hashed to a single worker, ensuring serialised sort key
updates only where necessary.
To support these changes, the back-pressure system has been changed to
account for all outstanding persist jobs in the system, regardless of
type or assigned worker - a logical, bounded queue is composed together
of a semaphore limiting the number of persist tasks overall, and a
series of physical, unbounded queues - one to each worker & the global
queue. The overall system remains bounded by the
INFLUXDB_IOX_PERSIST_QUEUE_DEPTH value, and is now simpler to reason
about (it is independent of the number of workers, etc).
2022-12-15 18:30:51 +01:00
c7e4bf3dd1
refactor(config): default persist queue depth=250
...
Allow up to 250 persist jobs to be enqueued for any one worker before
pausing.
With 5 workers, this gives a maximum outstanding persist jobs of 2,500.
2022-12-14 17:19:19 +01:00
1da9b63cce
fix(ingester2): persist deadlock
...
Removes the submission queue from the persist fan-out, instead the
PersistHandle now carries the shared state internally (cheaply cloned
via ref counts).
This also resolves the persist deadlock when under load.
2022-12-13 16:47:45 +01:00
5141cba1db
fix: Only switch into querier RPC write path if ingester addresses specified
...
This enables testing of the querier using the old path with the
rpc_write feature turned on.
2022-12-08 17:40:04 -05:00
b85130cb7c
fix: Make --ingester-addresses optional for the querier in RPC write mode
2022-12-08 17:22:52 -05:00
619a2d0856
fix: Remove conflicting arguments from the RouterRpcWriteConfig ( #6355 )
...
These were added in
https://github.com/influxdata/influxdb_iox/pull/6346 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 20:21:37 +00:00
6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics ( #6313 )
...
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics
Closes #6310 .
* refactor: rename and tune default exec mem limits
* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
b819a48d59
refactor(ingester2): rename concurrent limit var
...
The ingester will handle many types of requests - this limit applies to
queries only.
2022-12-02 14:36:42 +01:00
77cbc880f6
feat: Add cap limit on number of partitions to be compacted in parallel ( #6305 )
...
* feat: Add cap limit on number of partitions to be comapcted in parallel
* chore: cleanup
* chore: clearer comments
2022-12-01 21:23:44 +00:00
7c21db360f
fix: Make wal rotation period a CLI arg with a default of 300 s
2022-12-01 13:51:45 -05:00
b9e424582f
refactor: Extract a clap block for the ingester2 RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
fef3bc02cd
refactor: Extract a clap block for the router RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
d07658282c
feat: add router config parameter for retention ( #6278 )
...
* chore: remove unused/moved ns_autocreation dml handler
* feat(router): expose new ns retention as config
* fix: forgot to set default value for router retention arg
* chore: make new namespace retention param an option
2022-11-30 13:14:39 +00:00
caa595a6fc
chore(deps): Bump serde_json from 1.0.88 to 1.0.89 ( #6203 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.88 to 1.0.89.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.88...v1.0.89 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 09:28:31 +00:00
52c50c16e1
chore(deps): Bump serde_json from 1.0.87 to 1.0.88
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.87 to 1.0.88.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.87...v1.0.88 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-21 01:52:18 +00:00
7c813c170a
feat: reintroduce compactor first file in partition exception ( #6176 )
...
* feat: compactor ignores max file count for first file
chore: typo in comment in compactor
* feat: restore special first file in partition compaction logic; add limit
* fix: calculation in compaction max file count
chore: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 15:58:59 +00:00
62851afc27
feat: add querier->ingester circuit breaker ( #6147 )
...
* feat: add log ingester memory pressure persist
* feat: add querier->ingester circuit breaker
Closes #4608 .
* docs: explain high-level circuit breaker
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* test: add additional test assertion
* refactor: upgrade info to warning log
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-11-16 10:50:33 +00:00
43687a86d2
fix: Remove lots of needless borrows that Clippy can now identify
...
Except for in generated code that we don't control.
2022-11-09 10:54:18 -05:00
f9316decee
chore: expose compactor's hot compaction hours thresholds as cfg ( #6060 )
...
* chore: expose compactor's hot compaction hours thresholds as cfg
* fix: add missing compactor arg envar; fix some comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-07 15:29:17 +00:00
654ed98d1f
feat: config param to set when partition is cold ( #6044 )
...
* feat: config param to set when partition is cold
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* fix: make default 8 hours and avoid using 8 * 60 becasue it is a string, not expression which makes a test fail
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-03 15:03:56 +00:00
2e83e04eab
feat: Use workspace package metadata to reduce differences and repetition
2022-10-24 13:04:09 -04:00
bebb15d30f
chore(deps): Bump serde_json from 1.0.86 to 1.0.87
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.86 to 1.0.87.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.86...v1.0.87 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-20 07:52:33 +00:00
5d835d5047
revert: rdkafka/rskafka swapping ( #5844 )
...
This reverts commit 442a7ff2a4https://github.com/influxdata/influxdb_iox/pull/5800
2022-10-17 12:34:28 +02:00
ff7b571fae
feat: Log object store configuration on startup ( #5865 )
2022-10-14 18:10:20 +00:00
442a7ff2a4
revert: "revert: rdkafka/rskafka swapping ( #5800 )" ( #5844 )
...
* revert: "revert: rdkafka/rskafka swapping (#5800 )"
This reverts commit b77c3540e1
2022-10-13 09:33:06 +00:00
266b8f2a58
Merge branch 'main' into dependabot/cargo/clap-4.0.2
2022-10-12 14:01:28 +00:00
933493fab3
chore(deps): Bump object_store from 0.5.0 to 0.5.1
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.0 to 0.5.1.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.0...object_store_0.5.1 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-11 01:19:10 +00:00
2277fcf08a
chore(deps): Bump serde_json from 1.0.85 to 1.0.86
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.85 to 1.0.86.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.85...v1.0.86 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 01:42:37 +00:00
d0f8f0aa60
fix: Remove variadic space-separated support for --write-buffer-connection-config
...
We don't actually appear to use the space support anywhere, preferring
the comma version.
2022-09-30 16:59:28 -04:00
576d629ce4
fix: Remove leading `--` from long option names
2022-09-30 16:59:28 -04:00
50f84906e2
fix: Remove multiple_values = true; it's now implied because of Vec
...
See <https://docs.rs/clap/4.0.2/clap/_derive/index.html#arg-types >
> clap assumes some intent based on the type used:
>
> ...
>
> Vec<T> 0.. occurrences of argument .action(ArgAction::Append).required(false).num_args(1..)
2022-09-30 16:59:03 -04:00
73d7105f20
fix: Update from clap ArgEnum to ValueEnum
...
See <https://github.com/clap-rs/clap/pull/4127 >
2022-09-30 16:59:03 -04:00
kodiakhq[bot]
dependabot[bot]
Carol (Nichols || Goulding)
Dom Dwyer
Luke Bond
Carol (Nichols || Goulding)
Marco Neumann
Nga Tran
Andrew Lamb
Jake Goulding