96419b78e0
Merge pull request #5311 from influxdata/dom/instrument-kafka-produce
...
build: bump rskafka to latest
2022-08-04 15:20:45 +00:00
0ba3ae1e0d
Merge branch 'main' into dom/instrument-kafka-produce
2022-08-04 15:13:49 +00:00
600617ec08
Merge pull request #5307 from influxdata/dom/instrument-agg
...
feat: instrument kafka aggregated DML batch size
2022-08-04 14:57:40 +00:00
36d36c507c
ci: bump redpanda version
2022-08-04 16:57:28 +02:00
76d3a12dab
Merge branch 'main' into dom/instrument-agg
2022-08-04 14:49:10 +00:00
77fd967517
feat: instrument kafka aggregated DML batch size
...
The Kafka write buffer implementation (and only the Kafka impl) merges
together successive DML writes for the same namespace & partition within
a window of time.
This commit records the number of DML writes that have been merged
together to form a single batched op before it is dispatched to Kafka.
2022-08-04 16:48:56 +02:00
e8231b2986
chore(deps): Bump serde_json from 1.0.82 to 1.0.83 ( #5297 )
...
* chore(deps): Bump serde_json from 1.0.82 to 1.0.83
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.82 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.82...v1.0.83 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 14:28:29 +00:00
e24cecd926
fix: buffer allocation while reading parquet files ( #5312 )
...
Work around https://github.com/apache/arrow-rs/issues/2321 by limiting
reader batch size to number of rows (based on file-level metadata).
Fixes https://github.com/influxdata/conductor/issues/1103 .
2022-08-04 14:21:05 +00:00
3989ac1386
refactor: remove `split_members` and use `split_conjunction` from upstream DataFusion ( #5308 )
...
* refactor: remove split_members and use split_conjunction from datafusion
* fix: clippy
2022-08-04 13:58:59 +00:00
eea8270e83
fix: `compute_split_time` with small step sizes ( #5309 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 13:40:30 +00:00
34ccc9c7f5
chore: Revert "chore: Revert "refactor: bump batch size ( #5251 )" ( #5288 )" ( #5300 )
...
This reverts commit 471b8be92f
2022-08-04 13:19:46 +00:00
1cad7e13ec
build: bump rskafka to latest
...
Includes minor code changes needed to support the rskafka HEAD commit.
Breaking changes made in
https://github.com/influxdata/rskafka/issues/160
2022-08-04 15:02:11 +02:00
76171c5cea
refactor: simplify creating AND expressions ( #5298 )
...
* refactor: simplify creating AND expressions
* fix: fmt + clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 11:25:43 +00:00
5f9ade103f
chore: tweak rustc flags ( #5293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 09:12:21 +00:00
251af3f844
chore(deps): Bump serde from 1.0.140 to 1.0.142 ( #5296 )
...
* chore(deps): Bump serde from 1.0.140 to 1.0.142
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.140 to 1.0.142.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.140...v1.0.142 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-04 08:49:12 +00:00
07fe5c8af3
chore(deps): Bump libc from 0.2.126 to 0.2.127 ( #5306 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.126 to 0.2.127.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.126...0.2.127 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-04 08:40:23 +00:00
2ecf02e7b2
Merge pull request #5299 from influxdata/crepererum/issue5272
...
fix: compactor-related freeze and shutdown fixes
2022-08-03 21:13:09 +00:00
b3958321d3
Merge branch 'main' into crepererum/issue5272
2022-08-03 21:06:15 +00:00
55e1e2ec2b
chore(deps): Bump thiserror from 1.0.31 to 1.0.32 ( #5294 )
...
* chore(deps): Bump thiserror from 1.0.31 to 1.0.32
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.31 to 1.0.32.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.31...1.0.32 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 16:20:36 +00:00
039950b4fd
feat: ensure clean compactor executor shutdown
2022-08-03 18:07:00 +02:00
fd74f2639b
fix: do not attempt to poll future lists in compactor
...
It seems that the buffering / parallelization code cannot deal with
empty lists and just freezes forever (which blocks shutdown but will
also freeze the compactor forever).
2022-08-03 18:04:05 +02:00
4bd8977d55
refactor: add some main function debug logs
2022-08-03 18:00:28 +02:00
6011c4cd1f
chore: update datafusion pin ( #5290 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:43:18 +00:00
840e4801b8
feat: make querier RAM pool split a proper feature ( #5283 )
...
* feat: make querier RAM pool split a proper feature
- use propre pool names
- expose sizing via CLI/env
Closes https://github.com/influxdata/conductor/issues/1102 .
* refactor: improve naming and docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:27:23 +00:00
d0f88c664c
docs: improve some docstrings in `iox_query` ( #5291 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:20:12 +00:00
ee151c8b41
fix: make batch size back to 1024 to see if the OOM in the compactor go away ( #5289 )
...
* fix: make batch size back to 1024 to see if the OOM in the compactor go away
* fix: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Marco Neumann <marco@crepererum.net>
* fix: import needed constant
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 15:13:04 +00:00
663a20d743
refactor: remove `--ingster-address` ( #5255 )
...
Closes #5002 .
2022-08-03 15:05:01 +00:00
471b8be92f
chore: Revert "refactor: bump batch size ( #5251 )" ( #5288 )
...
This reverts commit bb172f8fa8
2022-08-03 14:23:45 +00:00
9fbc95c3ad
feat: add sequencer reset count metric and log to ingester ( #5286 )
...
Split out from #5253 .
Helps with #5128 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 13:00:36 +00:00
5fd59fcc43
chore(deps): Bump syn from 1.0.98 to 1.0.99 ( #5284 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.98 to 1.0.99.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.98...1.0.99 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-03 12:50:56 +00:00
273b3cc165
chore: replace `dotenv` with `dotenvy` ( #5285 )
...
The latter one is a maintained fork. This avoids having both crates
after #5282 .
2022-08-03 12:41:38 +00:00
7c67b93015
chore(deps): Bump sqlx from 0.6.0 to 0.6.1 ( #5282 )
...
Bumps [sqlx](https://github.com/launchbadge/sqlx ) from 0.6.0 to 0.6.1.
- [Release notes](https://github.com/launchbadge/sqlx/releases )
- [Changelog](https://github.com/launchbadge/sqlx/blob/main/CHANGELOG.md )
- [Commits](https://github.com/launchbadge/sqlx/commits )
---
updated-dependencies:
- dependency-name: sqlx
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 12:19:16 +00:00
772dd858a8
feat: support for regex comparisons in storage CLI ( #5281 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 10:08:44 +00:00
94fe5b4c10
chore(deps): Bump paste from 1.0.7 to 1.0.8 ( #5280 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.7 to 1.0.8.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.7...1.0.8 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-03 09:03:25 +00:00
964062b40c
feat: Add information about profiling using Instruments on macOS ( #5275 )
2022-08-03 08:45:33 +00:00
a4e2f880be
feat: Expose a C API for the IOx LP parser ( #5267 )
...
Can be useful to call the IOx LP parser from other processes, for example from Go.
I used it to run an online comparison of IOx and influxdb Go LP parser in order to identify compatibility
issues.
2022-08-02 15:44:41 +00:00
8e2443d879
feat: use two RAM pools in querier ( #5271 )
...
Quick&Dirty implementation of a RAM-pool split to see if this has any
effect. I expect the querier performance to improve due to this because
large read buffers can no longer evict precious metadata.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-02 15:14:26 +00:00
4812db9887
feat: fewer buckets but larger ranges for compaction duration histogram ( #5259 )
...
* chore: reduce log info
* feat: fewer buckets but larger ranges for compaction duration histogram
* chore: Apply suggestions from code review
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
* chore: run fmt after appying reviewer's suggestions
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-02 14:19:30 +00:00
9c9658ca38
test(influxdb_line_protocol): add value verification test ( #5270 )
2022-08-02 11:18:09 +00:00
ee491cbbfc
fix: re-enable querier read buffer cache ( #5268 )
...
This reverts commit 82913743f1#5252 .
I misjudged the cache hit ratio for the RB, see
https://github.com/influxdata/k8s-infra/pull/4548
So let's bring back the RB cache until we have some form of parquet
cache in place.
2022-08-02 08:37:30 +00:00
e57ae07db7
chore(deps): Bump serde from 1.0.140 to 1.0.141 ( #5260 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.140 to 1.0.141.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.140...v1.0.141 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-02 07:52:06 +00:00
a8f6d579c8
feat: add metric for predicate-based cache entry removal ( #5257 )
2022-08-02 07:44:53 +00:00
fec6b18d80
feat: add metric for TTL cache expiration ( #5256 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-02 07:00:30 +00:00
7d15bd6029
Merge pull request #5265 from influxdata/lp_fp
...
fix: Fix bug and incompatibility in floating point parsing of scientific notation
2022-08-02 06:23:16 +02:00
84a856069b
fix: Scientific notation without + or -
...
Closes #5264
2022-08-02 05:46:28 +02:00
a926996485
fix: Negative scientific notation without decimal parts
...
Closes #5263
2022-08-02 05:40:55 +02:00
8f1b6f2465
chore: reduce log info ( #5254 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-01 16:00:34 +00:00
82913743f1
refactor: disable querier read buffer cache ( #5252 )
...
Let's try and see how this performs in prod.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-01 15:43:22 +00:00
bb172f8fa8
refactor: bump batch size ( #5251 )
...
This is what DataFusion uses by default and I don't see a reason why we
should use such small batch sizes.
The affect is probably only visible in certain filter-aggregate queries
that don't focus on a single series (because there we likely end up with
1 or 2 batches only, esp. after #5250 ) for coarse-grained filters, esp.
when the filter key is not the first sort key.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-01 13:49:58 +00:00
b12ebe1109
fix: do not panic on invalid timestamp ranges ( #5249 )
...
Timestamp ranges come from "untrusted" inputs (via gRPC) and must not
lead to panics. The only case where this could happen is at `start >
end`. Let's just set `start = end` in this case. Reaonsing:
- Semantically this is a sound range, since this is only a somewhat
degenerated case of "empty".
- We already allow `start = end` to represent "empty" ranges.
- We already clamp (and therefore modify) `start` to the valid range.
Fixes https://github.com/influxdata/conductor/issues/1080 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-08-01 13:35:34 +00:00
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Marco Neumann
Andrew Lamb
Nga Tran
Raphael Taylor-Davies
Stuart Carnie
Marko Mikulicic