26b97482df
chore(compactor2): Split up tests into smaller modules ( #7072 )
2023-02-27 17:53:41 +00:00
5194999d62
feat: Use ? for id of uncreated parquet files ( #7066 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 16:35:51 +00:00
63bf663b94
Merge pull request #7056 from influxdata/cn/test-rpc-write-in-querier
...
test: Switch most querier tests to use the RPC write path
2023-02-27 15:24:39 +00:00
731a131a85
Merge branch 'main' into cn/test-rpc-write-in-querier
2023-02-27 15:17:51 +00:00
dd5d4f4435
chore(compactor2): document and test `split_percentage` and `percentage_max_file_size` knobs ( #7026 )
...
* chore: document and test split_percentage and percentage_max_file_size
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* chore: add test with both max file size and split percentage
* docs: whitespace engineering and small typo
---------
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-02-27 15:01:06 +00:00
b74bef5271
Merge pull request #7065 from influxdata/cn/cargo-hakari-update
...
chore: Update cargo-hakari and the workspace hack crate's contents
2023-02-27 11:02:44 +00:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
ab031bd929
chore: Regenerate workspace hack crate contents
2023-02-27 11:55:20 +01:00
a5ed647246
chore: Update cargo-hakari config
2023-02-27 11:51:28 +01:00
2ad05c9f64
chore: Move hakari config file to the newly-preferred location
2023-02-27 11:51:28 +01:00
8002d34fa2
feat: add "split dedup by partition" optimizer rule ( #7020 )
...
* feat: add "split dedup by partition" optimizer rule
- some additional testing infra
- includes config infra for optimizer passes
- not wired up yet since we still use the old plan generation
For #6098 .
* refactor: change default and improve docs
2023-02-27 10:27:48 +00:00
4cf066680e
chore(deps): Bump tempfile from 3.3.0 to 3.4.0 ( #7069 )
...
* chore(deps): Bump tempfile from 3.3.0 to 3.4.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 10:02:21 +00:00
9a9345551c
chore(deps): Bump tower-http from 0.3.5 to 0.4.0 ( #7070 )
...
* chore(deps): Bump tower-http from 0.3.5 to 0.4.0
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.5 to 0.4.0.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.5...tower-http-0.4.0 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-27 09:55:23 +00:00
5dd15c9a23
chore(deps): Bump syn from 1.0.108 to 1.0.109 ( #7068 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.108 to 1.0.109.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.108...1.0.109 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-27 09:46:17 +00:00
2ed5758ddb
feat: InfluxQL planner learns how to project multiple measurements ( #7063 )
...
* feat: Planner learns how to project multiple measurements
Closes #6896
* chore: Update test
* chore: PR feedback
2023-02-27 00:50:06 +00:00
accf59ad61
fix: Update prost-types to fix cargo_audit ( #7067 )
2023-02-26 22:15:54 +00:00
6e9b50fe65
chore(deps): Bump syn from 1.0.107 to 1.0.108 ( #7058 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.107 to 1.0.108.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.107...1.0.108 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-24 11:19:18 +00:00
00a63b765b
chore(deps): Bump is-terminal from 0.4.3 to 0.4.4 ( #7060 )
...
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/sunfishcode/is-terminal/releases )
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.3...v0.4.4 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-24 11:12:38 +00:00
5853cc1bd7
Merge pull request #7061 from influxdata/dependabot/cargo/prost-types-0.11.7
...
chore(deps): Bump prost-types from 0.11.6 to 0.11.7
2023-02-24 11:05:49 +00:00
b827fb2c75
chore(deps): Bump prost-types from 0.11.6 to 0.11.7
...
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/commits )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-24 01:21:05 +00:00
6fce8edb8e
Merge pull request #7055 from influxdata/dom/wal-syscalls
...
perf(wal): reduce syscalls / allocations
2023-02-23 20:29:27 +00:00
f19fbc2d88
Merge branch 'main' into dom/wal-syscalls
2023-02-23 20:21:53 +00:00
5e9e08a86d
test: Switch most querier tests to use the RPC write path
2023-02-23 14:48:51 -05:00
ed740e28a9
Merge pull request #7054 from influxdata/dom/query-buckets
...
refactor(metric): increase partition bucket range
2023-02-23 17:59:59 +00:00
0927031857
Merge branch 'main' into dom/query-buckets
2023-02-23 17:46:31 +00:00
3d1cb58ab1
Merge pull request #7053 from influxdata/dom/log-query-errors
...
refactor(iox): log query errors
2023-02-23 17:46:23 +00:00
bb74ffaef3
Merge branch 'main' into dom/wal-syscalls
2023-02-23 17:07:46 +00:00
0062663b24
Merge branch 'main' into dom/log-query-errors
2023-02-23 17:05:38 +00:00
b9f7f12c0c
perf(wal): avoid buffer allocation in writer
...
Eliminate buffer allocation (& growing) in the WAL file writer by
reusing a single buffer each time.
This implementation shrinks the buffer size down to 128KiB if it grows
above that amount to prevent one large write from consuming memory
forever more (128KiB should be plenty more than the common write size).
2023-02-23 18:05:06 +01:00
c180d3d8ac
perf(wal): reduce I/O syscall count
...
Each WAL entry is prepended by a two field header, followed by the
payload bytes. Previously a syscall was made for each header field, and
then another to write the payload bytes (or in reality, at least one
call is made).
This commit reduces the syscalls down to a single write call by building
the entire record in memory before calling write(). This adds 8 bytes to
the in-memory buffer size compared to prior to this commit.
This is effectively a reimplementation of a BufWriter but optimised for
our expected memory usage and (more importantly) capable of issuing the
fsync calls necessary for WAL durability.
2023-02-23 18:05:06 +01:00
a4114c9385
Merge branch 'main' into dom/query-buckets
2023-02-23 17:04:59 +00:00
b76b75e911
fix: do not panic for unsupported expressions ( #7052 )
...
We see this in one of our prod clusters ATM.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 17:03:43 +00:00
90f8166b44
refactor(metric): increase partition bucket range
...
More buckets -> more better.
2023-02-23 18:01:56 +01:00
21654126b2
Merge branch 'main' into dom/log-query-errors
2023-02-23 17:01:19 +00:00
57d0582368
refactor(iox): log query errors
...
Make sure all query errors are logged in the top-level query handler.
2023-02-23 17:59:04 +01:00
08578cded5
refactor: n_threads and n_target_partitions are non-zero ( #7047 )
...
* refactor: n_threads and n_target_partitions are non-zero
Zero values will just panic. Prevent that earlier.
* fix: typo
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
---------
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-02-23 16:57:00 +00:00
0282eb4750
feat: streaming implementation of gap filling ( #7037 )
...
* feat: streaming implementation of gap filling
* chore: cargo fmt
* refactor: when gapfilling, concatenate input batches less frequently
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 15:42:41 +00:00
228f947fac
Merge pull request #7040 from influxdata/dom/query-instrumentation
...
feat(metrics): instrument ingester query responses
2023-02-23 10:45:31 +00:00
fe66c60037
Merge branch 'main' into dom/query-instrumentation
2023-02-23 10:30:51 +00:00
ff432f9d16
refactor: clarify metric descriptions
...
Make it clear that the counters track the count of X sent to the client,
not the full possible set of X that could be sent (these values differ
in the query abort/error case).
2023-02-23 11:29:33 +01:00
a5750237b8
Merge pull request #7046 from influxdata/dom/wal-io-thread
...
perf(wal): dedicated file writer I/O thread
2023-02-23 10:24:09 +00:00
6d147ec008
refactor: warn! -> error! and spelling
...
Fix a typo, use "error" level instead of "warn".
2023-02-23 11:13:57 +01:00
e3498e3925
perf(wal): use dedicated writer I/O thread
...
Change the WAL buffer flusher to use a dedicated I/O thread instead of
performing serialisation & blocking file I/O on the async runtime
threads. This should reduce runtime blocking / latency variance on the
async threads.
The added overhead is 1 channel send, but this is per WAL batch of
writes (not per DML write, or worse, per file write). This impl also
amortises allocation of the serialisation buffer, rather than growing
one incrementally for each batch.
2023-02-23 11:13:56 +01:00
c72c9d2dba
refactor: derive Debug on WAL types
...
Deriving debug is highly encouraged so that Result::unwrap() and friends
can print the state of an object if it is causing a panic (it's
impossible to call unwrap() otherwise!)
2023-02-23 11:13:56 +01:00
0b424888fd
Merge pull request #7043 from influxdata/dom/panic-msg
...
refactor: better panic message for column lookup
2023-02-23 10:13:41 +00:00
55cef48597
Merge branch 'main' into dom/panic-msg
2023-02-23 10:03:21 +00:00
39c6cac0ca
Merge pull request #7050 from influxdata/dependabot/cargo/pprof-0.11.1
...
chore(deps): Bump pprof from 0.11.0 to 0.11.1
2023-02-23 10:03:09 +00:00
2da4fb6ac5
Merge branch 'main' into dom/panic-msg
2023-02-23 10:02:01 +00:00
7377f0bb27
chore: remove wrapper type from message
...
Unwrap the type wrapper.
2023-02-23 10:59:57 +01:00
1b95b60c04
chore(deps): Bump pprof from 0.11.0 to 0.11.1
...
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.11.0 to 0.11.1.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/compare/v0.11.0...v0.11.1 )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-23 01:21:47 +00:00
Andrew Lamb
kodiakhq[bot]
Carol (Nichols || Goulding)
Marco Neumann
dependabot[bot]
Stuart Carnie
Dom
Christopher M. Wolff