ab031bd929
chore: Regenerate workspace hack crate contents
2023-02-27 11:55:20 +01:00
a5ed647246
chore: Update cargo-hakari config
2023-02-27 11:51:28 +01:00
2ad05c9f64
chore: Move hakari config file to the newly-preferred location
2023-02-27 11:51:28 +01:00
8002d34fa2
feat: add "split dedup by partition" optimizer rule ( #7020 )
...
* feat: add "split dedup by partition" optimizer rule
- some additional testing infra
- includes config infra for optimizer passes
- not wired up yet since we still use the old plan generation
For #6098 .
* refactor: change default and improve docs
2023-02-27 10:27:48 +00:00
4cf066680e
chore(deps): Bump tempfile from 3.3.0 to 3.4.0 ( #7069 )
...
* chore(deps): Bump tempfile from 3.3.0 to 3.4.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 10:02:21 +00:00
9a9345551c
chore(deps): Bump tower-http from 0.3.5 to 0.4.0 ( #7070 )
...
* chore(deps): Bump tower-http from 0.3.5 to 0.4.0
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.3.5 to 0.4.0.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.3.5...tower-http-0.4.0 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-27 09:55:23 +00:00
5dd15c9a23
chore(deps): Bump syn from 1.0.108 to 1.0.109 ( #7068 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.108 to 1.0.109.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.108...1.0.109 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-27 09:46:17 +00:00
2ed5758ddb
feat: InfluxQL planner learns how to project multiple measurements ( #7063 )
...
* feat: Planner learns how to project multiple measurements
Closes #6896
* chore: Update test
* chore: PR feedback
2023-02-27 00:50:06 +00:00
accf59ad61
fix: Update prost-types to fix cargo_audit ( #7067 )
2023-02-26 22:15:54 +00:00
6e9b50fe65
chore(deps): Bump syn from 1.0.107 to 1.0.108 ( #7058 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.107 to 1.0.108.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.107...1.0.108 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-24 11:19:18 +00:00
00a63b765b
chore(deps): Bump is-terminal from 0.4.3 to 0.4.4 ( #7060 )
...
Bumps [is-terminal](https://github.com/sunfishcode/is-terminal ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/sunfishcode/is-terminal/releases )
- [Commits](https://github.com/sunfishcode/is-terminal/compare/v0.4.3...v0.4.4 )
---
updated-dependencies:
- dependency-name: is-terminal
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-02-24 11:12:38 +00:00
5853cc1bd7
Merge pull request #7061 from influxdata/dependabot/cargo/prost-types-0.11.7
...
chore(deps): Bump prost-types from 0.11.6 to 0.11.7
2023-02-24 11:05:49 +00:00
b827fb2c75
chore(deps): Bump prost-types from 0.11.6 to 0.11.7
...
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/commits )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-24 01:21:05 +00:00
6fce8edb8e
Merge pull request #7055 from influxdata/dom/wal-syscalls
...
perf(wal): reduce syscalls / allocations
2023-02-23 20:29:27 +00:00
f19fbc2d88
Merge branch 'main' into dom/wal-syscalls
2023-02-23 20:21:53 +00:00
ed740e28a9
Merge pull request #7054 from influxdata/dom/query-buckets
...
refactor(metric): increase partition bucket range
2023-02-23 17:59:59 +00:00
0927031857
Merge branch 'main' into dom/query-buckets
2023-02-23 17:46:31 +00:00
3d1cb58ab1
Merge pull request #7053 from influxdata/dom/log-query-errors
...
refactor(iox): log query errors
2023-02-23 17:46:23 +00:00
bb74ffaef3
Merge branch 'main' into dom/wal-syscalls
2023-02-23 17:07:46 +00:00
0062663b24
Merge branch 'main' into dom/log-query-errors
2023-02-23 17:05:38 +00:00
b9f7f12c0c
perf(wal): avoid buffer allocation in writer
...
Eliminate buffer allocation (& growing) in the WAL file writer by
reusing a single buffer each time.
This implementation shrinks the buffer size down to 128KiB if it grows
above that amount to prevent one large write from consuming memory
forever more (128KiB should be plenty more than the common write size).
2023-02-23 18:05:06 +01:00
c180d3d8ac
perf(wal): reduce I/O syscall count
...
Each WAL entry is prepended by a two field header, followed by the
payload bytes. Previously a syscall was made for each header field, and
then another to write the payload bytes (or in reality, at least one
call is made).
This commit reduces the syscalls down to a single write call by building
the entire record in memory before calling write(). This adds 8 bytes to
the in-memory buffer size compared to prior to this commit.
This is effectively a reimplementation of a BufWriter but optimised for
our expected memory usage and (more importantly) capable of issuing the
fsync calls necessary for WAL durability.
2023-02-23 18:05:06 +01:00
a4114c9385
Merge branch 'main' into dom/query-buckets
2023-02-23 17:04:59 +00:00
b76b75e911
fix: do not panic for unsupported expressions ( #7052 )
...
We see this in one of our prod clusters ATM.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 17:03:43 +00:00
90f8166b44
refactor(metric): increase partition bucket range
...
More buckets -> more better.
2023-02-23 18:01:56 +01:00
21654126b2
Merge branch 'main' into dom/log-query-errors
2023-02-23 17:01:19 +00:00
57d0582368
refactor(iox): log query errors
...
Make sure all query errors are logged in the top-level query handler.
2023-02-23 17:59:04 +01:00
08578cded5
refactor: n_threads and n_target_partitions are non-zero ( #7047 )
...
* refactor: n_threads and n_target_partitions are non-zero
Zero values will just panic. Prevent that earlier.
* fix: typo
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
---------
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-02-23 16:57:00 +00:00
0282eb4750
feat: streaming implementation of gap filling ( #7037 )
...
* feat: streaming implementation of gap filling
* chore: cargo fmt
* refactor: when gapfilling, concatenate input batches less frequently
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-23 15:42:41 +00:00
228f947fac
Merge pull request #7040 from influxdata/dom/query-instrumentation
...
feat(metrics): instrument ingester query responses
2023-02-23 10:45:31 +00:00
fe66c60037
Merge branch 'main' into dom/query-instrumentation
2023-02-23 10:30:51 +00:00
ff432f9d16
refactor: clarify metric descriptions
...
Make it clear that the counters track the count of X sent to the client,
not the full possible set of X that could be sent (these values differ
in the query abort/error case).
2023-02-23 11:29:33 +01:00
a5750237b8
Merge pull request #7046 from influxdata/dom/wal-io-thread
...
perf(wal): dedicated file writer I/O thread
2023-02-23 10:24:09 +00:00
6d147ec008
refactor: warn! -> error! and spelling
...
Fix a typo, use "error" level instead of "warn".
2023-02-23 11:13:57 +01:00
e3498e3925
perf(wal): use dedicated writer I/O thread
...
Change the WAL buffer flusher to use a dedicated I/O thread instead of
performing serialisation & blocking file I/O on the async runtime
threads. This should reduce runtime blocking / latency variance on the
async threads.
The added overhead is 1 channel send, but this is per WAL batch of
writes (not per DML write, or worse, per file write). This impl also
amortises allocation of the serialisation buffer, rather than growing
one incrementally for each batch.
2023-02-23 11:13:56 +01:00
c72c9d2dba
refactor: derive Debug on WAL types
...
Deriving debug is highly encouraged so that Result::unwrap() and friends
can print the state of an object if it is causing a panic (it's
impossible to call unwrap() otherwise!)
2023-02-23 11:13:56 +01:00
0b424888fd
Merge pull request #7043 from influxdata/dom/panic-msg
...
refactor: better panic message for column lookup
2023-02-23 10:13:41 +00:00
55cef48597
Merge branch 'main' into dom/panic-msg
2023-02-23 10:03:21 +00:00
39c6cac0ca
Merge pull request #7050 from influxdata/dependabot/cargo/pprof-0.11.1
...
chore(deps): Bump pprof from 0.11.0 to 0.11.1
2023-02-23 10:03:09 +00:00
2da4fb6ac5
Merge branch 'main' into dom/panic-msg
2023-02-23 10:02:01 +00:00
7377f0bb27
chore: remove wrapper type from message
...
Unwrap the type wrapper.
2023-02-23 10:59:57 +01:00
1b95b60c04
chore(deps): Bump pprof from 0.11.0 to 0.11.1
...
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.11.0 to 0.11.1.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/compare/v0.11.0...v0.11.1 )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-02-23 01:21:47 +00:00
7e31b2638d
fix: Understandable compactor2 config report ( #7028 )
...
* fix: Understandable compactor2 config report
* fix: do not log postgres dsn
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-22 23:43:31 +00:00
f93baf7693
chore: Update DataFusion and `arrow` / `arrow-flight` / `parquet` to `33.0.0` ( #7045 )
...
* chore: Update DataFusion and arrow/arrow-flight/parquet to 33.0.0
* fix: Update test output
* fix: update more test output
* fix: Update querier test output
* chore: Run cargo hakari tasks
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting
Fix formatting of batch pretty printing.
* test: fix formatting for selector tests
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: Christopher Wolff <chris.wolff@influxdata.com>
2023-02-22 21:24:20 +00:00
6fb93a7679
refactor: Make InfluxQL planning sync ( #7038 )
...
* refactor: Move statement parsing to separate fn
* refactor: Remove async from `InfluxQLToLogicalPlan`
Closes #6607
* chore: Remove async functions and tokio::test
* chore: Remove redundant attribute
* chore: Feedback, switch to dynamic dispatch vs generic implementation
2023-02-22 19:33:49 +00:00
902ef67d72
Merge pull request #7023 from influxdata/cn/less-shard-id
...
fix: Use shard_id in fewer protobuf serializations
2023-02-22 15:12:40 +00:00
48e6cce746
Merge branch 'main' into cn/less-shard-id
2023-02-22 15:05:37 +00:00
5b7bdf23a9
Merge pull request #7004 from influxdata/alamb/remove_old_algorithm
...
refactor(compactor): delete all at once algorithm
2023-02-22 08:37:48 -06:00
80ba49b611
Merge branch 'main' into alamb/remove_old_algorithm
2023-02-22 08:30:30 -06:00
18815b951d
chore: update TODO comment ( #7044 )
2023-02-22 14:17:30 +00:00
Carol (Nichols || Goulding)
Marco Neumann
dependabot[bot]
Stuart Carnie
Andrew Lamb
Dom
Christopher M. Wolff
kodiakhq[bot]
Joe-Blount