Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
11,185 Commits
205 Branches
453 Tags
382 MiB
Commit Graph

11185 Commits (07b7107f9aa50965c262b3a40585c05be81a6848)

Author SHA1 Message Date
Marco Neumann 07b7107f9a
feat: sub-traces for `create_chunks` (#7148) 
In one prod case the majority of this was NOT spend on creating the
child chunks. I suspect that the summary creation and the string cloning
involved in there are quite slow. So let's have slightly more detailed
tracing and see.
 2023-03-07 15:06:37 +00:00
dependabot[bot] e7369449f8
chore(deps): Bump io-lifetimes from 1.0.5 to 1.0.6 (#7144) 
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes) from 1.0.5 to 1.0.6.
- [Release notes](https://github.com/sunfishcode/io-lifetimes/releases)
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/v1.0.5...v1.0.6)

---
updated-dependencies:
- dependency-name: io-lifetimes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-07 10:08:36 +00:00
dependabot[bot] 600a353211
chore(deps): Bump sqlparser from 0.31.0 to 0.32.0 (#7142) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.31.0 to 0.32.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.31.0...v0.32.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-07 09:27:31 +00:00
Marco Neumann 91471fe568
fix: check schema when calculating sorting for `ParquetExec` (#7136) 
When combining sort keys, we have to check the schema of the chunk to
differentiate between "column does not exist within this chunk" and
"column exists but is not sorted".

This is unlikely an issue in prod at the moment (if there is not bug in
the ingester or compactor), but this was found while working on tests
for #6098. Overall this should improve robustness.

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-07 09:20:31 +00:00
Nga Tran 075264057f
fix: L1 files must be sorted in their min_time if they need to split before compacting (#7139) 
* fix: L1 files must be sorted in their min_time if they need to split before compacting

* chore: clearer comments

* chore: Apply suggestions from code review

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>

* chore: run fmt after applying review suggestions

---------

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
 2023-03-06 22:52:15 +00:00
Nga Tran 97bf8d3a89
test: reproducer for bug 17221 (#7137) 2023-03-06 19:22:30 +00:00
Andrew Lamb ed0704ac8d
chore: Update datafusion (#7100) 
* chore: Update datafusion

* chore: iox_query to compile for API changes + update tests

* chore: Run cargo hakari tasks

---------

Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-03-06 17:59:24 +00:00
Joe-Blount 87ae7e72cd
chore: add warnings to compaction simulator for excessively oversized files (#7126) 
* chore: add warnings to compaction simulator for excessively oversized file

* chore: Update comment in compactor2_test_utils/src/lib.rs

Co-authored-by: Andrew Lamb <alamb@influxdata.com>

---------

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
 2023-03-06 15:54:38 +00:00
dependabot[bot] 535f3d92a2
chore(deps): Bump serde_json from 1.0.93 to 1.0.94 (#7132) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.93 to 1.0.94.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.93...v1.0.94)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-06 12:12:23 +00:00
dependabot[bot] 8771dcb645
chore(deps): Bump thiserror from 1.0.38 to 1.0.39 (#7131) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.38 to 1.0.39.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.38...1.0.39)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-06 10:55:26 +00:00
Marco Neumann d79405b367
chore: add deploy tools to CI image (#7134) 
Adds `skopeo` and gcloud CLI to CI image. This should eventually replace
our manual installation during the `deploy_releases` step:

d8d097c183/.circleci/config.yml (L493-L527)

This step costs us 20min during CD, which is ridiculous. A follow-up PR
later this week will use the CI image instead of an ad hoc installation.

Also see https://github.com/influxdata/idpe/issues/17098 .

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-06 10:47:43 +00:00
dependabot[bot] 3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 (#7130) 
Bumps [paste](https://github.com/dtolnay/paste) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases)
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12)

---
updated-dependencies:
- dependency-name: paste
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-06 10:40:41 +00:00
dependabot[bot] 8f3a9396d0
chore(deps): Bump async-trait from 0.1.64 to 0.1.66 (#7129) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.64 to 0.1.66.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.64...0.1.66)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-06 10:13:29 +00:00
dependabot[bot] d8d097c183
chore(deps): Bump rustix from 0.36.8 to 0.37.1 (#7128) 
* chore(deps): Bump rustix from 0.36.8 to 0.37.1

Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.36.8 to 0.37.1.
- [Release notes](https://github.com/bytecodealliance/rustix/releases)
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.8...v0.37.1)

---
updated-dependencies:
- dependency-name: rustix
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: Run cargo hakari tasks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
 2023-03-06 09:48:01 +00:00
Andrew Lamb dfd87f3e20
test(compactor): Add test for large amounts of data with a single timestamp (#7123) 
* test(compactor): Add test for large amounts of data with a single timestamp

* fix: Update compactor2/tests/layouts/single_timestamp.rs

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>

---------

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
 2023-03-03 20:12:23 +00:00
kodiakhq[bot] 18e183a54b
Merge pull request #7121 from influxdata/dom/seqnum-set 
feat: SequenceNumberSet intersection + helpers
 2023-03-03 19:19:47 +00:00
Dom c74339ebd9
Merge branch 'main' into dom/seqnum-set 2023-03-03 16:28:28 +00:00
kodiakhq[bot] 069184a6d1
Merge pull request #7120 from influxdata/dom/nesting 
style: remove unnecessary nesting
 2023-03-03 16:28:18 +00:00
Dom 5b7a8beff8
Merge branch 'main' into dom/nesting 2023-03-03 16:21:55 +00:00
Dom Dwyer ac1b37c0f0
feat(data_types): SequenceNumberSet intersection 
Support computing the intersection of two SequenceNumberSet.
 2023-03-03 17:21:39 +01:00
Dom Dwyer 146494f619
perf: pre-allocation of SequenceNumberSet 
Support pre-allocation of SequenceNumberSet for known-length sets.
 2023-03-03 17:21:38 +01:00
Dom Dwyer 7801a63334
feat: apply RLE optimisation for SequenceNumberSet 
Allow a SequenceNumberSet to be space-optimised by transforming it to
use run length encoding.
 2023-03-03 17:21:38 +01:00
Dom Dwyer 043f3421ba
feat(data_types): PartialEq for SequenceNumberSet 
Derive + test partial equality matching for SequenceNumberSet.
 2023-03-03 17:21:38 +01:00
Christopher M. Wolff c15d789613
fix: account for memory in GapFill operator (#7115) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-03 16:21:13 +00:00
Dom Dwyer 5c86f0570e
style: remove unnecessary nesting 
I'm not sure why rustfmt didn't do this, but it removes a level of
indenting.
 2023-03-03 17:19:21 +01:00
Joe-Blount f4d64099bf
Merge pull request #7116 from influxdata/jrb_10_l1l2_overlap_tests 
chore: add tests for L1/L2 invariant violations in simulator
 2023-03-03 08:27:52 -06:00
Joe-Blount cd159fd08b
Merge branch 'main' into jrb_10_l1l2_overlap_tests 2023-03-03 08:21:33 -06:00
dependabot[bot] 1e80227df8
chore(deps): Bump mockito from 0.32.3 to 0.32.4 (#7117) 
Bumps [mockito](https://github.com/lipanski/mockito) from 0.32.3 to 0.32.4.
- [Release notes](https://github.com/lipanski/mockito/releases)
- [Commits](https://github.com/lipanski/mockito/compare/0.32.3...0.32.4)

---
updated-dependencies:
- dependency-name: mockito
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
 2023-03-03 10:14:19 +00:00
Dom c50bb82a75
Merge pull request #7118 from influxdata/dependabot/cargo/object_store-0.5.5 
chore(deps): Bump object_store from 0.5.4 to 0.5.5
 2023-03-03 10:07:22 +00:00
dependabot[bot] 3256fcc72e
chore(deps): Bump object_store from 0.5.4 to 0.5.5 
Bumps [object_store](https://github.com/apache/arrow-rs) from 0.5.4 to 0.5.5.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md)
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.4...object_store_0.5.5)

---
updated-dependencies:
- dependency-name: object_store
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-03-03 02:00:51 +00:00
Joe-Blount 5e0a0a028b chore: add tests for L1/L2 invariant violations in simulator 2023-03-02 16:10:12 -06:00
kodiakhq[bot] fe14f78e0c
Merge pull request #7112 from influxdata/dom/wal-cancellation 
fix(wal): write commit cancellation safety
 2023-03-02 21:17:39 +00:00
kodiakhq[bot] 6c807ad37a
Merge branch 'main' into dom/wal-cancellation 2023-03-02 21:10:59 +00:00
Joe-Blount caa6a84488
chore: verify split time invariants within simulator (#7114) 2023-03-02 20:58:09 +00:00
kodiakhq[bot] cb8d55c5b2
Merge pull request #7093 from influxdata/dom/wal-flusher-task-leak 
fix: wal flusher task / memory leak
 2023-03-02 20:50:25 +00:00
kodiakhq[bot] 0c530aa9f7
Merge branch 'main' into dom/wal-flusher-task-leak 2023-03-02 20:44:02 +00:00
Marco Neumann 999a5dae03
refactor: sort key cleanups (#7113) 
* refactor: remove unused `ColumnSort`

* refactor: remove invalid assertion

It is true that time SHOULD be the last sort key, but we absoletely
don't require that, esp. not in the query tier. The ingester will
currently always produce sort keys where time is last, but if we ever
going to deal w/ external data sources like bulk loaded parquet files,
this may not always be the case.

Found while constructing some edge case tests.

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-02 16:08:21 +00:00
Dom c3d9219afc
Merge pull request #7110 from influxdata/dom/record-wal-seqnum-sets 
feat(wal): track set of write IDs in a segment file
 2023-03-02 15:20:27 +00:00
Dom 8fe874a7f0
Merge branch 'main' into dom/record-wal-seqnum-sets 2023-03-02 15:13:36 +00:00
Dom 746d9e4db5
Merge pull request #7094 from influxdata/dom/perf-batch-buffer-reuse 
perf(wal): avoid batch buffer reallocations
 2023-03-02 15:03:19 +00:00
Dom c9338fb785
Merge branch 'main' into dom/perf-batch-buffer-reuse 2023-03-02 14:50:27 +00:00
Dom Dwyer bde300c988
fix: WAL/buffer commit cancellation safety 
Ensure that a write that is added to the WAL is always attempted to be
applied to the BufferTree.

This covers off the case of a user submitting a write, waiting long
enough for it to be added to the WAL buffer, and then disconnecting
before it is added to the BufferTree (and before they get a response).

This is a minor issue on its own, but fixing it is necessary for correct
reference counting of WAL files:

    https://github.com/influxdata/influxdb_iox/issues/6566

This also documents a low-risk opportunity for the WAL contents &
BufferTree to diverge, potentially leading to a crash-loop at startup:

    https://github.com/influxdata/influxdb_iox/issues/7111

In practice a crash loop is unlikely, as it would require broken
invariants elsewhere (no schema validation being applied).

Closes https://github.com/influxdata/influxdb_iox/issues/6281.
 2023-03-02 15:48:41 +01:00
Dom Dwyer e442957ae5
refactor: derive Clone on instrumentation wrappers 
If the inner types are Clone-able, the instrumentation wrappers become
clone-able too. Both wrappers are cheap to clone.
 2023-03-02 15:48:41 +01:00
Dom Dwyer cf228fa38b
feat(ingester): cancellation-safe future helper 
Allow a future to be wrapped such that it is always driven to completion
(obviously excluding deadlocks/process crashes).
 2023-03-02 15:48:41 +01:00
Andrew Lamb 9f0645a775
fix(compactor2): fix off by one error in time ranges of simulator (#7098) 
* fix(compactor2): fix off by one error in time ranges of simulator

* chore: update a test that were added recently and this PR fixes it

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: NGA-TRAN <nga-tran@live.com>
 2023-03-02 14:43:53 +00:00
Marco Neumann 2c4da24f73
feat: sort-related phys. optimizers (#7095) 
* feat: `SortPushdown` optimizer

* feat: `RedundantSort` optimizer

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-02 14:20:13 +00:00
Andrew Lamb 6b35af115e
chore(compactor2): Add tests for remaining file knobs (#7099) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-02 12:20:44 +00:00
Andrew Lamb eb488cf55a
chore(compactor2): Improve documentation on split times (#7105) 
* chore(compactor2): Improve documentation on split times

* fix: Apply suggestions from code review

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>

* fix: fmt

* fix: typo

---------

Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-02 12:00:55 +00:00
Andrew Lamb b2052c1b81
chore(compactor2): add documentation about the compactor2 file level invariants (#7101) 
* chore(compactor2): add documentation about the compactor2 file level invariants

* fix: Fix doc links

---------

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2023-03-02 11:24:22 +00:00
Dom 160e93ea48
Merge branch 'main' into dom/perf-batch-buffer-reuse 2023-03-02 10:43:06 +00:00