d79405b367
chore: add deploy tools to CI image ( #7134 )
...
Adds `skopeo` and gcloud CLI to CI image. This should eventually replace
our manual installation during the `deploy_releases` step:
d8d097c183/.circleci/config.yml (L493-L527)https://github.com/influxdata/idpe/issues/17098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-06 10:47:43 +00:00
3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 ( #7130 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
8f3a9396d0
chore(deps): Bump async-trait from 0.1.64 to 0.1.66 ( #7129 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.64 to 0.1.66.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.64...0.1.66 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:13:29 +00:00
d8d097c183
chore(deps): Bump rustix from 0.36.8 to 0.37.1 ( #7128 )
...
* chore(deps): Bump rustix from 0.36.8 to 0.37.1
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.8 to 0.37.1.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.8...v0.37.1 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-06 09:48:01 +00:00
dfd87f3e20
test(compactor): Add test for large amounts of data with a single timestamp ( #7123 )
...
* test(compactor): Add test for large amounts of data with a single timestamp
* fix: Update compactor2/tests/layouts/single_timestamp.rs
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
---------
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
2023-03-03 20:12:23 +00:00
18e183a54b
Merge pull request #7121 from influxdata/dom/seqnum-set
...
feat: SequenceNumberSet intersection + helpers
2023-03-03 19:19:47 +00:00
c74339ebd9
Merge branch 'main' into dom/seqnum-set
2023-03-03 16:28:28 +00:00
069184a6d1
Merge pull request #7120 from influxdata/dom/nesting
...
style: remove unnecessary nesting
2023-03-03 16:28:18 +00:00
5b7a8beff8
Merge branch 'main' into dom/nesting
2023-03-03 16:21:55 +00:00
ac1b37c0f0
feat(data_types): SequenceNumberSet intersection
...
Support computing the intersection of two SequenceNumberSet.
2023-03-03 17:21:39 +01:00
146494f619
perf: pre-allocation of SequenceNumberSet
...
Support pre-allocation of SequenceNumberSet for known-length sets.
2023-03-03 17:21:38 +01:00
7801a63334
feat: apply RLE optimisation for SequenceNumberSet
...
Allow a SequenceNumberSet to be space-optimised by transforming it to
use run length encoding.
2023-03-03 17:21:38 +01:00
043f3421ba
feat(data_types): PartialEq for SequenceNumberSet
...
Derive + test partial equality matching for SequenceNumberSet.
2023-03-03 17:21:38 +01:00
c15d789613
fix: account for memory in GapFill operator ( #7115 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-03 16:21:13 +00:00
5c86f0570e
style: remove unnecessary nesting
...
I'm not sure why rustfmt didn't do this, but it removes a level of
indenting.
2023-03-03 17:19:21 +01:00
f4d64099bf
Merge pull request #7116 from influxdata/jrb_10_l1l2_overlap_tests
...
chore: add tests for L1/L2 invariant violations in simulator
2023-03-03 08:27:52 -06:00
cd159fd08b
Merge branch 'main' into jrb_10_l1l2_overlap_tests
2023-03-03 08:21:33 -06:00
1e80227df8
chore(deps): Bump mockito from 0.32.3 to 0.32.4 ( #7117 )
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 0.32.3 to 0.32.4.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/0.32.3...0.32.4 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-03 10:14:19 +00:00
c50bb82a75
Merge pull request #7118 from influxdata/dependabot/cargo/object_store-0.5.5
...
chore(deps): Bump object_store from 0.5.4 to 0.5.5
2023-03-03 10:07:22 +00:00
3256fcc72e
chore(deps): Bump object_store from 0.5.4 to 0.5.5
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.4 to 0.5.5.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.4...object_store_0.5.5 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-03 02:00:51 +00:00
5e0a0a028b
chore: add tests for L1/L2 invariant violations in simulator
2023-03-02 16:10:12 -06:00
fe14f78e0c
Merge pull request #7112 from influxdata/dom/wal-cancellation
...
fix(wal): write commit cancellation safety
2023-03-02 21:17:39 +00:00
6c807ad37a
Merge branch 'main' into dom/wal-cancellation
2023-03-02 21:10:59 +00:00
caa6a84488
chore: verify split time invariants within simulator ( #7114 )
2023-03-02 20:58:09 +00:00
cb8d55c5b2
Merge pull request #7093 from influxdata/dom/wal-flusher-task-leak
...
fix: wal flusher task / memory leak
2023-03-02 20:50:25 +00:00
0c530aa9f7
Merge branch 'main' into dom/wal-flusher-task-leak
2023-03-02 20:44:02 +00:00
999a5dae03
refactor: sort key cleanups ( #7113 )
...
* refactor: remove unused `ColumnSort`
* refactor: remove invalid assertion
It is true that time SHOULD be the last sort key, but we absoletely
don't require that, esp. not in the query tier. The ingester will
currently always produce sort keys where time is last, but if we ever
going to deal w/ external data sources like bulk loaded parquet files,
this may not always be the case.
Found while constructing some edge case tests.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 16:08:21 +00:00
c3d9219afc
Merge pull request #7110 from influxdata/dom/record-wal-seqnum-sets
...
feat(wal): track set of write IDs in a segment file
2023-03-02 15:20:27 +00:00
8fe874a7f0
Merge branch 'main' into dom/record-wal-seqnum-sets
2023-03-02 15:13:36 +00:00
746d9e4db5
Merge pull request #7094 from influxdata/dom/perf-batch-buffer-reuse
...
perf(wal): avoid batch buffer reallocations
2023-03-02 15:03:19 +00:00
c9338fb785
Merge branch 'main' into dom/perf-batch-buffer-reuse
2023-03-02 14:50:27 +00:00
bde300c988
fix: WAL/buffer commit cancellation safety
...
Ensure that a write that is added to the WAL is always attempted to be
applied to the BufferTree.
This covers off the case of a user submitting a write, waiting long
enough for it to be added to the WAL buffer, and then disconnecting
before it is added to the BufferTree (and before they get a response).
This is a minor issue on its own, but fixing it is necessary for correct
reference counting of WAL files:
https://github.com/influxdata/influxdb_iox/issues/6566
This also documents a low-risk opportunity for the WAL contents &
BufferTree to diverge, potentially leading to a crash-loop at startup:
https://github.com/influxdata/influxdb_iox/issues/7111
In practice a crash loop is unlikely, as it would require broken
invariants elsewhere (no schema validation being applied).
Closes https://github.com/influxdata/influxdb_iox/issues/6281 .
2023-03-02 15:48:41 +01:00
e442957ae5
refactor: derive Clone on instrumentation wrappers
...
If the inner types are Clone-able, the instrumentation wrappers become
clone-able too. Both wrappers are cheap to clone.
2023-03-02 15:48:41 +01:00
cf228fa38b
feat(ingester): cancellation-safe future helper
...
Allow a future to be wrapped such that it is always driven to completion
(obviously excluding deadlocks/process crashes).
2023-03-02 15:48:41 +01:00
9f0645a775
fix(compactor2): fix off by one error in time ranges of simulator ( #7098 )
...
* fix(compactor2): fix off by one error in time ranges of simulator
* chore: update a test that were added recently and this PR fixes it
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: NGA-TRAN <nga-tran@live.com>
2023-03-02 14:43:53 +00:00
2c4da24f73
feat: sort-related phys. optimizers ( #7095 )
...
* feat: `SortPushdown` optimizer
* feat: `RedundantSort` optimizer
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 14:20:13 +00:00
6b35af115e
chore(compactor2): Add tests for remaining file knobs ( #7099 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 12:20:44 +00:00
eb488cf55a
chore(compactor2): Improve documentation on split times ( #7105 )
...
* chore(compactor2): Improve documentation on split times
* fix: Apply suggestions from code review
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
* fix: fmt
* fix: typo
---------
Co-authored-by: Joe-Blount <73478756+Joe-Blount@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 12:00:55 +00:00
b2052c1b81
chore(compactor2): add documentation about the compactor2 file level invariants ( #7101 )
...
* chore(compactor2): add documentation about the compactor2 file level invariants
* fix: Fix doc links
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 11:24:22 +00:00
160e93ea48
Merge branch 'main' into dom/perf-batch-buffer-reuse
2023-03-02 10:43:06 +00:00
f3caf604b5
refactor(wal): last batch length for preallocation
...
There's no need to sub 1 from the batch length to shrink the buffer over
time - the capacity of the new batch will be the length of the last. A
large batch followed by a small batch will cause the pre-allocated next
batch to be small too.
2023-03-02 11:40:38 +01:00
0b40e0d17c
feat(wal): SequenceNumberSet for rotated file
...
Changes Wal::rotate() to return the SequenceNumberSet containing the IDs
of all writes in the segment file that is rotated out.
2023-03-02 10:58:03 +01:00
b22643350f
refactor(wal): track segment sequence numbers
...
Changes the WAL to maintain a SequenceNumberSet containing every ID
wrote to the currently open segment file.
The sets are derived from batched data for efficiency, rather than
recorded per write, to prevent any overhead in the hot path. The batch
set is merged with the file set off the hot path, in a separate I/O
thread (not the async runtime).
2023-03-02 10:58:02 +01:00
6aa33ef380
feat: impl FromIterator for SequenceNumberSet
...
Allow a SequenceNumberSet to be instantiated from an iterator of
SequenceNumber.
2023-03-02 10:58:02 +01:00
6532fb752b
feat: impl Extend for SequenceNumberSet
...
Allow a SequenceNumberSet to be efficiently extended from any iterator
of SequenceNumber instances.
2023-03-02 10:58:01 +01:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
06b2c7a329
chore(deps): Bump sqlparser from 0.30.0 to 0.31.0 ( #7108 )
...
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs ) from 0.30.0 to 0.31.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases )
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.30.0...v0.31.0 )
---
updated-dependencies:
- dependency-name: sqlparser
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-02 09:43:43 +00:00
c95d078e46
feat: add `NestedUnion` opt ( #7092 )
...
* docs: typo
* feat: add `NestedUnion` opt
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:09:05 +00:00
f3267f992a
Merge pull request #7102 from influxdata/cn/share-normalization
...
fix: Use the same normalization code for explain tests as e2e tests do
2023-03-02 03:18:51 +00:00
f730991602
Merge branch 'main' into cn/share-normalization
2023-03-02 03:12:16 +00:00
Marco Neumann
dependabot[bot]
Andrew Lamb
kodiakhq[bot]
Dom
Christopher M. Wolff
Joe-Blount
Joe-Blount