6399d2b861
chore(deps): Bump uuid from 1.3.0 to 1.3.1 ( #7483 )
...
Bumps [uuid](https://github.com/uuid-rs/uuid ) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases )
- [Commits](https://github.com/uuid-rs/uuid/compare/1.3.0...1.3.1 )
---
updated-dependencies:
- dependency-name: uuid
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-10 17:32:37 +00:00
5f43f2a719
refactor: remove old query planning code ( #7449 )
...
Closes #7406 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-06 16:05:08 +00:00
66982f988b
chore(deps): Bump object_store from 0.5.5 to 0.5.6 ( #7433 )
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.5 to 0.5.6.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/commits )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-04-04 08:43:34 +00:00
275dad704e
chore(deps): Bump futures from 0.3.27 to 0.3.28 ( #7397 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.27 to 0.3.28.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.27...0.3.28 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-31 10:44:23 +00:00
86c55918ec
chore(deps): Bump tempfile from 3.4.0 to 3.5.0 ( #7376 )
...
* chore(deps): Bump tempfile from 3.4.0 to 3.5.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 11:15:54 +00:00
9ddca96336
chore(deps): Bump thiserror from 1.0.39 to 1.0.40 ( #7375 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.39 to 1.0.40.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.39...1.0.40 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-30 10:43:28 +00:00
4eedb7ea77
chore(deps): Bump async-trait from 0.1.66 to 0.1.68 ( #7374 )
...
* chore(deps): Bump async-trait from 0.1.66 to 0.1.68
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.66 to 0.1.68.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.66...0.1.68 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-03-30 10:14:36 +00:00
9cbcdc7672
chore(deps): Bump tokio from 1.26.0 to 1.27.0 ( #7373 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.26.0...tokio-1.27.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-30 09:36:04 +00:00
c7b3565990
refactor: Compare data inside the partition as persisted
...
Use tokio::task::yield_now to allow the persist task to
be enqueued instead of tokio::time::advance.
2023-03-28 11:28:53 +01:00
de043f93e5
feat(ingester2): Add test coverage for hot partition persistence
...
This commit adds a test for the hot partition persistence
post-write observer functionality.
2023-03-27 12:46:54 +01:00
3d545369ae
feat(persist): Do not capture partition ID for hot persist metric
...
Capturing this information has unbounded cardinality and is not
worth the performance and operational implications.
2023-03-23 11:33:17 +00:00
3fbab1a335
feat(persist): Add hot partition persistence counter metric
...
This counter tracks how many times each partition has been
enqueued for persistence as a result of hitting the pre-configured
limit.
2023-03-22 11:41:31 +00:00
eb72fbf3d3
test(ingester): graceful shutdown
...
Adds a ingester e2e test asserting the graceful shutdown behaviour;
specifically that:
* Buffered data is uploaded to object storage
* WAL files are cleaned up and empty (no replay on startup)
2023-03-21 14:21:56 +01:00
fba15cb521
refactor: wal file process completion notification
...
Allow a caller to await processing of a submitted WAL segment in the WAL
reference tracker.
This allows callers to (optionally) order operations that should happen
after the file has been processed.
2023-03-17 14:16:58 +01:00
1958541460
refactor: waker for empty WAL files
...
Implement a waker primitive, allowing a caller to be woken once there
are no inactive WAL files.
This enables a caller to sequence operations to happen after deletion of
inactive WAL files.
2023-03-17 14:16:58 +01:00
094ea0e73b
refactor(test): disambiguate waker
...
The "wal waker" is a waker that is invoked when a WAL file is deleted.
If there's more than one waker floating around, this can be a confusing
name.
2023-03-17 14:16:57 +01:00
20ec47b00b
feat: virtual chunk order col ( #7240 )
...
* feat: introduce `CHUNK_ORDER_COLUMN_NAME`
* feat: impl `ChunkOrder` everywhere
* feat: `ChunkOrder::get`
* feat: emit chunk order column for `RecordBatchesExec`
* feat: `chunk_order_field`
* feat: chunk order col for parquet chunks
* feat: optional chunk order col handling for dedup
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-17 09:39:21 +00:00
78c8429caa
chore: split reference_tracker module into submods
...
Breaks the reference_tracker module into sub-modules - this keeps the
code a little tidier. Soon I'll add more code, so this one file was
getting really big!
2023-03-16 14:35:29 +01:00
43d174b891
refactor: remove Clone derive on CompletedPersist
...
This type is exposed as an Arc-wrapped (reference counted) shared item
to avoid cloning the underlying data, and as such this type should not
be cloned directly.
2023-03-16 11:55:01 +01:00
8529e9551f
Merge branch 'main' into dom/wal-ref-metrics
2023-03-15 17:54:01 +00:00
3c97cb31a3
chore(ingester2): Simplify err match arm variable capture
2023-03-15 16:39:39 +00:00
882d087dd5
chore(ingester2): Log well-known query error at dbg
...
When the ingester handles a query for a table/namespace
that has been persisted and the data is not in the
buffer it logs at error level that the table/namespace
could not be found. This is a valid state and an
expected error (the data can be queried from object
storage), so we can make this less noisy.
2023-03-15 15:41:34 +00:00
5e94aee44a
feat(wal): reference tracker metrics
...
Adds three metrics to expose the internal state of the WAL file
reference tracker.
These metrics are mainly useful to identify why WAL files are not being
deleted, if any.
2023-03-15 15:55:15 +01:00
84a5f291ee
docs: use correct var name
...
Co-authored-by: Fraser Savage <fraser@savage.engineer>
2023-03-15 14:24:19 +00:00
bef29a6837
feat(wal): segement file reference counting
...
Reference count rotated WAL segment files, tracking the number of
unpersisted operations in each and deleting the file once all the data
within them has been persisted to object storage.
This commit contains the implementation of the reference counting logic,
and is currently unused. A follow-up PR will wire this into the various
places needed to feed it the necessary information.
Part of https://github.com/influxdata/influxdb_iox/issues/6566 .
2023-03-14 17:29:32 +01:00
f5c50094ca
refactor: owned persisted SequenceNumberSet copy
...
Allow an owned SequenceNumberSet to be destructured from an owned
CompletedPersist notification.
Additionally allow an owned SequenceNumberSet to be obtained from a
(potentially) shared Arc<CompletedPersist> in a memory efficient / no
cloning way. A CompletedPersist notification is typically wrapped in an
Arc, and at time of writing, typically referenced only once.
2023-03-14 17:29:16 +01:00
3a9ca8879b
chore(deps): Bump futures from 0.3.26 to 0.3.27 ( #7193 )
...
Bumps [futures](https://github.com/rust-lang/futures-rs ) from 0.3.26 to 0.3.27.
- [Release notes](https://github.com/rust-lang/futures-rs/releases )
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.26...0.3.27 )
---
updated-dependencies:
- dependency-name: futures
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-13 10:53:59 +00:00
efdb7bf72d
refactor: Enable reuse of a metric test macro
2023-03-08 14:52:52 -05:00
8771dcb645
chore(deps): Bump thiserror from 1.0.38 to 1.0.39 ( #7131 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.38 to 1.0.39.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.38...1.0.39 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-06 10:55:26 +00:00
3689827793
chore(deps): Bump paste from 1.0.11 to 1.0.12 ( #7130 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.11...1.0.12 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:40:41 +00:00
8f3a9396d0
chore(deps): Bump async-trait from 0.1.64 to 0.1.66 ( #7129 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.64 to 0.1.66.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.64...0.1.66 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-06 10:13:29 +00:00
5c86f0570e
style: remove unnecessary nesting
...
I'm not sure why rustfmt didn't do this, but it removes a level of
indenting.
2023-03-03 17:19:21 +01:00
3256fcc72e
chore(deps): Bump object_store from 0.5.4 to 0.5.5
...
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.4 to 0.5.5.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.4...object_store_0.5.5 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-03 02:00:51 +00:00
6c807ad37a
Merge branch 'main' into dom/wal-cancellation
2023-03-02 21:10:59 +00:00
bde300c988
fix: WAL/buffer commit cancellation safety
...
Ensure that a write that is added to the WAL is always attempted to be
applied to the BufferTree.
This covers off the case of a user submitting a write, waiting long
enough for it to be added to the WAL buffer, and then disconnecting
before it is added to the BufferTree (and before they get a response).
This is a minor issue on its own, but fixing it is necessary for correct
reference counting of WAL files:
https://github.com/influxdata/influxdb_iox/issues/6566
This also documents a low-risk opportunity for the WAL contents &
BufferTree to diverge, potentially leading to a crash-loop at startup:
https://github.com/influxdata/influxdb_iox/issues/7111
In practice a crash loop is unlikely, as it would require broken
invariants elsewhere (no schema validation being applied).
Closes https://github.com/influxdata/influxdb_iox/issues/6281 .
2023-03-02 15:48:41 +01:00
e442957ae5
refactor: derive Clone on instrumentation wrappers
...
If the inner types are Clone-able, the instrumentation wrappers become
clone-able too. Both wrappers are cheap to clone.
2023-03-02 15:48:41 +01:00
cf228fa38b
feat(ingester): cancellation-safe future helper
...
Allow a future to be wrapped such that it is always driven to completion
(obviously excluding deadlocks/process crashes).
2023-03-02 15:48:41 +01:00
0b40e0d17c
feat(wal): SequenceNumberSet for rotated file
...
Changes Wal::rotate() to return the SequenceNumberSet containing the IDs
of all writes in the segment file that is rotated out.
2023-03-02 10:58:03 +01:00
c538cac4ef
chore(deps): Bump tokio from 1.25.0 to 1.26.0 ( #7107 )
...
* chore(deps): Bump tokio from 1.25.0 to 1.26.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.25.0 to 1.26.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.25.0...tokio-1.26.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-02 09:50:39 +00:00
bbd471718d
docs: hot partition persistence
...
Document the hot partition persistence configuration values.
2023-03-01 14:27:07 +01:00
cfd377d12a
refactor: use as_ for cheap conversion methods
...
This follows with the naming conventions of rust - cheap conversions are
prefixed "as_" and not "to_".
2023-03-01 14:27:06 +01:00
e4089acbae
docs(persist): fix-up handle usage
...
Previously a PersistHandle was cloned for sharing, now it is not
(shareable by wrapping in an Arc).
This fixes the documentation to reflect the change in expected usage.
2023-03-01 14:27:06 +01:00
be661890c5
docs: module-level overviews
...
Adds one-liner documentation of what each module contains - this is
helpful to understand what is where, when looking at the rendered docs.
2023-03-01 14:27:05 +01:00
2a8731dd90
docs: ingester overview documentation
...
Adds "overview" documentation for the ingester, including the high-level
purpose & design. Each subsystem is briefly documented, with links for
jumping-off points into more specific documentation.
2023-03-01 14:27:05 +01:00
90ceb2c896
chore(deps): Bump crossbeam-utils from 0.8.14 to 0.8.15 ( #7089 )
...
Bumps [crossbeam-utils](https://github.com/crossbeam-rs/crossbeam ) from 0.8.14 to 0.8.15.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases )
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md )
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-utils-0.8.14...crossbeam-utils-0.8.15 )
---
updated-dependencies:
- dependency-name: crossbeam-utils
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-01 10:16:09 +00:00
faae5eb438
chore: Rerun cargo hakari manage-deps
2023-02-27 11:56:15 +01:00
4cf066680e
chore(deps): Bump tempfile from 3.3.0 to 3.4.0 ( #7069 )
...
* chore(deps): Bump tempfile from 3.3.0 to 3.4.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases )
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS )
- [Commits](https://github.com/Stebalien/tempfile/commits )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-02-27 10:02:21 +00:00
0927031857
Merge branch 'main' into dom/query-buckets
2023-02-23 17:46:31 +00:00
90f8166b44
refactor(metric): increase partition bucket range
...
More buckets -> more better.
2023-02-23 18:01:56 +01:00
21654126b2
Merge branch 'main' into dom/log-query-errors
2023-02-23 17:01:19 +00:00
dependabot[bot]
Marco Neumann
Fraser Savage
Dom Dwyer
Carol (Nichols || Goulding)
kodiakhq[bot]