fd8a89deea
feat(ingester): WIP - WAL rotate task uses reference tracker for delete
...
This is the first commit in line to connect the WAL segment reference
tracker actor up to the rest of the ingester. It removes the segment file
deletion and hacky sleep from the rotate task, deferring to the actor
for deletion tracking.
2023-07-05 15:23:37 +01:00
7b2ef53c7b
refactor(ingester): Notify `SequenceNumberSet` when tracking unbuffered writes
...
Writes now contain multiple sequence numbers, so the WAL reference
actor must be notified of *all* sequence numbers contained for a write
that failed to be applied to the buffer.
2023-07-05 15:13:29 +01:00
2da99f8032
refactor: Use `const` instead of unnecessary lazy_static
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-07-05 14:42:55 +01:00
e74a7a7dd4
test(wal): Test correct assignment of write per-partition sequence numbers
...
This adds extra test coverage for the ingester's WAL replay & RPC write
paths, as well as the WAL E2E tests, to ensure that all sequence numbers
present in a WriteOperation/WalOperation are encoded and present when
decoded.
2023-07-05 14:42:47 +01:00
e6e09d0c15
feat(ingester): Assign individual sequence numbers for writes per partition
...
This commit asks the oracle for a new sequence number for each table
batch of a write operation (and thus each partition of a write) when
handling an RPC write operation before appending the operation to the
WAL. The ingester now honours the sequence numbers per-partition when
WAL replay is performed.
2023-07-05 14:29:27 +01:00
54a8f7d007
feat(data_types): Add `Extend<SequenceNumberSet>` for `SequenceNumberSet`
...
Although callers could manually extend the sequence number set by continually
adding in an iterator loop or a fold expression, this enables other
combinator patterns when dealing with collections of sequence number
sets.
2023-07-05 14:23:18 +01:00
30939cfe96
refactor(wal): Remove op-level `sequence_number`, use per table map
...
This commit removes the op-level sequence number from the proto
definition, now reading and writing solely to the per table (and thus
per partition) sequence number map. Tables/partitions within the same
write op are still assigned the same number for now, so there should be
no semantic different
2023-07-05 14:20:43 +01:00
5815df5e6d
Merge pull request #8069 from influxdata/savage/use-u64-for-sequence-number
...
refactor(ingester): Use unsigned sequence number, remove its `Sqlx::Type`
2023-07-05 13:01:10 +00:00
70a6e60415
Merge branch 'main' into savage/use-u64-for-sequence-number
2023-07-05 12:55:44 +00:00
35d93f9475
fix: include `PartitionHashId` in size estimations ( #8153 )
...
As for the other types: size estimations are conservative, so we assume
the value behind the `Arc` is owned by the estimating party.
2023-07-05 10:42:39 +00:00
3827257f94
chore(deps): Bump thiserror from 1.0.40 to 1.0.41 ( #8149 )
...
Bumps [thiserror](https://github.com/dtolnay/thiserror ) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.40...1.0.41 )
---
updated-dependencies:
- dependency-name: thiserror
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-07-05 09:25:14 +00:00
9c65185068
refactor: normalize catalog metric names ( #8152 )
...
Use the same prefix for all metrics of the same repo type. This makes
reading dashboards way easier.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-05 09:18:39 +00:00
7c771ce6f3
chore(deps): Bump serde_json from 1.0.99 to 1.0.100 ( #8147 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.99 to 1.0.100.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.99...v1.0.100 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-05 09:12:11 +00:00
b5c9628f0f
chore(deps): Bump async-trait from 0.1.69 to 0.1.70 ( #8148 )
...
Bumps [async-trait](https://github.com/dtolnay/async-trait ) from 0.1.69 to 0.1.70.
- [Release notes](https://github.com/dtolnay/async-trait/releases )
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.69...0.1.70 )
---
updated-dependencies:
- dependency-name: async-trait
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-05 09:05:13 +00:00
9f6acfa3d9
chore: update `hermit-abi` to 0.3.2 ( #8150 )
...
0.3.1 got yanked.
2023-07-05 08:59:12 +00:00
4873093e1a
Merge pull request #8058 from influxdata/sgc/issue/test_conflict
...
chore: add test to demonstrate aliasing rules for conflict
2023-07-04 23:16:12 +00:00
2a5a13f840
Merge branch 'main' into sgc/issue/test_conflict
2023-07-05 09:10:47 +10:00
d2658d2e32
chore: update heappy to `1de977a241cdd768acc5b6c82c0728b30c7db7b4` ( #8145 )
...
Removes some redundant deps.
2023-07-04 14:14:46 +00:00
b827b76583
chore(deps): Bump console-subscriber from 0.1.9 to 0.1.10 ( #8142 )
...
Bumps [console-subscriber](https://github.com/tokio-rs/console ) from 0.1.9 to 0.1.10.
- [Release notes](https://github.com/tokio-rs/console/releases )
- [Commits](https://github.com/tokio-rs/console/compare/console-subscriber-v0.1.9...console-subscriber-v0.1.10 )
---
updated-dependencies:
- dependency-name: console-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 12:13:49 +00:00
36c253123a
chore(deps): Bump pprof from 0.11.1 to 0.12.0 ( #8141 )
...
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.11.1 to 0.12.0.
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 12:00:59 +00:00
71795f265f
chore(deps): Bump serde from 1.0.164 to 1.0.166 ( #8140 )
...
Bumps [serde](https://github.com/serde-rs/serde ) from 1.0.164 to 1.0.166.
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.164...v1.0.166 )
---
updated-dependencies:
- dependency-name: serde
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 10:56:56 +00:00
9a03d9c9fe
chore(deps): Bump paste from 1.0.12 to 1.0.13 ( #8139 )
...
Bumps [paste](https://github.com/dtolnay/paste ) from 1.0.12 to 1.0.13.
- [Release notes](https://github.com/dtolnay/paste/releases )
- [Commits](https://github.com/dtolnay/paste/compare/1.0.12...1.0.13 )
---
updated-dependencies:
- dependency-name: paste
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-04 07:57:41 +00:00
20238f30ab
feat(querier): Log errors that happen *during* query execution ( #8137 )
2023-07-03 19:19:54 +00:00
1acbf4a20d
Merge pull request #8125 from influxdata/cn+savage/ingester-partition-pruning
...
feat: partition pruning for ingester query responses
2023-07-03 16:38:07 +01:00
0297fe3651
refactor: less nesting in partition pruning logic
...
Improve readability by pulling the partition pruning logic into it's own
function and clean up some minor bits.
2023-07-03 17:25:03 +02:00
edf6686130
fix(test): custom partitioning template pruning
...
Configure the partition pruning test to use a partition template that
partitions on the "region" field. This will allow it to be used for
pruning at query time.
2023-07-03 17:25:03 +02:00
36ed914689
test: type coercion in ingester tests
2023-07-03 17:25:02 +02:00
171b2a14c7
fix: doc link
2023-07-03 17:25:01 +02:00
70b44f78ee
test: correctly decode ingester reponses in end2end tests
2023-07-03 17:25:01 +02:00
b1a4e3955e
test: `ingester_partition_pruning` must perform type coercion
2023-07-03 17:25:00 +02:00
e9b456df1f
fix: do not panic for pruning errors
2023-07-03 17:25:00 +02:00
0bcf85d48c
refactor: de-dup code
2023-07-03 17:24:59 +02:00
cd28bf0337
test: Query an ingester with a predicate that should prune partitions
2023-07-03 17:24:58 +02:00
8ebf390d9c
feat: Try to prune ingester partitions by partition key
...
This is hacktastic.
2023-07-03 17:24:58 +02:00
da34eb7b35
feat: Load both table name and partition template in the ingester
2023-07-03 17:24:57 +02:00
5f759528d3
test(ingester): Add `BufferTree` test for predicate-filtered queries
2023-07-03 17:24:56 +02:00
246c2b0749
refactor(ingester): Accept a predicate as parameter to `query_exec`
...
This will allow the ingester to apply a predicate when serving a query
and only stream back data that satisfies the predicate.
2023-07-03 17:24:56 +02:00
b76fdab1a4
refactor: Move querier::df_stats to iox_query::chunk_statistics so it can be shared with ingester
2023-07-03 17:24:55 +02:00
83ac267a97
Merge pull request #8136 from influxdata/dom/no-rustflags-mess
...
build: don't force tokio_unstable / override user config
2023-07-03 16:16:24 +01:00
e8065f871a
Merge branch 'main' into dom/no-rustflags-mess
2023-07-03 15:57:46 +01:00
dba8c553cf
Merge pull request #8135 from influxdata/dom/shutdown-file-discovery
...
test(e2e): ingester shutdown persist file discovery
2023-07-03 13:56:23 +00:00
05f5f80805
Merge branch 'main' into dom/shutdown-file-discovery
2023-07-03 13:51:19 +00:00
e5a9e1534a
test: assert 1 file persisted
...
There should be a single file persisted during graceful shutdown.
2023-07-03 15:51:02 +02:00
5d0c172e61
test(e2e): query shutdown-persisted files
...
Ensure buffered ingester data is persisted and remains queryable after a
graceful ingester shutdown.
2023-07-03 15:51:02 +02:00
4e38fbc887
feat(e2e): configurable querier circuit breakers
...
Allow the querier's circuit breaker thresholds to be configured in test
runs - this helps speed up tests that involve hitting offline ingesters.
2023-07-03 15:51:01 +02:00
21757d39ef
chore: don't doc behind tokio_unstable flags
...
Unfortunately there's no good way to conditionally document:
https://doc.rust-lang.org/beta/unstable-book/language-features/doc-cfg.html
Currently unstable.
2023-07-03 15:42:35 +02:00
3a8d234b1c
build: optional tokio_unstable support
...
This makes using the tokio_unstable feature optional - the build
configuration defines what is enabled, rather than the other way around.
By default, and for prod builds and users who have not set a RUSTFLAGS,
the tokio_unstable is enabled and this is leveraged to provide the
metrics exposed by the flag. For builds where tokio_unstable is not
enabled, these metrics are not included, without causing a compilation
error.
CI will against the flag-enabled/prod code by virtue of being the
default.
2023-07-03 15:28:59 +02:00
ea4aa9352f
chore: remove build-time tokio-unstable cfg check
...
This is incompatible with envs that set RUSTFLAGS (by design), however
this is a bit of a over step - people should be able to configure their
env!
2023-07-03 15:18:56 +02:00
de9392c019
feat(e2e): assert absolute parquet count
...
An existing pair of e2e steps allows the caller to assert an increase in
the number of persisted parquet files, but there was no primitive to
assert the current count.
2023-07-03 14:52:35 +02:00
c933c15325
feat(e2e): step to gracefully stop ingesters
...
Adds a step that gracefully stops all ingesters (stop - not restart -
they don't come back!)
2023-07-03 14:34:44 +02:00
Fraser Savage
kodiakhq[bot]
Marco Neumann
dependabot[bot]
Stuart Carnie
Andrew Lamb
Dom
Carol (Nichols || Goulding)