Selectors that are value-based (like "min" and "max") and that also
track "other" fields (so things apart from the timestamp and the value),
panicked when the saw the same min/max value in different batches but
the timestamp did was NOT smaller than the previously selected value. An
example would be:
Batch | time | value | other
----- | ---- | ----- | ------
1 | 0 | 0 | 0
2 | 0 | 0 | 1
This adds the fix and a test.
Closes https://github.com/influxdata/idpe/issues/17968 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds "topic" support, allowing a node to subscribe to one or more types
of application payloads independently.
A gossip node is optionally initialised with a set of topics (defaulting
to "all topics") and this set of topic interests is propagated
throughout the cluster via the usual PEX mechanism, alongside the
existing connection & identity information.
When broadcasting an application payload, the sender only transmits it
to nodes that had registered an interest in this payload type. This
prevents wasted network bandwidth and CPU for all nodes, and allows
multiple, distinct payload types to be propagated independently to keep
subsystems that rely on gossip decoupled from each other (no giant,
brittle payload enum type).
Defines a Topic enum in the common generated_types package (alongside
the proto definitions of the actual gossip payloads).
This Topic enum will be used to identify message types across the gossip
transport.
Adds an (unused) Topic and TopicSet type to the gossip crate.
A Topic is a number from 0 to 63 inclusive that uniquely identifies a
"type" of application payload. The TopicSet efficiently encodes the set
of topics a node is interested in as a 64-bit wide bitmap, with each
Topic encoded as a single bit, shifted right N times for the topic
value.
This allows for very cheap "is this node interested" set evaluation and
space-efficient propagation of interest sets during PEX.
1. Use `Display` instead of `Debug` for error formatting. The latter one
is pretty verbose (it's basically Rust struct syntax) but also misses
a LOT of documentation / wording that is important. Errors should
always be printed using `Display` (and we do that basically
everywhere in our codebase).
2. Use structured logging.
Ref https://github.com/influxdata/idpe/issues/17967 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* chore: Make it clear there is only a single DataFusion memory pool
* fix: assert there is a single bridge
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
`Predicate` is InfluxRPC specific and contains way more than just
filter expression.
Ref #8097.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: add tracking of why bytes are written in simulator
* chore: enable breakdown of why bytes are written in a few larger tests
* chore: enable writes breakdown in another test
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This commit hooks up the `disk_full_protection` module's
`guard_disk_capacity` task with the disk space metrics reporter attached
to the `IngesterGuard`.
This commit adds a `disk_full_protection` module for the ingester that
contains a task for guarding disk capacity, by setting the ingest error
state when at 90% disk used up, rotating the WAL, draining & persisting
the buffer tree and then tidying up the fully persisted closed segment
files.
This should prevent the CPU-bound DataFusion runtime from stalling our
main IO runtime.
This is similar to:
- `iox_query::exec::cross_rt_stream`
- https://github.com/apache/arrow-rs/pull/4015
- https://github.com/apache/arrow-rs/pull/4040
Note: I currently have no concrete evidence that this is an issue, but
worker stalling in tokio is really hard to debug and I would like to be
better safe than sorry.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Remove LIST operations from `CachedObjectStore` because:
- **not used/desired:** The querier should NEVER use LIST operations
on the object store. All the planning is done using catalog data.
- **misleading interface:** The `CachedObjectStore` -- that stores
parquet data -- should not implement uncached LIST operations,
because this is misleading. This operation will never be cached.
Or in other words: less code, less potential bugs.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Dom
kodiakhq[bot]
dependabot[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann
Fraser Savage
Joe-Blount
Joe-Blount
Nga Tran