* feat(idpe-17789): scheduler job_status() (#8121)
This block of work moves into the scheduler some of the specific downstream actions affiliated with compaction outcomes. Which responsibilities stay in the compactor, versus moved to the scheduler, roughly followed the heuristic of whether the action (a) had an impact on global catalog state (a.k.a. commits and partition skipping), (b) whether it's logging affiliated with compactor health (e.g. ParitionDoneSink logging outcomes) versus system health (e.g. logging commits), and (c) reporting to the scheduler on any errors encountered during compaction. This boundary is subject to change as we move forward.
Also, a noted caveat (TODO) on this commit. We have a CompactionJob which is used to track work handed off to each compactor. Currently it still uses the partition_id for tracking, but the followup PR will start moving the compactor to have more CompactionJob uuid awareness.
* feat(influxql): support TOP and BOTTOM functions
Add support for the TOP and BOTTOM functions which return the first
n rows in some ordered data set.
* fix: clippy
* refactor(influxql): use window aggregates for selectors
Change the implentation of ProjectionType::Selector to use a window
aggregate, rather than an aggregate with a custom selector function.
This is in preparation for implementing PERCENTILE.
* feat(influxql): PERCENTILE selector
Add a selector for the row containing the nth percentile of a
partition. This is the behaviour used when a single selector function
is used in an influxql query.
* feat(influxql): PERCENTILE aggregator
Add the PERCENTILE aggregation function for when the PERCENTILE
function is used in an aggregating projection. This implementation
buffers all non-null field values in memory in order to perform the
operation and therefore could be an expensive operation. This is
necessary for compatibility with earlier influxdb versions.
* refactor(influxql): move PERCENTILE implementation out of plan
The plan module is getting rather full of user-defined function
implementations. This breaks the new functions used to implement
percentile into some new top-level modules for aggregate and window
UDFs.
* fix: doc-lint
* chore: refactor `find_enumerated`
* chore: use `s` in format string
* chore: include the unexpected selector function in the error
* chore(influxql): review suggestions
Added some addition comments to help understanding.
Changed the handling os slector functions such that FIRST, LAST,
MAX & MIN behave the same as they did before PERCENTILE was added.
* chore(influxql): make percent_row_number a window UDF
Now that user-defined window functions are available make the
percent_row_number function be one of those. this allows the values
to be calculated for the entire window partition in one go.
For some reason the user-defined window function cannot return NULL
values. This function uses 0 where it would otherwise use NULL, as
row numbering starts at 1.
---------
Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Remove the logical complexity of error handling for an error that cannot
occur.
This was an artifact of pre-PR refactoring - the error being returned
SHOULD never be reached, as the only error returned is the "your message
is too big" error, and that's not possible because the message size is
validated in the GossipHandle::broadcast() method before it reaches the
reactor.
Emit metrics tracking the number of bytes sent / received, and number of
frames sent / received by the local node.
Track the number of discovered peers to record peer discovery rate and
current number of known peers per node.
Calculate the available byte size for a user payload sent via gossip,
and pro-actively check this limit earlier, when the caller is attempting
to send the frame, rather than later in the reactor where there's no
feedback to the caller.
DRY frame serialisation to simplify enforcement, and validate/refuse
oversized frames in the reactor so that frames are unlikely to be
truncated by receivers.
Adds a simple "gossip" implementation (more accurately described as a
pub/sub primitive currently) that supports broadcasting
application-level messages to the set of active peers.
This implementation uses UDP as a transport for best-effort delivery,
and enables zero-copy use of the payload using the Bytes crate.
Only peers explicitly provided as "seeds" when initialising will be
known to a gossip node - there's currently no peer exchange mechanism.
This implementation tolerates seeds changing their DNS entries when
restarting to point at new socket addresses (such as within Kubernetes
when pods move around).
Adds a proto definition and configures prost to build the rust types
from it.
The gossip framing is intended to be flexible and decoupled - the gossip
library will batch together one or more opaque application messages
and/or control frames, and uniquely identify each peer with a
per-instance UUID to detect crashes/restarts and track peers.
wiedld
Joe-Blount
Andrew Lamb
Dom
dependabot[bot]
Martin Hilton
Carol (Nichols || Goulding)
kodiakhq[bot]
Fraser Savage
Joe-Blount
Marko Mikulicic