- take span ctx directly instead of the execution context (see point
below)
- use the original trace ID (i.e. the one that we get via HTTP header),
NOT some internal span/trace because the latter is only available for
sampled requests, while the former one is generally more available (we
also do that for the stdout logs btw.)
- minor code clean ups
This is prep work for #8774.
* chore: Update DataFusion pin again
* chore: update for different type
* fix: statistics
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat(querier): convert timezone sent from ingester
In order to facilitate the change of default timezone from None to
UTC make the querier able to convert the timezone sent from the
ingester into its preferred type. This can convert from None to UTC
or UTC to None and should allow the interaction between ingesters
and queriers with differing settings for the default timezone.
To allow testing of both conversions, the type checking has been
made more liberal when converting an arrow schema to an IOx one.
* fix: fmt
* fix: lint
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
For #8350, we won't have all the record batches from the ingester during
planning but we'll stream them during the execution. Technically the
DF plan is already based on streams, it's just `QueryChunkData` that
required a materialized `Vec<RecordBatch>`. This change moves the stream
creation up so a chunk can decide to either use `QueryChunkData::in_mem`
(which conveniently creates the stream) or it can provide its own
stream.
There where like 3 layers (metrics, observer, pruner) that all only had
a single implementation. IIRC this is a leftover from older code where
`iox_query` was more involved in query pruning. With #8705 however the
chunk pruning is pushed even closer to the source (i.e. the querier
code) and it is just more practical to perform the metric management
directly in the querier code (this was the case already, it was just
somewhat hidden by the interfaces). This also allows us to add metrics
for #8705 more easily.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This change causes WAL replay to mimic the RPC write handler, mostly
respecting the `IngestState` before apply an op while replaying a WAL
file. The caveat is that `DiskFull` is ignored as WAL replay specifically
helps with this state.
This requires the `IngestState` and associated types to be public so
that WAL replay can be called by the benchmarker. The module containing
the `IngestState` is private and is only conditionally re-exported under
the benchmark feature as part of the `internal_implementation_details`
module.
Use output of #8725 within the column ranges of the querier. Currently
this won't have any effect since the column ranges are only used to
prune parquet files and parquet files come with their own, more precise
time range (and that information has priority). However for #8705 we
want to use it to prune partitions before needing to deal with the
parquet files.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
We shall ignore certain error cases during query processing. This layer
provides an easy interface for that.
Note that this is also done in the V1 client, just in a more hidden /
hard-to-test manner.
For #8349.
Defines an RPC service to be used by two peers to converge their schema
cache content by exchanging their serialised Merkle Search Tree pages (a
compact representation of the MST itself).
This will be used in the latter half of the following sync protocol:
┌─────┐ ┌────┐
│Local│ │Peer│
└──┬──┘ └─┬──┘
│ [1] Send content hash │
│────────────────────────>
│ │
│ ┌───────────────┐
│ │Compute hash, │
│ │stop if equal │
│ └───────────────┘
│ │
│ ╔════════════════╗ │
═══════════════════════╪═══╣ Switch to gRPC ╠═══╪═══════════════════════
│ ╚════════════════╝ │
│ │
│[2] Serialised MST pages│
│<────────────────────────
│ │
┌──────────────┐ │
│Perform diff │ │
└──────────────┘ │
│ [3] Inconsistent pages │
│────────────────────────>
│ │
│ │
╔═══════╤════╪════════════════════════╪════════════╗
║ LOOP │ For each inconsistent page │ ║
╟───────┘ │ │ ║
║ │ [4] Scheams │ ║
║ │<──────────────────────── ║
╚════════════╪════════════════════════╪════════════╝
┌──┴──┐ ┌─┴──┐
│Local│ │Peer│
└─────┘ └────┘
The initial consistency probe request [1] is sent over gossip and is
used to trigger a further sync of inconsistent MST content if necessary.
This message is a no-op if the MSTs are found to be fully consistent.
If an inconsistency is detected between the two peers, the protocol
switches to perform RPC over TCP, calling the AntiEntropyService defined
in this commit to complete the sync process.
The receiver of the consistency probe [1] calls GetTreeDiff and provides
the their MST pages [2], causing the local node to compute the diff
between the two MSTs, and return the set of inconsistent ranges [3] that
require convergence.
Once the set of inconsistent ranges have been identified, the peer pulls
all the schemas within those ranges and merges them into the local cache
to ensure it has all the content of the source node.
Once this protocol has run in both directions between two peers (and in
absence of further updates between runs) then these two peers are
guaranteed to have converged.
* feat: impl `PartialEq + Eq` for `TestError`
* feat: i->q V2 circuit breaker
This is a straight port from V1, it even uses the same test. The code is
copied though (instead of reusing the old one) because the interface in
the V2 client is so different and the new testing infra is also nicer
(IMHO).
For #8349.
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Marco Neumann
kodiakhq[bot]
Andrew Lamb
Joe-Blount
Nga Tran
Dom
Martin Hilton
Fraser Savage
dependabot[bot]
Carol (Nichols || Goulding)