3bd24b67ba
* feat: extend flight client to accept multiple (changing) schemas See #4849. Originally I intended not to use Flight at all for the new ingester<>querier protocol. However since flight also deals with dictionary batches and multiple batches and the gRPC protocol that I would write would look very similar, I will use Flight with a bit more flexible message types. The rough idea for the protocol is the following stream: - for each partition: 1. "none" message with partition metadata 2. for each chunk (can have different schemas under certain circumstances): 1. "schema" message (resets dictionary state) 2. (optional) dictionary batch messages 3. one or more "record batch" message The nice thing about it is that the same arrow client works also for the existing client<>querier protocol since there we just send: 1. "schema" message (no app metadata) 2. (optional) dictionary batch messages 3. zero, one or more "record batch" message (no app metadata) * refactor: separate high- and low-level flight client It is very unlikely that a user will use the high-level batch-producing functionality and the low-level stuff within the same session. So let's split this into to clients (high-level uses the low-level one internally) to avoid confusion. Also add documentation on our protocol handling. * refactor: enumerate all variants in match statement to better catch errors in the future |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| build.rs | ||