History

Carol (Nichols \|\| Goulding) c8108f01e7 chore: Upgrade to Rust 1.64 (#5727 ) * chore: Upgrade to Rust 1.64 * fix: Use iter find instead of a for loop, thanks clippy * fix: Remove some needless borrows, thanks clippy * fix: Use then_some rather than then with a closure, thanks clippy * fix: Use iter retain rather than filter collect, thanks clippy Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>		2022-09-22 18:04:00 +00:00
..
src	chore: Upgrade to Rust 1.64 (#5727 )	2022-09-22 18:04:00 +00:00
Cargo.toml	chore(deps): Bump itertools from 0.10.4 to 0.10.5 (#5707 )	2022-09-21 08:15:59 +00:00
README.md	ci: fix cargo deny (#4629 )	2022-05-18 09:38:35 +00:00

README.md

IOx Query Layer

The IOx query layer is responsible for translating query requests from different query languages and planning and executing them against Chunks stored across various IOx storage systems.

Query Frontends

SQL
Storage gRPC
Flux (possibly in the future)
InfluxQL (possibly in the future)
Others (possibly in the future)

Sources of Chunk data

ReadBuffer
MutableBuffer
Parquet Files
Others (possibly in the future, like Remote Chunk?)

The goal is to use the shared query / plan representation in order to avoid N*M combinations of language and Chunk source.

Thus query planning is implemented in terms of traits, and those traits are implemented by different chunk implementations.

Among other things, this means that this crate should not depend directly on the ReadBuffer or the MutableBuffer.

┌───────────────┐  ┌────────────────┐    ┌──────────────┐      ┌──────────────┐
│Mutable Buffer │  │  Read Buffer   │    │Parquet Files │  ... │Future Source │
│               │  │                │    │              │      │              │
└───────────────┘  └────────────────┘    └──────────────┘      └──────────────┘
        ▲                   ▲                    ▲                     ▲
        └───────────────────┴─────────┬──────────┴─────────────────────┘
                                      │
                                      │
                     ┌─────────────────────────────────┐
                     │          Shared Common          │
                     │   Predicate, Plans, Execution   │
                     └─────────────────────────────────┘
                                      ▲
                                      │
                                      │
               ┌──────────────────────┼─────────────────────────┐
               │                      │                         │
               │                      │                         │
               │                      │                         │
     ┌───────────────────┐  ┌──────────────────┐      ┌──────────────────┐
     │   SQL Frontend    │  │   gRPC Storage   │ ...  │ Future Frontend  │
     │                   │  │     Frontend     │      │ (e.g. InfluxQL)  │
     └───────────────────┘  └──────────────────┘      └──────────────────┘

We are trying to avoid ending up with something like this:

                          ┌─────────────────────────────────────────────────┐
                          │                                                 │
                          ▼                                                 │
                   ┌────────────┐                                           │
                   │Read Buffer │                  ┌────────────────────────┤
        ┌──────────┼────────────┼─────┬────────────┼────────────────────────┤
        │          └────────────┘     │            ▼                        │
        ▼                 ▲           │    ┌──────────────┐                 │
┌───────────────┐         │           │    │Parquet Files │                 │
│Mutable Buffer │         │           ├───▶│              │...              │
│               │◀────────┼───────────┤    └──────────────┘   ┌─────────────┼┐
└───────────────┘         │           │            ▲          │Future Source││
        ▲                 │           ├────────────┼─────────▶│             ││◀─┐
        │                 │           │            │          └─────────────┼┘  │
        │                 │           │            │                        │   │
        │                 │           │            │                        │   │
        │      ┌──────────┘           │            │                        │   │
        │      │                      │            │                        │   │
        │      ├──────────────────────┼────────────┘                        │   │
        └──────┤                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
               │                      │                                     │   │
     ┌───────────────────┐  ┌──────────────────┐      ┌──────────────────┐  │   │
     │   SQL Frontend    │  │   gRPC Storage   │ ...  │ Future Frontend  │  │   │
     │                   │  │     Frontend     │      │ (e.g. InfluxQL)  │──┴───┘
     └───────────────────┘  └──────────────────┘      └──────────────────┘