933ab1f8c7
This commit changes the behaviour of the persist system to enable optimal parallelism of persist operations, and improve the accuracy of the outstanding job bound / back-pressure. Previously all persist operations for a given partition were consistently hashed to a single worker task. This serialised persistence per partition, ensuring all updates to the partition sort key were serialised. However, this also unnecessarily serialises persist operations that do not need to update the sort key, reducing the potential throughput of the system; in the worst case of a single partition receiving all the writes, only one worker would be persisting, and the other N-1 workers would be idle. After this change, the sort key is inspected when enqueuing the persist operation and if it can be determined that no sort key update is necessary (the typical case), then the persist task is placed into a global work queue from which all workers consume. This allows for maximal parallelisation of these jobs, and the removes the per-worker head-of-line blocking. In the case that the sort key does need updating, these jobs continue to be consistently hashed to a single worker, ensuring serialised sort key updates only where necessary. To support these changes, the back-pressure system has been changed to account for all outstanding persist jobs in the system, regardless of type or assigned worker - a logical, bounded queue is composed together of a semaphore limiting the number of persist tasks overall, and a series of physical, unbounded queues - one to each worker & the global queue. The overall system remains bounded by the INFLUXDB_IOX_PERSIST_QUEUE_DEPTH value, and is now simpler to reason about (it is independent of the number of workers, etc). |
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.toml | ||
| README.md | ||
README.md
IOx Query Layer
The IOx query layer is responsible for translating query requests from different query languages and planning and executing them against Chunks stored across various IOx storage systems.
Query Frontends
- SQL
- Storage gRPC
- Flux (possibly in the future)
- InfluxQL (possibly in the future)
- Others (possibly in the future)
Sources of Chunk data
- ReadBuffer
- MutableBuffer
- Parquet Files
- Others (possibly in the future, like Remote Chunk?)
The goal is to use the shared query / plan representation in order to avoid N*M combinations of language and Chunk source.
Thus query planning is implemented in terms of traits, and those traits are implemented by different chunk implementations.
Among other things, this means that this crate should not depend directly on the ReadBuffer or the MutableBuffer.
┌───────────────┐ ┌────────────────┐ ┌──────────────┐ ┌──────────────┐
│Mutable Buffer │ │ Read Buffer │ │Parquet Files │ ... │Future Source │
│ │ │ │ │ │ │ │
└───────────────┘ └────────────────┘ └──────────────┘ └──────────────┘
▲ ▲ ▲ ▲
└───────────────────┴─────────┬──────────┴─────────────────────┘
│
│
┌─────────────────────────────────┐
│ Shared Common │
│ Predicate, Plans, Execution │
└─────────────────────────────────┘
▲
│
│
┌──────────────────────┼─────────────────────────┐
│ │ │
│ │ │
│ │ │
┌───────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ SQL Frontend │ │ gRPC Storage │ ... │ Future Frontend │
│ │ │ Frontend │ │ (e.g. InfluxQL) │
└───────────────────┘ └──────────────────┘ └──────────────────┘
We are trying to avoid ending up with something like this:
┌─────────────────────────────────────────────────┐
│ │
▼ │
┌────────────┐ │
│Read Buffer │ ┌────────────────────────┤
┌──────────┼────────────┼─────┬────────────┼────────────────────────┤
│ └────────────┘ │ ▼ │
▼ ▲ │ ┌──────────────┐ │
┌───────────────┐ │ │ │Parquet Files │ │
│Mutable Buffer │ │ ├───▶│ │... │
│ │◀────────┼───────────┤ └──────────────┘ ┌─────────────┼┐
└───────────────┘ │ │ ▲ │Future Source││
▲ │ ├────────────┼─────────▶│ ││◀─┐
│ │ │ │ └─────────────┼┘ │
│ │ │ │ │ │
│ │ │ │ │ │
│ ┌──────────┘ │ │ │ │
│ │ │ │ │ │
│ ├──────────────────────┼────────────┘ │ │
└──────┤ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │
┌───────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │
│ SQL Frontend │ │ gRPC Storage │ ... │ Future Frontend │ │ │
│ │ │ Frontend │ │ (e.g. InfluxQL) │──┴───┘
└───────────────────┘ └──────────────────┘ └──────────────────┘