diff --git a/content/influxdb/cloud-dedicated/reference/internals/storage-engine.md b/content/influxdb/cloud-dedicated/reference/internals/storage-engine.md index 508dab4ad..80de0c545 100644 --- a/content/influxdb/cloud-dedicated/reference/internals/storage-engine.md +++ b/content/influxdb/cloud-dedicated/reference/internals/storage-engine.md @@ -23,11 +23,13 @@ queries, and is optimized to reduce storage cost. - [Storage engine diagram](#storage-engine-diagram) - [Storage engine components](#storage-engine-components) + - [Router](#router) - [Ingester](#ingester) - [Querier](#querier) - [Catalog](#catalog) - [Object store](#object-store) - [Compactor](#compactor) + - [Garbage collector](#garbage-collector) - [Scaling strategies](#scaling-strategies) - [Vertical scaling](#vertical-scaling) - [Horizontal scaling](#horizontal-scaling) @@ -38,11 +40,29 @@ queries, and is optimized to reduce storage cost. ## Storage engine components +- [Router](#router) - [Ingester](#ingester) - [Querier](#querier) - [Catalog](#catalog) - [Object store](#object-store) - [Compactor](#compactor) +- [Garbage collector](#garbage-collector) + +### Router + +The Router (also known as the Ingest Router) parses incoming line +protocol and then routes it to [Ingesters](#ingester). +To ensure write durability, the Router replicates data to two or more of the +available Ingesters. + +##### Router scaling strategies + +The Router can be scaled both [vertically](#vertical-scaling) and +[horizontally](#horizontal-scaling). +Horizontal scaling increases write throughput and is typically the most +effective scaling strategy for the Router. +Vertical scaling (specifically increased CPU) improves the Router's ability to +parse incoming line protocol with lower latency. ### Ingester @@ -150,6 +170,20 @@ increasing the available CPU) is the most effective scaling strategy for the Com Horizontal scaling increases compaction throughput, but not as efficiently as vertical scaling. +### Garbage collector + +The Garbage collector runs background jobs that evict expired or deleted data, +remove obsolete compaction files, and reclaim space in both the [Catalog](#catalog) and the +[Object store](#object-store). + +##### Garbage collector scaling strategies + +The Garbage collector is not designed for distributed load and should _not_ be +scaled horizontally. The Garbage collector does not perform CPU- or +memory-intensive work, so [vertical scaling](#vertical-scaling) should only be +considered only if you observe very high CPU usage or if the container regularly +runs out of memory. + --- ## Scaling strategies diff --git a/content/influxdb/clustered/admin/scale-cluster.md b/content/influxdb/clustered/admin/scale-cluster.md index 061a3e702..31aa9042f 100644 --- a/content/influxdb/clustered/admin/scale-cluster.md +++ b/content/influxdb/clustered/admin/scale-cluster.md @@ -262,13 +262,22 @@ to automatically scale your cluster as needed. ## Recommended scaling strategies per component +- [Router](#router) - [Ingester](#ingester) - [Querier](#querier) -- [Router](#router) - [Compactor](#compactor) - [Catalog](#catalog) - [Object store](#object-store) +### Router + +The Router can be scaled both [vertically](#vertical-scaling) and +[horizontally](#horizontal-scaling). +Horizontal scaling increases write throughput and is typically the most +effective scaling strategy for the Router. +Vertical scaling (specifically increased CPU) improves the Router's ability to +parse incoming line protocol with lower latency. + ### Ingester The Ingester can be scaled both [vertically](#vertical-scaling) and @@ -322,13 +331,6 @@ Horizontal scaling increases query throughput to handle more concurrent queries. Vertical scaling improves the Querier’s ability to process computationally intensive queries. -### Router - -The Router can be scaled both [vertically](#vertical-scaling) and -[horizontally](#horizontal-scaling). -Horizontal scaling increases request throughput and is typically the most effective -scaling strategy for the Router. - ### Compactor The Compactor can be scaled both [vertically](#vertical-scaling) and diff --git a/content/influxdb/clustered/reference/internals/storage-engine.md b/content/influxdb/clustered/reference/internals/storage-engine.md index c26add10d..7425df27d 100644 --- a/content/influxdb/clustered/reference/internals/storage-engine.md +++ b/content/influxdb/clustered/reference/internals/storage-engine.md @@ -11,6 +11,7 @@ menu: parent: InfluxDB internals influxdb/clustered/tags: [storage, internals] related: + - /influxdb/clustered/admin/scale-cluster/ - /influxdb/clustered/admin/custom-partitions/ --- @@ -23,11 +24,13 @@ queries, and is optimized to reduce storage cost. - [Storage engine diagram](#storage-engine-diagram) - [Storage engine components](#storage-engine-components) + - [Router](#router) - [Ingester](#ingester) - [Querier](#querier) - [Catalog](#catalog) - [Object store](#object-store) - [Compactor](#compactor) + - [Garbage collector](#garbage-collector) ## Storage engine diagram @@ -35,11 +38,29 @@ queries, and is optimized to reduce storage cost. ## Storage engine components +- [Router](#router) - [Ingester](#ingester) - [Querier](#querier) - [Catalog](#catalog) - [Object store](#object-store) - [Compactor](#compactor) +- [Garbage collector](#garbage-collector) + +### Router + +The Router (also known as the Ingest Router) parses incoming line +protocol and then routes it to [Ingesters](#ingester). +To ensure write durability, the Router replicates data to two or more of the +available Ingesters. + +##### Router scaling strategies + +The Router can be scaled both [vertically](/influxdb/clustered/admin/scale-cluster/#vertical-scaling) +and [horizontally](/influxdb/clustered/admin/scale-cluster/#horizontal-scaling). +Horizontal scaling increases write throughput and is typically the most +effective scaling strategy for the Router. +Vertical scaling (specifically increased CPU) improves the Router's ability to +parse incoming line protocol with lower latency. ### Ingester @@ -149,3 +170,18 @@ Because compaction is a compute-heavy process, vertical scaling (especially increasing the available CPU) is the most effective scaling strategy for the Compactor. Horizontal scaling increases compaction throughput, but not as efficiently as vertical scaling. + +### Garbage collector + +The Garbage collector runs background jobs that evict expired or deleted data, +remove obsolete compaction files, and reclaim space in both the [Catalog](#catalog) and the +[Object store](#object-store). + +##### Garbage collector scaling strategies + + +The Garbage collector is not designed for distributed load and should _not_ be +scaled horizontally. The Garbage collector does not perform CPU- or +memory-intensive work, so [vertical scaling](/influxdb/clustered/admin/scale-cluster/#vertical-scaling) +should only be considered only if you observe very high CPU usage or +if the container regularly runs out of memory. diff --git a/static/svgs/v3-storage-architecture.svg b/static/svgs/v3-storage-architecture.svg index f22289a03..f15cc5ca5 100644 --- a/static/svgs/v3-storage-architecture.svg +++ b/static/svgs/v3-storage-architecture.svg @@ -1,127 +1,107 @@ - - - - - - - - - -Ingester - - - - -Querier -Object StorageTime series data stored inApache Parquet format - -CatalogRelational metadataservice - + + + + + + + + + +Ingester + + + +Router + + + + +Querier +Object Storage +Time series data stored in +Apache Parquet format + - Compactor + Catalog + Relational metadata + service + + +Compactor + + + + - - - - - - - - - + + - - - - - - + + + - - - - - - - - - + + - - - - - - + + + + + + + + - - - - - - + + + - +Query yet-to-be-persisted data + +WAL +Short-term +persistence +Write requests +Query requests - - - - - - - - - - -Query yet-to-be-persisted data - - - -WALShort-termpersistence - - Write requests + + + + + + + + - Query requests + + + + + + + - - - - - - + + - - - - - - - - - - - - - - - - - - - +Garbage Collector +