diff --git a/content/influxdb/cloud/reference/internals/_index.md b/content/influxdb/cloud/reference/internals/_index.md new file mode 100644 index 000000000..80aec7f2b --- /dev/null +++ b/content/influxdb/cloud/reference/internals/_index.md @@ -0,0 +1,9 @@ +--- +title: InfluxDB Internals +menu: + influxdb_cloud_ref: + name: InfluxDB Cloud internals +weight: 7 +--- + +{{< children >}} diff --git a/content/influxdb/cloud/reference/internals/durability.md b/content/influxdb/cloud/reference/internals/durability.md new file mode 100644 index 000000000..1662055c1 --- /dev/null +++ b/content/influxdb/cloud/reference/internals/durability.md @@ -0,0 +1,69 @@ +--- +title: InfluxDB Cloud data durability +description: > + InfluxDB Cloud ensures the durability of all stored data by automatically creating + backups, replicating data across geographic locations, and verifying replicated + data is consistent and backups are readable. +weight: 101 +menu: + influxdb_cloud_ref: + name: Data durability + parent: InfluxDB Cloud internals +influxdb/cloud/tags: [backups, internals] +--- + +InfluxDB Cloud ensures the durability of all stored data by automatically creating +backups, replicating data across geographic locations, and verifying replicated +data is consistent and backups are readable. + +##### On this page + +- [Backup processes](#backup-processes) +- [Recovery](#recovery) +- [Data verification](#data-verification) + +## Backup processes +InfluxDB Cloud stores **two out-of-band backup copies** of all data. +The following processes each generate backups: + +- [Backup on write](#backup-on-write) +- [Backup after compaction](#backup-after-compaction) + +### Backup on write +All inbound write requests to InfluxDB Cloud queue in a durable message queue. +The message queue stores the raw [line protocol](/influxdb/cloud/reference/glossary/#line-protocol) +of each write request in an object storage location as an out-of-band backup. +The message queue also writes data to the live storage service, replicates it twice +across two geographic locations, and makes it available to the InfluxDB Cloud service. + +The durability of the message queue is 72 hours, meaning InfluxDB Cloud can sustain +a failure of its underlying storage tier or object storage services for up to 72 hours. + +To minimize potential data loss due to defects introduced within the broader InfluxDB Cloud service, +InfluxData minimizes code between the data ingest and backup processes. + +### Backup after compaction +The InfluxDB storage engine compresses data over time in a process known as +[compaction](/influxdb/cloud/reference/glossary/#compaction). +When each compaction cycle completes, InfluxDB Cloud stores compressed +[TSM](/influxdb/cloud/reference/glossary/#tsm-time-structured-merge-tree) files +in an object storage location. + +## Recovery +InfluxDB Cloud uses out-of-band backups stored in object storage to recover data: + +- **Message queue backup:** line protocol based on inbound write requests +- **Historic backup:** compressed TSM files + +The Recovery Point Objective (RPO) is any accepted write. +The Recovery Time Objective (RTO) is harder to definitively predict as potential failure modes can vary. +While most common failure modes can be resolved within minutes or hours, +catastrophic or extreme failure modes may take longer. +For example, if we need to rebuild all data from the message queue backup, +it could take 24 hours or longer. + +## Data verification +InfluxDB Cloud has two data verification services running 24x7: + +- **Entropy detection:** ensures that replicated data is consistent +- **Backup verification:** validates and ensures backups in object storage are readable diff --git a/content/influxdb/v2.0/reference/glossary.md b/content/influxdb/v2.0/reference/glossary.md index 3fe1d5787..83d04ed06 100644 --- a/content/influxdb/v2.0/reference/glossary.md +++ b/content/influxdb/v2.0/reference/glossary.md @@ -188,6 +188,10 @@ Use comments with Flux statements to describe your functions. A standardized text file format used by the InfluxDB web server to create log entries when generating server log files. +### compaction + +The process of optimizing disk usage by compressing time series data. + ### continuous query (CQ) Continuous queries are the predecessor to tasks in InfluxDB 2.0.