Update delete docs, add data retention info, update schema functions (#3830)

* update delete docs, add data retention info, update schema fucntions with retention, closes #3223, closes #3786, closes #3730, closes #2571

* updated wording around querying expired data in cloud

* updated language about deleting fields

* Apply suggestions from code review

Co-authored-by: Jason Stirnaman <jstirnaman@influxdata.com>

Co-authored-by: Jason Stirnaman <jstirnaman@influxdata.com>
pull/3838/head
Scott Anderson 2022-03-07 16:23:02 -07:00 committed by GitHub
parent 348c099f29
commit c95eb4d7bd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
27 changed files with 399 additions and 78 deletions

View File

@ -34,5 +34,7 @@
pre {
display: block;
line-height: 1.75rem;
@import "article/code-api-methods";
}
}

View File

@ -0,0 +1,14 @@
.api {
margin-right: .35rem;
padding: .15rem .5rem .25rem;
border-radius: $radius;
color: $g20-white;
font-weight: bold;
font-size: 1rem;
&.get { background: $gr-viridian; }
&.post { background: $b-ocean; }
&.patch { background: $y-topaz; }
&.delete { background: $r-ruby; }
&.put {background: $br-pulsar; }
}

View File

@ -70,21 +70,8 @@ pre {
line-height: 1.7rem;
white-space: pre;
}
}
pre .api {
margin-right: .35rem;
padding: .15rem .5rem .25rem;
border-radius: $radius;
color: $g20-white;
font-weight: bold;
font-size: .9rem;
&.get { background: $gr-viridian; }
&.post { background: $b-ocean; }
&.patch { background: $y-topaz; }
&.delete { background: $r-ruby; }
&.put {background: $br-pulsar; }
@import "code-api-methods";
}
///////////////////////// Codeblocks fullscreen toggle /////////////////////////

View File

@ -43,7 +43,7 @@
}
///////////////////////////////// Shard diagram ////////////////////////////////
#shard-diagram {
#shard-diagram, #data-retention {
display: flex;
flex-direction: column;
max-width: 550px;
@ -71,6 +71,26 @@
border-left: 1px solid $article-text;
}
}
.one-quarter {width: 25%; height: .75rem;}
.three-quarters {width: 75%; height: .75rem;}
.border-left {border-left: 1px solid $article-text;}
.retention-label {
position: relative;
&:before {
content: "";
display: inline-block;
width: .65rem;
margin-right: .5rem;
border-top: 1px solid $article-text;
vertical-align: middle;
}
}
.deleted-label {
color: $r-ruby;
text-align: center;
font-size: .9rem;
}
}
.shard-groups {
display: flex;
@ -96,9 +116,14 @@
padding: .65rem 1rem;
color: #fff;
border-radius: .25rem;
@include gradient($article-table-header, 90deg)
@include gradient($article-table-header, 90deg);
background-attachment: fixed;
}
&.deleted {
opacity: .3;
.shard {@include gradient($grad-red-dark)}
}
}
}
}

View File

@ -30,6 +30,19 @@ schema.fieldKeys(
)
```
{{% note %}}
#### Deleted fields
Fields [deleted from InfluxDB Cloud using the `/api/v2/delete` endpoint or the `influx delete` command](/influxdb/cloud/write-data/delete-data/)
**do not** appear in results.
#### Expired fields
- **InfluxDB Cloud**: field keys associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: field keys associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -30,6 +30,19 @@ schema.measurementFieldKeys(
)
```
{{% note %}}
#### Deleted fields
Fields [explicitly deleted from InfluxDB Cloud](/influxdb/cloud/write-data/delete-data/)
**do not** appear in results.
#### Expired fields
- **InfluxDB Cloud**: field keys associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: field keys associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -29,6 +29,19 @@ schema.measurementTagKeys(
)
```
{{% note %}}
#### Deleted tags
Tags [explicitly deleted from InfluxDB](/{{< latest "influxdb" >}}/write-data/delete-data/)
**do not** appear in results.
#### Expired tags
- **InfluxDB Cloud**: tags associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: tags associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -32,6 +32,19 @@ schema.measurementTagValues(
)
```
{{% note %}}
#### Deleted tags
Tags [explicitly deleted from InfluxDB](/{{< latest "influxdb" >}}/write-data/delete-data/)
**do not** appear in results.
#### Expired tags
- **InfluxDB Cloud**: tags associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: tags associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -30,6 +30,19 @@ schema.tagKeys(
)
```
{{% note %}}
#### Deleted tags
Tags [explicitly deleted from InfluxDB](/{{< latest "influxdb" >}}/write-data/delete-data/)
**do not** appear in results.
#### Expired tags
- **InfluxDB Cloud**: tags associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: tags associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -31,6 +31,19 @@ schema.tagValues(
)
```
{{% note %}}
#### Deleted tags
Tags [explicitly deleted from InfluxDB](/{{< latest "influxdb" >}}/write-data/delete-data/)
**do not** appear in results.
#### Expired tags
- **InfluxDB Cloud**: tags associated with points outside of the bucket's
retention policy **may** appear in results up to an hour after expiring.
- **InfluxDB OSS**: tags associated with points outside of the bucket's
retention policy **may** appear in results.
For more information, see [Data retention in InfluxDB OSS](/{{< latest "influxdb" >}}/reference/internals/data-retention/).
{{% /note %}}
## Parameters
### bucket {data-type="string"}

View File

@ -59,6 +59,10 @@ to create a new bucket. A bucket requires the following:
- days (`d`)
- weeks (`w`)
{{% note %}}
The minimum retention period is **one hour**.
{{% /note %}}
```sh
# Syntax
influx bucket create -n <bucket-name> -o <org-name> -r <retention-period-duration>

View File

@ -71,7 +71,20 @@ influx bucket update -i 034ad714fdd6f000 -n my-new-bucket
##### Update a bucket's retention period
Valid retention period duration units are nanoseconds (`ns`), microseconds (`us` or `µs`), milliseconds (`ms`), seconds (`s`), minutes (`m`), hours (`h`), days (`d`), or weeks (`w`).
Valid retention period duration units:
- nanoseconds (`ns`)
- microseconds (`us` or `µs`)
- milliseconds (`ms`)
- seconds (`s`)
- minutes (`m`)
- hours (`h`)
- days (`d`)
- weeks (`w`)
{{% note %}}
The minimum retention period is **one hour**.
{{% /note %}}
```sh
# Syntax

View File

@ -0,0 +1,42 @@
---
title: Data retention in InfluxDB Cloud
description: >
The InfluxDB Cloud retention service checks for and removes data with timestamps
beyond the defined retention period of the bucket the data is stored in.
weight: 103
menu:
influxdb_cloud_ref:
name: Data retention
parent: InfluxDB Cloud internals
influxdb/cloud/tags: [internals]
---
The **InfluxDB Cloud retention enforcement service** checks for and removes data
with timestamps beyond the defined retention period of the
[bucket](/influxdb/cloud/reference/glossary/#bucket) the data is stored in.
This service is designed to automatically delete "expired" data and optimize disk
usage without any user intervention.
- [Bucket retention period](#bucket-retention-period)
- [When does data actually get deleted?](#when-does-data-actually-get-deleted)
## Bucket retention period
A **bucket retention period** is the duration of time that a bucket retains data.
Retention periods can be as short as an hour or infinite.
[Points](/influxdb/cloud/reference/glossary/#point) in a bucket with timestamps
beyond the defined retention period (relative to now) are flagged for deletion
(also known as "tombstoned").
{{% note %}}
#### View bucket retention periods
Use the [`influx bucket list` command](/influxdb/cloud/reference/cli/influx/bucket/list/)
to view the retention period buckets in your organization.
{{% /note %}}
## When does data actually get deleted?
The InfluxDB Cloud retention enforcement service **runs hourly** and tombstones
all points with timestamps beyond the bucket retention period.
Tombstoned points persist on disk, but are filtered from all query results until
the next [compaction](/influxdb/cloud/reference/glossary/#compaction) cycle,
when they are removed from disk.
Compaction cycle intervals vary.

View File

@ -2,7 +2,8 @@
title: Delete data
list_title: Delete data
description: >
Delete data in the InfluxDB CLI and API.
Use the `influx` CLI or the InfluxDB API `/api/v2/delete` endpoint to delete
data from an InfluxDB Bucket.
menu:
influxdb_cloud:
name: Delete data

View File

@ -66,6 +66,10 @@ to create a new bucket. A bucket requires the following:
- days (`d`)
- weeks (`w`)
{{% note %}}
The minimum retention period is **one hour**.
{{% /note %}}
```sh
# Syntax
influx bucket create -n <bucket-name> -o <org-name> -r <retention-period-duration>

View File

@ -64,7 +64,20 @@ influx bucket update -i 034ad714fdd6f000 -n my-new-bucket
##### Update a bucket's retention period
Valid retention period duration units are nanoseconds (`ns`), microseconds (`us` or `µs`), milliseconds (`ms`), seconds (`s`), minutes (`m`), hours (`h`), days (`d`), or weeks (`w`).
Valid retention period duration units:
- nanoseconds (`ns`)
- microseconds (`us` or `µs`)
- milliseconds (`ms`)
- seconds (`s`)
- minutes (`m`)
- hours (`h`)
- days (`d`)
- weeks (`w`)
{{% note %}}
The minimum retention period is **one hour**.
{{% /note %}}
```sh
# Syntax

View File

@ -46,8 +46,17 @@ influx bucket create [flags]
| `-t` | `--token` | API token | string | `INFLUX_TOKEN` |
{{% note %}}
Valid `--retention` units are nanoseconds (`ns`), microseconds (`us` or `µs`),
milliseconds (`ms`), seconds (`s`), minutes (`m`), hours (`h`), days (`d`), and weeks (`w`).
#### Retention periods
The minimum retention period is **one hour**. Valid `--retention` units:
- nanoseconds (`ns`)
- microseconds (`us` or `µs`)
- milliseconds (`ms`)
- seconds (`s`)
- minutes (`m`)
- hours (`h`)
- days (`d`)
- weeks (`w`)
{{% /note %}}
## Examples

View File

@ -41,8 +41,17 @@ influx bucket update [flags]
| `-t` | `--token` | API token | string | `INFLUX_TOKEN` |
{{% note %}}
Valid `--retention` units are nanoseconds (`ns`), microseconds (`us` or `µs`),
milliseconds (`ms`), seconds (`s`), minutes (`m`), hours (`h`), days (`d`), and weeks (`w`).
#### Retention periods
The minimum retention period is **one hour**. Valid `--retention` units:
- nanoseconds (`ns`)
- microseconds (`us` or `µs`)
- milliseconds (`ms`)
- seconds (`s`)
- minutes (`m`)
- hours (`h`)
- days (`d`)
- weeks (`w`)
{{% /note %}}
## Examples

View File

@ -804,8 +804,9 @@ In a batch of data, points that InfluxDB couldn't write to a bucket.
Field type conflicts are a common cause of rejected points.
### retention period
The duration of time that a bucket retains data.
Points with timestamps older than their bucket's retention period are dropped.
The [duration](#duration) of time that a bucket retains data.
InfluxDB drops points with timestamps older than their bucket's retention period.
The minimum retention period is **one hour**.
Related entries: [bucket](#bucket), [shard group duration](#shard-group-duration)

View File

@ -0,0 +1,81 @@
---
title: Data retention in InfluxDB
description: >
The InfluxDB retention service checks for and removes data with timestamps beyond
the defined retention period of the bucket the data is stored in.
weight: 103
menu:
influxdb_2_1_ref:
name: Data retention
parent: InfluxDB internals
influxdb/v2.1/tags: [storage, internals]
related:
- /influxdb/v2.1/reference/internals/shards/
- /influxdb/v2.1/reference/internals/storage-engine/
---
The **InfluxDB retention enforcement service** checks for and removes data with
timestamps beyond the defined retention period of the
[bucket](/influxdb/v2.1/reference/glossary/#bucket) the data is stored in.
This service is designed to automatically delete "expired" data and optimize disk
usage without any user intervention.
By default, the retention enforcement service runs every 30 minutes.
You can
configure this interval with the
[`storage-retention-check-interval`](/influxdb/v2.1/reference/config-options/#storage-retention-check-interval)
configuration option.
- [Bucket retention period](#bucket-retention-period)
- [Shard group duration](#shard-group-duration)
- [When does data actually get deleted?](#when-does-data-actually-get-deleted)
## Bucket retention period
A **bucket retention period** is the duration of time that a bucket retains data.
Retention periods can be as short as an hour or infinite.
[Points](/influxdb/v2.1/reference/glossary/#point) in a bucket with timestamps
beyond the defined retention period (relative to now) are _eligible_ for deletion.
## Shard group duration
InfluxDB stores data on disk in [shards](/influxdb/v2.1/reference/glossary/#shard).
Each shard belongs to a shard group and each shard group has a shard group duration.
The **shard group duration** defines the duration of time that each
shard in the shard group covers.
Each shard contains only points with timestamps in specific time range defined
by the shard group duration.
By default, shard group durations are set automatically based on the bucket retention
period, but can also be explicitly defined when creating or updating a bucket.
_For more information, see [InfluxDB shard group duration](/influxdb/v2.1/reference/internals/shards/#shard-group-duration)._
{{% note %}}
#### View bucket retention periods and shard group durations
Use the [`influx bucket list` command](/influxdb/v2.1/reference/cli/influx/bucket/list/)
to view the retention period and shard group duration of buckets in your organization.
{{% /note %}}
## When does data actually get deleted?
The InfluxDB retention enforcement service runs at regular intervals and deletes
[shard groups](/influxdb/v2.1/reference/internals/shards/#shard-groups), not individual points.
The service will only delete a shard group when the entire time range covered by
the shard group is beyond the bucket retention period.
{{% note %}}
#### Data is queryable until deleted
Even though data may be older than the specified bucket retention period,
it is queryable until removed by the retention enforcement service.
{{% /note %}}
To calculate the possible time data will persist before being deleted:
- **minimum**: `bucket-retention-period`
- **maximum** `bucket-retention-period + shard-group-duration`
For example, if your bucket retention period is three days (`3d`) and your
shard group duration is one day (`1d`), the retention enforcement service
deletes all shard groups with data that is **three to four days old** the next
time the service runs.
{{< html-diagram/data-retention >}}

View File

@ -8,7 +8,7 @@ menu:
influxdb_2_1_ref:
name: Shards & shard groups
parent: InfluxDB internals
weight: 102
weight: 103
influxdb/v2.1/tags: [storage, internals]
related:
- /influxdb/v2.1/reference/internals/storage-engine/

View File

@ -31,9 +31,11 @@ Use [logical operators](#logical-operators) to combine two or more predicate exp
```sql
key1="value1" AND key2="value"
```
{{% note %}}
Predicate expressions can use any column or tag except `_time` or `_value`.
{{% /note %}}
{{% warn %}}
With **InfluxDB {{< current-version >}}**, delete predicates can use any column
or tag **except** `_time`{{% oss-only %}}, `_field`, {{% /oss-only %}}or `_value`.
{{% /warn %}}
## Logical operators
Logical operators join two or more predicate expressions.
@ -51,22 +53,29 @@ Comparison operators compare left and right operands and return `true` or `false
## Delete predicate examples
### Delete points with a specific measurement
- [Delete points by measurement](#delete-points-by-measurement)
- {{% cloud-only %}}[Delete points by field](#delete-points-by-field){{% /cloud-only %}}
- [Delete points by tag set](#delete-points-by-tag-set)
### Delete points by measurement
The following will delete points in the `sensorData` measurement:
```sql
_measurement="sensorData"
```
<!--
### Delete points with a specific field
{{% cloud-only %}}
### Delete points by field
The following will delete points with the `temperature` field:
```sql
_field="temperature"
```
-->
### Delete points with a specific tag set
{{% /cloud-only %}}
### Delete points by tag set
The following will delete points from the `prod-1.4` host in the `us-west` region:
```sql
@ -79,4 +88,5 @@ The delete predicate syntax has the following limitations.
- Delete predicates do not support regular expressions.
- Delete predicates do not support the `OR` logical operator.
- Delete predicates only support equality (`=`), not inequality (`!=`).
- Delete predicates can use any column or tag except `_time` or `_value`.
- Delete predicates can use any column or tag **except** `_time`
{{% oss-only %}}, `_field`, {{% /oss-only %}}or `_value`.

View File

@ -2,7 +2,8 @@
title: Delete data
list_title: Delete data
description: >
Delete data in the InfluxDB CLI and API.
Use the `influx` CLI or the InfluxDB API `/api/v2/delete` endpoint to delete
data from an InfluxDB bucket.
menu:
influxdb_2_1:
name: Delete data
@ -14,47 +15,30 @@ related:
- /influxdb/v2.1/reference/cli/influx/delete/
---
<!--
## Delete data in the InfluxDB UI
Delete data from buckets you've created. You cannot delete data from system buckets.
### Delete data from buckets
1. Click **Load Data** in the navigation bar.
{{< nav-icon "load data" >}}
2. Select **Buckets**.
3. Next to the bucket with data you want to delete, click **Delete Data by Filter**.
4. In the **Delete Data** window that appears:
- Select a **Target Bucket** to delete data from.
- Enter a **Time Range** to delete data from.
- Click **+ Add Filter** to filter by tag key and value pair.
- Select **I understand that this cannot be undone**.
5. Click **Confirm Delete** to delete the selected data.
### Delete data from the Data Explorer
1. Click the **Data Explorer** icon in the sidebar.
{{< nav-icon "data-explorer" >}}
2. Click **Delete Data** in the top navigation bar.
3. In the **Delete Data** window that appears:
- Select a **Target Bucket** to delete data from.
- Enter a **Time Range** to delete data from.
- Click **+ Add Filter** to filter by tag key-value pairs.
- Select **I understand that this cannot be undone**.
4. Click **Confirm Delete** to delete the selected data.
!-->
Use the [`influx` CLI](/influxdb/v2.1/reference/cli/influx/) or the InfluxDB API
[`/api/v2/delete`](/influxdb/v2.1/api/#operation/PostDelete) endpoint to delete data.
[`/api/v2/delete`](/influxdb/v2.1/api/#operation/PostDelete) endpoint to delete
data from an InfluxDB bucket.
- [Delete data using the influx CLI](#delete-data-using-the-influx-cli)
- [Delete data using the API](#delete-data-using-the-api)
InfluxDB {{< current-version >}} supports deleting data by the following:
- time range
- measurement (`_measurement`)
- tag
- {{% cloud-only %}}field (`_field`){{% /cloud-only %}}
{{% oss-only %}}
{{% warn %}}
InfluxDB {{< current-version >}} does not support deleting data by field.
{{% /warn %}}
{{% /oss-only %}}
Once a delete request completes successfully, the deleted data is no longer queryable,
but will remain on disk until the compaction service runs.
## Delete data using the influx CLI
@ -80,8 +64,8 @@ deletes all data in the specified bucket with timestamps between the specified `
##### Delete points in a specific measurement with a specific tag value
```sh
influx delete --bucket example-bucket \
 --start '1970-01-01T00:00:00Z' \
  --stop $(date +"%Y-%m-%dT%H:%M:%SZ") \
--start '1970-01-01T00:00:00Z' \
--stop $(date +"%Y-%m-%dT%H:%M:%SZ") \
--predicate '_measurement="example-measurement" AND exampleTag="exampleTagValue"'
```
@ -95,6 +79,9 @@ influx delete --bucket example-bucket \
## Delete data using the API
Use the InfluxDB API [`/api/v2/delete` endpoint](/influxdb/v2.1/api/#operation/PostDelete)
to delete points from InfluxDB.
{{< api-endpoint method="post" endpoint="http://localhost:8086/api/v2/delete" >}}
Include the following:
- **Request method:** `POST`
@ -115,7 +102,7 @@ Deleting data without a [delete predicate](/influxdb/v2.1/reference/syntax/delet
deletes all data in the specified bucket with timestamps between the specified `start` and `stop` times.
{{% /warn %}}
#### Examples
### Examples
##### Delete points in a specific measurement with a specific tag value
```sh

View File

@ -1,3 +1,3 @@
{{- $productPathData := findRE "[^/]+.*?" .Page.RelPermalink -}}
{{- $currentVersion := replaceRE "v" "" (index $productPathData 1) -}}
{{ $currentVersion }}
<span class="current-version">{{ $currentVersion }}</span>

View File

@ -4,5 +4,5 @@
{{ $page := .Site.GetPage $path }}
{{ with $page }}
{{ .Content | replaceRE `\/influxdb\/v2\.[0-9]{1,2}\/` "/influxdb/cloud/" | safeHTML }}
{{ .Content | replaceRE `\/influxdb\/v2\.[0-9]{1,2}\/` "/influxdb/cloud/" | replaceRE `<span class="current-version">.*<\/span>` "Cloud" | safeHTML }}
{{ end }}

View File

@ -0,0 +1,41 @@
<div id="data-retention">
<div class="periods">
<p>4d</p>
<p>3d</p>
<p>2d</p>
<p>1d</p>
<p>0d</p>
</div>
<div class="timeline">
<div class="interval"></div>
<div class="interval"></div>
<div class="interval"></div>
<div class="interval"></div>
</div>
<div class="shard-groups">
<div class="shard-group deleted">
<p>Shard group</p>
<p class="shard">Shard</p>
</div>
<div class="shard-group">
<p>Shard group</p>
<p class="shard">Shard</p>
</div>
<div class="shard-group">
<p>Shard group</p>
<p class="shard">Shard</p>
</div>
<div class="shard-group">
<p>Shard group</p>
<p class="shard">Shard</p>
</div>
</div>
<div class="timeline">
<div class="one-quarter">
<p class="deleted-label all-caps">Deleted</p>
</div>
<div class="three-quarters border-left">
<p class=retention-label>3d retention period</p>
</div>
</div>
</div>

View File

@ -1,3 +1,3 @@
influx bucket create \
--name my_schema_bucket \
--schema-type explicit
--schema-type explicit