Write data from InfluxDB OSS to InfluxDB Cloud (#3419)
* updated Flux location option with correct syntax, closes #3393 * WIP write oss to cloud * WIP write from oss to cloud * finalize write data from oss to cloud doc * duplicated in cloud, updated nav weights, closes #3322 * lowercase data * Apply suggestions from code review Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com> * updated write oss to cloud to address PR feedback * Apply suggestions from code review Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com> * Update content/influxdb/v2.1/write-data/oss-to-cloud.md Co-authored-by: Kelly <kelly@influxdata.com> Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com>pull/3392/head^2
Scott Anderson
GitHub
parent
e14d0255d4
commit
67158869f6
|
|
@ -13,7 +13,7 @@ v2: /influxdb/v2.0/process-data/common-tasks/downsample-data/
|
|||
InfluxDB can handle hundreds of thousands of data points per second. Working with that much data over a long period of time can create storage concerns.
|
||||
A natural solution is to downsample the data; keep the high precision raw data for only a limited time, and store the lower precision, summarized data longer.
|
||||
This guide describes how to automate the process of downsampling data and expiring old data using InfluxQL. To downsample and retain data using Flux and InfluxDB 2.0,
|
||||
see [Process Data with InfluxDB tasks](/influxdb/v2.0/process-data/).
|
||||
see [Process data with InfluxDB tasks](/influxdb/v2.0/process-data/).
|
||||
|
||||
### Definitions
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Process Data with InfluxDB tasks
|
||||
seotitle: Process Data with InfluxDB tasks
|
||||
title: Process data with InfluxDB tasks
|
||||
seotitle: Process data with InfluxDB tasks
|
||||
description: >
|
||||
InfluxDB's task engine runs scheduled Flux tasks that process and analyze data.
|
||||
This collection of articles provides information about creating and managing InfluxDB tasks.
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ menu:
|
|||
influxdb_cloud:
|
||||
name: Delete data
|
||||
parent: Write data
|
||||
weight: 106
|
||||
weight: 107
|
||||
influxdb/cloud/tags: [delete]
|
||||
related:
|
||||
- /influxdb/v2.0/reference/syntax/delete-predicate/
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
---
|
||||
title: Write data from InfluxDB OSS to InfluxDB Cloud
|
||||
description: >
|
||||
Use `to()` or `experimental.to()` to write data from InfluxDB OSS to InfluxDB Cloud.
|
||||
Selectively write data or process data before writing it to InfluxDB Cloud.
|
||||
menu:
|
||||
influxdb_cloud:
|
||||
name: Write from OSS to Cloud
|
||||
parent: Write data
|
||||
weight: 105
|
||||
influxdb/cloud/tags: [write]
|
||||
---
|
||||
|
||||
{{< duplicate-oss >}}
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
title: Troubleshoot issues writing data
|
||||
seotitle: Troubleshoot issues writing data
|
||||
list_title: Troubleshoot issues writing data
|
||||
weight: 105
|
||||
weight: 106
|
||||
description: >
|
||||
Troubleshoot issues writing data. Find response codes for failed writes. Discover how writes fail, from exceeding rate or payload limits, to syntax errors and schema conflicts. Find suggestions to fix failures.
|
||||
menu:
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ v2: /influxdb/v2.0/process-data/common-tasks/downsample-data/
|
|||
InfluxDB can handle hundreds of thousands of data points per second. Working with that much data over a long period of time can create storage concerns.
|
||||
A natural solution is to downsample the data; keep the high precision raw data for only a limited time, and store the lower precision, summarized data longer.
|
||||
This guide describes how to automate the process of downsampling data and expiring old data using InfluxQL. To downsample and retain data using Flux and InfluxDB 2.0,
|
||||
see [Process Data with InfluxDB tasks](/influxdb/v2.0/process-data/).
|
||||
see [Process data with InfluxDB tasks](/influxdb/v2.0/process-data/).
|
||||
|
||||
### Definitions
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Process Data with InfluxDB tasks
|
||||
seotitle: Process Data with InfluxDB tasks
|
||||
title: Process data with InfluxDB tasks
|
||||
seotitle: Process data with InfluxDB tasks
|
||||
description: >
|
||||
InfluxDB's task engine runs scheduled Flux tasks that process and analyze data.
|
||||
This collection of articles provides information about creating and managing InfluxDB tasks.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Process Data with InfluxDB tasks
|
||||
seotitle: Process Data with InfluxDB tasks
|
||||
title: Process data with InfluxDB tasks
|
||||
seotitle: Process data with InfluxDB tasks
|
||||
description: >
|
||||
InfluxDB's task engine runs scheduled Flux tasks that process and analyze data.
|
||||
This collection of articles provides information about creating and managing InfluxDB tasks.
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ menu:
|
|||
influxdb_2_1:
|
||||
name: Delete data
|
||||
parent: Write data
|
||||
weight: 106
|
||||
weight: 107
|
||||
influxdb/v2.1/tags: [delete]
|
||||
related:
|
||||
- /influxdb/v2.1/reference/syntax/delete-predicate/
|
||||
|
|
|
|||
|
|
@ -0,0 +1,227 @@
|
|||
---
|
||||
title: Write data from InfluxDB OSS to InfluxDB Cloud
|
||||
description: >
|
||||
Use `to()` or `experimental.to()` to write data from InfluxDB OSS to InfluxDB Cloud.
|
||||
Selectively write data or process data before writing it to InfluxDB Cloud.
|
||||
menu:
|
||||
influxdb_2_1:
|
||||
name: Write from OSS to Cloud
|
||||
parent: Write data
|
||||
weight: 105
|
||||
influxdb/v2.1/tags: [write, ]
|
||||
---
|
||||
|
||||
To write data from InfluxDB OSS to InfluxDB Cloud, use the Flux
|
||||
[`to()`](/flux/v0.x/stdlib/influxdata/influxdb/to/) or
|
||||
[`experimental.to()`](/flux/v0.x/stdlib/experimental/to/) functions.
|
||||
Write data once with a single query execution or use [InfluxDB tasks](/influxdb/v2.1/process-data/)
|
||||
to [routinely write data to InfluxDB Cloud](#automate-writing-data-from-influxdb-oss-to-influxdb-cloud).
|
||||
|
||||
{{% cloud %}}
|
||||
#### InfluxDB Cloud rate limits
|
||||
Write requests to InfluxDB Cloud are subject to the rate limits associated with your
|
||||
[InfluxDB Cloud pricing plan](/influxdb/cloud/account-management/pricing-plans/).
|
||||
{{% /cloud %}}
|
||||
|
||||
1. Query data from InfluxDB OSS.
|
||||
2. _(Optional)_ [Filter](/{{% latest "flux" %}}/stdlib/universe/filter/) or process data to write to InfluxDB Cloud.
|
||||
3. Use `to` or `experimental.to` to write data to InfluxDB Cloud.
|
||||
For most use cases, `to()` is the correct function to use, but depending on
|
||||
the structure of the data you're writing, `experimental.to` may be required.
|
||||
|
||||
**Use the following guidelines**:
|
||||
|
||||
- **to()**: Use to write data in field keys to the `_field` column and field values to the `_value` column.
|
||||
|
||||
- **experimental.to()**: Use to write data in column names to corresponding field keys and column values to field values.
|
||||
|
||||
_See [input and output examples for `to()` functions](#input-and-output-data-for-to-functions)._
|
||||
|
||||
4. Provide the following parameters to either function:
|
||||
|
||||
- **bucket**: InfluxDB Cloud bucket to write to
|
||||
- **host**: InfluxDB Cloud region URL
|
||||
- **org**: InfluxDB Cloud organization
|
||||
- **token**: InfluxDB Cloud API Token
|
||||
|
||||
5. ({{< req "Recommended" >}}) To keep your raw API token out of queries, store
|
||||
your InfluxDB Cloud API token as an [InfluxDB secret](/influxdb/v2.1/security/secrets/)
|
||||
in your InfluxDB OSS instance and use [`secrets.get()`](/flux/v0.x/stdlib/influxdata/influxdb/secrets/get/)
|
||||
to retrieve the secret value as shown in the following example
|
||||
(select the function you're using to see the correct format):
|
||||
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[to()](#)
|
||||
[experimental.to()](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```js
|
||||
import "influxdata/influxdb/secrets"
|
||||
|
||||
cloudToken = secrets.get(key: "INFLUX_CLOUD_API_TOKEN")
|
||||
|
||||
from(bucket: "example-oss-bucket")
|
||||
|> range(start: -10m)
|
||||
|> filter(fn: (r) => r._measurement == "example-measurement")
|
||||
|> to(
|
||||
bucket: "example-cloud-bucket",
|
||||
host: "https://cloud2.influxdata.com",
|
||||
org: "example-org",
|
||||
token: cloudToken,
|
||||
)
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```js
|
||||
import "experimental"
|
||||
import "influxdata/influxdb/secrets"
|
||||
|
||||
cloudToken = secrets.get(key: "INFLUX_CLOUD_API_TOKEN")
|
||||
|
||||
from(bucket: "example-oss-bucket")
|
||||
|> range(start: -10m)
|
||||
|> filter(fn: (r) => r._measurement == "example-measurement")
|
||||
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|
||||
|> experimental.to(
|
||||
bucket: "example-cloud-bucket",
|
||||
host: "https://cloud2.influxdata.com",
|
||||
org: "example-org",
|
||||
token: cloudToken,
|
||||
)
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
## Input and output data for to() functions
|
||||
|
||||
{{< tabs-wrapper >}}
|
||||
{{% tabs %}}
|
||||
[to()](#)
|
||||
[experimental.to()](#)
|
||||
{{% /tabs %}}
|
||||
{{% tab-content %}}
|
||||
|
||||
- `to()` requires `_time`, `_measurement`, `_field`, and `_value` columns.
|
||||
- `to()` writes all other columns as tags where the column name is the tag key
|
||||
and the column value is the tag value.
|
||||
|
||||
#### Input data
|
||||
| _time | _measurement | exampleTag | _field | _value |
|
||||
| :------------------- | :----------- | :--------: | :----- | -----: |
|
||||
| 2021-01-01T00:00:00Z | example-m | A | temp | 80.0 |
|
||||
| 2021-01-01T00:01:00Z | example-m | A | temp | 80.3 |
|
||||
| 2021-01-01T00:02:00Z | example-m | A | temp | 81.1 |
|
||||
|
||||
| _time | _measurement | exampleTag | _field | _value |
|
||||
| :------------------- | :----------- | :--------: | :----- | -----: |
|
||||
| 2021-01-01T00:00:00Z | example-m | A | rpm | 4023 |
|
||||
| 2021-01-01T00:01:00Z | example-m | A | rpm | 4542 |
|
||||
| 2021-01-01T00:02:00Z | example-m | A | rpm | 4901 |
|
||||
|
||||
#### Output line protocol
|
||||
```
|
||||
example-m,exampleTag=A temp=80.0,rpm=4023i 1609459200000000000
|
||||
example-m,exampleTag=A temp=80.3,rpm=4542i 1609459260000000000
|
||||
example-m,exampleTag=A temp=81.1,rpm=4901i 1609459320000000000
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
- `experimental.to()` requires `_time` and `_measurement` columns.
|
||||
- Columns **in** the [group key](/flux/v0.x/get-started/data-model/#grouop-key)
|
||||
(other than `_measurement`) are parsed as tags where the column name is the
|
||||
tag key and the column value is the tag value.
|
||||
- Columns **not in** the group key (other than `_time_`) are parsed as fields
|
||||
where the column name is the field key and the column value is the field value.
|
||||
|
||||
#### Input data {id="experimental-input-data"}
|
||||
{{< flux/group-key "[_measurement, exampleTag]">}}
|
||||
| _time | _measurement | exampleTag | temp | rpm |
|
||||
| :------------------- | :----------- | :--------: | ---: | ---: |
|
||||
| 2021-01-01T00:00:00Z | example-m | A | 80.0 | 4023 |
|
||||
| 2021-01-01T00:01:00Z | example-m | A | 80.3 | 4542 |
|
||||
| 2021-01-01T00:02:00Z | example-m | A | 81.1 | 4901 |
|
||||
|
||||
#### Output line protocol {id="experimental-output-line-protocol"}
|
||||
```
|
||||
example-m,exampleTag=A temp=80.0,rpm=4023i 1609459200000000000
|
||||
example-m,exampleTag=A temp=80.3,rpm=4542i 1609459260000000000
|
||||
example-m,exampleTag=A temp=81.1,rpm=4901i 1609459320000000000
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
{{< /tabs-wrapper >}}
|
||||
|
||||
## Examples
|
||||
|
||||
- [Downsample and write data to InfluxDB Cloud](#downsample-and-write-data-to-influxdb-cloud)
|
||||
- [Write min, max, and mean values to InfluxDB Cloud](#write-min-max-and-mean-values-to-influxdb-cloud)
|
||||
|
||||
#### Downsample and write data to InfluxDB Cloud
|
||||
```js
|
||||
import "influxdata/influxdb/secrets"
|
||||
|
||||
cloudToken = secrets.get(key: "INFLUX_CLOUD_API_TOKEN")
|
||||
|
||||
from(bucket: "example-oss-bucket")
|
||||
|> range(start: -10m)
|
||||
|> filter(fn: (r) => r._measurement == "example-measurement")
|
||||
|> aggregateWindow(every: 1m, fn: last)
|
||||
|> to(
|
||||
bucket: "example-cloud-bucket",
|
||||
host: "https://cloud2.influxdata.com",
|
||||
org: "example-org",
|
||||
token: cloudToken,
|
||||
)
|
||||
```
|
||||
|
||||
#### Write min, max, and mean values to InfluxDB Cloud
|
||||
```js
|
||||
import "influxdata/influxdb/secrets"
|
||||
|
||||
cloudToken = secrets.get(key: "INFLUX_CLOUD_API_TOKEN")
|
||||
|
||||
data = from(bucket: "example-oss-bucket")
|
||||
|> range(start: -30m)
|
||||
|> filter(fn: (r) => r._measurement == "example-measurement")
|
||||
|
||||
min = data |> aggregateWindow(every: 10m, fn: min) |> map(fn: (r) => ({ r with _field: "{$r._field}_min" }))
|
||||
max = data |> aggregateWindow(every: 10m, fn: max) |> map(fn: (r) => ({ r with _field: "{$r._field}_max" }))
|
||||
mean = data |> aggregateWindow(every: 10m, fn: mean) |> map(fn: (r) => ({ r with _field: "{$r._field}_mean" }))
|
||||
|
||||
union(tables: [min, max, mean])
|
||||
|> to(
|
||||
bucket: "example-cloud-bucket",
|
||||
host: "https://cloud2.influxdata.com",
|
||||
org: "example-org",
|
||||
token: cloudToken,
|
||||
)
|
||||
```
|
||||
|
||||
## Automate writing data from InfluxDB OSS to InfluxDB Cloud
|
||||
To automatically and routinely write data from InfluxDB OSS to InfluxDB Cloud,
|
||||
[create a task](/influxdb/v2.1/process-data/manage-tasks/create-task/) in your
|
||||
InfluxDB OSS instance that regularly queries, processes, and writes data to
|
||||
InfluxDB Cloud.
|
||||
|
||||
```js
|
||||
import "influxdata/influxdb/tasks"
|
||||
|
||||
option task = {
|
||||
name: "Downsample to InfluxDB Cloud",
|
||||
every: 1h,
|
||||
}
|
||||
|
||||
from(bucket: "example-oss-bucket")
|
||||
|> range(start: -10m)
|
||||
|> filter(fn: (r) => r._measurement == "example-measurement")
|
||||
|> aggregateWindow(every: 1m, fn: last)
|
||||
|> to(
|
||||
bucket: "example-cloud-bucket",
|
||||
host: "https://cloud2.influxdata.com",
|
||||
org: "example-org",
|
||||
token: cloudToken,
|
||||
)
|
||||
```
|
||||
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
title: Troubleshoot issues writing data
|
||||
seotitle: Troubleshoot issues writing data
|
||||
list_title: Troubleshoot issues writing data
|
||||
weight: 106 >
|
||||
seotitle: Troubleshoot issues writing data to InfluxDB
|
||||
weight: 106
|
||||
description: >
|
||||
Troubleshoot issues writing data. Find response codes for failed writes. Discover how writes fail, from exceeding rate or payload limits, to syntax errors and schema conflicts.
|
||||
menu:
|
||||
influxdb_2_1:
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
{{ $expandLabel := .Get 0 }}
|
||||
<div class="expand" id="{{ $expandLabel | anchorize }}">
|
||||
<p class="expand-label">
|
||||
<span class="expand-toggle"></span><span>{{ $expandLabel | safeHTML }}</span>
|
||||
<span class="expand-toggle"></span><span>{{ $expandLabel | markdownify | safeHTML }}</span>
|
||||
</p>
|
||||
<div class="expand-content" style="display: none;" >
|
||||
{{ .Inner }}
|
||||
|
|
|
|||
Loading…
Reference in New Issue