Flux migration troubleshooting and IOx naming cleanup (#5249)
* adds error to migration troubleshoot, some IOx cleanup, closes influxdata/DAR#387 * Apply suggestions from code review Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com> * Apply suggestions from code review --------- Co-authored-by: Jason Stirnaman <stirnamanj@gmail.com>pull/5246/head^2
Scott Anderson
GitHub
parent
42e2a7a4f4
commit
92db741681
|
|
@ -310,7 +310,7 @@ Replace the following:
|
|||
|
||||
##### Other Telegraf configuration options
|
||||
|
||||
`influx_uint_support`: supported in InfluxDB IOx.
|
||||
`influx_uint_support`: supported in InfluxDB v3.
|
||||
|
||||
For more plugin options, see [`influxdb`](https://github.com/influxdata/telegraf/blob/master/plugins/outputs/influxdb/README.md) on GitHub.
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ The following questions will help guide your decision to migrate.
|
|||
**Yes, you should migrate**. Series cardinality is a major limiting factor with
|
||||
the InfluxDB TSM storage engine. The more unique series in your data, the less
|
||||
performant your database.
|
||||
The IOx storage engine supports near limitless series cardinality and is, without
|
||||
The InfluxDB v3 storage engine supports limitless series cardinality and is, without
|
||||
question, the better solution for high series cardinality workloads.
|
||||
|
||||
#### Do you want to use SQL to query your data?
|
||||
|
|
@ -62,8 +62,8 @@ from the following providers:
|
|||
{{< cloud_regions type=iox-list >}}
|
||||
|
||||
If your deployment requires other cloud providers or regions, you may need to
|
||||
wait until the IOx storage engine is available in a region that meets your requirements.
|
||||
We are currently working to make InfluxDB IOx available on more providers and
|
||||
wait until the v3 storage engine is available in a region that meets your requirements.
|
||||
We are currently working to make InfluxDB v3 available on more providers and
|
||||
in more regions around the world.
|
||||
|
||||
#### Are you reliant on Flux queries and Flux tasks?
|
||||
|
|
|
|||
|
|
@ -385,7 +385,7 @@ Write data with your existing workloads that already use the InfluxDB v1 `/write
|
|||
|
||||
{{% note %}}
|
||||
|
||||
If migrating data from InfluxDB 1.x, see the [Migrate data from InfluxDB 1.x to InfluxDB {{% product-name %}}](/influxdb/cloud-serverless/guides/migrate-data/migrate-1x-to-iox/) guide.
|
||||
If migrating data from InfluxDB 1.x, see the [Migrate data from InfluxDB 1.x to InfluxDB {{% product-name %}}](/influxdb/cloud-serverless/guides/migrate-data/migrate-1x-to-serverless/) guide.
|
||||
|
||||
{{% /note %}}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,8 @@ menu:
|
|||
parent: Migrate data
|
||||
weight: 103
|
||||
aliases:
|
||||
- /influxdb/cloud-serverless/write-data/migrate-data/migrate-1x-to-iox
|
||||
- /influxdb/cloud-serverless/write-data/migrate-data/migrate-1x-to-iox/
|
||||
- /influxdb/cloud-serverless/guides/migrate-data/migrate-1x-to-iox/
|
||||
---
|
||||
|
||||
To migrate data from an InfluxDB 1.x OSS or Enterprise instance to InfluxDB Cloud
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
title: Migrate data from TSM to IOx in InfluxDB Cloud
|
||||
title: Migrate data from TSM to InfluxDB Cloud Serverless
|
||||
description: >
|
||||
To migrate data from a TSM-powered InfluxDB Cloud organization to an InfluxDB
|
||||
Cloud Serverless organization powered by the v3 storage engine, query the data in
|
||||
|
|
@ -12,6 +12,7 @@ menu:
|
|||
weight: 102
|
||||
aliases:
|
||||
- /influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox
|
||||
- /influxdb/cloud-serverless/guides/migrate-data/migrate-tsm-to-iox
|
||||
alt_links:
|
||||
cloud: /influxdb/cloud/write-data/migrate-data/migrate-cloud-to-cloud/
|
||||
---
|
||||
|
|
@ -339,6 +340,7 @@ to allow for variation between batches.
|
|||
So in this example, **it would be best to set your `batchInterval` to `4d`**.
|
||||
|
||||
##### Important things to note
|
||||
|
||||
- This assumes no other queries are running in your source InfluxDB Cloud organization.
|
||||
- This assumes no other writes are happening in your destination InfluxDB Cloud Serverless organization.
|
||||
{{% /expand %}}
|
||||
|
|
@ -346,7 +348,8 @@ So in this example, **it would be best to set your `batchInterval` to `4d`**.
|
|||
{{< /expand-wrapper >}}
|
||||
|
||||
## Monitor the migration progress
|
||||
The [InfluxDB TSM to IOx Migration Community template](https://github.com/influxdata/community-templates/tree/master/influxdb-tsm-iox-migration/)
|
||||
|
||||
The [InfluxDB TSM to Serverless Migration Community template](https://github.com/influxdata/community-templates/tree/master/influxdb-tsm-iox-migration/)
|
||||
installs the migration task outlined in this guide as well as a dashboard
|
||||
for monitoring running data migrations.
|
||||
|
||||
|
|
@ -355,14 +358,17 @@ for monitoring running data migrations.
|
|||
<a class="btn" href="https://github.com/influxdata/community-templates/tree/master/influxdb-tsm-iox-migration/#quick-install">Install the InfluxDB Cloud Migration template</a>
|
||||
|
||||
## Troubleshoot migration task failures
|
||||
|
||||
If the migration task fails, [view your task logs](/influxdb/cloud/process-data/manage-tasks/task-run-history/)
|
||||
to identify the specific error. Below are common causes of migration task failures.
|
||||
|
||||
- [Exceeded rate limits](#exceeded-rate-limits)
|
||||
- [Invalid API token](#invalid-api-token)
|
||||
- [Query timeout](#query-timeout)
|
||||
- [Batch size is too large](#batch-size-is-too-large)
|
||||
|
||||
### Exceeded rate limits
|
||||
|
||||
If your data migration causes you to exceed your InfluxDB Cloud organization's
|
||||
limits and quotas, the task will return an error similar to:
|
||||
|
||||
|
|
@ -375,20 +381,22 @@ too many requests
|
|||
a smaller interval. Each batch will then query less data.
|
||||
|
||||
### Invalid API token
|
||||
If the API token you add as the `INFLUXDB_SERVERLESS_SECRET` doesn't have write access
|
||||
to your InfluxDB Cloud Serverless bucket, the task will return an error similar to:
|
||||
|
||||
If the API token you add as the `INFLUXDB_CLOUD_SECRET` doesn't have read access to
|
||||
your InfluxDB Cloud bucket, the task will return an error similar to:
|
||||
|
||||
```
|
||||
unauthorized access
|
||||
```
|
||||
|
||||
**Possible solutions**:
|
||||
- Ensure the API token has write access to your InfluxDB Cloud Serverless bucket.
|
||||
- Generate a new API token with write access to the bucket you want to migrate to.
|
||||
Then, update the `INFLUXDB_SERVERLESS_TOKEN` secret in your InfluxDB Cloud (TSM)
|
||||
instance with the new token.
|
||||
- Ensure the API token has read access to your InfluxDB Cloud bucket.
|
||||
- Generate a new InfluxDB Cloud API token with read access to the bucket you
|
||||
want to migrate. Then, update the `INFLUXDB_CLOUD_TOKEN` secret in your
|
||||
InfluxDB OSS instance with the new token.
|
||||
|
||||
### Query timeout
|
||||
|
||||
The InfluxDB Cloud query timeout is 90 seconds. If it takes longer than this to
|
||||
return the data from the batch interval, the query will time out and the
|
||||
task will fail.
|
||||
|
|
@ -397,3 +405,15 @@ task will fail.
|
|||
- Update the `migration.batchInterval` setting in your migration task to use
|
||||
a smaller interval. Each batch will then query less data and take less time
|
||||
to return results.
|
||||
|
||||
### Batch size is too large
|
||||
|
||||
If your batch size is too large, the task returns an error similar to the following:
|
||||
|
||||
```
|
||||
internal error: error calling function "metadata" @97:1-97:11: error calling function "findRecord" @67:32-67:69: wrong number of fields
|
||||
```
|
||||
|
||||
**Possible solutions**:
|
||||
- Update the `migration.batchInterval` setting in your migration task to use
|
||||
a smaller interval and retrieve less data per batch.
|
||||
|
|
@ -23,18 +23,19 @@ To switch {{< product-name "short" >}} organizations:
|
|||
3. Select the organization you want to switch to from the drop-down list.
|
||||
|
||||
{{% note %}}
|
||||
#### Migrate to IOx
|
||||
#### Migrate to InfluxDB Cloud Serverless
|
||||
|
||||
To benefit from IOx's unlimited cardinality and support for SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/).
|
||||
To unlock the benefits of the InfluxDB v3 storage engine, including unlimited
|
||||
cardinality and SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/).
|
||||
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and [organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and
|
||||
[organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
[cloud2.influxdata.com](https://cloud2.influxdata.com) on or after **January 31, 2023**
|
||||
are powered by the InfluxDB IOx storage engine.
|
||||
are on InfluxDB Cloud Serverless and are powered by the InfluxDB v3 storage engine.
|
||||
|
||||
To see which storage engine your organization uses,
|
||||
find the **InfluxDB Cloud powered by** link in your
|
||||
[InfluxDB Cloud organization homepage](https://cloud2.influxdata.com) version information.
|
||||
If your organization is using TSM, you'll see **TSM** followed by the version number.
|
||||
If IOx, you'll see
|
||||
**InfluxDB Cloud Serverless** followed by the version number.
|
||||
To see which storage engine your organization uses, find the **InfluxDB Cloud powered by**
|
||||
link in your [InfluxDB Cloud organization homepage](https://cloud2.influxdata.com)
|
||||
version information. If your organization is using TSM, you'll see **TSM**
|
||||
followed by the version number. If Serverless, you'll see **InfluxDB Cloud Serverless**
|
||||
followed by the version number.
|
||||
{{% /note %}}
|
||||
|
|
|
|||
|
|
@ -26,18 +26,19 @@ The following articles provide information about managing organizations:
|
|||
{{< children >}}
|
||||
|
||||
{{% note %}}
|
||||
#### Migrate to IOx
|
||||
#### Migrate to InfluxDB Cloud Serverless
|
||||
|
||||
To benefit from IOx's unlimited cardinality and support for SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/).
|
||||
To unlock the benefits of the InfluxDB v3 storage engine, including unlimited
|
||||
cardinality and SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-serverless/).
|
||||
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and [organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and
|
||||
[organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
[cloud2.influxdata.com](https://cloud2.influxdata.com) on or after **January 31, 2023**
|
||||
are powered by the InfluxDB IOx storage engine.
|
||||
are on InfluxDB Cloud Serverless and are powered by the InfluxDB v3 storage engine.
|
||||
|
||||
To see which storage engine your organization is using,
|
||||
find the **InfluxDB Cloud powered by** link in your
|
||||
[InfluxDB Cloud organization homepage](https://cloud2.influxdata.com) version information.
|
||||
If your organization is using TSM, you'll see **TSM** followed by the version number.
|
||||
If IOx, you'll see
|
||||
**InfluxDB Cloud Serverless** followed by the version number.
|
||||
{{% /note %}}
|
||||
To see which storage engine your organization uses, find the **InfluxDB Cloud powered by**
|
||||
link in your [InfluxDB Cloud organization homepage](https://cloud2.influxdata.com)
|
||||
version information. If your organization is using TSM, you'll see **TSM**
|
||||
followed by the version number. If Serverless, you'll see **InfluxDB Cloud Serverless**
|
||||
followed by the version number.
|
||||
{{% /note %}}
|
||||
|
|
|
|||
|
|
@ -22,20 +22,21 @@ walks through the migration.
|
|||
The specific process varies depending on whether your destination account is
|
||||
powered by our current database engine,
|
||||
[Time-Structured Merge Tree (TSM)](/influxdb/v2/reference/internals/storage-engine/#time-structured-merge-tree-tsm)
|
||||
or [our new database engine, InfluxDB IOx](/blog/announcing-general-availability-new-database-engine/).
|
||||
or [our new database engine, InfluxDB v3](/blog/announcing-general-availability-new-database-engine/).
|
||||
|
||||
To benefit from IOx's unlimited cardinality and support for SQL, migrate your data to IOx.
|
||||
To benefit from the InfluxDB v3 storage engine's unlimited cardinality and
|
||||
support for SQL, migrate your data to InfluxDB Cloud Serverless.
|
||||
|
||||
- [Migrate data TSM to IOx](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/)
|
||||
- [Migrate data TSM to Serverless](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-serverless/)
|
||||
- [Migrate data from TSM to TSM](/influxdb/cloud/write-data/migrate-data/migrate-cloud-to-cloud/).
|
||||
|
||||
To see which storage engine your organization is using,
|
||||
find the **InfluxDB Cloud powered by** link in your
|
||||
[InfluxDB Cloud organization homepage](https://cloud2.influxdata.com) version information.
|
||||
If your organization is using TSM, you'll see **TSM** followed by the version number.
|
||||
If IOx, you'll see
|
||||
**InfluxDB Cloud Serverless** followed by the version number.
|
||||
To see which storage engine your organization uses, find the **InfluxDB Cloud powered by**
|
||||
link in your [InfluxDB Cloud organization homepage](https://cloud2.influxdata.com)
|
||||
version information. If your organization is using TSM, you'll see **TSM**
|
||||
followed by the version number. If Serverless, you'll see **InfluxDB Cloud Serverless**
|
||||
followed by the version number.
|
||||
{{% note %}}
|
||||
|
||||
#### Dual write into both organizations
|
||||
|
||||
Depending on the duration of your retention policy for storing data it may be
|
||||
|
|
|
|||
|
|
@ -11,20 +11,21 @@ weight: 11
|
|||
---
|
||||
|
||||
{{% note %}}
|
||||
#### Migrate to IOx
|
||||
#### Migrate to InfluxDB Cloud Serverless
|
||||
|
||||
To benefit from IOx's unlimited cardinality and support for SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/).
|
||||
To unlock the benefits of the InfluxDB v3 storage engine, including unlimited
|
||||
cardinality and SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-serverless/).
|
||||
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and [organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and
|
||||
[organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
[cloud2.influxdata.com](https://cloud2.influxdata.com) on or after **January 31, 2023**
|
||||
are powered by the InfluxDB IOx storage engine.
|
||||
are on InfluxDB Cloud Serverless and are powered by the InfluxDB v3 storage engine.
|
||||
|
||||
To see which storage engine your organization uses,
|
||||
find the **InfluxDB Cloud powered by** link in your
|
||||
[InfluxDB Cloud organization homepage](https://cloud2.influxdata.com) version information.
|
||||
If your organization is using TSM, you'll see **TSM** followed by the version number.
|
||||
If IOx, you'll see
|
||||
**InfluxDB Cloud Serverless** followed by the version number.
|
||||
To see which storage engine your organization uses, find the **InfluxDB Cloud powered by**
|
||||
link in your [InfluxDB Cloud organization homepage](https://cloud2.influxdata.com)
|
||||
version information. If your organization is using TSM, you'll see **TSM**
|
||||
followed by the version number. If Serverless, you'll see **InfluxDB Cloud Serverless**
|
||||
followed by the version number.
|
||||
{{% /note %}}
|
||||
|
||||
To upgrade from **InfluxDB OSS 2.x** to **InfluxDB Cloud**:
|
||||
|
|
@ -40,6 +41,7 @@ To upgrade from **InfluxDB OSS 2.x** to **InfluxDB Cloud**:
|
|||
|
||||
{{% note %}}
|
||||
#### Consider when upgrading
|
||||
|
||||
- InfluxDB Cloud requires token authentication, and you must create all new API tokens.
|
||||
- InfluxDB Cloud does not support:
|
||||
- Multiple [organizations](/influxdb/cloud/reference/glossary/#organization) per account.
|
||||
|
|
@ -51,12 +53,14 @@ To upgrade from **InfluxDB OSS 2.x** to **InfluxDB Cloud**:
|
|||
{{% /note %}}
|
||||
|
||||
## Create an InfluxDB Cloud account
|
||||
|
||||
Do one of the following to create an InfluxDB Cloud account:
|
||||
|
||||
- [Subscribe through InfluxData](/influxdb/cloud/sign-up/#subscribe-through-influxdata) and start for free.
|
||||
- [Subscribe through your cloud provider](/influxdb/cloud/sign-up/#subscribe-through-a-cloud-provider).
|
||||
|
||||
## Create an All-Access API token
|
||||
|
||||
InfluxDB API tokens are unique to each organization.
|
||||
Create an **All-Access** token in your InfluxDB Cloud user interface (UI) to use
|
||||
for the upgrade process.
|
||||
|
|
@ -76,6 +80,7 @@ You cannot migrate tokens from InfluxDB 2.x to InfluxDB Cloud.
|
|||
_For more information about managing tokens and token types, see [Manage tokens](/influxdb/cloud/admin/tokens/)._
|
||||
|
||||
## Set up influx CLI connection configurations
|
||||
|
||||
The `influx` command line interface (CLI) lets you create connection configurations
|
||||
that automatically provides **host**, **organization**, and **API token**
|
||||
credentials to CLI commands.
|
||||
|
|
@ -100,6 +105,7 @@ Include the following flags for each configuration:
|
|||
- **InfluxDB OSS 2.x**: Provide an [Operator token](/influxdb/v2/admin/tokens/#operator-token).
|
||||
|
||||
##### Create an InfluxDB Cloud connection configuration
|
||||
|
||||
```sh
|
||||
# Example cloud connection configuration
|
||||
influx config create \
|
||||
|
|
@ -110,6 +116,7 @@ influx config create \
|
|||
```
|
||||
|
||||
##### Create an InfluxDB 2.x connection configuration
|
||||
|
||||
{{< keep-url >}}
|
||||
```sh
|
||||
# Example 2.x connection configuration
|
||||
|
|
@ -121,6 +128,7 @@ influx config create \
|
|||
```
|
||||
|
||||
## Use templates to migrate InfluxDB resources
|
||||
|
||||
[InfluxDB templates](/influxdb/cloud/tools/influxdb-templates/) let you export InfluxDB
|
||||
[resources](/influxdb/cloud/tools/influxdb-templates/#template-resources) such as buckets,
|
||||
dashboards, labels, tasks, and more and import them into another InfluxDB instance.
|
||||
|
|
@ -128,7 +136,9 @@ Export resources from your **InfluxDB 2.x** instance and migrate them to
|
|||
your **InfluxDB Cloud** instance.
|
||||
|
||||
{{% note %}}
|
||||
|
||||
#### InfluxDB Cloud Free Plan resource limits
|
||||
|
||||
If upgrading to an InfluxDB Cloud Free Plan, you are only able to create a [limited number of resources](/influxdb/cloud/account-management/limits/#free-plan-limits).
|
||||
If your exported template exceeds these limits, the resource migration will fail.
|
||||
{{% /note %}}
|
||||
|
|
@ -179,7 +189,9 @@ influx apply --active-config cloud
|
|||
and [`influx export all`](/influxdb/cloud/reference/cli/influx/export/all#examples) documentation.
|
||||
|
||||
{{% note %}}
|
||||
|
||||
#### Update hardcoded InfluxDB URLs
|
||||
|
||||
If any of your migrated resources contain hardcoded InfluxDB URLs (`http://localhost:8086`), do one of the following to update these URLs to your [InfluxDB Cloud region URL](/influxdb/cloud/reference/regions/):
|
||||
|
||||
- Migrate your resources to InfluxDB Cloud, and then update URLs in the InfluxDB Cloud UI.
|
||||
|
|
@ -187,6 +199,7 @@ If any of your migrated resources contain hardcoded InfluxDB URLs (`http://local
|
|||
{{% /note %}}
|
||||
|
||||
## Migrate DBRP mappings
|
||||
|
||||
InfluxDB database and retention policy (DBRP) mappings let you query InfluxDB Cloud
|
||||
buckets with InfluxQL and the InfluxDB 1.x DBRP convention.
|
||||
**If you have DBRP mappings in your InfluxDB 2.x instance**, migrate them
|
||||
|
|
@ -223,11 +236,13 @@ to your InfluxDB Cloud instance.
|
|||
{{< /expand-wrapper >}}
|
||||
|
||||
## Dual write to InfluxDB 2.x and InfluxDB Cloud
|
||||
|
||||
Update external clients to write to your InfluxDB Cloud instance.
|
||||
**We recommend writing data to both InfluxDB 2.x and InfluxDB Cloud until you
|
||||
finish [migrating your existing time series data](#migrate-time-series-data)**. For step-by-step instructions, see [Dual write](/influxdb/cloud/write-data/no-code/use-telegraf/dual-write/).
|
||||
|
||||
## Migrate time series data
|
||||
|
||||
To migrate your time series data from your InfluxDB 2.x instance to your
|
||||
InfluxDB Cloud instance, do the following:
|
||||
|
||||
|
|
@ -278,7 +293,9 @@ InfluxDB Cloud instance, do the following:
|
|||
4. Repeat steps 2-3 for each bucket.
|
||||
|
||||
{{% note %}}
|
||||
|
||||
#### InfluxDB Cloud write rate limits
|
||||
|
||||
Write requests are subject to rate limits associated with your
|
||||
[InfluxDB Cloud pricing plan](/influxdb/cloud/account-management/pricing-plans/).
|
||||
If your exported line protocol size potentially exceeds your rate limits,
|
||||
|
|
@ -313,6 +330,7 @@ rate limits track the payload size of the **uncompressed** line protocol.
|
|||
{{% /note %}}
|
||||
|
||||
#### Migrate system buckets
|
||||
|
||||
InfluxDB [system buckets](/influxdb/cloud/reference/internals/system-buckets/)
|
||||
contain data related to the InfluxDB monitoring and alerting system.
|
||||
Although the retention period for system buckets in both InfluxDB Cloud and
|
||||
|
|
@ -320,6 +338,7 @@ InfluxDB 2.x is only seven days, if you want to migrate this data,
|
|||
use the same method described above [to migrate time series data](#migrate-time-series-data).
|
||||
|
||||
#### Export and write data in a single command
|
||||
|
||||
If your data and rate limits allow, you can export and write data in a single
|
||||
command without writing a line protocol export file to disk.
|
||||
The `influxd inspect export-lp` command can output to **stdout** and the `influx write`
|
||||
|
|
@ -343,5 +362,6 @@ influx write \
|
|||
{{< /expand-wrapper >}}
|
||||
|
||||
## Collaborate with other users
|
||||
|
||||
To collaborate with other users in your InfluxDB Cloud organization,
|
||||
[invite users to join your organization](/influxdb/cloud/admin/organizations/users/#invite-a-user-to-your-organization/).
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ aliases:
|
|||
- /influxdb/cloud/migrate-data/migrate-cloud-to-cloud/
|
||||
weight: 102
|
||||
alt_links:
|
||||
cloud-serverless: /influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/
|
||||
cloud-serverless: /influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-serverless/
|
||||
---
|
||||
|
||||
To migrate data from one InfluxDB Cloud organization to another, query the
|
||||
|
|
@ -37,7 +37,7 @@ All query and write requests are subject to your InfluxDB Cloud organization's
|
|||
- [Configuration help](#configuration-help)
|
||||
- [Monitor the migration progress](#monitor-the-migration-progress)
|
||||
- [Troubleshoot migration task failures](#troubleshoot-migration-task-failures)
|
||||
- [Migrate to InfluxDB Cloud Serverless powered by IOx](#migrate-to-influxdb-cloud-serverless-powered-by-iox)
|
||||
- [Migrate to InfluxDB Cloud Serverless](#migrate-to-influxdb-cloud-serverless)
|
||||
|
||||
## Set up the migration
|
||||
|
||||
|
|
@ -326,6 +326,7 @@ to allow for variation between batches.
|
|||
So in this example, **it would be best to set your `batchInterval` to `4d`**.
|
||||
|
||||
##### Important things to note
|
||||
|
||||
- This assumes no other queries are running in your source InfluxDB Cloud organization.
|
||||
- This assumes no other writes are happening in your destination InfluxDB Cloud organization.
|
||||
{{% /expand %}}
|
||||
|
|
@ -333,6 +334,7 @@ So in this example, **it would be best to set your `batchInterval` to `4d`**.
|
|||
{{< /expand-wrapper >}}
|
||||
|
||||
## Monitor the migration progress
|
||||
|
||||
The [InfluxDB Cloud Migration Community template](https://github.com/influxdata/community-templates/tree/master/influxdb-cloud-oss-migration/)
|
||||
installs the migration task outlined in this guide as well as a dashboard
|
||||
for monitoring running data migrations.
|
||||
|
|
@ -342,14 +344,17 @@ for monitoring running data migrations.
|
|||
<a class="btn" href="https://github.com/influxdata/community-templates/tree/master/influxdb-cloud-oss-migration/#quick-install">Install the InfluxDB Cloud Migration template</a>
|
||||
|
||||
## Troubleshoot migration task failures
|
||||
|
||||
If the migration task fails, [view your task logs](/influxdb/cloud/process-data/manage-tasks/task-run-history/)
|
||||
to identify the specific error. Below are common causes of migration task failures.
|
||||
|
||||
- [Exceeded rate limits](#exceeded-rate-limits)
|
||||
- [Invalid API token](#invalid-api-token)
|
||||
- [Query timeout](#query-timeout)
|
||||
- [Batch size is too large](#batch-size-is-too-large)
|
||||
|
||||
### Exceeded rate limits
|
||||
|
||||
If your data migration causes you to exceed your InfluxDB Cloud organization's
|
||||
limits and quotas, the task will return an error similar to:
|
||||
|
||||
|
|
@ -362,6 +367,7 @@ too many requests
|
|||
a smaller interval. Each batch will then query less data.
|
||||
|
||||
### Invalid API token
|
||||
|
||||
If the API token you add as the `INFLUXDB_CLOUD_SECRET` doesn't have read access to
|
||||
your InfluxDB Cloud bucket, the task will return an error similar to:
|
||||
|
||||
|
|
@ -376,6 +382,7 @@ unauthorized access
|
|||
InfluxDB OSS instance with the new token.
|
||||
|
||||
### Query timeout
|
||||
|
||||
The InfluxDB Cloud query timeout is 90 seconds. If it takes longer than this to
|
||||
return the data from the batch interval, the query will time out and the
|
||||
task will fail.
|
||||
|
|
@ -385,17 +392,30 @@ task will fail.
|
|||
a smaller interval. Each batch will then query less data and take less time
|
||||
to return results.
|
||||
|
||||
### Migrate to InfluxDB Cloud Serverless powered by IOx
|
||||
### Batch size is too large
|
||||
|
||||
To unlock the benefits of the IOx storage engine, including unlimited cardinality and SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-iox/).
|
||||
If your batch size is too large, the task returns an error similar to the following:
|
||||
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and [organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
```
|
||||
internal error: error calling function "metadata" @97:1-97:11: error calling function "findRecord" @67:32-67:69: wrong number of fields
|
||||
```
|
||||
|
||||
**Possible solutions**:
|
||||
- Update the `migration.batchInterval` setting in your migration task to use
|
||||
a smaller interval and retrieve less data per batch.
|
||||
|
||||
### Migrate to InfluxDB Cloud Serverless
|
||||
|
||||
To unlock the benefits of the InfluxDB v3 storage engine, including unlimited
|
||||
cardinality and SQL, [migrate your data to an InfluxDB Cloud Serverless organization](/influxdb/cloud-serverless/write-data/migrate-data/migrate-tsm-to-serverless/).
|
||||
|
||||
All InfluxDB Cloud [accounts](/influxdb/cloud-serverless/admin/accounts/) and
|
||||
[organizations](/influxdb/cloud-serverless/admin/organizations/) created through
|
||||
[cloud2.influxdata.com](https://cloud2.influxdata.com) on or after **January 31, 2023**
|
||||
are powered by the InfluxDB IOx storage engine.
|
||||
are on InfluxDB Cloud Serverless and are powered by the InfluxDB v3 storage engine.
|
||||
|
||||
To see which storage engine your organization uses,
|
||||
find the **InfluxDB Cloud powered by** link in your
|
||||
[InfluxDB Cloud organization homepage](https://cloud2.influxdata.com) version information.
|
||||
If your organization is using TSM, you'll see **TSM** followed by the version number.
|
||||
If IOx, you'll see
|
||||
**InfluxDB Cloud Serverless** followed by the version number.
|
||||
To see which storage engine your organization uses, find the **InfluxDB Cloud powered by**
|
||||
link in your [InfluxDB Cloud organization homepage](https://cloud2.influxdata.com)
|
||||
version information. If your organization is using TSM, you'll see **TSM**
|
||||
followed by the version number. If Serverless, you'll see **InfluxDB Cloud Serverless**
|
||||
followed by the version number.
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ All queries against data in InfluxDB Cloud are subject to your organization's
|
|||
- [Troubleshoot migration task failures](#troubleshoot-migration-task-failures)
|
||||
|
||||
## Set up the migration
|
||||
|
||||
1. [Install and set up InfluxDB OSS](/influxdb/{{< current-version-link >}}/install/).
|
||||
|
||||
2. **In InfluxDB Cloud**, [create an API token](/influxdb/cloud/admin/tokens/create-token/)
|
||||
|
|
@ -69,6 +70,7 @@ Batch range is beyond the migration range. Migration is complete.
|
|||
## Migration task
|
||||
|
||||
### Configure the migration
|
||||
|
||||
1. Specify how often you want the task to run using the `task.every` option.
|
||||
_See [Determine your task interval](#determine-your-task-interval)._
|
||||
|
||||
|
|
@ -313,6 +315,7 @@ to allow for variation between batches.
|
|||
So in this example, **it would be best to set your `batchInterval` to `35d`**.
|
||||
|
||||
##### Important things to note
|
||||
|
||||
- This assumes no other queries are running in your InfluxDB Cloud organization.
|
||||
- You should also consider your network speeds and whether a batch can be fully
|
||||
downloaded within the [task interval](#determine-your-task-interval).
|
||||
|
|
@ -322,6 +325,7 @@ So in this example, **it would be best to set your `batchInterval` to `35d`**.
|
|||
{{< /expand-wrapper >}}
|
||||
|
||||
## Monitor the migration progress
|
||||
|
||||
The [InfluxDB Cloud Migration Community template](https://github.com/influxdata/community-templates/tree/master/influxdb-cloud-oss-migration/)
|
||||
installs the migration task outlined in this guide as well as a dashboard
|
||||
for monitoring running data migrations.
|
||||
|
|
@ -331,14 +335,17 @@ for monitoring running data migrations.
|
|||
<a class="btn" href="https://github.com/influxdata/community-templates/tree/master/influxdb-cloud-oss-migration/#quick-install">Install the InfluxDB Cloud Migration template</a>
|
||||
|
||||
## Troubleshoot migration task failures
|
||||
|
||||
If the migration task fails, [view your task logs](/influxdb/v2/process-data/manage-tasks/task-run-history/)
|
||||
to identify the specific error. Below are common causes of migration task failures.
|
||||
|
||||
- [Exceeded rate limits](#exceeded-rate-limits)
|
||||
- [Invalid API token](#invalid-api-token)
|
||||
- [Query timeout](#query-timeout)
|
||||
- [Batch size is too large](#batch-size-is-too-large)
|
||||
|
||||
### Exceeded rate limits
|
||||
|
||||
If your data migration causes you to exceed your InfluxDB Cloud organization's
|
||||
limits and quotas, the task will return an error similar to:
|
||||
|
||||
|
|
@ -351,6 +358,7 @@ too many requests
|
|||
a smaller interval. Each batch will then query less data.
|
||||
|
||||
### Invalid API token
|
||||
|
||||
If the API token you add as the `INFLUXDB_CLOUD_SECRET` doesn't have read access to
|
||||
your InfluxDB Cloud bucket, the task will return an error similar to:
|
||||
|
||||
|
|
@ -365,6 +373,7 @@ unauthorized access
|
|||
InfluxDB OSS instance with the new token.
|
||||
|
||||
### Query timeout
|
||||
|
||||
The InfluxDB Cloud query timeout is 90 seconds. If it takes longer than this to
|
||||
return the data from the batch interval, the query will time out and the
|
||||
task will fail.
|
||||
|
|
@ -373,3 +382,15 @@ task will fail.
|
|||
- Update the `migration.batchInterval` setting in your migration task to use
|
||||
a smaller interval. Each batch will then query less data and take less time
|
||||
to return results.
|
||||
|
||||
### Batch size is too large
|
||||
|
||||
If your batch size is too large, the task returns an error similar to the following:
|
||||
|
||||
```
|
||||
internal error: error calling function "metadata" @97:1-97:11: error calling function "findRecord" @67:32-67:69: wrong number of fields
|
||||
```
|
||||
|
||||
**Possible solutions**:
|
||||
- Update the `migration.batchInterval` setting in your migration task to use
|
||||
a smaller interval and retrieve less data per batch.
|
||||
|
|
|
|||
Loading…
Reference in New Issue