Influxdata
/
docs-v2
mirror of https://github.com/influxdata/docs-v2.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
docs-v2/content/influxdb3/clustered/guides/migrate-data/migrate-1x-to-clustered.md

12 KiB
Raw Permalink Blame History

title description menu weight related alt_links
Migrate data from InfluxDB 1.x to InfluxDB Clustered To migrate data from a TSM-powered InfluxDB 1.x (OSS or Enterprise) to an InfluxDB cluster, export the data as line protocol and write the exported data to your InfluxDB Clustered database.
influxdb3_clustered
name parent
Migrate from 1.x to Clustered Migrate data
103
/influxdb3/clustered/admin/databases/
/influxdb3/clustered/admin/tokens/
/influxdb3/clustered/primers/api/v1/
/influxdb3/clustered/primers/api/v2/
cloud-serverless cloud-dedicated
/influxdb3/cloud-serverless/guides/migrate-data/migrate-1x-to-serverless/ /influxdb3/cloud-dedicated/guides/migrate-data/migrate-1x-to-cloud-dedicated/

To migrate data from an InfluxDB 1.x OSS or Enterprise instance to InfluxDB Clustered, export the data as line protocol and write the exported data to an InfluxDB database.

Before you migrate

Before you migrate from InfluxDB 1.x to {{< product-name >}}, there are schema design practices supported by the TSM storage engine that are not supported in the InfluxDB 3 storage engine. Specifically, InfluxDB 3 enforces the following schema restrictions:

  • You can't use duplicate names for tags and fields.
  • By default, measurements can contain up to 250 columns where each column represents time, a field, or a tag.

For more information, see Schema restrictions.

If your schema does not adhere to these restrictions, you must update your schema before migrating to {{< product-name >}}.

{{< expand-wrapper >}} {{% expand "Fix duplicate tag and field names" %}}

If your current schema in InfluxDB 1.x includes tags and fields with the same name, rename either the duplicate tag key or field key. The simplest way to do this is to directly modify the line protocol exported in step 1 of Migrate data below before writing it to {{< product-name >}}.

For example, the following line protocol includes both a tag and field named temp.

home,room=Kitchen,temp=F co=0i,hum=56.6,temp=71.0 1672531200000000000

To be able to write this line protocol to {{< product-name >}}, update the temp tag key to tempScale:

home,room=Kitchen,tempScale=F co=0i,hum=56.6,temp=71.0 1672531200000000000

{{% /expand %}} {{% expand "Fix measurements with more than 250 total columns" %}}

If in your current schema, the total number of tags, fields, and time columns in a single measurement exceeds 250, we recommend updating your schema before migrating to {{< product-name >}}.

Although you can increase the column limit per measurement when creating a database, it may adversely affect query performance.

Because tags are metadata used to identify specific series, we recommend splitting groups of fields across multiple measurements.

In your InfluxDB 1.x instance:

  1. Create a new database to store the modified data in.
  2. Identify what fields could be grouped together in a measurement and not exceed the 250 column limit.
  3. Query each group of fields from the existing database and write them into a unique measurement in the new database:

{{< flex >}} {{% flex-content "half" %}}

Write one set of fields to a new measurement
SELECT 
  field1,
  field2,
  field3,
  field4
INTO
  "new-database".."new-measurement-1"
FROM
  "example-measurement"
GROUP BY *

{{% /flex-content %}} {{% flex-content "half" %}}

Write another set of fields to new measurement
SELECT 
  field5,
  field6,
  field7,
  field8
INTO
  "new-database".."new-measurement-2"
FROM
  "example-measurement"
GROUP BY *

{{% /flex-content %}} {{< /flex >}}

When exporting your data as line protocol in step 1 of Migrate data below, only export the newly created database that contains measurements with separate groups of fields.

If duplicating data is not feasible

If duplicating data into a new database isn't feasible, you can directly modify the exported line protocol to group certain fields into unique measurements. For example:

example-measurement field1=0,field2=0,field3=0,field4=0,field5=0,field6=0,field7=0,field8=0 1672531200000000000

Would become:

new-measurement-1 field1=0,field2=0,field3=0,field4=0 1672531200000000000
new-measurement-2 field5=0,field6=0,field7=0,field8=0 1672531200000000000

{{% /expand %}} {{< /expand-wrapper >}}

Tools to use

The migration process uses the following tools:

Migrate data

  1. Export data from your InfluxDB 1.x instance as line protocol.

    Use the InfluxDB 1.x influx_inspect export utility to export data as line protocol and store it in a file. Include the following:

    • ({{< req "Required" >}}) -lponly flag to export line protocol without InfluxQL DDL or DML.
    • ({{< req "Required" >}}) -out flag with a path to an output file. Default is ~/.influxdb/export. Any subsequent export commands without the output file defined will overwrite the existing export file.
    • -compress flag to use gzip to compress the output.
    • -datadir flag with the path to your InfluxDB 1.x data directory. Only required if the data directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
    • -waldir flag with the path to your InfluxDB 1.x wal directory. Only required if the wal directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
    • -database flag with a specific database name to export. By default, all databases are exported.
    • -retention flag with a specific retention policy to export. By default, all retention policies are exported.
    • -start flag with an RFC3339 timestamp that defines the earliest time to export. Default is 1677-09-20T16:27:54-07:44.
    • -end flag with an RFC3339 timestamp that defines the latest time to export. Default is 2262-04-11T16:47:16-07:00.

    {{% note %}} We recommend exporting each database and retention policy combination separately to easily write the exported data into corresponding InfluxDB Clustered databases. {{% /note %}}

    Export all data in a database and retention policy to a file
    influx_inspect export \
  -lponly \
  -database example-db \
  -retention example-rp \
  -out path/to/export-file.lp
    View more export command examples:

    {{< expand-wrapper >}} {{% expand "Export all data to a file" %}}

influx_inspect export \
  -lponly \
  -out path/to/export-file.lp.gzip

{{% /expand %}}

{{% expand "Export all data to a compressed file" %}}

influx_inspect export \
  -lponly \
  -compress \
  -out path/to/export-file.lp.gzip

{{% /expand %}}

{{% expand "Export data within time bounds to a file" %}}

influx_inspect export \
  -lponly \
  -start 2020-01-01T00:00:00Z \
  -end 2023-01-01T00:00:00Z \
  -out path/to/export-file.lp

{{% /expand %}}

{{% expand "Export a database and all its retention policies to a file" %}}

influx_inspect export \
  -lponly \
  -database example-db \
  -out path/to/export-file.lp

{{% /expand %}}

{{% expand "Export a specific database and retention policy to a file" %}}

influx_inspect export \
  -lponly \
  -database example-db \
  -retention example-rp \
  -out path/to/export-file.lp

{{% /expand %}}

{{% expand "Export all data from non-default data and wal directories" %}}

influx_inspect export \
  -lponly \
  -datadir path/to/influxdb/data/ \
  -waldir path/to/influxdb/wal/ \
  -out path/to/export-file.lp

{{% /expand %}} {{< /expand-wrapper >}}

  1. Create InfluxDB Clustered databases for each InfluxDB 1.x database and retention policy combination.

    {{% note %}} If coming from InfluxDB v1, the concepts of databases and retention policies have been combined into a single concept--database. Retention policies are no longer part of the InfluxDB data model. However, InfluxDB Clustered does support InfluxQL, which requires databases and retention policies. See InfluxQL DBRP naming convention.

If coming from InfluxDB v2, InfluxDB Cloud (TSM), or InfluxDB Cloud Serverless, database and bucket are synonymous. {{% /note %}}

{{< expand-wrapper >}}

{{% expand "View example 1.x databases and retention policies as InfluxDB Clustered databases" %}} If you have the following InfluxDB 1.x data structure:

  • example-db (database)
    • autogen (retention policy)
    • historical-1mo (retention policy)
    • historical-6mo (retention policy)
    • historical-1y (retention policy)

You would create the following InfluxDB {{< current-version >}} databases:

  • example-db/autogen
  • example-db/historical-1mo
  • example-db/historical-6mo
  • example-db/historical-1y

{{% /expand %}} {{< /expand-wrapper >}}

Use the [`influxctl database create` command](/influxdb3/clustered/reference/cli/influxctl/database/create/)
to [create a database](/influxdb3/clustered/admin/databases/create/) in your InfluxDB cluster.

Provide the following arguments:

- _(Optional)_ Database [retention period](/influxdb3/clustered/admin/databases/#retention-periods)
  (default is infinite)
- Database name _(see [Database naming restrictions](#database-naming-restrictions))_

```sh
influxctl database create --retention-period 30d <DATABASE_NAME>
```

To learn more about databases in InfluxDB Clustered, see [Manage databases](/influxdb3/clustered/admin/databases/).

  1. Create a database token for writing to your InfluxDB Clustered database.

    Use the influxctl token create command to create a database token with write permission to your database.

    Provide the following:

    • Permission grants
      • --read-database: Grants read access to a database
      • --write-database Grants write access to a database
    • Token description
    influxctl token create \
  --read-database example-db \
  --write-database example-db \
  "Read/write token for example-db database"

  2. Write the exported line protocol to your InfluxDB cluster.

    Use the v1 API or v2 API endpoints to write data to your InfluxDB cluster.

    Choose from the following options:

    Write each export file to the target database.

    {{% warn %}}

v2.x influx CLI not supported

Don't use the influx CLI with InfluxDB Clustered. While it may coincidentally work, it isn't officially supported.

For help finding the best workflow for your situation, contact Support. {{% /warn %}}