feat(dedicated): iceberg for snowflake:
- Merge fixes - Provide instruction for checking snapshot status - Cleanupinceberg
Jason Stirnaman
Jason Stirnaman
parent
a488a4fd1d
commit
0cd0cdeac3
|
|
@ -1,19 +0,0 @@
|
|||
---
|
||||
title: Export in Iceberg format for Snowflake
|
||||
seotitle: Export time series data to Iceberg format for Snowflake
|
||||
description: >
|
||||
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
|
||||
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
|
||||
menu:
|
||||
influxdb3_cloud_dedicated:
|
||||
name: Snowflake with Iceberg
|
||||
parent: Export data
|
||||
weight: 101
|
||||
influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export]
|
||||
source: /shared/influxdb3-query-guides/snapshots/snowflake.md
|
||||
---
|
||||
|
||||
<!-- //FIXME The setup instructions are specific to Dedicated. After the content
|
||||
is approved, let's move it from content/shared to here -->
|
||||
<!-- //SOURCE - content/shared/influxdb3-query-guides/snapshots/snowflake.md -->
|
||||
|
||||
|
|
@ -1,14 +1,15 @@
|
|||
---
|
||||
title: Export data
|
||||
title: Use integrations to export and query data
|
||||
description: >
|
||||
Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems
|
||||
and avoid complex ETL processes.
|
||||
menu:
|
||||
influxdb3_cloud_dedicated:
|
||||
parent: Process & visualize data
|
||||
parent: Query data
|
||||
name: Use integrations
|
||||
weight: 103
|
||||
---
|
||||
Use InfluxData export tools to:
|
||||
Use {{% product-name %}} integrations:
|
||||
|
||||
- Integrate time series data stored in {{% product-name %}} with other tools and systems
|
||||
- Avoid complex ETL (Extract, Transform, Load) processes
|
||||
|
|
@ -1,3 +1,17 @@
|
|||
---
|
||||
title: Export Iceberg for Snowflake
|
||||
seotitle: Export time series data to Iceberg format for Snowflake
|
||||
description: >
|
||||
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
|
||||
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
|
||||
menu:
|
||||
influxdb3_cloud_dedicated:
|
||||
name: Export Iceberg for Snowflake
|
||||
parent: Use integrations
|
||||
weight: 101
|
||||
influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export]
|
||||
---
|
||||
|
||||
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
|
||||
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
|
||||
|
||||
|
|
@ -40,16 +54,18 @@ Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/c
|
|||
|
||||
The {{% product-name %}} Iceberg integration requires setup by the InfluxData team.
|
||||
After you have contacted sales and enabled Iceberg integration, you'll specify which
|
||||
tables you want to make available for exporting.
|
||||
InfluxData support engineers will help you with the integration steps, which include:
|
||||
tables you want to make available for export.
|
||||
InfluxData support engineers will work with you to set up the integration by:
|
||||
|
||||
- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details.
|
||||
This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata.
|
||||
- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently.
|
||||
For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
|
||||
- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export.
|
||||
- Configuring a Snowflake external stage using the `CREATE STAGE` SQL command with your storage provider details.
|
||||
This creates an external storage location (such as AWS S3) for Iceberg table data and metadata.
|
||||
- Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables.
|
||||
For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
|
||||
- Creating an exports configuration that specifies which tables are available for export.
|
||||
|
||||
<!-- Customer-triggered exports aren't yet supported.
|
||||
After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake.
|
||||
-->
|
||||
|
||||
> [!Important]
|
||||
> #### Export tables are read-only in Snowflake
|
||||
|
|
@ -60,6 +76,14 @@ After the setup is complete, you can export data snapshots to Iceberg format and
|
|||
|
||||
## Export data to Iceberg format
|
||||
|
||||
After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots.
|
||||
The exported data is stored in Iceberg format in the external storage location you specified.
|
||||
|
||||
_To refresh the exported data with new or changed data, submit a request to the
|
||||
[InfluxData Support team](https://support.influxdata.com) each time a refresh is needed.
|
||||
Currently, customer-triggered exports aren't supported._
|
||||
|
||||
<!-- Customer-triggered exports aren't yet supported.
|
||||
Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format.
|
||||
|
||||
> [!Important]
|
||||
|
|
@ -82,8 +106,8 @@ influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME
|
|||
|
||||
Replace the following:
|
||||
|
||||
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. <!-- Namespace might need more explanation -->
|
||||
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export
|
||||
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
|
||||
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export
|
||||
|
||||
{{% /tab-content %}}
|
||||
{{% tab-content %}}
|
||||
|
|
@ -112,14 +136,19 @@ curl -X POST https://{{% influxdb/host %}}/snapshots/export \
|
|||
|
||||
Replace the following:
|
||||
|
||||
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. <!--Might need more explanation-->
|
||||
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export
|
||||
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
|
||||
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export
|
||||
|
||||
{{% /tab-content %}}
|
||||
{{< /tabs-wrapper >}}
|
||||
-- Customer-triggered exports aren't supported -->
|
||||
|
||||
## Check snapshot status
|
||||
|
||||
To check the status of a snapshot export, contact the [InfluxData Support team](https://support.influxdata.com).
|
||||
|
||||
<!-- API isn't yet available to customers.
|
||||
|
||||
Use the HTTP API to check the status of an ongoing or completed snapshot export.
|
||||
|
||||
{{% api-endpoint method="GET" endpoint="/snapshots/status" %}}
|
||||
|
|
@ -127,6 +156,7 @@ Use the HTTP API to check the status of an ongoing or completed snapshot export.
|
|||
```bash
|
||||
curl -X GET https://{{% influxdb/host %}}/snapshots/status
|
||||
```
|
||||
-- API isn't yet available to customers -->
|
||||
|
||||
## Query the Iceberg table from Snowflake
|
||||
|
||||
|
|
@ -136,8 +166,11 @@ Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >
|
|||
|
||||
When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind:
|
||||
|
||||
- **Export controls limited to InfluxData**: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes.
|
||||
Submit a request to the [InfluxData Support team](https://support.influxdata.com) to:
|
||||
- Update your Snowflake tables with new or changed data
|
||||
- Check the status of a snapshot export
|
||||
- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine.
|
||||
- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB.
|
||||
- **Performance**: Query performance may vary based on data size and query complexity.
|
||||
- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration.
|
||||
|
||||
Loading…
Reference in New Issue