From 54f39f7935174e8eefc7f833fb92a96ae972c109 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 19 Mar 2025 12:18:06 -0700 Subject: [PATCH 01/62] Iceberg Integration --- .../snapshots/snowflake.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index e69de29bb2..a974c53317 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -0,0 +1,152 @@ +# Integrating InfluxDB with Snowflake Using Apache Iceberg + +## Overview + +Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. + +### Key Benefits + +- **Efficient data access**: Query InfluxDB data directly from Snowflake. +- **Cost-effective storage**: Optimize data retention and minimize storage costs. +- **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. + +## Prerequisites + +Before you begin, ensure you have the following: + +- A **Snowflake account** with necessary permissions. +- Access to an **external object store** (such as AWS S3). +- Familiarity with **Apache Iceberg** and **Snowflake**. + +## Step-by-step guide + +### Step 1: Configure external storage + +Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. + +#### Example: Configure an S3 stage in Snowflake + +```sql +CREATE STAGE my_s3_stage +URL='s3://my-bucket/' +STORAGE_INTEGRATION=my_storage_integration; +``` + +For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). + +### Step 2: Set up a catalog integration in Snowflake + +Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. + +#### Example: Create a catalog integration in Snowflake + +```sql +CREATE CATALOG INTEGRATION my_catalog_integration + CATALOG_SOURCE = 'OBJECT_STORE' + TABLE_FORMAT = 'ICEBERG' + ENABLED = TRUE; +``` + +For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + +### Step 3: Export InfluxDB data to Iceberg format + +Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. + +#### Example: Export data using the Iceberg Exporter + +```sh +# Clone the Iceberg exporter repository +git clone https://github.com/influxdata/influxdb_iox.git +cd influxdb_iox/iceberg_exporter +``` + +- Configure the exporter with your InfluxDB data source and target Iceberg table location. +- Run the exporter to generate Iceberg-compatible Parquet files. + +For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). + +### Step 4: Create an Iceberg table in Snowflake + +After exporting the data, create an Iceberg table in Snowflake. + +#### Example: Create an Iceberg table in Snowflake + +```sql +CREATE ICEBERG TABLE my_iceberg_table + EXTERNAL_VOLUME = 'my_external_volume' + METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; +``` + +Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. + +### Step 5: Query the Iceberg table from Snowflake + +Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. + +#### Example: Query the Iceberg table + +```sql +SELECT * FROM my_iceberg_table +WHERE timestamp > '2025-01-01'; +``` + +## Interfaces for using Iceberg integration + +- **CLI**: `Influx CTL` enables users to trigger snapshot exports. +- **API**: Provides REST endpoints to manage and configure snapshots. +- **SQL (Snowflake)**: Query Iceberg tables using standard SQL. + +## CLI and API reference + +### CLI commands + +#### Example: Enable Iceberg feature and export a snapshot + +```sh +# Enable Iceberg feature +influxctl enable-iceberg + +# Export a snapshot +influxctl export --namespace foo --table bar +``` + +### API endpoints + +#### Example: Export a snapshot + +- **Method**: `POST` +- **Endpoint**: `/snapshots/export` +- **Request body**: + +```json +{ + "namespace": "foo", + "table": "bar" +} +``` + +#### Example: Check snapshot status + +- **Method**: `GET` +- **Endpoint**: `/snapshots/status` + +## Considerations and limitations + +- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. +- **Performance**: Query performance may vary based on data size and query complexity. +- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. + +## Next steps + +- Enhance REST Catalog support. +- Optimize AWS S3 access control and security. +- Improve Grafana dashboards and alerting. +- Expand compatibility testing with other Iceberg engines. + +## References + +- **InfluxDB Iceberg Exporter**: [GitHub Repository](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter) +- **Snowflake Iceberg Tables**: [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg) + + From 0d708335d9836f97d0afe24035492cd9719f5ab7 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:12 -0700 Subject: [PATCH 02/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- .../influxdb3-query-guides/snapshots/snowflake.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a974c53317..91afffb3f2 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -137,16 +137,5 @@ influxctl export --namespace foo --table bar - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. -## Next steps - -- Enhance REST Catalog support. -- Optimize AWS S3 access control and security. -- Improve Grafana dashboards and alerting. -- Expand compatibility testing with other Iceberg engines. - -## References - -- **InfluxDB Iceberg Exporter**: [GitHub Repository](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter) -- **Snowflake Iceberg Tables**: [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg) From 882547b7afa661d9f7e8ac65467c34d12004571c Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:21 -0700 Subject: [PATCH 03/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 91afffb3f2..0495ba4870 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -111,7 +111,9 @@ influxctl enable-iceberg influxctl export --namespace foo --table bar ``` -### API endpoints +### Use the API to manage and configure snapshots + +Use the {{% product-name %}} HTTP API to export snapshots and check status. #### Example: Export a snapshot From 1bbbc744fb0137f3350475caa55189a48f82cdfe Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:39 -0700 Subject: [PATCH 04/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0495ba4870..4199425d50 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,5 @@ # Integrating InfluxDB with Snowflake Using Apache Iceberg -## Overview Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. From 6962de95e1645ea085a2cc831fa8921a0b866c4c Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:56 -0700 Subject: [PATCH 05/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 4199425d50..fdfe8f9a20 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,7 +1,8 @@ # Integrating InfluxDB with Snowflake Using Apache Iceberg -Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. +Export time-series data snapshots from InfluxDB into Apache Iceberg format. +Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. ### Key Benefits From d2a5bcbde8de6cc662ac46ce4272e15a5b12d662 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:15 -0700 Subject: [PATCH 06/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index fdfe8f9a20..6a4bf29d7f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,4 @@ -# Integrating InfluxDB with Snowflake Using Apache Iceberg +# Integrate with Snowflake using Apache Iceberg Export time-series data snapshots from InfluxDB into Apache Iceberg format. From c647a73cb9333a073c60f603cbd16fff2ef59f17 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:23 -0700 Subject: [PATCH 07/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6a4bf29d7f..f3a92ec76c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -18,7 +18,6 @@ Before you begin, ensure you have the following: - Access to an **external object store** (such as AWS S3). - Familiarity with **Apache Iceberg** and **Snowflake**. -## Step-by-step guide ### Step 1: Configure external storage From 7dc8f54eb7b1033468365fab1024a9e6bfa94986 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:35 -0700 Subject: [PATCH 08/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index f3a92ec76c..1079ec9446 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -19,7 +19,7 @@ Before you begin, ensure you have the following: - Familiarity with **Apache Iceberg** and **Snowflake**. -### Step 1: Configure external storage +## Step 1: Configure external storage Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. From d77a65159e3269b4b29ffeae1923660a4d89d234 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:47 -0700 Subject: [PATCH 09/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1079ec9446..4a10a87b48 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -92,9 +92,9 @@ WHERE timestamp > '2025-01-01'; ## Interfaces for using Iceberg integration -- **CLI**: `Influx CTL` enables users to trigger snapshot exports. -- **API**: Provides REST endpoints to manage and configure snapshots. -- **SQL (Snowflake)**: Query Iceberg tables using standard SQL. +- [Use the CLI to trigger snapshot exports](#use-the-CLI-to-trigger-snapshot-exports) +- [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) +- [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) ## CLI and API reference From d25621de0af6792fecc942af66c37f5b7c2c2cb7 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:54 -0700 Subject: [PATCH 10/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 4a10a87b48..c4151a105f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -98,7 +98,7 @@ WHERE timestamp > '2025-01-01'; ## CLI and API reference -### CLI commands +### Use the CLI to trigger snapshot exports #### Example: Enable Iceberg feature and export a snapshot From 66128e9e0106f3f6f1ac1e6d0f60f16cee7daaaa Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:47:06 -0700 Subject: [PATCH 11/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c4151a105f..a66cb2272d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -96,8 +96,6 @@ WHERE timestamp > '2025-01-01'; - [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) - [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) -## CLI and API reference - ### Use the CLI to trigger snapshot exports #### Example: Enable Iceberg feature and export a snapshot From 8ed0dcc7858f94552af1d179201a68167cce754b Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:27:51 -0700 Subject: [PATCH 12/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a66cb2272d..62c9e1c22a 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -23,7 +23,7 @@ Before you begin, ensure you have the following: Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. -#### Example: Configure an S3 stage in Snowflake +### Example: Configure an S3 stage in Snowflake ```sql CREATE STAGE my_s3_stage From 2a1c09bdd4aea285116d7fb5cc718503d61f8aa4 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:02 -0700 Subject: [PATCH 13/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 62c9e1c22a..494168403d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -33,7 +33,7 @@ STORAGE_INTEGRATION=my_storage_integration; For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). -### Step 2: Set up a catalog integration in Snowflake +## Step 2: Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. From 712796f2b4b82e8791789818238077990740f9cc Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:13 -0700 Subject: [PATCH 14/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 494168403d..0a63768b75 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -37,7 +37,7 @@ For more details, refer to the [Snowflake documentation](https://docs.snowflake. Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -#### Example: Create a catalog integration in Snowflake +### Example: Create a catalog integration in Snowflake ```sql CREATE CATALOG INTEGRATION my_catalog_integration From fe9790be6f6027b5293f3ae9a6f6092a62eb6679 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:22 -0700 Subject: [PATCH 15/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0a63768b75..58309eb2b7 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -46,7 +46,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration ENABLED = TRUE; ``` -For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). ### Step 3: Export InfluxDB data to Iceberg format From b6466e097407a69605b7b15a2b01d2602664b6bd Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:31 -0700 Subject: [PATCH 16/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 58309eb2b7..3a9f55d7db 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -48,7 +48,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -### Step 3: Export InfluxDB data to Iceberg format +## Step 3: Export InfluxDB data to Iceberg format Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. From 1dfc787f900d3f721909dd0b74313b5b00de7cdf Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:57 -0700 Subject: [PATCH 17/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3a9f55d7db..40d4616812 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -50,7 +50,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Step 3: Export InfluxDB data to Iceberg format -Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. +Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. #### Example: Export data using the Iceberg Exporter From 2a9023e6719159cb6dc756ec7ac83f0ee54a3858 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:07 -0700 Subject: [PATCH 18/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 40d4616812..be3907ed25 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -52,7 +52,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. -#### Example: Export data using the Iceberg Exporter +### Example: Export data using the Iceberg exporter ```sh # Clone the Iceberg exporter repository From 8e021c686732f8931ba250ad589e9b046190242a Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:33 -0700 Subject: [PATCH 19/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index be3907ed25..fbcd27d256 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -79,7 +79,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -### Step 5: Query the Iceberg table from Snowflake +## Step 5: Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From 1e5c81eed58d44134e05f2fba0c5b978202443d4 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:50 -0700 Subject: [PATCH 20/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index fbcd27d256..6dacdbe00c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -83,7 +83,7 @@ Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external st Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. -#### Example: Query the Iceberg table +### Example: Query the Iceberg table ```sql SELECT * FROM my_iceberg_table From 44e58769ed4f7162dab1c572df97848a4ba92c3a Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 17:12:58 -0700 Subject: [PATCH 21/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6dacdbe00c..b835158553 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -6,7 +6,7 @@ Integrate data with Snowflake and other Iceberg-compatible tools without the nee ### Key Benefits -- **Efficient data access**: Query InfluxDB data directly from Snowflake. +- **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. From b5a8857b335593131d077dfbb1322d8c0c9de6f8 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 17:13:16 -0700 Subject: [PATCH 22/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index b835158553..9b4c7f5067 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -50,7 +50,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Step 3: Export InfluxDB data to Iceberg format -Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. +Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. ### Example: Export data using the Iceberg exporter From c5e3e7a3352a26ac76d8ec58b7cac253725ef374 Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 20 Mar 2025 18:10:09 -0700 Subject: [PATCH 23/62] updating headings --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 9b4c7f5067..5aebcfc240 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -19,7 +19,7 @@ Before you begin, ensure you have the following: - Familiarity with **Apache Iceberg** and **Snowflake**. -## Step 1: Configure external storage +## Configure external storage Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. @@ -33,7 +33,7 @@ STORAGE_INTEGRATION=my_storage_integration; For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). -## Step 2: Set up a catalog integration in Snowflake +## Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -48,7 +48,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -## Step 3: Export InfluxDB data to Iceberg format +## Export InfluxDB data to Iceberg format Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. @@ -65,7 +65,7 @@ cd influxdb_iox/iceberg_exporter For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). -### Step 4: Create an Iceberg table in Snowflake +### Create an Iceberg table in Snowflake After exporting the data, create an Iceberg table in Snowflake. @@ -79,7 +79,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -## Step 5: Query the Iceberg table from Snowflake +## Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From 24cf2cf14c6b5e416684e36c1e75a380f29fd89b Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 20 Mar 2025 18:18:42 -0700 Subject: [PATCH 24/62] Adding a numbered-list TOC --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5aebcfc240..eb8f316660 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -10,6 +10,16 @@ Integrate data with Snowflake and other Iceberg-compatible tools without the nee - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. +## Implementation steps + +Follow these steps to integrate InfluxDB 3 with Snowflake using Apache Iceberg: + +1. [Configure external storage](#configure-external-storage) +2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) +3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) +4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +5. [Query your data in Snowflake](#query-your-data-in-snowflake) + ## Prerequisites Before you begin, ensure you have the following: From 72534686762bad8c6322f41467658545939eb0e3 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 10:13:59 -0700 Subject: [PATCH 25/62] Added an export command example --- .../snapshots/snowflake.md | 57 ++++++++++++++++--- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index eb8f316660..1118ca50b9 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -62,18 +62,57 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. -### Example: Export data using the Iceberg exporter +### Example: Export data using Iceberg exporter -```sh -# Clone the Iceberg exporter repository -git clone https://github.com/influxdata/influxdb_iox.git -cd influxdb_iox/iceberg_exporter -``` +This example assumes: +- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data) +- You've configured compaction to trigger more quickly with these environment variables: + + - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` + - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` +- You have a `config.json` -- Configure the exporter with your InfluxDB data source and target Iceberg table location. -- Run the exporter to generate Iceberg-compatible Parquet files. +#### Example `config.json` -For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). +```json +{ + "exports": [ + { + "namespace": "company_sensors", + "table_name": "cpu" + } + ] +} +``` +#### Running the export command + +```console +$ influxdb_iox iceberg export \ + --catalog-dsn postgresql://postgres@localhost:5432/postgres \ + --source-object-store file + --source-data-dir ~/.influxdb_iox/object_store \ + --sink-object-store file \ + --sink-data-dir /tmp/iceberg \ + --export-config-path config.json +``` +The export command outputs an absolute path to an Iceberg metadata file: + +`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json +` +#### Example: Querying the exported metadata using duckdb + +```console +$ duckdb +D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; +┌───────────┬──────────────────────┬─────────────────────┬─────────────┬───┬────────────┬───────────────┬─────────────┬────────────────────┬────────────────────┐ +│ cpu │ host │ time │ usage_guest │ … │ usage_nice │ usage_softirq │ usage_steal │ usage_system │ usage_user │ +│ varchar │ varchar │ timestamp │ double │ │ double │ double │ double │ double │ double │ +├───────────┼──────────────────────┼─────────────────────┼─────────────┼───┼────────────┼───────────────┼─────────────┼────────────────────┼────────────────────┤ +│ cpu-total │ Andrews-MBP.hsd1.m… │ 2020-06-11 16:52:00 │ 0.0 │ … │ 0.0 │ 0.0 │ 0.0 │ 1.1173184357541899 │ 0.9435133457479826 │ +├───────────┴──────────────────────┴─────────────────────┴─────────────┴───┴────────────┴───────────────┴─────────────┴────────────────────┴────────────────────┤ +│ 1 rows 13 columns (9 shown) │ +└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` ### Create an Iceberg table in Snowflake From b3d5f81cd011c4d659c59293538c73ddffbd5a6b Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 10:52:16 -0700 Subject: [PATCH 26/62] Proof read: improving grammar and clarity --- .../snapshots/snowflake.md | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1118ca50b9..bbc075708b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,5 @@ # Integrate with Snowflake using Apache Iceberg - Export time-series data snapshots from InfluxDB into Apache Iceberg format. Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. @@ -60,17 +59,17 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Export InfluxDB data to Iceberg format -Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. +Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. ### Example: Export data using Iceberg exporter -This example assumes: -- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data) -- You've configured compaction to trigger more quickly with these environment variables: +This example assumes the following: +- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data). +- You've configured compaction to trigger more quickly with these environment variables: - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` -- You have a `config.json` +- You have a `config.json`. #### Example `config.json` @@ -84,6 +83,7 @@ This example assumes: ] } ``` + #### Running the export command ```console @@ -95,11 +95,12 @@ $ influxdb_iox iceberg export \ --sink-data-dir /tmp/iceberg \ --export-config-path config.json ``` + The export command outputs an absolute path to an Iceberg metadata file: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json ` -#### Example: Querying the exported metadata using duckdb +#### Example: Querying the exported metadata using DuckDB ```console $ duckdb @@ -114,6 +115,8 @@ D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` +Next, create an Iceberg table in Snowflake. + ### Create an Iceberg table in Snowflake After exporting the data, create an Iceberg table in Snowflake. @@ -151,10 +154,10 @@ WHERE timestamp > '2025-01-01'; ```sh # Enable Iceberg feature -influxctl enable-iceberg +$ influxctl enable-iceberg # Export a snapshot -influxctl export --namespace foo --table bar +$ influxctl export --namespace foo --table bar ``` ### Use the API to manage and configure snapshots From 4b1f3c201aa4bf29c4506f200a2d52c303450eb9 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 11:28:30 -0700 Subject: [PATCH 27/62] Adding explanations for examples --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index bbc075708b..6e75d61f70 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -166,6 +166,8 @@ Use the {{% product-name %}} HTTP API to export snapshots and check status. #### Example: Export a snapshot +This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. + - **Method**: `POST` - **Endpoint**: `/snapshots/export` - **Request body**: @@ -176,17 +178,21 @@ Use the {{% product-name %}} HTTP API to export snapshots and check status. "table": "bar" } ``` +The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. #### Example: Check snapshot status +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. + - **Method**: `GET` - **Endpoint**: `/snapshots/status` +In this example, the GET request to the /snapshots/status endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. + ## Considerations and limitations +When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: + - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. - - - From 8e83dd53c4c506b56ec679cd57efea940c9c4e9e Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 11:41:50 -0700 Subject: [PATCH 28/62] Updating examples --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6e75d61f70..5c5d5a397e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -182,12 +182,12 @@ The `POST` request to the `/snapshots/export` endpoint triggers the export of da #### Example: Check snapshot status -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - **Method**: `GET` - **Endpoint**: `/snapshots/status` -In this example, the GET request to the /snapshots/status endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. +The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. ## Considerations and limitations From 0476969b07798c936a060d509b45d5b308c75501 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:58:51 -0700 Subject: [PATCH 29/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5c5d5a397e..99b22c0fe4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,3 @@ -# Integrate with Snowflake using Apache Iceberg Export time-series data snapshots from InfluxDB into Apache Iceberg format. Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. From bc4a8e0d53a63f8edcefa7ba1a9964256c9457fb Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:12 -0700 Subject: [PATCH 30/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 99b22c0fe4..d081697511 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,6 @@ -Export time-series data snapshots from InfluxDB into Apache Iceberg format. -Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. +Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. +Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. ### Key Benefits From 981dcb662da9b2d910f4410b62a8170d6808a4e2 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:27 -0700 Subject: [PATCH 31/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../snapshots/snowflake.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index d081697511..a0dfea7088 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -8,16 +8,6 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. -## Implementation steps - -Follow these steps to integrate InfluxDB 3 with Snowflake using Apache Iceberg: - -1. [Configure external storage](#configure-external-storage) -2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) -3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) -4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -5. [Query your data in Snowflake](#query-your-data-in-snowflake) - ## Prerequisites Before you begin, ensure you have the following: @@ -26,6 +16,14 @@ Before you begin, ensure you have the following: - Access to an **external object store** (such as AWS S3). - Familiarity with **Apache Iceberg** and **Snowflake**. +## Integrate InfluxDB 3 with Snowflake + +1. [Configure external storage](#configure-external-storage) +2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) +3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) +4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +5. [Query your data in Snowflake](#query-your-data-in-snowflake) + ## Configure external storage From 6165ac3235364b0a2ebe9f2f491fa025dffc6c31 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:49 -0700 Subject: [PATCH 32/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a0dfea7088..3846ae0746 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -25,19 +25,17 @@ Before you begin, ensure you have the following: 5. [Query your data in Snowflake](#query-your-data-in-snowflake) -## Configure external storage +### Create a Snowflake external stage -Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. +Use the `CREATE STAGE` Snowflake SQL command to set up an external storage location +(such as AWS S3) to store Iceberg table data and metadata--for example: -### Example: Configure an S3 stage in Snowflake +#### Example: Configure an S3 stage in Snowflake ```sql CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -``` - -For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). ## Set up a catalog integration in Snowflake From 09ef8c23b11288ba1777d4f7a3a779028177b6d0 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:00:07 -0700 Subject: [PATCH 33/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3846ae0746..8452aa1483 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -37,11 +37,11 @@ CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -## Set up a catalog integration in Snowflake +### Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -### Example: Create a catalog integration in Snowflake +#### Example: Create a catalog integration in Snowflake ```sql CREATE CATALOG INTEGRATION my_catalog_integration From be368f4919556d8e7b139a95672c2fc8608d37b6 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:01:43 -0700 Subject: [PATCH 34/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8452aa1483..e441e1c71f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -52,7 +52,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -## Export InfluxDB data to Iceberg format +### Export InfluxDB time series data to Iceberg format Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. From 56b15512cb9961f22723ca2496084723f6142d7e Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:01:57 -0700 Subject: [PATCH 35/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index e441e1c71f..5d3db0f74e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -54,7 +54,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format -Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. ### Example: Export data using Iceberg exporter From 36c3fcb9f4031d73b659c1ea36e87a9b58e7179b Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:02:11 -0700 Subject: [PATCH 36/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5d3db0f74e..8d8edec05d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -56,7 +56,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -### Example: Export data using Iceberg exporter +#### Example: Export data using Iceberg exporter This example assumes the following: From 46acc006a173b12d35712943442ffca78dcbe337 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:02:30 -0700 Subject: [PATCH 37/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8d8edec05d..accdb4af6d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -60,7 +60,6 @@ Use the InfluxDB Iceberg exporter to convert and export your time-series data fr This example assumes the following: -- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data). - You've configured compaction to trigger more quickly with these environment variables: - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` From 0237a45ed48b88c1e0a85b288363f627e67ceeb0 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:03 -0700 Subject: [PATCH 38/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index accdb4af6d..63bcdaffe4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -78,7 +78,7 @@ This example assumes the following: } ``` -#### Running the export command +### Run the export command ```console $ influxdb_iox iceberg export \ From 902a704ccb3ea664514a99471002ca79ae57e134 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:22 -0700 Subject: [PATCH 39/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../snapshots/snowflake.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 63bcdaffe4..54c82b8437 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -94,22 +94,6 @@ The export command outputs an absolute path to an Iceberg metadata file: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json ` -#### Example: Querying the exported metadata using DuckDB - -```console -$ duckdb -D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; -┌───────────┬──────────────────────┬─────────────────────┬─────────────┬───┬────────────┬───────────────┬─────────────┬────────────────────┬────────────────────┐ -│ cpu │ host │ time │ usage_guest │ … │ usage_nice │ usage_softirq │ usage_steal │ usage_system │ usage_user │ -│ varchar │ varchar │ timestamp │ double │ │ double │ double │ double │ double │ double │ -├───────────┼──────────────────────┼─────────────────────┼─────────────┼───┼────────────┼───────────────┼─────────────┼────────────────────┼────────────────────┤ -│ cpu-total │ Andrews-MBP.hsd1.m… │ 2020-06-11 16:52:00 │ 0.0 │ … │ 0.0 │ 0.0 │ 0.0 │ 1.1173184357541899 │ 0.9435133457479826 │ -├───────────┴──────────────────────┴─────────────────────┴─────────────┴───┴────────────┴───────────────┴─────────────┴────────────────────┴────────────────────┤ -│ 1 rows 13 columns (9 shown) │ -└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ -``` - -Next, create an Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From d945e1c6f2ae4b780dc9828ab22ae93f5d82d9b4 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:35 -0700 Subject: [PATCH 40/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 54c82b8437..44097d32f2 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -109,7 +109,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -## Query the Iceberg table from Snowflake +### Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From 4cf8dd7fdb6550daad008b1387ffb07da4637eba Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:46 -0700 Subject: [PATCH 41/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 44097d32f2..3318eb80fc 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -113,7 +113,7 @@ Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external st Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. -### Example: Query the Iceberg table +#### Example: Query the Iceberg table ```sql SELECT * FROM my_iceberg_table From 3be23a256fb0f07cdfabfa7309a4b444a5b92aeb Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 10:48:33 -0700 Subject: [PATCH 42/62] removing run export example and adding --- .../snapshots/snowflake.md | 21 +++---------------- 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3318eb80fc..af8f2e3e0a 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,3 @@ - Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. @@ -60,9 +59,6 @@ Use the InfluxDB Iceberg exporter to convert and export your time-series data fr This example assumes the following: -- You've configured compaction to trigger more quickly with these environment variables: - - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` - You have a `config.json`. #### Example `config.json` @@ -78,22 +74,11 @@ This example assumes the following: } ``` -### Run the export command - -```console -$ influxdb_iox iceberg export \ - --catalog-dsn postgresql://postgres@localhost:5432/postgres \ - --source-object-store file - --source-data-dir ~/.influxdb_iox/object_store \ - --sink-object-store file \ - --sink-data-dir /tmp/iceberg \ - --export-config-path config.json -``` +After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: -The export command outputs an absolute path to an Iceberg metadata file: +`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` -`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json -` +This metadata file is what you'll reference when creating your Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From 43c2e5e6e8e16836734560058a1255cd32a1fc47 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:04:51 -0700 Subject: [PATCH 43/62] Changes to- Example: Export data using Iceberg exporter --- .../influxdb3-query-guides/snapshots/snowflake.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index af8f2e3e0a..c2de54b1c9 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -55,13 +55,12 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -#### Example: Export data using Iceberg exporter +#### Creating a configuration file -This example assumes the following: +Before running the export command, you need to create a configuration file that specifies which tables to export: -- You have a `config.json`. - -#### Example `config.json` +1. Create a file named `config.json` in your working directory +2. Add the following JSON content, adjusting the namespace and table names to match your data: ```json { @@ -74,6 +73,8 @@ This example assumes the following: } ``` +This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. + After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` From e55ce49f4cf080b73e6c1a16c934b36086300112 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:32:05 -0700 Subject: [PATCH 44/62] Adding note to- Export InfluxDB time series data to Iceberg format --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c2de54b1c9..52e29f2830 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -54,7 +54,13 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + +### Export InfluxDB time series data to Iceberg format +> **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. + +Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. #### Creating a configuration file Before running the export command, you need to create a configuration file that specifies which tables to export: From bd320e6f6015e9b53ae45271e616527c82b81a47 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:39:38 -0700 Subject: [PATCH 45/62] removing reduandant sections and information --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 52e29f2830..8ffd7f0564 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -53,14 +53,11 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format -Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). - -### Export InfluxDB time series data to Iceberg format - > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + #### Creating a configuration file Before running the export command, you need to create a configuration file that specifies which tables to export: From b7d2687860ae4919d1237d256390a646170b41b5 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:53:21 -0700 Subject: [PATCH 46/62] removing IOX references --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8ffd7f0564..1b5e1517d0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -56,7 +56,6 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). #### Creating a configuration file @@ -78,11 +77,7 @@ Before running the export command, you need to create a configuration file that This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. -After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: - -`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` - -This metadata file is what you'll reference when creating your Iceberg table in Snowflake. +After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From b8ac4c6c28a4153176477bfac254f8e8df739efb Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:03:01 -0700 Subject: [PATCH 47/62] Rewriting section to focus and clarify that Iceberg table is done with support assistance --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1b5e1517d0..55aa53dec5 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -81,9 +81,11 @@ After configuring the export settings in the `config.json` file, the system auto ### Create an Iceberg table in Snowflake -After exporting the data, create an Iceberg table in Snowflake. +After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: -#### Example: Create an Iceberg table in Snowflake +1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. + +2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: ```sql CREATE ICEBERG TABLE my_iceberg_table @@ -91,7 +93,9 @@ CREATE ICEBERG TABLE my_iceberg_table METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; ``` -Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. +3. Your support engineer will help ensure the EXTERNAL_VOLUME and METADATA_FILE_PATH parameters correctly point to your external storage and metadata file. + +This creates a table in Snowflake that reads directly from the Iceberg-formatted data exported from your InfluxDB instance. ### Query the Iceberg table from Snowflake From 27f2fbb51ceed5d35223251797492d4e01022e98 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:11:48 -0700 Subject: [PATCH 48/62] removing entier section: Interfaces for using Iceberg integration section --- .../snapshots/snowflake.md | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 55aa53dec5..8c048b67ca 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -108,24 +108,6 @@ SELECT * FROM my_iceberg_table WHERE timestamp > '2025-01-01'; ``` -## Interfaces for using Iceberg integration - -- [Use the CLI to trigger snapshot exports](#use-the-CLI-to-trigger-snapshot-exports) -- [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) -- [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) - -### Use the CLI to trigger snapshot exports - -#### Example: Enable Iceberg feature and export a snapshot - -```sh -# Enable Iceberg feature -$ influxctl enable-iceberg - -# Export a snapshot -$ influxctl export --namespace foo --table bar -``` - ### Use the API to manage and configure snapshots Use the {{% product-name %}} HTTP API to export snapshots and check status. From 7ea9afcea5e203de83d28109c32aad4df6f99c1f Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:28:12 -0700 Subject: [PATCH 49/62] Adding read-only note to section --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8c048b67ca..733cd6aafa 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -81,8 +81,14 @@ After configuring the export settings in the `config.json` file, the system auto ### Create an Iceberg table in Snowflake +After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: + After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: +> **⚠️ Important**: **Tables created through this integration are read-only.** +> **You cannot write directly to these tables using Snowflake or any other engine.** +> They serve as an analytics interface to your InfluxDB data. + 1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. 2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: From 569122a00da4e486772f844beaed3aad2a6da00f Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:56:27 -0700 Subject: [PATCH 50/62] updating TOC numbe4r list --- .../influxdb3-query-guides/snapshots/snowflake.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 733cd6aafa..b621908cb5 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -17,11 +17,10 @@ Before you begin, ensure you have the following: ## Integrate InfluxDB 3 with Snowflake -1. [Configure external storage](#configure-external-storage) -2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) -3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) -4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -5. [Query your data in Snowflake](#query-your-data-in-snowflake) +1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) +2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) +3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) ### Create a Snowflake external stage @@ -36,7 +35,7 @@ CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -### Set up a catalog integration in Snowflake +### Set up a catalog integration in Snowflake {#set-up-a-catalog-integration-in-snowflake} Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -79,7 +78,7 @@ This configuration file tells the exporter which InfluxDB tables to convert to I After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. -### Create an Iceberg table in Snowflake +### Create an Iceberg table in Snowflake After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: From 25f2d8332073ca0b093ff2d546e60d6a4725a033 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 13:06:51 -0700 Subject: [PATCH 51/62] Updating who to contact for Snowflake Integration --- .../influxdb3-query-guides/snapshots/snowflake.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index b621908cb5..db5589c5f6 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,10 @@ +# Snowflake Integration + Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. +> **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. + ### Key Benefits - **Efficient data access**: Query your data directly from Snowflake. @@ -17,16 +21,16 @@ Before you begin, ensure you have the following: ## Integrate InfluxDB 3 with Snowflake +Once you've contacted sales and enabled Iceberg integration, your InfluxData support engineers will help you with the following steps: + 1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) 2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) 3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) 4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) - ### Create a Snowflake external stage -Use the `CREATE STAGE` Snowflake SQL command to set up an external storage location -(such as AWS S3) to store Iceberg table data and metadata--for example: +Your support engineer will guide you through setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command. This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. #### Example: Configure an S3 stage in Snowflake @@ -34,8 +38,9 @@ Use the `CREATE STAGE` Snowflake SQL command to set up an external storage locat CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; +``` -### Set up a catalog integration in Snowflake {#set-up-a-catalog-integration-in-snowflake} +### Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -84,7 +89,7 @@ After the export process is complete, you can work with your InfluxData support After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: -> **⚠️ Important**: **Tables created through this integration are read-only.** +> **Note**: **Tables created through this integration are read-only.** > **You cannot write directly to these tables using Snowflake or any other engine.** > They serve as an analytics interface to your InfluxDB data. From f524cda671a1929fbac1f810715720abe9ada0ef Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:39:40 -0700 Subject: [PATCH 52/62] Restructuring, reving, and proof reading edit --- .../snapshots/snowflake.md | 103 ++++++++++++------ 1 file changed, 69 insertions(+), 34 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index db5589c5f6..59bc2957c3 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -15,9 +15,22 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q Before you begin, ensure you have the following: -- A **Snowflake account** with necessary permissions. -- Access to an **external object store** (such as AWS S3). -- Familiarity with **Apache Iceberg** and **Snowflake**. +- **InfluxDB Cloud Dedicated plan** or compatible environment +- **Snowflake account** with appropriate permissions +- **External object store** (AWS S3, Azure Blob Storage, or GCP) + +## Request Iceberg Integration + +1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. + +2. Be prepared to provide: + - Your organization ID + - Customer name + - Cloud provider and region + - External storage details (bucket name, region, permissions) + - Technical contact information + +> **Note**: Iceberg integration is a premium feature that requires setup by the InfluxData team. ## Integrate InfluxDB 3 with Snowflake @@ -42,7 +55,7 @@ STORAGE_INTEGRATION=my_storage_integration; ### Set up a catalog integration in Snowflake -Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +Your support engineer will help you set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. #### Example: Create a catalog integration in Snowflake @@ -59,7 +72,58 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. -Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#tab-cli) +[API](#tab-api) +{{% /tabs %}} + +{{% tab-content %}} +{{% tab-pane id="tab-cli" %}} + +#### Using the CLI + +Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: + +```sh +influxctl snapshot export --namespace foo --table bar +``` + +{{% /tab-pane %}} +{{% tab-pane id="tab-api" %}} + +#### Using the API + +Use the {{% product-name %}} HTTP API to export snapshots and check status. + +##### Example: Export a snapshot + +This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. + +- **Method**: `POST` +- **Endpoint**: `/snapshots/export` +- **Request body**: + +```json +{ + "namespace": "foo", + "table": "bar" +} +``` +The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. + +##### Example: Check snapshot status + +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. + +- **Method**: `GET` +- **Endpoint**: `/snapshots/status` + +The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. + +{{% /tab-pane %}} +{{% /tab-content %}} +{{< /tabs-wrapper >}} #### Creating a configuration file @@ -118,35 +182,6 @@ SELECT * FROM my_iceberg_table WHERE timestamp > '2025-01-01'; ``` -### Use the API to manage and configure snapshots - -Use the {{% product-name %}} HTTP API to export snapshots and check status. - -#### Example: Export a snapshot - -This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. - -- **Method**: `POST` -- **Endpoint**: `/snapshots/export` -- **Request body**: - -```json -{ - "namespace": "foo", - "table": "bar" -} -``` -The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. - -#### Example: Check snapshot status - -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - -- **Method**: `GET` -- **Endpoint**: `/snapshots/status` - -The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. - ## Considerations and limitations When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: From 4b18dba5edf044fd467acce23afc9f933f2c1831 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:42:36 -0700 Subject: [PATCH 53/62] google dev style read through --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 59bc2957c3..0a6572d8f4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,11 +1,11 @@ -# Snowflake Integration +# Snowflake integration -Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. -Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. +Integrate {{< product-name >}} with Snowflake without complex ETL processes. +Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. > **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. -### Key Benefits +## Key benefits - **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. @@ -15,11 +15,11 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q Before you begin, ensure you have the following: -- **InfluxDB Cloud Dedicated plan** or compatible environment +- **{{< product-name >}} Cloud Dedicated plan** or compatible environment - **Snowflake account** with appropriate permissions - **External object store** (AWS S3, Azure Blob Storage, or GCP) -## Request Iceberg Integration +## Request Iceberg integration 1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. From 08008246dc4b3941929349dc0212cd85e31b5d7a Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:47:09 -0700 Subject: [PATCH 54/62] updating prereqs --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0a6572d8f4..c65e207d2c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -16,8 +16,6 @@ Export time series data snapshots into Apache Iceberg format and query directly Before you begin, ensure you have the following: - **{{< product-name >}} Cloud Dedicated plan** or compatible environment -- **Snowflake account** with appropriate permissions -- **External object store** (AWS S3, Azure Blob Storage, or GCP) ## Request Iceberg integration From 5a5ca2359091a659fe0bb74877e039560901427b Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 16:00:07 -0700 Subject: [PATCH 55/62] wip: removing tab shortcode to figure out error --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c65e207d2c..570f99827e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -70,15 +70,6 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. -{{< tabs-wrapper >}} -{{% tabs %}} -[CLI](#tab-cli) -[API](#tab-api) -{{% /tabs %}} - -{{% tab-content %}} -{{% tab-pane id="tab-cli" %}} - #### Using the CLI Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: @@ -87,9 +78,6 @@ Use the `influxctl` command to export InfluxDB time-series data to Iceberg forma influxctl snapshot export --namespace foo --table bar ``` -{{% /tab-pane %}} -{{% tab-pane id="tab-api" %}} - #### Using the API Use the {{% product-name %}} HTTP API to export snapshots and check status. From 5d0d7823ed9a454ef23819fb218e038813f480aa Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 16:36:06 -0700 Subject: [PATCH 56/62] adding shortcode structure back --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 570f99827e..ec669a045f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -70,6 +70,14 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#tab-cli) +[API](#tab-api) +{{% /tabs %}} + +{{% tab-content %}} +{{% tab-pane id="tab-cli" %}} #### Using the CLI Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: @@ -77,7 +85,8 @@ Use the `influxctl` command to export InfluxDB time-series data to Iceberg forma ```sh influxctl snapshot export --namespace foo --table bar ``` - +{{% /tab-pane %}} +{{% tab-pane id="tab-api" %}} #### Using the API Use the {{% product-name %}} HTTP API to export snapshots and check status. @@ -106,7 +115,6 @@ This example shows how to check the status of an ongoing or completed snapshot e - **Endpoint**: `/snapshots/status` The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. - {{% /tab-pane %}} {{% /tab-content %}} {{< /tabs-wrapper >}} From 30cd321c5ecdce3f74a10d24fee917d9b0822860 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 12:23:34 -0500 Subject: [PATCH 57/62] feat(dedicated): iceberg export for snowflake: - Add directory and frontmatter - Revise content to be Cloud Dedicated specific, omitting implementation and config detail that will be handled by InfluxData Support - Fix code samples - Organize, clean up headings - TODO: Comment out influxctl and API code if necessary (until features are completed) - TODO: Verify details with PM - TODO: Move content as noted in the iceberg-for-snowflake.md comment --- .../process-data/export/_index.md | 16 ++ .../export/iceberg-for-snowflake.md | 19 ++ .../snapshots/snowflake.md | 198 +++++++----------- 3 files changed, 113 insertions(+), 120 deletions(-) create mode 100644 content/influxdb3/cloud-dedicated/process-data/export/_index.md create mode 100644 content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md diff --git a/content/influxdb3/cloud-dedicated/process-data/export/_index.md b/content/influxdb3/cloud-dedicated/process-data/export/_index.md new file mode 100644 index 0000000000..f00e2ed286 --- /dev/null +++ b/content/influxdb3/cloud-dedicated/process-data/export/_index.md @@ -0,0 +1,16 @@ +--- +title: Export data +description: > + Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems + and avoid complex ETL processes. +menu: + influxdb3_cloud_dedicated: + parent: Process & visualize data +weight: 103 +--- +Use InfluxData export tools to: + +- Integrate time series data stored in {{% product-name %}} with other tools and systems +- Avoid complex ETL (Extract, Transform, Load) processes + +{{< children >}} \ No newline at end of file diff --git a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md new file mode 100644 index 0000000000..6cbf1447d3 --- /dev/null +++ b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md @@ -0,0 +1,19 @@ +--- +title: Export in Iceberg format for Snowflake +seotitle: Export time series data to Iceberg format for Snowflake +description: > + Integrate {{< product-name >}} with Snowflake without complex ETL processes. + Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. +menu: + influxdb3_cloud_dedicated: + name: Snowflake with Iceberg + parent: Export data +weight: 101 +influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] +source: /shared/influxdb3-query-guides/snapshots/snowflake.md +--- + + + + diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index ec669a045f..07d4a9663b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,185 +1,143 @@ -# Snowflake integration - Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. -> **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. - ## Key benefits - **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. -## Prerequisites +> [!Note] +> #### Contact sales to enable Iceberg integration +> +> Iceberg integration is a premium feature that requires setup by the InfluxData team. +> [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. + +- [Prerequisite](#prerequisite) +- [Request Iceberg integration](#request-iceberg-integration) +- [Integrate InfluxDB 3 with Snowflake](#integrate-influxdb-3-with-snowflake) +- [Export data to Iceberg format](#export-data-to-iceberg-format) +- [Check snapshot status](#check-snapshot-status) +- [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) +- [Considerations and limitations](#considerations-and-limitations) -Before you begin, ensure you have the following: +### Prerequisite -- **{{< product-name >}} Cloud Dedicated plan** or compatible environment +Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/cloud-dedicated/get-started/setup/). -## Request Iceberg integration +### Request Iceberg integration -1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. +1. [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration for your {{% product-name %}} cluster. -2. Be prepared to provide: +2. Be prepared to provide the following information: - Your organization ID - Customer name - Cloud provider and region - External storage details (bucket name, region, permissions) - Technical contact information -> **Note**: Iceberg integration is a premium feature that requires setup by the InfluxData team. - -## Integrate InfluxDB 3 with Snowflake - -Once you've contacted sales and enabled Iceberg integration, your InfluxData support engineers will help you with the following steps: - -1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) -2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) -3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) - -### Create a Snowflake external stage - -Your support engineer will guide you through setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command. This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. - -#### Example: Configure an S3 stage in Snowflake - -```sql -CREATE STAGE my_s3_stage -URL='s3://my-bucket/' -STORAGE_INTEGRATION=my_storage_integration; -``` +### Setup Snowflake and configure your cluster -### Set up a catalog integration in Snowflake +Iceberg integration for {{% product-name %}} requires setup by the InfluxData team. +After you have contacted sales and enabled Iceberg integration, you'll specify which +tables you want to make available for exporting. +InfluxData support engineers will help you with the integration steps, which include: -Your support engineer will help you set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details. +This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. +- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export. -#### Example: Create a catalog integration in Snowflake +After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake. -```sql -CREATE CATALOG INTEGRATION my_catalog_integration - CATALOG_SOURCE = 'OBJECT_STORE' - TABLE_FORMAT = 'ICEBERG' - ENABLED = TRUE; -``` +> [!Important] +> #### Export tables are read-only in Snowflake +> +> Tables created through this integration are _read-only_. +> You cannot write directly to these tables using Snowflake or any other engine. +> -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +## Export data to Iceberg format -### Export InfluxDB time series data to Iceberg format +Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg format. -> **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. +> [!Important] +> Before you can export data to Iceberg format, the InfluxData Support Team needs +> to have set up your tables for exporting. +> For more information, [contact the InfluxData Sales team]({{< cta-link >}}). {{< tabs-wrapper >}} {{% tabs %}} -[CLI](#tab-cli) -[API](#tab-api) +[CLI](#cli) +[API](#api) {{% /tabs %}} - {{% tab-content %}} -{{% tab-pane id="tab-cli" %}} -#### Using the CLI -Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: - -```sh -influxctl snapshot export --namespace foo --table bar +{{% code-placeholders "NAMESPACE|TABLE_NAME" %}} +```bash +influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME ``` -{{% /tab-pane %}} -{{% tab-pane id="tab-api" %}} -#### Using the API - -Use the {{% product-name %}} HTTP API to export snapshots and check status. +{{% /code-placeholders %}} -##### Example: Export a snapshot +Replace the following: -This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export -- **Method**: `POST` -- **Endpoint**: `/snapshots/export` -- **Request body**: - -```json -{ - "namespace": "foo", - "table": "bar" -} -``` -The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. - -##### Example: Check snapshot status - -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - -- **Method**: `GET` -- **Endpoint**: `/snapshots/status` - -The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. -{{% /tab-pane %}} {{% /tab-content %}} -{{< /tabs-wrapper >}} - -#### Creating a configuration file +{{% tab-content %}} -Before running the export command, you need to create a configuration file that specifies which tables to export: +**Endpoint**: -1. Create a file named `config.json` in your working directory -2. Add the following JSON content, adjusting the namespace and table names to match your data: +{{% api-endpoint method="POST" endpoint="/snapshots/export" %}} +**Request body**: + ```json { - "exports": [ - { - "namespace": "company_sensors", - "table_name": "cpu" - } - ] + "namespace": "NAMESPACE", + "table": "TABLE_NAME" } ``` -This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. +The following example shows how to use cURL with the HTTP API: -After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. +{{% code-placeholders "NAMESPACE|TABLE_NAME" %}} +```bash +curl -X POST https://{{% influxdb/host %}}/snapshots/export \ + -d '{"namespace": "NAMESPACE", "table": "TABLE_NAME"}' +``` +{{% /code-placeholders %}} -### Create an Iceberg table in Snowflake +Replace the following: -After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export -After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: +{{% /tab-content %}} +{{< /tabs-wrapper >}} -> **Note**: **Tables created through this integration are read-only.** -> **You cannot write directly to these tables using Snowflake or any other engine.** -> They serve as an analytics interface to your InfluxDB data. +## Check snapshot status -1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. +Use the HTTP API to check the status of an ongoing or completed snapshot export. -2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: +{{% api-endpoint method="GET" endpoint="/snapshots/status" %}} -```sql -CREATE ICEBERG TABLE my_iceberg_table - EXTERNAL_VOLUME = 'my_external_volume' - METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; +```bash +curl -X GET https://{{% influxdb/host %}}/snapshots/status ``` -3. Your support engineer will help ensure the EXTERNAL_VOLUME and METADATA_FILE_PATH parameters correctly point to your external storage and metadata file. - -This creates a table in Snowflake that reads directly from the Iceberg-formatted data exported from your InfluxDB instance. - -### Query the Iceberg table from Snowflake +## Query the Iceberg table from Snowflake -Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. - -#### Example: Query the Iceberg table - -```sql -SELECT * FROM my_iceberg_table -WHERE timestamp > '2025-01-01'; -``` +Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}. ## Considerations and limitations When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: +- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine. - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. + From 0be333c33f3a1b7e1e965cdcdb25d81e03e5d259 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:51:38 -0500 Subject: [PATCH 58/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 07d4a9663b..65f2065af0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -64,7 +64,7 @@ Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg f > [!Important] > Before you can export data to Iceberg format, the InfluxData Support Team needs -> to have set up your tables for exporting. +> to set up your tables for exporting. > For more information, [contact the InfluxData Sales team]({{< cta-link >}}). {{< tabs-wrapper >}} From 4fabcc917828606d6ead1d4839f12c4998d22a77 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:51:56 -0500 Subject: [PATCH 59/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 65f2065af0..8ca9a72a42 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -8,7 +8,7 @@ Export time series data snapshots into Apache Iceberg format and query directly - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. > [!Note] -> #### Contact sales to enable Iceberg integration +> #### Contact sales to enable the InfluxDB Iceberg integration > > Iceberg integration is a premium feature that requires setup by the InfluxData team. > [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. From d287d39788c7b4b9be6f4781535b52419fe78670 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:52:16 -0500 Subject: [PATCH 60/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8ca9a72a42..da56408d2b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -10,7 +10,7 @@ Export time series data snapshots into Apache Iceberg format and query directly > [!Note] > #### Contact sales to enable the InfluxDB Iceberg integration > -> Iceberg integration is a premium feature that requires setup by the InfluxData team. +> The InfluxDB Iceberg integration is a premium feature. > [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. - [Prerequisite](#prerequisite) From a488a4fd1dc1940e14412571932c691756202653 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:54:04 -0500 Subject: [PATCH 61/62] Apply suggestions from code review Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index da56408d2b..a7c06ca894 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -38,7 +38,7 @@ Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/c ### Setup Snowflake and configure your cluster -Iceberg integration for {{% product-name %}} requires setup by the InfluxData team. +The {{% product-name %}} Iceberg integration requires setup by the InfluxData team. After you have contacted sales and enabled Iceberg integration, you'll specify which tables you want to make available for exporting. InfluxData support engineers will help you with the integration steps, which include: @@ -60,7 +60,7 @@ After the setup is complete, you can export data snapshots to Iceberg format and ## Export data to Iceberg format -Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg format. +Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format. > [!Important] > Before you can export data to Iceberg format, the InfluxData Support Team needs From 0cd0cdeac37f7073df0ab7666b8a3a90be2100f1 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 16:17:08 -0500 Subject: [PATCH 62/62] feat(dedicated): iceberg for snowflake: - Merge fixes - Provide instruction for checking snapshot status - Cleanup --- .../export/iceberg-for-snowflake.md | 19 ------- .../use-integrations}/_index.md | 7 ++- .../iceberg-for-snowflake.md} | 57 +++++++++++++++---- 3 files changed, 49 insertions(+), 34 deletions(-) delete mode 100644 content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md rename content/influxdb3/cloud-dedicated/{process-data/export => query-data/use-integrations}/_index.md (73%) rename content/{shared/influxdb3-query-guides/snapshots/snowflake.md => influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md} (64%) diff --git a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md deleted file mode 100644 index 6cbf1447d3..0000000000 --- a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Export in Iceberg format for Snowflake -seotitle: Export time series data to Iceberg format for Snowflake -description: > - Integrate {{< product-name >}} with Snowflake without complex ETL processes. - Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. -menu: - influxdb3_cloud_dedicated: - name: Snowflake with Iceberg - parent: Export data -weight: 101 -influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] -source: /shared/influxdb3-query-guides/snapshots/snowflake.md ---- - - - - diff --git a/content/influxdb3/cloud-dedicated/process-data/export/_index.md b/content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md similarity index 73% rename from content/influxdb3/cloud-dedicated/process-data/export/_index.md rename to content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md index f00e2ed286..59425da14f 100644 --- a/content/influxdb3/cloud-dedicated/process-data/export/_index.md +++ b/content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md @@ -1,14 +1,15 @@ --- -title: Export data +title: Use integrations to export and query data description: > Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems and avoid complex ETL processes. menu: influxdb3_cloud_dedicated: - parent: Process & visualize data + parent: Query data + name: Use integrations weight: 103 --- -Use InfluxData export tools to: +Use {{% product-name %}} integrations: - Integrate time series data stored in {{% product-name %}} with other tools and systems - Avoid complex ETL (Extract, Transform, Load) processes diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md similarity index 64% rename from content/shared/influxdb3-query-guides/snapshots/snowflake.md rename to content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md index a7c06ca894..63aca4d2a0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md @@ -1,3 +1,17 @@ +--- +title: Export Iceberg for Snowflake +seotitle: Export time series data to Iceberg format for Snowflake +description: > + Integrate {{< product-name >}} with Snowflake without complex ETL processes. + Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. +menu: + influxdb3_cloud_dedicated: + name: Export Iceberg for Snowflake + parent: Use integrations +weight: 101 +influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] +--- + Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. @@ -40,16 +54,18 @@ Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/c The {{% product-name %}} Iceberg integration requires setup by the InfluxData team. After you have contacted sales and enabled Iceberg integration, you'll specify which -tables you want to make available for exporting. -InfluxData support engineers will help you with the integration steps, which include: +tables you want to make available for export. +InfluxData support engineers will work with you to set up the integration by: -- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details. -This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. -- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export. +- Configuring a Snowflake external stage using the `CREATE STAGE` SQL command with your storage provider details. +This creates an external storage location (such as AWS S3) for Iceberg table data and metadata. +- Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables. +For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +- Creating an exports configuration that specifies which tables are available for export. + > [!Important] > #### Export tables are read-only in Snowflake @@ -60,6 +76,14 @@ After the setup is complete, you can export data snapshots to Iceberg format and ## Export data to Iceberg format +After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots. +The exported data is stored in Iceberg format in the external storage location you specified. + +_To refresh the exported data with new or changed data, submit a request to the +[InfluxData Support team](https://support.influxdata.com) each time a refresh is needed. +Currently, customer-triggered exports aren't supported._ + + -- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export {{% /tab-content %}} {{% tab-content %}} @@ -112,14 +136,19 @@ curl -X POST https://{{% influxdb/host %}}/snapshots/export \ Replace the following: -- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. -- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export {{% /tab-content %}} {{< /tabs-wrapper >}} +-- Customer-triggered exports aren't supported --> ## Check snapshot status +To check the status of a snapshot export, contact the [InfluxData Support team](https://support.influxdata.com). + + ## Query the Iceberg table from Snowflake @@ -136,8 +166,11 @@ Use SQL in Snowflake to query the Iceberg table exported from {{< product-name > When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: +- **Export controls limited to InfluxData**: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes. + Submit a request to the [InfluxData Support team](https://support.influxdata.com) to: + - Update your Snowflake tables with new or changed data + - Check the status of a snapshot export - **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine. - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. -