Skip to content

Iceberg Integration documentation #5918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 62 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
54f39f7
Iceberg Integration
MeelahMe Mar 19, 2025
0d70833
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
882547b
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
1bbbc74
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
6962de9
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
d2a5bcb
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
c647a73
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
7dc8f54
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
d77a651
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
d25621d
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
66128e9
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
8ed0dcc
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
2a1c09b
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
712796f
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
fe9790b
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
b6466e0
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
1dfc787
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
2a9023e
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
8e021c6
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
1e5c81e
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
44e5876
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
b5a8857
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
c5e3e7a
updating headings
MeelahMe Mar 21, 2025
24cf2cf
Adding a numbered-list TOC
MeelahMe Mar 21, 2025
7253468
Added an export command example
MeelahMe Mar 21, 2025
b3d5f81
Proof read: improving grammar and clarity
MeelahMe Mar 21, 2025
4b1f3c2
Adding explanations for examples
MeelahMe Mar 21, 2025
8e83dd5
Updating examples
MeelahMe Mar 21, 2025
0476969
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
bc4a8e0
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
981dcb6
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
6165ac3
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
09ef8c2
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
be368f4
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
56b1551
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
36c3fcb
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
46acc00
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
0237a45
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
902a704
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
d945e1c
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
4cf8dd7
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
3be23a2
removing run export example and adding
MeelahMe Mar 26, 2025
43c2e5e
Changes to- Example: Export data using Iceberg exporter
MeelahMe Mar 26, 2025
e55ce49
Adding note to- Export InfluxDB time series data to Iceberg format
MeelahMe Mar 26, 2025
bd320e6
removing reduandant sections and information
MeelahMe Mar 26, 2025
b7d2687
removing IOX references
MeelahMe Mar 26, 2025
b8ac4c6
Rewriting section to focus and clarify that Iceberg table is done wit…
MeelahMe Mar 26, 2025
27f2fbb
removing entier section: Interfaces for using Iceberg integration sec…
MeelahMe Mar 26, 2025
7ea9afc
Adding read-only note to section
MeelahMe Mar 26, 2025
569122a
updating TOC numbe4r list
MeelahMe Mar 26, 2025
25f2d83
Updating who to contact for Snowflake Integration
MeelahMe Mar 26, 2025
f524cda
Restructuring, reving, and proof reading edit
MeelahMe Mar 26, 2025
4b18dba
google dev style read through
MeelahMe Mar 26, 2025
0800824
updating prereqs
MeelahMe Mar 26, 2025
5a5ca23
wip: removing tab shortcode to figure out error
MeelahMe Mar 26, 2025
5d0d782
adding shortcode structure back
MeelahMe Mar 26, 2025
30cd321
feat(dedicated): iceberg export for snowflake:
jstirnaman Mar 27, 2025
0be333c
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
4fabcc9
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
d287d39
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
a488a4f
Apply suggestions from code review
jstirnaman Mar 27, 2025
0cd0cde
feat(dedicated): iceberg for snowflake:
jstirnaman Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: Use integrations to export and query data
description: >
Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems
and avoid complex ETL processes.
menu:
influxdb3_cloud_dedicated:
parent: Query data
name: Use integrations
weight: 103
---
Use {{% product-name %}} integrations:

- Integrate time series data stored in {{% product-name %}} with other tools and systems
- Avoid complex ETL (Extract, Transform, Load) processes

{{< children >}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
title: Export Iceberg for Snowflake
seotitle: Export time series data to Iceberg format for Snowflake
description: >
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
menu:
influxdb3_cloud_dedicated:
name: Export Iceberg for Snowflake
parent: Use integrations
weight: 101
influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export]
---

Integrate {{< product-name >}} with Snowflake without complex ETL processes.
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.

## Key benefits

- **Efficient data access**: Query your data directly from Snowflake.
- **Cost-effective storage**: Optimize data retention and minimize storage costs.
- **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake.

> [!Note]
> #### Contact sales to enable the InfluxDB Iceberg integration
>
> The InfluxDB Iceberg integration is a premium feature.
> [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account.

- [Prerequisite](#prerequisite)
- [Request Iceberg integration](#request-iceberg-integration)
- [Integrate InfluxDB 3 with Snowflake](#integrate-influxdb-3-with-snowflake)
- [Export data to Iceberg format](#export-data-to-iceberg-format)
- [Check snapshot status](#check-snapshot-status)
- [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake)
- [Considerations and limitations](#considerations-and-limitations)

### Prerequisite

Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/cloud-dedicated/get-started/setup/).

### Request Iceberg integration

1. [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration for your {{% product-name %}} cluster.

2. Be prepared to provide the following information:
- Your organization ID
- Customer name
- Cloud provider and region
- External storage details (bucket name, region, permissions)
- Technical contact information

### Setup Snowflake and configure your cluster

The {{% product-name %}} Iceberg integration requires setup by the InfluxData team.
After you have contacted sales and enabled Iceberg integration, you'll specify which
tables you want to make available for export.
InfluxData support engineers will work with you to set up the integration by:

- Configuring a Snowflake external stage using the `CREATE STAGE` SQL command with your storage provider details.
This creates an external storage location (such as AWS S3) for Iceberg table data and metadata.
- Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables.
For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
- Creating an exports configuration that specifies which tables are available for export.

<!-- Customer-triggered exports aren't yet supported.
After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake.
-->

> [!Important]
> #### Export tables are read-only in Snowflake
>
> Tables created through this integration are _read-only_.
> You cannot write directly to these tables using Snowflake or any other engine.
>

## Export data to Iceberg format

After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots.
The exported data is stored in Iceberg format in the external storage location you specified.

_To refresh the exported data with new or changed data, submit a request to the
[InfluxData Support team](https://support.influxdata.com) each time a refresh is needed.
Currently, customer-triggered exports aren't supported._

<!-- Customer-triggered exports aren't yet supported.
Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format.

> [!Important]
> Before you can export data to Iceberg format, the InfluxData Support Team needs
> to set up your tables for exporting.
> For more information, [contact the InfluxData Sales team]({{< cta-link >}}).

{{< tabs-wrapper >}}
{{% tabs %}}
[CLI](#cli)
[API](#api)
{{% /tabs %}}
{{% tab-content %}}

{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
```bash
influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME
```
{{% /code-placeholders %}}

Replace the following:

- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export

{{% /tab-content %}}
{{% tab-content %}}

**Endpoint**:

{{% api-endpoint method="POST" endpoint="/snapshots/export" %}}

**Request body**:

```json
{
"namespace": "NAMESPACE",
"table": "TABLE_NAME"
}
```

The following example shows how to use cURL with the HTTP API:

{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
```bash
curl -X POST https://{{% influxdb/host %}}/snapshots/export \
-d '{"namespace": "NAMESPACE", "table": "TABLE_NAME"}'
```
{{% /code-placeholders %}}

Replace the following:

- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export

{{% /tab-content %}}
{{< /tabs-wrapper >}}
-- Customer-triggered exports aren't supported -->

## Check snapshot status

To check the status of a snapshot export, contact the [InfluxData Support team](https://support.influxdata.com).

<!-- API isn't yet available to customers.

Use the HTTP API to check the status of an ongoing or completed snapshot export.

{{% api-endpoint method="GET" endpoint="/snapshots/status" %}}

```bash
curl -X GET https://{{% influxdb/host %}}/snapshots/status
```
-- API isn't yet available to customers -->

## Query the Iceberg table from Snowflake

Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}.

## Considerations and limitations

When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind:

- **Export controls limited to InfluxData**: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes.
Submit a request to the [InfluxData Support team](https://support.influxdata.com) to:
- Update your Snowflake tables with new or changed data
- Check the status of a snapshot export
- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine.
- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB.
- **Performance**: Query performance may vary based on data size and query complexity.
- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration.
Empty file.