The Snowflake streaming source is available in the API to users who have purchased Real-Time CDP Ultimate.
You can now use the Snowflake streaming source when running Adobe Experience Platform on Amazon Web Services (AWS). Experience Platform running on AWS is currently available to a limited number of customers. To learn more about the supported Experience Platform infrastructure, see the Experience Platform multi-cloud overview.
Adobe Experience Platform allows data to be ingested from external sources while providing you with the ability to structure, label, and enhance incoming data using Platform services. You can ingest data from a variety of sources such as Adobe applications, cloud-based storage, databases, and many others.
Experience Platform provides support for streaming data from a Snowflake database.
The Snowflake streaming source works by having data loaded by periodically executing an SQL query and creating an output record for each row in the resulting set.
By using Kafka Connect, the Snowflake streaming source tracks the latest record that it receives from each table, so that it can start in the correct location for the next iteration. The source uses this functionality to filter data and only get the updated rows from a table on each iteration.
The following section outlines prerequisite steps to complete before you can stream data from your Snowflake database to Experience Platform:
A list of IP addresses must be added to an allow list prior to working with source connectors. Failing to add your region-specific IP addresses to your allow list may lead to errors or non-performance when using sources. See the IP address allow list page for more information.
The documentation below provides information on how to connect Amazon Redshift to Platform using APIs or the user interface:
In order for Flow Service to connect with Snowflake, you must provide the following connection properties:
Credential | Description |
---|---|
account |
The full account identifier (account name or account locator) of your Snowflake account appended with the suffix snowflakecomputing.com . The account identifier can be of different formats:
|
warehouse |
The Snowflake warehouse manages the query execution process for the application. Each Snowflake warehouse is independent from one another and must be accessed individually when bringing data over to Platform. |
database |
The Snowflake database contains the data you want to bring the Platform. |
username |
The username for the Snowflake account. |
password |
The password for the Snowflake user account. |
role |
(Optional) A custom-defined role that can be provided for a user, for a given connection. If unprovided, this value defaults to public . |
connectionSpec.id |
The connection specification returns a source’s connector properties, including authentication specifications related to creating the base and source connections. The connection specification ID for Snowflake is 51ae16c2-bdad-42fd-9fce-8d5dfddaf140 . |
You must configure privileges to a role, even if the default public role is assigned, to allow your source connection to access the relevant Snowflake database, schema, and table. The various privileges for different Snowflake entities is as follows:
Snowflake entity | Require role privilege |
---|---|
Warehouse | OPERATE, USAGE |
Database | USAGE |
Schema | USAGE |
Table | SELECT |
Auto-resume and auto-suspend must be enabled in the advanced settings configuration of your warehouse.
For more information on role and privilege management, refer to the Snowflake API reference.
backfill
boolean flag for your Snowflake source when creating a source connection.
TIMESTAMP_LTZ
or TIMESTAMP_NTZ
. If the timestamp column is set to TIMESTAMP_NTZ
, then the corresponding timezone in which the values are stored should be passed via the timezoneValue
parameter. If unprovided, the value will default to UTC.
TIMESTAMP_TZ
cannot be used a timestamp column or in a mapping.The following tutorial provides steps on how to connect your Snowflake streaming source to Experience Platform using the API: