Ultimate

Amazon Kinesis source

Last update: 2025-01-08
  • Topics:
  • Sources
    View more on this topic
  • Created for:
  • Developer
IMPORTANT
  • The Amazon Kinesis source is available in the sources catalog to users who have purchased Real-Time CDP Ultimate.

  • You can now use the Amazon Kinesis source when running Adobe Experience Platform on Amazon Web Services (AWS). Experience Platform running on AWS is currently available to a limited number of customers. To learn more about the supported Experience Platform infrastructure, see the Experience Platform multi-cloud overview.

Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.

Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Amazon Kinesis in real time.

NOTE

The scale factor for Kinesis must be increased if you need to ingest high volume data. Currently, maximum volume of data that you can bring from your Kinesis account to Platform is 4000 records per second. To scale up and ingest higher volume data, please contact your Adobe representative.

Prerequisites

The following section provides further information on prerequisite set up required before you can create a Kinesis source connection.

Set up access policy

A Kinesis stream requires the following permissions to create a source connection:

  • GetShardIterator
  • GetRecords
  • DescribeStream
  • ListStreams

These permissions are arranged through the Kinesis console and are checked by Platform once you enter your credentials and select your data stream.

The example below displays the minimum access rights required to create a Kinesis source connection.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kinesis:DescribeStream",
                "kinesis:ListStreams"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-2:901341027596:stream/*"
            ]
        }
    ]
}
Property Description
kinesis:GetShardIterator An action required to traverse through records.
kinesis:GetRecords An action required to get records from a specific offset or shard ID.
kinesis:DescribeStream An action that returns information regarding the stream including the shard map, which is needed to generate a shard ID.
kinesis:ListStreams An action required to list out available streams that you can select from the UI.

For more information on controlling access for Kinesis data streams, see the following Kinesis document.

Configure iterator type

Kinesis supports the following iterator types to allow you to specify the order of how your data is read:

Iterator type Description
AT_SEQUENCE_NUMBER The data is read starting from a position identified by a specific sequence number.
AFTER_SEQUENCE_NUMBER The data is read starting after position identified by a specific sequence number.
AT_TIMESTAMP The data is read starting from a position identified by a specific timestamp.
TRIM_HORIZON The data is read starting from the oldest data record.
LATEST The data is read starting from the most recent data record.

A Kinesis UI source currently only supports TRIM_HORIZON, while the API supports both TRIM_HORIZON and LATEST as modes to get data. The default iterator value that Platform uses for the Kinesis source is TRIM_HORIZON.

For more information on iterator types, see the following Kinesis document.

Connect Amazon Kinesis to Platform

The documentation below provides information on how to connect Amazon Kinesis to Platform using APIs or the user interface:

Using APIs

Using the UI

On this page