The Amazon Kinesis source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.
This tutorial walks you through the steps to connect Amazon Kinesis (hereinafter referred to as “Kinesis”) to Experience Platform, using the Flow Service API.
This guide requires a working understanding of the following components of Adobe Experience Platform:
The following sections provide additional information that you will need to know in order to successfully connect Kinesis to Platform using the Flow Service API.
In order for Flow Service to connect with your Amazon Kinesis account, you must provide values for the following connection properties:
Credential | Description |
---|---|
accessKeyId |
The access key ID is one half of the access key pair used to authenticate your Kinesis account to Platform. |
secretKey |
The secret access key is the other half of the access key pair used to authenticate your Kinesis account to Platform. |
region |
The region for your Kinesis account. See the guide on adding IP addresses to your allow list for more information on regions. |
connectionSpec.id |
The connection specification returns a source’s connector properties, including authentication specifications related to creating the base and source connections. The Kinesis connection specification ID is: 86043421-563b-46ec-8e6c-e23184711bf6 . |
For more information on Kinesis access keys and how to generate them, refer to this AWS guide on managing access keys for IAM users.
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
The first step in creating a source connection is to authenticate your Kinesis source and generate a base connection ID. A base connection ID allows you to explore and navigate files from within your source and identify specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your Kinesis authentication credentials as part of the request parameters.
API format
POST /connections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Amazon Kinesis connection",
"description": "Connector for Amazon Kinesis",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"auth": {
"specName": "Aws Kinesis authentication credentials",
"params": {
"accessKeyId": "{ACCESS_KEY_ID}",
"secretKey": "{SECRET_KEY}",
"region": "{REGION}"
}
},
"connectionSpec": {
"id": "86043421-563b-46ec-8e6c-e23184711bf6",
"version": "1.0"
}
}'
Property | Description |
---|---|
auth.params.accessKeyId |
The access key ID for your Kinesis account. |
auth.params.secretKey |
The secret access key for your Kinesis account. |
auth.params.region |
The region for your Kinesis account. |
connectionSpec.id |
The Kinesis connection specification ID: 86043421-563b-46ec-8e6c-e23184711bf6 |
Response
A successful response returns details of the newly created base connection, including its unique identifier (id
). This ID is required in the next step to create a source connection.
{
"id": "4cb0c374-d3bb-4557-b139-5712880adc55",
"etag": "\"6507cfd8-0000-0200-0000-5e18fc600000\""
}
A source connection creates and manages the connection to the external source from where data is ingested. A source connection consists of information like data source, data format, and the source connection ID needed to create a dataflow. A source connection instance is specific to a tenant and organization.
To create a source connection, make a POST request to the /sourceConnections
endpoint of the Flow Service API.
API format
POST /sourceConnections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "AWS Kinesis source connection",
"description": "A source connection for AWS Kinesis",
"baseConnectionId": "4cb0c374-d3bb-4557-b139-5712880adc55",
"connectionSpec": {
"id": "86043421-563b-46ec-8e6c-e23184711bf6",
"version": "1.0"
},
"data": {
"format": "json"
},
"params": {
"stream": "{STREAM}",
"dataType": "raw",
"reset": "latest"
}
}'
Property | Description |
---|---|
name |
The name of your source connection. Ensure that the name of your source connection is descriptive as you can use this to look up information on your source connection. |
description |
An optional value that you can provide to include more information on your source connection. |
baseConnectionId |
The base connection ID of your Kinesis source that was generated in the previous step. |
connectionSpec.id |
The fixed connection specification ID for Kinesis. This ID is: 86043421-563b-46ec-8e6c-e23184711bf6 |
data.format |
The format of the Kinesis data that you want to ingest. Currently, the only supported data format is json . |
params.stream |
The name of the data stream to pull records from. |
params.dataType |
This parameter defines the type of the data that is being ingested. Supported data types include: raw and xdm . |
params.reset |
This parameter defines how the data will be read. Use latest to start reading from the most recent data, and use earliest to start reading from the first available data in the stream. |
Response
A successful response returns the unique identifier (id
) of the newly created source connection. This ID is required in the next tutorial to create a dataflow.
{
"id": "e96d6135-4b50-446e-922c-6dd66672b6b2",
"etag": "\"66013508-0000-0200-0000-5f6e2ae70000\""
}
By following this tutorial, you have created a Kinesis source connection using the Flow Service API. You can use this source connection ID in the next tutorial to create a streaming dataflow using the Flow Service API.