Amazon S3 connection

Destination changelog

Connect to your Amazon S3 storage through API or UI

• To connect to your Amazon S3 storage location using the Experience Platform user interface, read the sections Connect to the destination and Activate audiences to this destination below.
• To connect to your Amazon S3 storage location programmatically, read the guide on how to activate audiences to file-based destinations by using the Flow Service API tutorial.

Supported audiences

This section describes which types of audiences you can export to this destination.

Export type and frequency

Refer to the table below for information about the destination export type and frequency.

Export datasets

This destination supports dataset exports. For complete information on how to set up dataset exports, read the tutorials:

• How to export datasets using the Experience Platform user interface.
• How to export datasets programmatically using the Flow Service API.

File format of the exported data

When exporting audience data, Experience Platform creates a .csv, parquet, or .json file in the storage location that you provided. For more information about the files, see the supported file formats for export section in the audience activation tutorial.

When exporting datasets, Experience Platform creates a .parquet or .json file in the storage location that you provided. For more information about the files, see the verify successful dataset export section in the export datasets tutorial.

Connect to the destination

To connect to this destination, follow the steps described in the destination configuration tutorial. In the destination configuration workflow, fill in the fields listed in the two sections below.

Authenticate to destination

To authenticate to the destination, fill in the required fields and select Connect to destination. The Amazon S3 destination supports two authentication methods:

• Access key and secret key authentication
• Assumed role authentication

Authentication with S3 access key and secret key

Use this authentication method when you want to input your Amazon S3 access key and secret key to allow Experience Platform to export data to your Amazon S3 properties.

• Amazon S3 access key and Amazon S3 secret key: In Amazon S3, generate an access key - secret access key pair to grant Experience Platform access to your Amazon S3 account. Learn more in the Amazon Web Services documentation.

• Encryption key: Optionally, you can attach your RSA-formatted public key to add encryption to your exported files. View an example of a correctly formatted encryption key in the image below.

  

Authentication with S3 assumed role

Use this authentication type if you prefer not to share account keys and secret keys with Adobe. Instead, Experience Platform connects to your Amazon S3 location using role-based access.

• Role: Paste the ARN of the role that you created in AWS for the Adobe user. The pattern is similar to arn:aws:iam::800873819705:role/destinations-role-customer. See the steps below for detailed guidance on how to configure S3 access correctly.
• Encryption key: Optionally, you can attach your RSA-formatted public key to add encryption to your exported files. View an example of a correctly formatted encryption key in the image below.

To do this, you need to create in the AWS console an assumed role for Adobe with the right required permissions to write to your Amazon S3 buckets.

Create a policy with the required permissions

1. Open the AWS Console and go to IAM > Policies > Create policy

2. Select Policy Editor > JSON and add the permissions below.

   
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:GetBucketLocation",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": "arn:aws:s3:::bucket/folder/*"
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket"
               ],
               "Resource": "arn:aws:s3:::bucket"
           }
       ]
   }
   

3. On the next page, enter a name for your policy and save it for reference. You’ll need this policy name when creating the role in the next step.

Create user role in your S3 customer account

1. Open the AWS Console and go to IAM > Roles > Create new role
2. Select Trusted entity type > AWS account
3. Select An AWS account > Another AWS account and enter the Adobe account ID: 670664943635
4. Add permissions using the policy created earlier
5. Enter a role name (for example, destinations-role-customer). The role name should be treated as confidential, similar to a password. It can be up to 64 characters long and can contain alphanumeric characters and the following special characters: +=,.@-_. Then verify that:
   • The Adobe account ID 670664943635 is present in the Select trusted entities section
   • The policy created earlier is present in Permissions policy summary

Provide the role for Adobe to assume

After creating the role in AWS, you need to provide the role ARN to Adobe. The ARN follows this pattern: arn:aws:iam::800873819705:role/destinations-role-customer

You can find the ARN on the main page after creating the role in the AWS console. You will use this ARN when creating the destination.

Verify role permissions and trust relationships

Ensure that your role has the following configuration:

• Permissions: The role should have permissions to access S3 (either full access or the minimal permissions provided in the Create a policy with the required permissions step above)
• Trust relationships: The role should have the root Adobe account (670664943635) in its trust relationships

Alternative: Restrict to specific Adobe user (Optional)

If you prefer not to allow the entire Adobe account, you can restrict access to only the specific Adobe user. To do this, edit the trust policy with the following configuration:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::670664943635:user/destinations-adobe-user"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

For more information, refer to the AWS documentation on creating roles.

Fill in destination details

To configure details for the destination, fill in the required and optional fields below. An asterisk next to a field in the UI indicates that the field is required.

• Name: Enter a name that will help you identify this destination.
• Description: Enter a description of this destination.
• Bucket name: Enter the name of the Amazon S3 bucket to be used by this destination.
• Folder path: Enter the path to the destination folder that will host the exported files.
• File type: Select the format Experience Platform should use for the exported files. When selecting the CSV option, you can also configure the file formatting options.
• Compression format: Select the compression type that Experience Platform should use for the exported files.
• Include manifest file: Toggle this option on if you’d like the exports to include a manifest JSON file that contains information about the export location, export size, and more. The manifest is named using the format manifest-<<destinationId>>-<<dataflowRunId>>.json. View a sample manifest file. The manifest file includes the following fields:
  • flowRunId: The dataflow run which generated the exported file.
  • scheduledTime: The time in UTC when the file was exported.
  • exportResults.sinkPath: The path in your storage location where the exported file is deposited.
  • exportResults.name: The name of the exported file.
  • size: The size of the exported file, in bytes.

Enable alerts

You can enable alerts to receive notifications on the status of the dataflow to your destination. Select an alert from the list to subscribe to receive notifications on the status of your dataflow. For more information on alerts, see the guide on subscribing to destinations alerts using the UI.

When you are finished providing details for your destination connection, select Next.

Required Amazon S3 permissions

To successfully connect and export data to your Amazon S3 storage location, create an Identity and Access Management (IAM) user for Experience Platform in Amazon S3 and assign permissions for the following actions:

• s3:DeleteObject
• s3:GetBucketLocation
• s3:GetObject
• s3:ListBucket
• s3:PutObject
• s3:ListMultipartUploadParts

Minimum required permissions for IAM assumed role authentication

When configuring the IAM role as a customer, make sure that the permission policy associated with the role includes the required actions to the target folder in the bucket and the s3:ListBucket action for the root of the bucket. View below an example of the minimum permissions policy for this authentication type:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::bucket/folder/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::bucket"
        }
    ]
}

Activate audiences to this destination

See Activate audience data to batch profile export destinations for instructions on activating audiences to this destination.

Validate successful data export

To verify if data has been exported successfully, check your Amazon S3 storage and make sure that the exported files contain the expected profile populations.

IP address allowlist

Refer to the IP address allowlist article if you need to add Adobe IPs to an allowlist.