There are several destination types in Experience Platform, as shown in the diagram below. These destinations have slightly different export patterns with regards to what triggers a destination export and what is included in an export, as described in the sections further below.
This documentation page only describes the profile export behavior for the connections highlighted at the bottom of the diagram.
Before diving into specific information per destination type, it is important to understand the concept of message aggregation for streaming destinations.
Experience Platform destinations export data to API-based integrations as HTTPS calls. Once the destinations service is notified by other upstream services that profiles have been updated as a result of batch ingestion, streaming ingestion, batch segmentation, streaming segmentation or identity graph changes, data is exported and sent to streaming destinations.
Profiles are aggregated into HTTPS messages before being dispatched to destination API endpoints.
Take the Facebook destination with a configurable aggregation policy as an example - data is sent in an aggregated fashion, where the destinations service takes all the incoming data from the profile service upstream and aggregates it by one of the following, before dispatching it to Facebook:
Whichever of the thresholds above is first met triggers an export to Facebook. So, in the Facebook Custom Audiences dashboard, you might see audiences coming in from Experience Platform in 10,000 record increments. You might be seeing 10,000 records every 2-3 minutes because the data gets processed and aggregated faster than the 300 seconds export interval, and gets sent faster, so about every 2-3 minutes until all records have been processed. If there are insufficient records to make up a 10,000 batch, then the current number of records will be sent as is when the time window threshold is met, so you might see smaller batches sent to Facebook as well.
As another example, consider the HTTP API destination, which has a best effort aggregation policy, with maxUsersPerRequest: 10
. This means that a maximum of ten profiles will be aggregated before an HTTP call is fired to this destination, but Experience Platform tries to dispatch profiles to the destination as soon as the destinations service receives updated re-evaluation information from an upstream service.
The aggregation policy is configurable, and destination developers can decide how to configure the aggregation policy to best meet the rate limitations of the API endpoints downstream. Read more about aggregation policy in the Destination SDK documentation.
Enterprise destinations are available only to Adobe Real-Time Customer Data Platform Ultimate customers.
The enterprise destinations in Experience Platform are Amazon Kinesis, Azure Event Hubs, and HTTP API.
Experience Platform optimizes the profile export behavior to your enterprise destination, to only export data to your API endpoint when relevant updates to a profile have occurred following audience qualification or other significant events. Profiles are exported to your destination in the following situations:
In all the cases described above, only the profiles where relevant updates have occurred are exported to your destination. For example, if an audience mapped to the destination flow has a hundred members, and five new profiles qualify for the segment, the export to your destination is incremental and only includes the five new profiles.
Note that all the mapped attributes are exported for a profile, no matter where the changes lie. So, in the example above all the mapped attributes for those five new profiles will be exported even if the attributes themselves haven’t changed.
Regarding the data that is exported for a given profile, it is important to understand the two different concepts of what determines a data export to your enterprise destination and which data is included in the export.
What determines a destination export | What is included in the destination export |
---|---|
|
|
Enterprise destinations stream backfill data when activating profiles to a destination. This means that the first data export after configuring an activation workflow to a destination will include profiles who qualified for the activated audience prior to the audience being mapped to the destination.
For example, consider this dataflow to an HTTP destination where three audiences are selected in the dataflow, and four attributes are mapped to the destination.
A profile export to the destination can be determined by a profile qualifying for or exiting one of the three mapped segments. However, in the data export, in the segmentMembership
object, other unmapped audiences might appear, if that particular profile is a member of them and if these share the same merge policy as the audience that triggered the export. If a profile qualifies for the Customer with DeLorean Cars audience but is also a member of the Watched “Back to the Future” movie and Science fiction fans segments, then these other two audiences will also be present in the segmentMembership
object of the data export, even though these are not mapped in the dataflow, if these share the same merge policy with the Customer with DeLorean Cars segment.
From a profile attributes point of view, any changes to the four attributes mapped above will determine a destination export and any of the four mapped attributes present on the profile will be present in the data export.
You can see examples of exported data to various enterprise destinations in the Amazon Kinesis, Azure Event Hubs, and HTTP API destination documentation pages.
The profile export behavior for streaming destinations such as Facebook, Trade Desk, and other API-based integrations is very similar to the behavior described above for enterprise destinations.
Examples of streaming destinations are the destinations belonging to the social and advertising categories in the catalog.
Experience Platform optimizes the profile export behavior to your streaming destination, to only export data to streaming API-based destinations when relevant updates to a profile have occurred following audience qualification or other significant events. Profiles are exported to your destination in the following situations:
In all the cases described above, only the profiles where relevant updates have occurred are exported to your destination. For example, if an audience mapped to the destination flow has a hundred members, and five new profiles qualify for the segment, the export to your destination is incremental and only includes the five new profiles.
Note that all the mapped attributes are exported for a profile, no matter where the changes lie. So, in the example above all the mapped attributes for those five new profiles will be exported even if the attributes themselves haven’t changed.
Regarding the data that is exported for a given profile, it is important to understand the two different concepts of what determines a data export to your streaming API destination and which data is included in the export.
What determines a destination export | What is included in the destination export |
---|---|
|
|
Streaming API destinations stream backfill data when activating profiles to a destination. This means that the first data export after configuring an activation workflow to a destination will include profiles who qualified for the activated audience prior to the audience being mapped to the destination.
For example, consider this dataflow to a streaming destination where three audiences are selected in the dataflow.
A profile export to the destination can be determined by a profile qualifying for or exiting one of the three mapped segments. If a profile qualified for the Customer with DeLorean Cars segment, this will trigger an export. The other audiences (City - Dallas and Basic Site Active) might be exported as well in case the profile has that audience present with one of the possible statuses (realized
or exited
). Unmapped audiences (like Science fiction fans) will not be exported.
From a profile attributes point of view, any changes to the three attributes mapped above will determine a destination export.
When exporting profiles to file-based destinations in Experience Platform, there are three types of schedules (listed below) and two file export options (full or incremental files) that you can use. All these settings are set on an audience level, even when multiple audiences are mapped to a single destination dataflow.
In any of the export situations above, the exported files include the profiles that qualified for the export, alongside the columns you selected as XDM attributes for export.
When a streaming audience is mapped to a batch destination, there is a higher likelihood that the number of profiles in the exported file is closer to the number of users in the segment. This is because there is a higher chance that the latest audience evaluation has run closer to the export time.
Not all updates on a profile qualify a profile to be included in incremental file exports. For example, if an attribute was added to or removed from a profile, that does not include the profile in the export.
However, when the segmentMembership
attribute on a profile changes, the profile will be included in exported files. In other words, if the profile becomes part of the audience or is removed from the audience, it is included in incremental file exports.
Similarly, if a new identity (new email address, phone number, ECID, and so on) is added to a profile in the identity graph, that will trigger the profile to be included in a new incremental file export.
If a new audience is added to a destination mapping, that does not affect qualifications and exports for another segment. Export schedules are configured individually per audience and files are exported separately for every segment, even if the audiences have been added to the same destination dataflow.
For example, in the export setting illustrated below, where an audience is exporting incremental file updates, note the following circumstances where a profile is included in an incremental file export or not:
xdm: loyalty.points
, xdm: loyalty.tier
, xdm: personalEmail.address
is updated on a profile.segmentMembership.status
XDM field is mapped in the destination activation workflow, profiles exiting the audience are also included in exported incremental files, with an exited
status.Based on the information in the section above, the profile export behavior to file-based destinations can be summarized as described below:
Full file exports
The full active population of the audience is exported every day.
What determines a destination export | What is included in the exported file |
---|---|
|
In full file exports, the entire active profile population of a segment, based on the latest audience evaluation, is included with each file export. The latest values for each XDM attribute selected for export are also included as columns in each file. Note that profiles in exited status are not included in the file export. |
Incremental file exports
In the first file export after setting up the activation workflow, the entire population of the audience is exported. In subsequent exports, only the modified profiles are exported.
What determines a destination export | What is included in the exported file |
---|---|
|
The profiles for which the audience membership has changed, along with the latest information for each XDM attribute selected for export. Profiles with the exited status are included in destination exports, if the |
As a reminder, changes in identity maps for a profile qualify it to be included in an incremental file export. Changes in attribute values do not qualify it to be included in an incremental file export.
After reading this document, you now know what to expect to see in profile exports to streaming, enterprise, and file-based destinations.
Next, you can read about how identities are handled in the activation workflow.