Flow runs represent an instance of flow execution. For example, if a flow is scheduled to run hourly at 9:00 AM, 10:00 AM, and 11:00 AM, then you would have three instances of a flow run. Flow runs are specific to your particular organization.
On-demand ingestion provides you with the capability to create a flow run against a given dataflow. This allows your users to create a flow run, based on given parameters and create an ingestion cycle, without service tokens. Support for on-demand ingestion is available only for batch sources.
This tutorial covers the steps on how to use on-demand ingestion and create a flow run using the Flow Service API.
In order to create a flow run, you must first have the flow ID of a dataflow that is scheduled for one-time ingestion.
This tutorial requires you to have a working understanding of the following components of Adobe Experience Platform:
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
To create a flow for a table-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against, as well as values for start time, end time, and delta column.
Table-based sources include the following source categories: advertising, analytics, consent and preferences, CRMs, customer success, database, marketing automation, payments, and protocols.
API format
POST /runs/
Request
The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de
.
You only need to provide the deltaColumn
when creating your first flow run. After that, deltaColumn
will be patched as part of copy
transformation in the flow and will be treated as the source of truth. Any attempts to change the deltaColumn
value through the flow run parameters will result in an error.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/runs' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json'
-d '{
"flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
"params": {
"startTime": "1663735590",
"windowStartTime": "1651584991",
"windowEndTime": "16515859567",
"deltaColumn": {
"name": "DOB"
}
}
}'
Parameter | Description |
---|---|
flowId |
The ID of the flow in which your flow run will be created against. |
params.startTime |
The scheduled time of when the on-demand flow run will begin. This value is represented in unix time. |
params.windowStartTime |
The earliest date and time that data will be retrieved from. This value is represented in unix time. |
params.windowEndTime |
The date and time that data will be retrieved up to. This value is represented in unix time. |
params.deltaColumn |
The delta column is required to partition the data and separate newly ingested data from historic data. Note: The deltaColumn is only needed when creating your firs flow run. |
params.deltaColumn.name |
The name of the delta column. |
Response
A successful response returns the details of the newly created flow run, including its unique run id
.
{
"items": [
{
"id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
"etag": "\"1100c53e-0000-0200-0000-627138980000\""
}
]
}
Property | Description |
---|---|
id |
The ID of the newly created flow run. See the guide on retrieving flow specifications for more information on table-based run specifications. |
etag |
The resource version of the flow run. |
To create a flow for a file-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against and values for start time and end time.
File-based sources include all cloud storage sources.
API format
POST /runs/
Request
The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de
.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/runs' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json'
-d '{
"flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
"params": {
"startTime": "1663735590",
"windowStartTime": "1651584991",
"windowEndTime": "16515859567"
}
}'
Parameter | Description |
---|---|
flowId |
The ID of the flow in which your flow run will be created against. |
params.startTime |
The scheduled time of when the on-demand flow run will begin. This value is represented in unix time. |
params.windowStartTime |
The earliest date and time that data will be retrieved from. This value is represented in unix time. |
params.windowEndTime |
The date and time that data will be retrieved up to. This value is represented in unix time. |
Response
A successful response returns the details of the newly created flow run, including its unique run id
.
{
"items": [
{
"id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
"etag": "\"1100c53e-0000-0200-0000-627138980000\""
}
]
}
Property | Description |
---|---|
id |
The ID of the newly created flow run. See the guide on retrieving flow specifications for more information on table-based run specifications. |
etag |
The resource version of the flow run. |
Once your flow run has been created, you can monitor the data that is being ingested through it to see information on flow runs, completion status, and errors. To monitor your flow runs using the API, see the tutorial on monitoring dataflows in the API . To monitor your flow runs using Platform UI, see the guide on monitoring sources dataflows using the monitoring dashboard.