Create a flow run for on-demand ingestion using the Flow Service API

Last update: 2024-07-31
  • Topics:
  • Sources
    View more on this topic
  • Created for:
  • Developer

Flow runs represent an instance of flow execution. For example, if a flow is scheduled to run hourly at 9:00 AM, 10:00 AM, and 11:00 AM, then you would have three instances of a flow run. Flow runs are specific to your particular organization.

On-demand ingestion provides you with the capability to create a flow run against a given dataflow. This allows your users to create a flow run, based on given parameters and create an ingestion cycle, without service tokens. Support for on-demand ingestion is available only for batch sources.

This tutorial covers the steps on how to use on-demand ingestion and create a flow run using the Flow Service API.

Getting started

NOTE

In order to create a flow run, you must first have the flow ID of a dataflow that is scheduled for one-time ingestion.

This tutorial requires you to have a working understanding of the following components of Adobe Experience Platform:

  • Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
  • Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.

Using Platform APIs

For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.

Create a flow run for a table-based source

To create a flow for a table-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against, as well as values for start time, end time, and delta column.

TIP

Table-based sources include the following source categories: advertising, analytics, consent and preferences, CRMs, customer success, database, marketing automation, payments, and protocols.

API format

POST /runs/

Request

The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de.

NOTE

You only need to provide the deltaColumn when creating your first flow run. After that, deltaColumn will be patched as part of copy transformation in the flow and will be treated as the source of truth. Any attempts to change the deltaColumn value through the flow run parameters will result in an error.

curl -X POST \
  'https://platform.adobe.io/data/foundation/flowservice/runs' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json'
  -d '{
      "flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
      "params": {
          "startTime": "1663735590",
          "windowStartTime": "1651584991",
          "windowEndTime": "16515859567",
          "deltaColumn": {
              "name": "DOB"
          }
      }
  }'
Parameter Description
flowId The ID of the flow in which your flow run will be created against.
params.startTime The scheduled time of when the on-demand flow run will begin. This value is represented in unix time.
params.windowStartTime The earliest date and time that data will be retrieved from. This value is represented in unix time.
params.windowEndTime The date and time that data will be retrieved up to. This value is represented in unix time.
params.deltaColumn The delta column is required to partition the data and separate newly ingested data from historic data. Note: The deltaColumn is only needed when creating your firs flow run.
params.deltaColumn.name The name of the delta column.

Response

A successful response returns the details of the newly created flow run, including its unique run id.

{
    "items": [
        {
            "id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
            "etag": "\"1100c53e-0000-0200-0000-627138980000\""
        }
    ]
}
Property Description
id The ID of the newly created flow run. See the guide on retrieving flow specifications for more information on table-based run specifications.
etag The resource version of the flow run.

Create a flow run for a file-based source

To create a flow for a file-based source, make a POST request to the Flow Service API while providing the ID of the flow you want to create the run against and values for start time and end time.

TIP

File-based sources include all cloud storage sources.

API format

POST /runs/

Request

The following request creates a flow run for flow ID 3abea21c-7e36-4be1-bec1-d3bad0e3e0de.

curl -X POST \
  'https://platform.adobe.io/data/foundation/flowservice/runs' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json'
  -d '{
      "flowId": "3abea21c-7e36-4be1-bec1-d3bad0e3e0de",
      "params": {
          "startTime": "1663735590",
          "windowStartTime": "1651584991",
          "windowEndTime": "16515859567"
      }
  }'
Parameter Description
flowId The ID of the flow in which your flow run will be created against.
params.startTime The scheduled time of when the on-demand flow run will begin. This value is represented in unix time.
params.windowStartTime The earliest date and time that data will be retrieved from. This value is represented in unix time.
params.windowEndTime The date and time that data will be retrieved up to. This value is represented in unix time.

Response

A successful response returns the details of the newly created flow run, including its unique run id.

{
    "items": [
        {
            "id": "3fb0418e-1804-45d6-8d56-dd51f05c0baf",
            "etag": "\"1100c53e-0000-0200-0000-627138980000\""
        }
    ]
}
Property Description
id The ID of the newly created flow run. See the guide on retrieving flow specifications for more information on table-based run specifications.
etag The resource version of the flow run.

Monitor your flow runs

Once your flow run has been created, you can monitor the data that is being ingested through it to see information on flow runs, completion status, and errors. To monitor your flow runs using the API, see the tutorial on monitoring dataflows in the API . To monitor your flow runs using Platform UI, see the guide on monitoring sources dataflows using the monitoring dashboard.

On this page