This tutorial covers the steps for updating a dataflow, including its basic information, schedule, and mapping sets using the Flow Service API.
Your source connection and your target connection should be mapped to a single dataflow. You should not update your source and target connections separately, as the changes will not be reflected in their corresponding dataflow. If your use case requires an update of your source and target connections, then you must create a new pair of source and target connections, as well as a new dataflow.
This tutorial requires you to have a valid flow ID. If you do not have a valid flow ID, select your connector of choice from the sources overview and follow the steps outlined before attempting this tutorial.
This tutorial also requires you to have a working understanding of the following components of Adobe Experience Platform:
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
The first step in updating your dataflow is to retrieve dataflow details using your flow ID. You can view the current details of an existing dataflow by making a GET request to the /flows
endpoint.
API format
GET /flows/{FLOW_ID}
Parameter | Description |
---|---|
{FLOW_ID} |
The unique id value for the dataflow you want to retrieve. |
Request
The following request retrieves updated information regarding your flow ID.
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns the current details of your dataflow including its version, schedule, and unique identifier (id
).
{
"items": [
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"createdAt": 1612310475905,
"updatedAt": 1614122324830,
"createdBy": "{CREATED_BY}",
"updatedBy": "{UPDATED_BY}",
"createdClient": "{CREATED_CLIENT}",
"updatedClient": "{UPDATED_CLIENT}",
"sandboxId": "{SANDBOX_ID}",
"sandboxName": "{SANDBOX_NAME}",
"imsOrgId": "{ORG_ID}",
"name": "Database dataflow using BigQuery",
"description": "collecting test1.Mytable from Google BigQuery",
"flowSpec": {
"id": "14518937-270c-4525-bdec-c2ba7cce3860",
"version": "1.0"
},
"state": "enabled",
"version": "\"5400d99c-0000-0200-0000-60358d540000\"",
"etag": "\"5400d99c-0000-0200-0000-60358d540000\"",
"sourceConnectionIds": [
"b7581b59-c603-4df1-a689-d23d7ac440f3"
],
"targetConnectionIds": [
"320f119a-5ac1-4ab1-88ea-eb19e674ea2e"
],
"inheritedAttributes": {
"sourceConnections": [
{
"id": "b7581b59-c603-4df1-a689-d23d7ac440f3",
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
},
"baseConnection": {
"id": "6990abad-977d-41b9-a85d-17ea8cf1c0e4",
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
}
}
}
],
"targetConnections": [
{
"id": "320f119a-5ac1-4ab1-88ea-eb19e674ea2e",
"connectionSpec": {
"id": "c604ff05-7f1a-43c0-8e18-33bf874cb11c",
"version": "1.0"
}
}
]
},
"scheduleParams": {
"startTime": "1612310466",
"frequency": "week",
"interval": "15",
"backfill": "true"
},
"transformations": [
{
"name": "Copy",
"params": {
"deltaColumn": {
"name": "Datefield",
"dateFormat": "YYYY-MM-DD",
"timezone": "UTC"
}
}
},
{
"name": "Mapping",
"params": {
"mappingId": "0b090130b58b4819afc78b6dc98b484d",
"mappingVersion": "0"
}
}
],
"runs": "/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c/runs",
"lastOperation": {
"started": 1614122316652,
"updated": 1614122324830,
"percentCompleted": 100.0,
"status": {
"value": "completed",
"errors": []
},
"ops": [
{
"op": "replace",
"path": "/scheduleParams/frequency",
"value": "week"
}
],
"operation": "update"
},
"lastRunDetails": {
"id": "a10cc80b-fbea-4c6b-873e-d7fd32f4d12d",
"state": "success",
"startedAtUTC": 1613079975512,
"completedAtUTC": 1613080027511
}
}
]
}
To update your dataflow’s run schedule, name, and description, perform a PATCH request to the Flow Service API while providing your flow ID, version, and the new schedule you want to use.
The If-Match
header is required when making a PATCH request. The value for this header is the unique version of the connection you want to update. The etag value updates with every successful update of a dataflow.
API format
PATCH /flows/{FLOW_ID}
Request
The following request updates your flow run schedule, as well as your dataflow’s name and description.
curl -X PATCH \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-H 'If-Match: "1a0037e4-0000-0200-0000-602e06f60000"' \
-d '[
{
"op": "replace",
"path": "/scheduleParams/frequency",
"value": "day"
},
{
"op": "replace",
"path": "/name",
"value": "Database Dataflow Feb2021"
},
{
"op": "replace",
"path": "/description",
"value": "Database dataflow for testing update API"
}
]'
Property | Description |
---|---|
op |
The operation call used to define the action needed to update the dataflow. Operations include: add , replace , and remove . |
path |
Defines the part of the flow that is to be updated. |
value |
The new value you want to update your parameter with. |
Response
A successful response returns your flow ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your flow ID.
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"etag": "\"50014cc8-0000-0200-0000-6036eb720000\""
}
You can update the mapping set of an existing dataflow by making a PATCH request to the Flow Service API and providing updated values for your mappingId
and mappingVersion
.
API format
PATCH /flows/{FLOW_ID}
Request
The following request updates your dataflow’s mapping set.
curl -X PATCH \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-H 'If-Match: "50014cc8-0000-0200-0000-6036eb720000"' \
-d '[
{
"op": "replace",
"path": "/transformations/0",
"value": {
"name": "Mapping",
"params": {
"mappingId": "c5f22f04e09f44498e528901546a83b1",
"mappingVersion": 2
}
}
}
]'
Property | Description |
---|---|
op |
The operation call used to define the action needed to update the dataflow. Operations include: add , replace , and remove . |
path |
Defines the part of the flow that is to be updated. In this example, transformations is being updated. |
value.name |
The name of the property that is to be updated. |
value.params.mappingId |
The new mapping ID to be used to update the mapping set of the dataflow. |
value.params.mappingVersion |
The new mapping version associated with the updated mapping ID. |
Response
A successful response returns your flow ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your flow ID.
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"etag": "\"2c000802-0000-0200-0000-613976440000\""
}
By following this tutorial, you have updated the basic information, schedule, and mapping sets of your dataflow using the Flow Service API. For more information on using source connectors, see the sources overview.