You can use the mode=draft
query parameter in the Flow Service API to set your Flow Service entities such as your base connections, source connections, target connections, and dataflows to a draft state.
Drafts can be updated later with new information and then published once they are ready, by using the op=publish
query parameter.
This tutorial provides steps on how to set your Flow Service entities to a draft state and allow you to pause and save your workflows for completion at a later time.
This tutorial requires you to have a working understanding of the following components of Adobe Experience Platform:
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
You must also check if the connection spec ID and corresponding flow spec ID of the source you are using is enabled for draft mode.
The following request retrieves the connection spec information for Azure File Storage:
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/connectionSpecs/be5ec48c-5b78-49d5-b8fa-7c89ec4569b8' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
A successful response returns the connection spec information for your source. To verify if draft mode is supported for your source, check that the items[0].attributes.isDraftModeSupported
has a value of true
.
{
"items": [
{
"id": "be5ec48c-5b78-49d5-b8fa-7c89ec4569b8",
"name": "azure-file-storage",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"authSpec": [
{
"name": "Basic Authentication",
"type": "basicAuthentication",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines auth params",
"properties": {
"host": {
"type": "string",
"description": "Specifies the Azure File Storage endpoint."
},
"userid": {
"type": "string",
"description": "Specify the user to access the Azure File Storage."
},
"password": {
"type": "string",
"description": "Specify the storage access key",
"format": "password"
}
},
"required": [
"host",
"userid",
"password"
]
}
}
],
"sourceSpec": {
"name": "CloudStorage",
"type": "CloudStorage",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "input path for copying files, can be a folder path, file path or a wildcard pattern"
},
"recursive": {
"type": "boolean",
"description": "indicates recursive copy in case of folder or wild card path, default is false"
}
},
"required": [
"path"
]
},
"attributes": {
"uiAttributes": {
"documentationLink": "http://www.adobe.com/go/sources-azure-file-storage-en",
"isSource": true,
"category": {
"key": "cloudStorage"
},
"icon": {
"key": "azureFileStorage"
},
"description": {
"key": "azureFileStorageDescription"
},
"label": {
"key": "azureFileStorageLabel"
}
}
}
},
"exploreSpec": {
"name": "FileSystem",
"type": "FileSystem",
"requestSpec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines explorable objects",
"properties": {
"objectType": {
"type": "string",
"enum": [
"file",
"folder",
"root"
]
}
},
"allOf": [
{
"if": {
"properties": {
"objectType": {
"enum": [
"file"
]
}
}
},
"then": {
"properties": {
"object": {
"type": "string",
"description": "defines file to get schema or preview of."
},
"fileType": {
"type": "string",
"enum": [
"delimited"
]
},
"preview": {
"type": "boolean"
}
},
"required": [
"object",
"fileType"
]
}
},
{
"if": {
"properties": {
"objectType": {
"enum": [
"folder"
]
}
}
},
"then": {
"properties": {
"object": {
"type": "string"
}
},
"required": [
"object"
]
}
}
]
},
"responseSpec": {
"root": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"description": "lists tables/items under the database/container requested.",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"description": "defines type of an item."
},
"name": {
"type": "string",
"description": "defines display name of an item."
},
"path": {
"type": "string",
"description": "defines path of an item."
},
"canPreview": {
"type": "boolean",
"default": false,
"description": "defines whether an item is previewable or not."
},
"canFetchSchema": {
"type": "boolean",
"default": false,
"description": "defines whether schema can be fetched for an item or not."
}
}
}
},
"folder": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string"
},
"name": {
"type": "string"
},
"path": {
"type": "string"
},
"canPreview": {
"type": "boolean",
"default": false
},
"canFetchSchema": {
"type": "boolean",
"default": false
}
}
}
},
"file": {
"delimited": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"format": {
"type": "string"
},
"schema": {
"type": "object",
"properties": {
"columns": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
}
}
}
}
},
"data": {
"type": "array",
"items": {
"type": "object"
}
}
}
}
}
}
},
"attributes": {
"category": "Cloud Storage",
"connectorName": "Azure File Storage",
"isSource": true,
"isDraftModeSupported": true,
"uiAttributes": {
"apiFeatures": {
"explorePaginationSupported": false
}
}
},
"permissionsInfo": {
"manage": [
{
"@type": "lowLevel",
"name": "EnterpriseSource",
"permissions": [
"write"
]
}
],
"view": [
{
"@type": "lowLevel",
"name": "EnterpriseSource",
"permissions": [
"read"
]
}
]
}
}
]
}
The following request retrieves the flow spec details for a cloud storage source:
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/flowSpecs?property=name==%22CloudStorageToAEP%22' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
A successful response returns the flow spec information for your source. To verify if draft mode is supported for your source, check that the items[0].attributes.isDraftModeSupported
has a value of true
.
{
"items": [
{
"id": "9753525b-82c7-4dce-8a9b-5ccfce2b9876",
"name": "CloudStorageToAEP",
"providerId": "0ed90a81-07f4-4586-8190-b40eccef1c5a",
"version": "1.0",
"sourceConnectionSpecIds": [
"b3ba5556-48be-44b7-8b85-ff2b69b46dc4",
"ecadc60c-7455-4d87-84dc-2a0e293d997b",
"b7829c2f-2eb0-4f49-a6ee-55e33008b629",
"4c10e202-c428-4796-9208-5f1f5732b1cf",
"fb2e94c9-c031-467d-8103-6bd6e0a432f2",
"32e8f412-cdf7-464c-9885-78184cb113fd",
"b7bf2577-4520-42c9-bae9-cad01560f7bc",
"998b8ae3-cec0-43b7-8abe-40b1eb4ee069",
"be5ec48c-5b78-49d5-b8fa-7c89ec4569b8",
"54e221aa-d342-4707-bcff-7a4bceef0001",
"c85f9425-fb21-426c-ad0b-405e9bd8a46c",
"26f526f2-58f4-4712-961d-e41bf1ccc0e8"
],
"targetConnectionSpecIds": [
"c604ff05-7f1a-43c0-8e18-33bf874cb11c"
],
"optionSpec": {
"name": "OptionSpec",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"errorDiagnosticsEnabled": {
"title": "Error diagnostics.",
"description": "Flag to enable detailed and sample error diagnostics summary.",
"type": "boolean",
"default": false
},
"partialIngestionPercent": {
"title": "Partial ingestion threshold.",
"description": "Percentage which defines the threshold of errors allowed before the run is marked as failed.",
"type": "number",
"exclusiveMinimum": 0
}
}
}
},
"transformationSpecs": [
{
"name": "Mapping",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines various params required for different mapping from source to target",
"properties": {
"mappingId": {
"type": "string"
},
"mappingVersion": {
"type": "string"
}
}
}
},
{
"name": "Encryption",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"description": "defines various params required for encrypted data ingestion",
"properties": {
"publicKeyId": {
"type": "string",
"description": "publicKeyId returned in encryptionKey creation API. One must use the publicKeyId corresponding to the same publicKey they used for encrypting the files"
}
}
}
}
],
"scheduleSpec": {
"name": "PeriodicSchedule",
"type": "Periodic",
"spec": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"startTime": {
"description": "epoch time",
"type": "integer"
},
"frequency": {
"type": "string",
"enum": [
"once",
"minute",
"hour",
"day",
"week"
]
},
"interval": {
"type": "integer"
},
"backfill": {
"type": "boolean",
"default": true
}
},
"required": [
"startTime",
"frequency"
],
"if": {
"properties": {
"frequency": {
"const": "once"
}
}
},
"then": {
"allOf": [
{
"not": {
"required": [
"interval"
]
}
},
{
"not": {
"required": [
"backfill"
]
}
}
]
},
"else": {
"required": [
"interval"
],
"if": {
"properties": {
"frequency": {
"const": "minute"
}
}
},
"then": {
"properties": {
"interval": {
"minimum": 15
}
}
},
"else": {
"properties": {
"interval": {
"minimum": 1
}
}
}
}
}
},
"attributes": {
"isSourceFlow": true,
"flacValidationSupported": true,
"isDraftModeSupported": true,
"frequency": "batch",
"notification": {
"category": "sources",
"flowRun": {
"enabled": true
}
}
},
"permissionsInfo": {
"manage": [
{
"@type": "lowLevel",
"name": "EnterpriseSource",
"permissions": [
"write"
]
}
],
"view": [
{
"@type": "lowLevel",
"name": "EnterpriseSource",
"permissions": [
"read"
]
}
]
}
}
]
}
To create a draft base connection, make a POST request to the /connections
endpoint of the Flow Service API and provide mode=draft
as a query parameter.
API format
POST /connections?mode=draft
Parameter | Description |
---|---|
mode |
A user-supplied query parameter that decides the state of the base connection. To set a base connection as a draft, set mode to draft . |
Request
The following request creates draft base connection for the Azure File Storage source:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections?mode=draft' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "ACME Azure File Storage Base Connection",
"description": "Azure File Storage base connection for ACME data (DRAFT)",
"auth": {
"specName": "Basic Authentication",
"params": {
"host": "{HOST}",
"userId": "{USER_ID}",
"password": "{PASSWORD}"
}
},
"connectionSpec": {
"id": "be5ec48c-5b78-49d5-b8fa-7c89ec4569b8",
"version": "1.0"
}
}'
Response
A successful response returns the base connection ID and corresponding etag for your draft base connection. You can use this ID later to update and publish your base connection.
{
"id": "f9377f50-607a-4818-b77f-50607a181860",
"etag": "\"2f0276fa-0000-0200-0000-5eab3abb0000\""
}
Once your draft is ready to be published, make a POST request to the /connections
endpoint and provide the ID of the draft base connection that you want to publish, as well as an action operation for publishing.
API format
POST /connections/{BASE_CONNECTION_ID}/action?op=publish
Parameter | Description |
---|---|
op |
An action operation that updates the state of the queried base connection. To publish a draft base connection, set op to publish . |
Request
The following request publishes the draft base connection for Azure File Storage that was created in an earlier step.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections/f9377f50-607a-4818-b77f-50607a181860/action?op=publish' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
Response
A successful response returns the ID and corresponding etag for your published base connection.
{
"id": "f9377f50-607a-4818-b77f-50607a181860",
"etag": "\"2f0276fa-0000-0200-0000-5eab3abb0000\""
}
To create a draft source connection, make a POST request to the /sourceConnections
endpoint of the Flow Service API and provide mode=draft
as a query parameter.
API format
POST /sourceConnections?mode=draft
Parameter | Description |
---|---|
mode |
A user-supplied query parameter that decides the state of the source connection. To set a source connection as a draft, set mode to draft . |
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections?mode=draft' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "ACME Azure File Storage Source Connection",
"description: "Azure File Storage source connection for ACME data (DRAFT)",
"baseConnectionId": "f9377f50-607a-4818-b77f-50607a181860",
"data": {
"format": "delimited",
},
"params": {
"path": "/acme/summerCampaign/account.csv",
"type": "file"
},
"connectionSpec": {
"id": "be5ec48c-5b78-49d5-b8fa-7c89ec4569b8",
"version": "1.0"
}
}'
Response
A successful response returns the source connection ID and corresponding etag for your draft source connection. You can use this ID later to update and publish your source connection.
{
"id": "26b53912-1005-49f0-b539-12100559f0e2",
"etag": "\"11004d97-0000-0200-0000-5f3c3b140000\""
}
You cannot publish a source connection if its associated base connection is still in draft state. Please ensure that your base connection is published first, before publishing your source connection.
Once your draft is ready to be published, make a POST request to the /sourceConnections
endpoint and provide the ID of the draft source connection that you want to publish, as well as an action operation for publishing.
API format
POST /sourceConnections/{SOURCE_CONNECTION_ID}/action?op=publish
Parameter | Description |
---|---|
op |
An action operation that updates the state of the queried source connection. To publish a draft source connection, set op to publish . |
Request
The following request publishes the draft source connection for Azure File Storage that was created in an earlier step.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections/26b53912-1005-49f0-b539-12100559f0e2/action?op=publish' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
Response
A successful response returns the ID and corresponding etag for your published source connection.
{
"id": "26b53912-1005-49f0-b539-12100559f0e2",
"etag": "\"11004d97-0000-0200-0000-5f3c3b140000\""
}
To create a draft target connection, make a POST request to the /targetConnections
endpoint of the Flow Service API and provide mode=draft
as a query parameter.
API format
POST /targetConnections?mode=draft
Parameter | Description |
---|---|
mode |
A user-supplied query parameter that decides the state of the target connection. To set a target connection as a draft, set mode to draft . |
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/targetConnections?mode=draft' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "ACME Azure File Storage Target Connection",
"description": "Azure File Storage target connection ACME data (DRAFT)",
"data": {
"schema": {
"id": "{SCHEMA_ID}",
"version": "application/vnd.adobe.xed-full+json;version=1"
}
},
"params": {
"dataSetId": "{DATASET_ID}"
},
"connectionSpec": {
"id": "c604ff05-7f1a-43c0-8e18-33bf874cb11c",
"version": "1.0"
}
}'
Response
A successful response returns the target connection ID and corresponding etag for your draft target connection. You can use this ID later to update and publish your target connection.
{
"id": "dbc5c132-bc2a-4625-85c1-32bc2a262558",
"etag": "\"8e000533-0000-0200-0000-5f3c40fd0000\""
}
You cannot publish a target connection if its associated base connection is still in draft state. Please ensure that your base connection is published first, before publishing your target connection.
Once your draft is ready to be published, make a POST request to the /targetConnections
endpoint and provide the ID of the draft target connection that you want to publish, as well as an action operation for publishing.
API format
POST /targetConnections/{TARGET_CONNECTION_ID}/action?op=publish
Parameter | Description |
---|---|
op |
An action operation that updates the state of the queried target connection. To publish a draft target connection, set op to publish . |
Request
The following request publishes the draft target connection for Azure File Storage that was created in an earlier step.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections/dbc5c132-bc2a-4625-85c1-32bc2a262558/action?op=publish' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
Response
A successful response returns the ID and corresponding etag for your published target connection.
{
"id": "dbc5c132-bc2a-4625-85c1-32bc2a262558",
"etag": "\"8e000533-0000-0200-0000-5f3c40fd0000\""
}
To set a dataflow as a draft, make a POST request to the /flows
endpoint while adding the mode=draft
as a query parameter. This allows you to create a dataflow and save it as a draft.
API format
POST /flows?mode=draft
Parameter | Description |
---|---|
mode |
A user-supplied query parameter that decides the state of the dataflow. To set a dataflow as a draft, set mode to draft . |
Request
The following request creates a draft dataflow.
'https://platform.adobe.io/data/foundation/flowservice/flows?mode=draft' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-d '{
"name": "ACME Azure File Storage Dataflow",
"description": "Azure File Storage dataflow for ACME data (DRAFT)",
"sourceConnectionIds": [
"26b53912-1005-49f0-b539-12100559f0e2"
],
"targetConnectionIds": [
"dbc5c132-bc2a-4625-85c1-32bc2a262558"
],
"flowSpec": {
"id": "9753525b-82c7-4dce-8a9b-5ccfce2b9876",
"version": "1.0"
}
}'
Response
A successful response returns the flow ID and corresponding etag for your draft dataflow. You can use this ID later to update and publish your dataflow.
{
"id": "c9751426-dff8-49b0-965f-50defcf4187b",
"etag": "\"69057131-0000-0200-0000-640f48320000\""
}
You cannot publish a dataflow if its associated source and target connections are still in draft state. Please ensure that your source and target connections are published first, before publishing your dataflow.
Once your draft is ready to be published, make a POST request to the /flows
endpoint while providing the ID of the draft dataflow that you want to publish, as well as an action operation for publishing.
API format
POST /flows/{FLOW_ID}/action?op=publish
Parameter | Description |
---|---|
op |
An action operation that updates the state of the queried dataflow. To publish a draft dataflow, set op to publish . |
Request
The following request publishes your draft dataflow.
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/flows/c9751426-dff8-49b0-965f-50defcf4187b/action?op=publish' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
Response
A successful response returns the ID and corresponding etag
of your dataflow.
{
"id": "c9751426-dff8-49b0-965f-50defcf4187b",
"etag": "\"69057131-0000-0200-0000-640f48320000\""
}
By following this tutorial, you have learned how to create drafts of your Flow Service entities as well as publish these drafts. For more information on sources, please read the sources overview.