Dataset expiration endpoint

Last update: 2024-07-19
  • Created for:
  • Developer

The /ttl endpoint in the Data Hygiene API allows you to schedule expiration dates for datasets in Adobe Experience Platform.

A dataset expiration is only a timed-delayed delete operation. The dataset is not protected in the interim, so it may be be deleted by other means before its expiry is reached.

NOTE

Although the expiry is specified as a specific instant in time, there may be up to 24 hours of delay after the expiry before the actual deletion is initiated. Once the delete is initiated, it can take up to seven days before all traces of the dataset have been removed from Platform systems.

At any time before the dataset-delete is actually initiated, you can cancel the expiration or modify its trigger time. After cancelling a dataset expiration, you can reopen it by setting a new expiry.

Once the dataset deletion is initiated, its expiration job will be marked as executing, and it may not be further altered. The dataset itself may be recoverable for up to seven days, but only through a manual process initiated through an Adobe service request. As the request executes, the data lake, Identity Service, and Real-Time Customer Profile begin separate processes to remove the dataset’s contents from their respective services. Once the data is deleted from all three services, the expiration is marked as completed.

WARNING

If a dataset is set to expire, you must manually change any dataflows that may be ingesting data into that dataset so that your downstream workflows are not negatively affected.

Advanced Data Lifecycle Management supports dataset deletions through the dataset expiration endpoint and ID deletions (row-level data) using primary identities via the workorder endpoint. You can also manage dataset expirations and record deletions through the Platform UI. See the linked documentation for more information.

NOTE

Data Lifecycle does not support batch deletion.

Getting started

The endpoint used in this guide is part of the Data Hygiene API. Before continuing, please review the API guide for information on required headers for CRUD operations, error messages, Postman collections, and how to read sample API calls.

IMPORTANT

When making calls to the data Hygiene API, you must use the -H x-sandbox-name: {SANDBOX_NAME} header.

List dataset expirations

You can list all dataset expirations for your organization by making a GET request. Query parameters can be used to filter the response for appropriate results.

API format

GET /ttl?{QUERY_PARAMETERS}
Parameter Description
{QUERY_PARAMETERS} A list of optional query parameters, with multiple parameters separated by & characters. Common parameters include limit and page for pagination purposes. For a full list of supported query parameters, refer to the appendix section.

Request

curl -X GET \
  https://platform.adobe.io/data/core/hygiene/ttl?updatedToDate=2021-08-01&author=LIKE%20%25Jane%20Doe%25 \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response lists the resulting dataset expirations. The following example has been truncated for space.

IMPORTANT

The ttlId in the response is also referred to as the {DATASET_EXPIRATION_ID}. They both refer to the unique identifier for the dataset expiration.

{
  "results": [
    {
      "ttlId": "SD-b16c8b48-a15a-45c8-9215-587ea89369bf",
      "datasetId": "629bd9125b31471b2da7645c",
      "datasetName": "Sample Acme dataset",
      "sandboxName": "hygiene-beta",
      "imsOrg": "A2A5*EF06164773A8A49418C@AdobeOrg",
      "status": "pending",
      "expiry": "2050-01-01T00:00:00Z",
      "updatedAt": "2023-06-09T16:52:44.136028Z",
      "updatedBy": "Jane Doe <jdoe@adobe.com> 77A51F696282E48C0A494 012@64d18d6361fae88d49412d.e"
    }
  ],
  "current_page": 0,
  "total_pages": 1,
  "total_count": 1
}
Property Description
total_count The count of dataset expirations that matched the listing call’s parameters.
results Contains the details of the returned dataset expirations. For more details on the properties of a dataset expiration, see the response section for making a lookup call.

Look up a dataset expiration

To lookup a dataset expiration, make a GET request with either the {DATASET_ID} or the {DATASET_EXPIRATION_ID}.

IMPORTANT

The {DATASET_EXPIRATION_ID} is referred to as the ttlId in the response. They both refer to the unique identifier for the dataset expiration.

API format

GET /ttl/{DATASET_ID}?include=history
GET /ttl/{DATASET_EXPIRATION_ID}
Parameter Description
{DATASET_ID} The ID of the dataset whose expiration you want to look up.
{DATASET_EXPIRATION_ID} The ID of the dataset expiration.

Request

The following request looks up the expiration details for dataset 62759f2ede9e601b63a2ee14:

curl -X GET \
  https://platform.adobe.io/data/core/hygiene/ttl/62759f2ede9e601b63a2ee14 \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response returns the details of the dataset expiration.

{
    "ttlId": "SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f",
    "datasetId": "62759f2ede9e601b63a2ee14",
    "datasetName": "XtVRwq9-38734",
    "sandboxName": "prod",
    "imsOrg": "A2A5*EF06164773A8A49418C@AdobeOrg",
    "status": "pending",
    "expiry": "2024-12-31T23:59:59Z",
    "updatedAt": "2024-05-11T15:12:40.393115Z",
    "updatedBy": "Jane Doe <jdoe@adobe.com> 77A51F696282E48C0A494 012@64d18d6361fae88d49412d.e",
    "displayName": "Delete Acme Data before 2025",
    "description": "The Acme information in this dataset is licensed for our use through the end of 2024."
}
Property Description
ttlId The ID of the dataset expiration.
datasetId The ID of the dataset that this expiration applies to.
datasetName The display name for the dataset this expiration applies to.
sandboxName The name of the sandbox that the target dataset is located under.
imsOrg Your organization’s ID.
status The current status of the dataset expiration.
expiry The scheduled date and time when the dataset will be deleted.
updatedAt A timestamp of when the expiration was last updated.
updatedBy The user who last updated the expiration.
displayName The display name for the expiration request.
description A description for the expiration request.

Catalog expiry tags

When using the Catalog API to look up dataset details, if the dataset has an active expiration it will be listed under tags.adobe/hygiene/ttl.

The following JSON represents a truncated response for a dataset’s details from Catalog, which an expiration value of 32503680000000. The tag’s value encodes the expiry as an integer number of milliseconds since the beginning of the Unix epoch.

{
  "63212313c308d51b997858ba": {
    "name": "Test Dataset",
    "description": "A piecrust promise, made to be broken",
    "imsOrg": "0FCC747E56F59C747F000101@AdobeOrg",
    "sandboxId": "8dc51b90-d0f9-11e9-b164-ed6a398c8b35",
    "tags": {
      "adobe/hygiene/ttl": [ "32503680000000" ],
      ...
    },
    ...
  }
}

Create a dataset expiration

To ensure that data is removed from the system after a specified period, schedule an expiration for a specific dataset by providing the dataset ID and the expiry date and time in ISO 8601 format.

To create a dataset expiration, perform a POST request as shown below and provide the values mentioned below within the payload.

NOTE

If you receive a 404 error, ensure that the request has no additional forward slashes. A trailing slash can cause a POST request to fail.

API format

POST /ttl

Request

curl -X POST \
  https://platform.adobe.io/data/core/hygiene/ttl \
  -H `Authorization: Bearer {ACCESS_TOKEN}`
  -H `x-gw-ims-org-id: {ORG_ID}`
  -H `x-api-key: {API_KEY}`
  -H `Accept: application/json`
  -d {
      "datasetId": "5b020a27e7040801dedbf46e",
      "expiry": "2030-12-31T23:59:59Z"
      "displayName": "Delete Acme Data before 2025",
      "description": "The Acme information in this dataset is licensed for our use through the end of 2024."
      }
Property Description
datasetId Required The ID of the target dataset that you want to schedule an expiration for.
expiry Required A date and time in ISO 8601 format. If the string has no explicit time zone offset, the time zone is assumed to be UTC. The lifespan of data within the system is set according to the provided expiry value.
Note:
  • The request will fail if a dataset expiration already exists for the dataset.
  • This date and time must be at least 24 hours in the future.
displayName An optional display name for the dataset expiration request.
description An optional description for the expiration request.

Response

A successful response returns an HTTP 201 (Created) status and the new state of the dataset expiration.

{
  "ttlId":       "SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f",
  "datasetId":   "5b020a27e7040801dedbf46e",
  "datasetName": "Acme licensed data",
  "sandboxName": "prod",
  "imsOrg":      "{ORG_ID}",
  "status":      "pending",
  "expiry":      "2030-12-31T23:59:59Z",
  "updatedAt":   "2021-08-19T11:14:16Z",
  "updatedBy":   "Jane Doe <jdoe@adobe.com> 77A51F696282E48C0A494 012@64d18d6361fae88d49412d.e",
  "displayName": "Delete Acme Data before 2031",
  "description": "The Acme information in this dataset is licensed for our use through the end of 2030."
}
Property Description
ttlId The ID of the dataset expiration.
datasetId The ID of the dataset that this expiration applies to.
datasetName The display name for the dataset this expiration applies to.
sandboxName The name of the sandbox that the target dataset is located under.
imsOrg Your organization’s ID.
status The current status of the dataset expiration.
expiry The scheduled date and time when the dataset will be deleted.
updatedAt A timestamp of when the expiration was last updated.
updatedBy The user who last updated the expiration.
displayName A display name for the expiration request.
description An description for the expiration request.

A 400 (Bad Request) HTTP status occurs if a dataset expiration already exists for the dataset. An unsuccessful response returns a 404 (Not Found) HTTP status if no such dataset expiration exists (or you do not have access to the dataset).

Update a dataset expiration

To update an expiration date for a dataset, use a PUT request and the ttlId. You can update the displayName, description, and/or expiry information.

NOTE

If you change the expiration date and time, it must be at least 24 hours in the future. This enforced delay provides an opportunity for you to cancel or re-schedule the expiration and avoid any accidental loss of data.

API format

PUT /ttl/{DATASET_EXPIRATION_ID}
Parameter Description
{DATASET_EXPIRATION_ID} The ID of the dataset expiration that you want to change. Note: This is referred to as the ttlId in the response.

Request

The following request reschedules a dataset expiration SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f to occur at the end of 2024 (Greenwich Mean Time). If the existing dataset expiration is found, that expiration is updated with the new expiry value.

curl -X PUT \
  https://platform.adobe.io/data/core/hygiene/ttl/SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json' \
  -d '{
        "expiry": "2024-12-31T23:59:59Z",
        "displayName": "Delete Acme Data before 2025",
        "description": "The Acme information in this dataset is licensed for our use through the end of 2024."
      }'
Property Description
expiry Required A date and time in ISO 8601 format. If the string has no explicit time zone offset, the time zone is assumed to be UTC. The lifespan of data within the system is set according to the provided expiry value. Any previous expiration timestamp for the same dataset is to be replaced by the new expiration value you have provided. This date and time must be at least 24 hours in the future.
displayName A display name for the expiration request.
description An optional description for the expiration request.

Response

A successful response returns the new state of the dataset expiration and an HTTP status 200 (OK) if a pre-existing expiration was updated.

{
    "ttlId": "SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f",
    "datasetId": "5b020a27e7040801dedbf46e",
    "imsOrg": "A2A5*EF06164773A8A49418C@AdobeOrg",
    "status": "pending",
    "expiry": "2024-12-31T23:59:59Z",
    "updatedAt": "2022-05-09T22:38:40.393115Z",
    "updatedBy": "Jane Doe <jdoe@adobe.com> 77A51F696282E48C0A494 012@64d18d6361fae88d49412d.e",
    "displayName": "Delete Acme Data before 2025",
    "description": "The Acme information in this dataset is licensed for our use through the end of 2024."
}
Property Description
ttlId The ID of the dataset expiration.
datasetId The ID of the dataset that this expiration applies to.
imsOrg Your organization’s ID.
status The current status of the dataset expiration.
expiry The scheduled date and time when the dataset will be deleted.
updatedAt A timestamp of when the expiration was last updated.
updatedBy The user who last updated the expiration.

An unsuccessful response returns a 404 (Not Found) HTTP status if no such dataset expiration exists.

Cancel a dataset expiration

You can cancel a dataset expiration by making a DELETE request.

NOTE

Only dataset expirations that have a status of pending can be canceled. Attempting to cancel an expiration that has executed or is already canceled returns an HTTP 404 error.

API format

DELETE /ttl/{EXPIRATION_ID}
Parameter Description
{EXPIRATION_ID} The ttlId of the dataset expiration that you want to cancel.

Request

The following request cancels a dataset expiration with ID SD-b16c8b48-a15a-45c8-9215-587ea89369bf:

curl -X DELETE \
  https://platform.adobe.io/data/core/hygiene/ttl/SD-b16c8b48-a15a-45c8-9215-587ea89369bf \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'

Response

A successful response returns HTTP status 204 (No Content), and the expiration’s status attribute is set to cancelled.

Appendix

Accepted query parameters

The following table outlines the available query parameters when listing dataset expirations:

NOTE

The description, displayName, and datasetName parameters all contain the ability to searched for by LIKE values. This means that you can find scheduled dataset expirations named: “Name123”, “Name183”, “DisplayName1234” by searching for the string “Name1”.

Parameter Description Example
author Use the author query parameter to find the person who most recently updated the dataset expiration. If no updates have been made since its creation, this will match the original creator of the expiration. This parameter matches expirations where the created_by field corresponds to the search string.
If the search string begins with LIKE or NOT LIKE, the remainder is treated as an SQL search pattern. Otherwise, the entire search string is treated as a literal string that must exactly match the entire content of a created_by field.		 author=LIKE %john%, author=John Q. Public
datasetId Matches expirations that apply to specific dataset. datasetId=62b3925ff20f8e1b990a7434
datasetName Matches expirations whose dataset name contains the provided search string. The match is case-insensitive. datasetName=Acme
description description=Handle expiration of Acme information through the end of 2024.
displayName Matches expirations whose display name contains the provided search string. The match is case-insensitive. displayName=License Expiry
executedDate / executedFromDate / executedToDate Filters results based on an exact execution date, an ending date for execution, or a starting date for execution. They are used to retrieve data or records associated with the execution of an operation on a specific date, before a particular date, or after a particular date. executedDate=2023-02-05T19:34:40.383615Z
expiryDate Matches expirations that occurred in the 24-hour window of the specified date. 2024-01-01
expiryToDate / expiryFromDate Matches expirations that are due to be executed, or have already been executed, during the specified interval. expiryFromDate=2099-01-01&expiryToDate=2100-01-01
limit An integer between 1 and 100 that indicates the maximum number of expirations to return. Defaults to 25. limit=50
orderBy The orderBy query parameter specifies the sorting order of the results returned by the API. Use it to arrange the data based on one or more fields, either in ascending (ASC) or descending (DESC) order. Use the + or - prefix to signify ASC, DESC respectively. The following values are accepted: displayName, description, datasetName, id, updatedBy, updatedAt, expiry, status. -datasetName
orgId Matches datasets expirations whose organization ID matches that of the parameter. This value defaults to that of the x-gw-ims-org-id headers, and is ignored unless the request supplies a service token. orgId=885737B25DC460C50A49411B@AdobeOrg
page An integer that indicates which page of expirations to return. page=3
sandboxName Matches dataset expirations whose sandbox name exactly matches the argument. Defaults to the sandbox name in the request’s x-sandbox-name header. Use sandboxName=* to include dataset expirations from all sandboxes. sandboxName=dev1
search Matches expirations where the specified string is an exact match for the expiration ID, or is contained in any of these fields:
  • author
  • display name
  • description
  • display name
  • dataset name
 search=TESTING
status A comma-separated list of statuses. When included, the response matches dataset expirations whose current status is among those listed. status=pending,cancelled
ttlId Matches the expiration request with the given ID. ttlID=SD-c8c75921-2416-4be7-9cfd-9ab01de66c5f
updatedDate Matches expirations that were updated in the 24-hour window of the specified date. 2024-01-01
updatedToDate / updatedFromDate Matches expirations that were updated in the 24-hour window starting at the stated time.

An expiration is considered updated on every edit, including when it is created, cancelled, or executed.		 updatedDate=2022-01-01
Previous page
Next page

On this page