Manage Experience Event Dataset Retention in the data lake using TTL

Efficient data management is critical for optimal performance, cost control, and data integrity. Use Experience Event Dataset Retention Time-To-Live (TTL) to enforce row-level expiration, automatically removing outdated records from datasets in the data lake while ensuring optimal storage efficiency and data relevance.

This guide explains how to evaluate, set, and manage TTL using the Catalog Service API. You’ll learn when and why to apply TTL, how to configure and update TTL values using API calls, and best practices to ensure effective implementation.

Why use TTL for row-level data management

As datasets grow, efficient data management becomes increasingly important to preserve performance, control costs, and keep data relevant. TTL-based row-level data expiration automates data cleanup by removing outdated records without manual intervention to help optimize storage and improve system efficiency.

TTL is useful when managing time-sensitive data that loses relevance over time. Consider implementing TTL if you need to:

• Reduce storage costs by automatically removing outdated records.
• Improve query performance by minimizing irrelevant data.
• Maintain data hygiene by keeping only relevant information.
• Optimize data retention to support business objectives.

Industry example

As an example, consider a video streaming service that tracks user interactions, such as video views, searches, and recommendations. While recent engagement data is crucial for personalization, older activity logs (for example, interactions from over a year ago) lose relevance. By using row-level expiration, Experience Platform automatically removes outdated logs, ensuring only current and meaningful data is used for analytics and recommendations.

Evaluate TTL suitability

Before applying a retention policy, assess whether your dataset is a good candidate for row-level expiration. Consider the following:

• Data relevance over time: Does older data provide value, or does it become obsolete?
• Impact on downstream processes: Will removing data affect reporting, analytics, or integrations?
• Storage cost versus retention value: Does the value of older data justify the cost of storing it?

If historical records are essential for long-term analysis or business operations, TTL may not be the right approach. Reviewing these factors ensures that TTL aligns with your data retention needs without negatively affecting data availability.

Plan your queries

Before applying TTL, it’s important to assess dataset size and data relevance, and to evaluate how much historical data should be retained. The following visual outlines the full process of implementing TTL, from planning queries to monitoring retention effectiveness.

Running targeted queries helps determine how much data would be retained or removed under different TTL configurations. For example, the following SQL query counts the number of records created within the last 30 days:

SELECT COUNT(1) FROM [datasetName] WHERE timestamp > date_sub(now(), INTERVAL 30 DAY);

Running similar queries for different time intervals helps validate TTL settings and ensure they balance storage efficiency and data accessibility.

Get started with TTL management

Before you can evaluate, set, and manage Experience Event Dataset Retention using the Catalog Service API, you must understand how to format your requests correctly. This includes knowing the API paths, providing required headers, and formatting request payloads. Refer to the Catalog Service API getting started guide for this essential information.

How to check current TTL settings

To begin your TTL management, first check current TTL settings. Make a GET request to the /ttl/{datasetId} endpoint to retrieve the default, maximum, and minimum TTL settings for a dataset. This step is necessary because TTL rules can vary based on the dataset type.

API format

GET /ttl/{DATASET_ID}

Request

The following request retrieves your organization’s TTL settings for a particular dataset.

curl -X GET \
  'https://platform.adobe.io/data/foundation/catalog/ttl/5ba9452f7de80408007fc52a' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}'
  -H 'x-sandbox-id: {SANDBOX_ID}'

Response

A successful response returns the TTL configuration for the dataset, including the default, maximum, and minimum TTL values for both adobe_lakeHouse and adobe_unifiedProfile storage.

{
    "67976f0b4878252ab887ccd9": {
        "name": "Acme Sales Data",
        "description": "This dataset contains sales transaction records for Acme Corporation.",
        "imsOrg": "{ORG_ID}",
        "sandboxId": "{SANDBOX_ID}",
        "tags": {
            "adobe/pqs/table": [
                "acme_sales_20250127_113331_106"
            ],
            "adobe/siphon/table/format": [
                "delta"
            ]
        },
        "extensions": {
            "adobe_lakeHouse": {
                "rowExpiration": {
                    "defaultValue": "P12M",
                    "maxValue": "P12M",
                    "minValue": "P30D"
                }
            },
            "adobe_unifiedProfile": {
                "rowExpiration": {
                    "defaultValue": "P12M",
                    "maxValue": "P12M",
                    "minValue": "P7D"
                }
            }
        },
        "version": "1.0.0",
        "created": 1737977611118,
        "updated": 1737977611118,
        "createdClient": "acme_data_pipeline",
        "createdUser": "john.snow@acmecorp.com",
        "updatedUser": "arya.stark@acmecorp.com",
        "classification": {
            "managedBy": "CUSTOMER"
        }
    }
}

How to set TTL for a dataset

To configure Experience Event Dataset Retention for your dataset, set a new TTL value by making a PATCH request to the /v2/datasets/{ID} endpoint.

API format

PATCH /v2/datasets/{DATASET_ID}

Request

In the example request below, the ttlValue is set to P3M. This ensures that records older than three months are automatically deleted. You can adjust the retention period to suit your business needs using values such as P6M for six months or P12M for one year.

curl -X PATCH \
  'https://platform.adobe.io/data/foundation/catalog/v2/datasets/{DATASET_ID}' \
  -h 'Authorization: Bearer {ACCESS_TOKEN}' \
  -h 'Content-Type: application/json' \
  -h 'x-api-key: {API_KEY}' \
  -h 'x-gw-ims-org-id: {ORG_ID}' \
  -d '{
    "extensions": {
        "adobe_lakeHouse": {
            "rowExpiration": {
                "ttlValue": "P3M"  // A 3 month retention period
            }
        }
    }
}

Response

A successful response shows the TTL configuration for the dataset. It includes details on row-level expiration settings for both adobe_lakeHouse and adobe_unifiedProfile storage.

{
    "67976f0b4878252ab887ccd9": {
        "name": "Acme Sales Data",
        "description": "This dataset contains sales transaction records for Acme Corporation.",
        "imsOrg": "{ORG_ID}",
        "sandboxId": "{SANDBOX_ID}",
        "tags": {
            "adobe/pqs/table": [
                "acme_sales_20250127_113331_106"
            ],
            "adobe/siphon/table/format": [
                "delta"
            ]
        },
        "extensions": {
            "adobe_lakeHouse": {
                "rowExpiration": {
                "ttlValue": "P3M",
                    "valueStatus": "custom",
                    "setBy": "user",
                    "updated": 1737977766499
                }
            },
            "adobe_unifiedProfile": {
                "rowExpiration": {
                    "ttlValue": "P3M",
                    "valueStatus": "custom",
                    "setBy": "user",
                    "updated": 1737977766499
                }
            }
        },
        "version": "1.0.0",
        "created": 1737977611118,
        "updated": 1737977611118,
        "createdClient": "acme_data_pipeline",
        "createdUser": "john.snow@acmecorp.com",
        "updatedUser": "arya.stark@acmecorp.com",
        "classification": {
            "managedBy": "CUSTOMER"
        }
    }
}

How to update TTL

Extend or shorten the retention period to suit your business needs by adjusting the TTL. For example, when considering the video streaming platform mentioned earlier, the platform may initially set the TTL to three months to ensure fresh engagement data for personalization. However, if their analysis shows that interaction patterns older than three months still provide valuable insights, they can extend the TTL period to six months to keep older records for better recommendation models.

To modify an existing TTL value, use the PATCH method on the /v2/datasets/{DATASET_ID} endpoint.

API format

PATCH /v2/datasets/{DATASET_ID}

Request

In the following request, the TTL is updated to six months (P6M) extending the record retention period before automatic deletion.

curl -X PATCH \
  'https://platform.adobe.io/data/foundation/catalog/v2/datasets/{DATASET_ID}' \
  -h 'Authorization: Bearer {ACCESS_TOKEN}' \
  -h 'Content-Type: application/json' \
  -h 'x-api-key: {API_KEY}' \
  -h 'x-gw-ims-org-id: {ORG_ID}' \
  -d '{
    "extensions": {
        "adobe_lakeHouse": {
            "rowExpiration": {
                "ttlValue": "P6M"  // Extend to 6 months
            }
        }
    }
}

Response

{  "extensions": {
        "adobe_lakeHouse": {
            "rowExpiration": {
              "ttlValue": "P6M",
              "valueStatus": "custom",
              "setBy": "user",
              "updated": "1737977766499"
            }
        },
        "adobe_unifiedProfile": {
            "rowExpiration": {
                "ttlValue": "P3M",
                "valueStatus": "custom",
                "setBy": "user",
                "updated": "17379754766355"
            }
        }
    }
}

Best practices for setting TTL

Choosing the right TTL value is crucial to ensuring that your Experience Event Dataset Retention policy balances data retention, storage efficiency, and analytical needs. A TTL that is too short may cause data loss, while one that is too long can increase storage costs and unnecessary data accumulation. Ensure that the TTL aligns with your dataset’s purpose by considering how often the data is accessed and how long it remains relevant.

The table below provides common TTL recommendations based on dataset type and usage patterns:

Review TTL settings periodically to ensure they continue to align with your storage policies, analytical needs, and business requirements.

Key considerations when setting TTL

Follow these best practices to ensure that TTL settings align with your data retention strategy:

• Audit TTL changes regularly. Every TTL update triggers an audit event. Use audit logs to track TTL modifications for compliance, data governance, and troubleshooting purposes.
• Remove TTL if data must be retained indefinitely. To disable TTL, set ttlValue to null. This prevents automatic expiration and retains all records permanently. Consider the storage implications before making this change.

Limitations of TTL

Be aware of the following limitations when using TTL:

• Experience Event Dataset Retention using TTL applies to row-level expiration, not dataset deletion. TTL removes records based on a defined retention period but does not delete entire datasets. To remove a dataset, use the dataset expiration endpoint or manual deletion.
• TTL cannot be removed, only updated. Once applied, TTL cannot be deleted, but you can modify the retention period to extend or shorten it. To retain data indefinitely, set a sufficiently long TTL instead of attempting to remove it.
• TTL is not a compliance tool. TTL optimizes storage and data lifecycle management but does not meet regulatory data retention requirements. For compliance, implement broader data governance strategies.

Dataset retention policy FAQs

This section provides answers for commonly asked questions about Dataset Retention policies in Adobe Experience Platform.

What types of datasets can I apply retention policy rules to?

How soon will the Dataset Retention job delete data from data lake services?

How soon will the Dataset Retention job delete data from Profile services?

Can I set different retention policies for data lake and Profile services?

How can I check my current dataset usage?

How can I verify if the data retention job was successful?

Can I recover deleted data?

What is the minimum TTL I can configure on a data lake Experience Event dataset?

What if I need to retain some data lake fields longer than my TTL policy allows?

Next steps

Now that you’ve learned how to manage TTL settings for row-level expiration, review the following documentation to further your understanding of TTL management:

• Retention jobs: Learn to schedule and automate dataset expirations in the Experience Platform UI with the data lifecycle UI guide, or check Dataset Retention configurations and verify that expired records are deleted.
• Dataset Expiration API endpoint guide: Discover how to delete entire datasets rather than just rows. Learn how to schedule, manage, and automate dataset expiration using the API to ensure efficient data retention.
• Data usage policies overview: Learn how to align your data retention strategy with broader compliance requirements and marketing use restrictions.