The Catalog Service API allows response data to be filtered through the use of request query parameters. Part of best practices for Catalog is to use filters in all API calls, as they reduce the load on the API and help to improve overall performance.
This document outlines the most common methods for filtering Catalog objects in the API. It is recommended that you reference this document while reading the Catalog developer guide to learn more about how to interact with the Catalog API. For more general information on Catalog Service, see the Catalog overview.
The limit
query parameter constrains the number of objects returned in a response. Catalog responses are automatically metered according to configured limits:
limit
parameter is not specified, the maximum number of objects per response payload is 20.observableSchema
is requested using the properties
query parameter, the maximum number of datasets returned is 20.limit
parameters (including limit=0
) result in 400-level error responses that outline proper ranges.API format
GET /{OBJECT_TYPE}?limit={LIMIT}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{LIMIT} |
An integer indicating the number of objects to return, ranging from 1 to 100. |
Request
The following request retrieves a list of datasets while limiting the response to three objects.
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?limit=3 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of datasets, limited to the number indicated by the limit
query parameter.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset 1",
"description": "Description of dataset.",
"files": "@/dataSetFiles?dataSetId=5ba9452f7de80400007fc52a"
},
"5bb276b03a14440000971552": {
"name": "Sample Dataset 2",
"description": "Description of dataset.",
"files": "@/dataSetFiles?dataSetId=5bb276b03a14440000971552"
},
"5bceaa4c26c115000039b24b": {
"name": "Sample Dataset 3"
}
}
Even when filtering the number of objects returned using the limit
parameter, the returned objects themselves can often contain more information than you actually need. To further reduce the load on the system, it is best practice to filter responses to include only the properties that you need.
The properties
parameter filters response objects to return only a set of specified properties. The parameter can be set to return one or multiple properties.
The properties
parameter can accept any level object properties. sampleKey
can be extracted using ?properties=subItem.sampleKey
.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset",
"description": "Sample dataset containing important data",
"subitem": {
"sampleKey": "sampleValue"
}
}
}
API format
GET /{OBJECT_TYPE}?properties={PROPERTY}
GET /{OBJECT_TYPE}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}
GET /{OBJECT_TYPE}/{OBJECT_ID}?properties={PROPERTY_1},{PROPERTY_2},{PROPERTY_3}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{PROPERTY} |
The name of an attribute to include in the response body. |
{OBJECT_ID} |
The unique identifier of a specific Catalog object being retrieved. |
Request
The following request retrieves a list of datasets. The comma-separated list of property names provided under the properties
parameter indicates the properties to be returned in the response. A limit
parameter is also included, which limits the number of datasets returned. If the request did not include a limit
parameter, the response would contain a maximum of 20 objects.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?limit=4&properties=name,schemaRef' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of Catalog objects with only the requested properties displayed.
{
"Dataset1": {
"name": "Dataset 1",
"schemaRef": {
"id": "https://ns.adobe.com/{TENANT_ID}/schemas/bc82c518380478b59a95c63e0f843121",
"contentType": "application/vnd.adobe.xed+json;version=1"
}
},
"Dataset2": {},
"Dataset3": {
"name": {},
},
"Dataset4": {
"name": "Dataset 4",
"schemaRef": {
"id": "https://ns.adobe.com/{TENANT_ID}/schemas/142afb78d8b368a5ba97a6cc8fc7e033",
"contentType": "application/vnd.adobe.xed+json;version=1"
}
}
}
Based on the response above, the following can be inferred:
Dataset1
)Dataset2
)Dataset3
)Dataset4
)In the schemaRef
property for each dataset, the version number indicates the latest minor version of the schema. See the section on schema versioning in the XDM API guide for more information.
The start
query parameter offsets the response list forward by a specified number, using zero-based numbering. For example, start=2
would offset the response to start on the third listed object.
If the start
parameter is not paired with a limit
parameter, the maximum number of objects returned is 20.
API format
GET /{OBJECT_TYPE}?start={OFFSET}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{OFFSET} |
An integer indicating the number of objects to offset the response by. |
Request
The following request retrieves a list of datasets, offsetting to the fifth object (start=4
) and limiting the response to two returned datasets (limit=2
).
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?start=4&limit=2 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
The response includes a JSON object containing two top-level items (limit=2
), one for each dataset and their details (details have been condensed in the example). The response is shifted by four (start=4
), meaning the datasets shown are number five and six chronologically.
{
"Dataset5": {},
"Dataset6": {}
}
Some Catalog objects support the use of a tags
attribute. Tags can attach information to an object and then be used later to retrieve that object. The choice of what tags to use and how to apply them depends on your organizational processes.
There are a few limitations to consider when using tags:
unifiedProfile
: This tag name is reserved for datasets to be ingested by Real-Time Customer Profile.unifiedIdentity
: This tag name is reserved for datasets to be ingested by Identity Service.Below is an example of a dataset that contains a tags
property. The tags within that property take the form of key-value pairs, with each tag value appearing as an array containing a single string:
{
"5be1f2ecc73c1714ceba66e2": {
"imsOrg": "{ORG_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"sample_tag_value"
]
},
"name": "Sample Dataset",
"description": "Same dataset containing sample data.",
"createdUser": "{CREATED_USER}",
"createdClient": "{CREATED_CLIENT}",
"updatedUser": "{UPDATED_USER}",
"version": "1.0.0",
"created": 1541534444286,
"updated": 1541534444286
}
}
API format
Values for the tags
parameter take the form of key-value pairs, using the format {TAG_NAME}:{TAG_VALUE}
. Multiple key-value pairs can be provided in the form of a comma-separated list. When multiple tags are provided, an AND relationship is assumed.
The parameter supports wildcard characters (*
) for tag values. For example, a search string of test*
returns any object where the tag value begins with “test”. A search string consisting solely of a wildcard can be used to filter objects based on whether or not they contain a specific tag, regardless of its value.
GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}
GET /{OBJECT_TYPE}?tags={TAG_NAME_1}:{TAG_VALUE_1},{TAG_NAME_2}:{TAG_VALUE_2}
GET /{OBJECT_TYPE}?tags={TAG_NAME}:{TAG_VALUE}*
GET /{OBJECT_TYPE}?tags={TAG_NAME}:*
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{TAG_NAME} |
The name of the tag to filter by. |
{TAG_VALUE} |
The value of the tag to filter by. Supports wildcard characters (* ). |
Request
The following request retrieves a list of datasets, filtering by one tag having a specific value AND the second tag being present.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?tags=sampleTag:123456,secondTag:* \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns a list of datasets that contain sampleTag
with a value of “123456”, AND secondTag
with any value. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 1",
"created": 1533539550237,
"updated": 1533539552416,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"Example tag value"
]
},
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.0",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 2",
"created": 1533539550237,
"updated": 1533539552416,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
"tags": {
"sampleTag": [
"123456"
],
"secondTag": [
"A different tag value"
],
"anotherTag": [
"2.0"
]
},
}
}
Some endpoints in the Catalog API have query parameters that allow for ranged queries, most often in the case of dates.
API format
GET /batches?createdAfter={TIMESTAMP_1}&createdBefore={TIMESTAMP_2}
Parameter | Description |
---|---|
{TIMESTAMP } |
A datetime integer in Unix Epoch time. |
Request
The following request retrieves a list of batches created during the month of April 2019.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/batches?createdAfter=1554076800000&createdBefore=1556668799000' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of Catalog objects that fall within the specified date range. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 1",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.0",
"imsOrg": "{ORG_ID}",
"name": "Example Dataset 2",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
The orderBy
query parameter allows you to sort (order) response data based on a specified property value. This parameter requires a “direction” (asc
for ascending or desc
for descending), followed by a colon (:
) and then a property to sort the results by. If a direction is not specified, the default direction is ascending.
Multiple sorting properties can be provided in a comma-separated list. If the first sorting property produces multiple objects that contain the same value for that property, the second sorting property is then used to further sort those matching objects.
For example, consider the following query: orderBy=name,desc:created
. Results are sorted in ascending order based on the first sorting property, name
. In cases where multiple records share the same name
property, those matching records are then sorted by the second sorting property, created
. If no returned records share the same name
, the created
property does not factor into the sorting.
API format
GET /{OBJECT_TYPE}?orderBy=asc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy=desc:{PROPERTY_NAME}
GET /{OBJECT_TYPE}?orderBy={PROPERTY_NAME_1},desc:{PROPERTY_NAME_2}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{PROPERTY_NAME} |
The name of a property to sort the results by. |
Request
The following request retrieves a list of datasets sorted by their name
property. If any datasets share the same name
, those datasets will in turn be ordered by their updated
property in descending order.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?orderBy=name,desc:updated' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of Catalog objects that are sorted according to the orderBy
parameter. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "0405",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.3",
"imsOrg": "{ORG_ID}",
"name": "AAM Dataset",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5cd3a129ec106214b722a939": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "AAM Dataset",
"created": 1554028394852,
"updated": 1554130582960,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Catalog provides two methods of filtering by property, which are further outlined in the sections that follow:
Simple filters allow you to filter responses based on specific property values. A simple filter takes the form of {PROPERTY_NAME}={VALUE}
.
For example, the query name=exampleName
returns only objects whose name
property contains a value of “exampleName”. By contrast, the query name=!exampleName
returns only objects whose name
property is not “exampleName”.
In addition, simple filters support the ability to query for multiple values for a single property. When multiple values are provided, the response returns objects whose property matches any of the values in the provided list. You can invert a multi-value query by prefixing a !
character to the list, returning only objects whose property value is not in the provided list (for example, name=!exampleName,anotherName
).
API format
GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}={VALUE_1},{VALUE_2},{VALUE_3}
GET /{OBJECT_TYPE}?{PROPERTY_NAME}=!{VALUE_1},{VALUE_2},{VALUE_3}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{PROPERTY_NAME} |
The name of the property whose value you want to filter by. |
{VALUE} |
A property value that determines which results to include (or exclude, depending on the query). |
Request
The following request retrieves a list of datasets, filtered to include only datasets whose name
property has a value of “exampleName” or “anotherName”.
curl -X GET \
'https://platform.adobe.io/data/foundation/catalog/dataSets?name=exampleName,anotherName' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of datasets, excluding any datasets whose name
is “exampleName” or “anotherName”. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.0.2",
"imsOrg": "{ORG_ID}",
"name": "exampleName",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.3",
"imsOrg": "{ORG_ID}",
"name": "anotherName",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
property
parameterThe property
query parameter provides more flexibility for property-based filtering than simple filters. In addition to filtering based on whether a property has a specific value, the property
parameter can use other comparison operators (such as “more-than” (>
) and “less-than” (<
)) as well as regular expressions to filter by property values. It can also filter by whether or not a property exists, regardless of its value.
The property
parameter can accept any level object properties. sampleKey
can be used for filtering using ?properties=subItem.sampleKey
.
{
"5ba9452f7de80400007fc52a": {
"name": "Sample Dataset",
"description": "Sample dataset containing important data",
"subitem": {
"sampleKey": "sampleValue"
}
}
}
API format
GET /{OBJECT_TYPE}?property={CONDITION}
Parameter | Description |
---|---|
{OBJECT_TYPE} |
The type of Catalog object to be retrieved. Valid objects are:
|
{CONDITION} |
A conditional expression that indicates which property to query for, and how its value is to be evaluated. Examples are provided below. |
The value of the property
parameter supports several different kinds of conditional expressions. The following table outlines the basic syntax for supported expressions:
Symbol(s) | Description | Example |
---|---|---|
(None) | Stating the property name with no operator returns only objects where the property exists, regardless of its value. | property=name |
! | Prefixing a “! ” to the value of a property parameter returns only objects where the property does not exist. |
property=!name |
~ | Returns only objects whose property values (string) match a regular expression provided after the tilde (~ ) symbol. |
property=name~^example |
== | Returns only objects whose property values exactly match the string provided after the double-equals symbol (== ). |
property=name==exampleName |
!= | Returns only objects whose property values do not match string provided after the not-equals symbol (!= ). |
property=name!=exampleName |
< | Returns only objects whose property values are less than (but not equal to) a stated amount. | property=version<1.0.0 |
<= | Returns only objects whose property values are less than (or equal to) a stated amount. | property=version<=1.0.0 |
> | Returns only objects whose property values are greater than (but not equal to) a stated amount. | property=version>1.0.0 |
>= | Returns only objects whose property values are greater than (or equal to) a stated amount. | property=version>=1.0.0 |
The name
property supports the use of a wildcard *
, either as the entire search string or as a part of it. Wildcards match empty characters, such that the search string te*st
will match the value “test”. Asterisks are escaped by doubling them (**
). A double-asterisk in a search string represents a single asterisk as a literal string.
Request
The following request will return any datasets with a version number greater than 1.0.3.
curl -X GET \
https://platform.adobe.io/data/foundation/catalog/dataSets?property=version>1.0.3 \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response contains a list of datasets whose version numbers are greater than 1.0.3. Unless a limit is also specified, the response contains a maximum of 20 objects.
{
"5b67f4dd9f6e710000ea9da4": {
"version": "1.1.2",
"imsOrg": "{ORG_ID}",
"name": "sampleDataset",
"created": 1554930967705,
"updated": 1554931119718,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5b1e3c867e6d2600003d5b49": {
"version": "1.0.6",
"imsOrg": "{ORG_ID}",
"name": "exampleDataset",
"created": 1554974386247,
"updated": 1554974386268,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
},
"5cd3a129ec106214b722a939": {
"version": "1.0.4",
"imsOrg": "{ORG_ID}",
"name": "anotherDataset",
"created": 1554028394852,
"updated": 1554130582960,
"createdClient": "{API_KEY}",
"createdUser": "{USER_ID}",
"updatedUser": "{USER_ID}",
}
}
Using an ampersand (&
), you can combine multiple filters in a single request. When additional conditions are added to a request, an AND relationship is assumed.
API format
GET /{OBJECT_TYPE}?{FILTER_1}={VALUE}&{FILTER_2}={VALUE}&{FILTER_3}={VALUE}