The Google BigQuery source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.
A base connection represents the authenticated connection between a source and Adobe Experience Platform.
Read this guide to learn how to create a base connection for Google BigQuery using the Flow Service API.
This guide requires a working understanding of the following components of Experience Platform:
The following sections provide additional information that you will need to know in order to successfully connect to Google BigQuery using the Flow Service API.
Read the Google BigQuery authentication guide for detailed steps on gathering your required credentials.
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
A base connection retains information between your source and Platform, including your source’s authentication credentials, the current state of the connection, and your unique base connection ID. The base connection ID allows you to explore and navigate files from within your source and identify the specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your Google BigQuery authentication credentials as part of the request parameters.
API format
POST /connections
Request
The following request creates a base connection for Google BigQuery using basic authentication:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Google BigQuery connection with basic authentication",
"description": "Google BigQuery connection with basic authentication",
"auth": {
"specName": "Basic Authentication",
"type": "OAuth2.0",
"params": {
"project": "{PROJECT}",
"clientId": "{CLIENT_ID},
"clientSecret": "{CLIENT_SECRET}",
"refreshToken": "{REFRESH_TOKEN}"
}
},
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
}
}'
Property | Description |
---|---|
auth.params.project |
The project ID of the default Google BigQuery project to query. against. |
auth.params.clientId |
The ID value used to generate the refresh token. |
auth.params.clientSecret |
The client value used to generate the refresh token. |
auth.params.refreshToken |
The refresh token obtained from Google used to authorize access to Google BigQuery. |
connectionSpec.id |
The Google BigQuery connection specification ID: 3c9b37f8-13a6-43d8-bad3-b863b941fedd . |
Response
A successful response returns details of the newly created connection, including its unique identifier (id
). This ID is required to explore your data in the next tutorial.
{
"id": "6990abad-977d-41b9-a85d-17ea8cf1c0e4",
"etag": "\"ca00acbf-0000-0200-0000-60149e1e0000\""
}
Request
The following request creates a base connection for Google BigQuery using service authentication:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Google BigQuery base connection with service account",
"description": "Google BigQuery connection with service account",
"auth": {
"specName": "Service Authentication",
"params": {
"projectId": "{PROJECT_ID}",
"keyFileContent": "{KEY_FILE_CONTENT},
"largeResultsDataSetId": "{LARGE_RESULTS_DATASET_ID}"
}
},
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
}
}'
Property | Description |
---|---|
auth.params.projectId |
The project ID of the default Google BigQuery project to query. against. |
auth.params.keyFileContent |
The key file that is used to authenticate the service account. You must encode the key file content in Base64. |
auth.params.largeResultsDataSetId |
(Optional) The pre-created Google BigQuery dataset ID that is required in order to enable support for large result sets. |
Response
A successful response returns details of the newly created connection, including its unique identifier (id
). This ID is required to explore your data in the next tutorial.
{
"id": "6990abad-977d-41b9-a85d-17ea8cf1c0e4",
"etag": "\"ca00acbf-0000-0200-0000-60149e1e0000\""
}
By following this tutorial, you have created a Google BigQuery base connection using the Flow Service API. You can use this base connection ID in the following tutorials: