How to Run Transformations After a Sync Using the REST API

Follow this REST API tutorial to complete the following workflow:

Create a dbt transformation project.
Create a transformation linked to a connection sync.
Trigger a transformation run manually, and poll for completion.

For dbt project guidance, including models, sources, scheduling , and dbt_project.yml, see Transformations for dbt.

Prerequisites

You have a scoped API key and secret.
You have a Fivetran destination group and at least one connection in that group.
You have a dbt project hosted in a Git repository accessible via SSH.
You have jq installed.

For help in authoring dbt models and structuring your dbt project, see Transformations for dbt.

Endpoints used in this workflow

POST /v1/transformation-projects
POST /v1/transformations
POST /v1/transformations/{transformationId}/run
GET /v1/transformations/{transformationId}

Set your environment variables

export FIVETRAN_API_KEY="<your_api_key>"
export FIVETRAN_API_SECRET="<your_api_secret>"
export FIVETRAN_BASE_URL="https://api.fivetran.com/v1"
export FIVETRAN_AUTH_HEADER=$(printf "%s" "$FIVETRAN_API_KEY:$FIVETRAN_API_SECRET" | base64 | tr -d '\n')

# The destination group that owns the connection and transformation project.
export GROUP_ID="<your_group_id>"

# The connection ID whose sync will trigger the transformation.
export CONNECTION_ID="<your_connection_id>"

# dbt project configuration.
export DBT_GIT_REMOTE_URL="git@github.com:<org>/<repo>.git"
export DBT_VERSION="1.8.9"
export DBT_DEFAULT_SCHEMA="<your_default_schema>"

Create transformation project

Use the Create a Transformation Project endpoint to create a dbt project and link it to your destination group. The project_config.git_remote_url must point to a Git repository that Fivetran can access via the SSH deploy key returned in the response.

CREATE_PROJECT_RESPONSE=$(curl --silent --show-error \
  --request POST \
  --url "$FIVETRAN_BASE_URL/transformation-projects" \
  --header "Accept: application/json" \
  --header "Authorization: Basic $FIVETRAN_AUTH_HEADER" \
  --header "Content-Type: application/json" \
  --data @- <<JSON
{
  "group_id": "$GROUP_ID",
  "type": "DBT_CORE",
  "run_tests": false,
  "project_config": {
    "git_remote_url": "$DBT_GIT_REMOTE_URL",
    "dbt_version": "$DBT_VERSION",
    "default_schema": "$DBT_DEFAULT_SCHEMA",
    "git_branch": "main",
    "folder_path": "/",
    "threads": 4,
    "target_name": "prod"
  }
}
JSON
)

echo "$CREATE_PROJECT_RESPONSE"
export TRANSFORMATION_PROJECT_ID=$(echo "$CREATE_PROJECT_RESPONSE" | jq -r '.data.id')
export SSH_PUBLIC_KEY=$(echo "$CREATE_PROJECT_RESPONSE" | jq -r '.data.project_config.public_key')
echo "TRANSFORMATION_PROJECT_ID=$TRANSFORMATION_PROJECT_ID"
echo "SSH_PUBLIC_KEY=$SSH_PUBLIC_KEY"

Add the public_key from the response as a deploy key in your Git repository before running the transformation. Without it, we can't clone your dbt project.

project_config fields

Field	Required	Description
`git_remote_url`	Yes	SSH URL of your dbt Git repository.
`dbt_version`	Yes	dbt version to use (for example, `1.8.9`).
`default_schema`	Yes	Default schema for dbt model output in your destination.
`git_branch`	No	Git branch to check out. Defaults to the repository default branch.
`folder_path`	No	Path within the repository where `dbt_project.yml` is located. Defaults to `/`.
`threads`	No	Number of threads dbt uses when running models. Defaults to `1`.
`target_name`	No	dbt target name, which maps to the `target` in `profiles.yml`.
`environment_vars`	No	List of environment variable strings in `KEY=VALUE` format passed to dbt.

Create transformation

Use the Create a Transformation endpoint to create a transformation job.

Set schedule.schedule_type to INTEGRATED to link the transformation to a connection sync. When the connection sync completes successfully, Fivetran automatically queues the transformation.

CREATE_TRANSFORMATION_RESPONSE=$(curl --silent --show-error \
  --request POST \
  --url "$FIVETRAN_BASE_URL/transformations" \
  --header "Accept: application/json" \
  --header "Authorization: Basic $FIVETRAN_AUTH_HEADER" \
  --header "Content-Type: application/json" \
  --data @- <<JSON
{
  "type": "DBT_CORE",
  "paused": false,
  "schedule": {
    "schedule_type": "INTEGRATED",
    "connection_ids": ["$CONNECTION_ID"]
  },
  "transformation_config": {
    "project_id": "$TRANSFORMATION_PROJECT_ID",
    "name": "My dbt Transformation",
    "steps": [
      {
        "name": "Run all models",
        "command": "dbt run"
      }
    ]
  }
}
JSON
)

echo "$CREATE_TRANSFORMATION_RESPONSE"
export TRANSFORMATION_ID=$(echo "$CREATE_TRANSFORMATION_RESPONSE" | jq -r '.data.id')
echo "TRANSFORMATION_ID=$TRANSFORMATION_ID"

To have the transformation follow its own schedule instead of running after a connection sync, set schedule_type to INTERVAL, TIME_OF_DAY, or CRON. You can still trigger a manual run with POST /run regardless of the configured schedule. The next step explains how to trigger a transformation run manually.

`schedule` fields for INTEGRATED type

Field	Required	Description
`schedule_type`	Yes	Set to `INTEGRATED` to run after a sync completes. Other options: `INTERVAL`, `TIME_OF_DAY`, `CRON`.
`connection_ids`	Conditional	List of connection IDs whose sync completion triggers this transformation. Required when `schedule_type` is `INTEGRATED` and `transformation_ids` is not set.
`transformation_ids`	Conditional	List of transformation IDs whose completion triggers this transformation. Required when `schedule_type` is `INTEGRATED` and `connection_ids` is not set.

`transformation_config` fields

Field	Required	Description
`project_id`	Yes	ID of the transformation project created in the previous step.
`name`	Yes	Display name for this transformation.
`steps`	Yes	Ordered list of dbt commands to run. Each step has a `name` label and a `command` dbt CLI command, for example, `dbt run`.

(Optional) Run transformation manually

Use the Run a Transformation endpoint to trigger an immediate run regardless of the configured schedule. Use this for on-demand execution, CI pipelines, or testing.

The examples in this tutorial provide reference control flow. Adapt how your workflow handles each terminal status to your business requirements.

RUN_RESPONSE=$(curl --silent --show-error \
  --write-out "\n%{http_code}" \
  --request POST \
  --url "$FIVETRAN_BASE_URL/transformations/$TRANSFORMATION_ID/run" \
  --header "Accept: application/json" \
  --header "Authorization: Basic $FIVETRAN_AUTH_HEADER" \
  --header "Content-Type: application/json")

RUN_HTTP_STATUS=$(echo "$RUN_RESPONSE" | tail -n 1)
RUN_RESPONSE_BODY=$(echo "$RUN_RESPONSE" | sed '$d')

echo "HTTP status: $RUN_HTTP_STATUS"
echo "$RUN_RESPONSE_BODY" | jq

A successful response returns HTTP 200. HTTP 409 can indicate either that the transformation is paused or that another run is already in progress. Check the response message to distinguish these cases. If another run is already in progress, see the Handle 409 — Transformation Already Running section below. If the transformation is paused, resume it before triggering a run.

Poll for run completion

Use the Retrieve Transformation Details endpoint and poll GET /v1/transformations/{transformationId} until the status field reaches a terminal state. In-progress statuses are NEW, SCHEDULING, RUNNING, and TERMINATING. Terminal statuses are SUCCEEDED, FAILED, CANCELED, BLOCKED, and PARTIALLY_SUCCEEDED.

The following example demonstrates how to distinguish in-progress, terminal, and unexpected statuses. Adapt the terminal-status branches to your business requirements.

TIMED_OUT=true
for i in {1..60}; do
  STATUS_RESPONSE=$(curl --silent --show-error \
    --request GET \
    --url "$FIVETRAN_BASE_URL/transformations/$TRANSFORMATION_ID" \
    --header "Accept: application/json" \
    --header "Authorization: Basic $FIVETRAN_AUTH_HEADER")

  TRANSFORMATION_STATUS=$(echo "$STATUS_RESPONSE" | jq -r '.data.status')
  echo "Attempt $i: status=$TRANSFORMATION_STATUS"

  case "$TRANSFORMATION_STATUS" in
    SUCCEEDED)
      echo "Transformation completed successfully."
      TIMED_OUT=false
      break
      ;;
    PARTIALLY_SUCCEEDED)
      echo "Transformation partially succeeded. Review the dbt logs and decide how to handle this status in your workflow."
      TIMED_OUT=false
      break
      ;;
    FAILED)
      echo "Transformation failed."
      exit 1
      ;;
    CANCELED)
      echo "Transformation was canceled."
      exit 1
      ;;
    BLOCKED)
      echo "Transformation is blocked and cannot run. Check the Fivetran dashboard for configuration issues."
      exit 1
      ;;
    RUNNING|SCHEDULING|NEW|TERMINATING)
      # Still in progress — wait and retry.
      sleep 10
      ;;
    *)
      echo "Unexpected status: $TRANSFORMATION_STATUS"
      exit 1
      ;;
  esac
done

if [ "$TIMED_OUT" = "true" ]; then
  echo "Timed out waiting for transformation to complete. Last status: $TRANSFORMATION_STATUS"
  exit 1
fi

Status values

Status	Meaning
`NEW`	Transformation is queued but not yet started.
`SCHEDULING`	Fivetran is preparing to start the run.
`RUNNING`	dbt commands are executing.
`TERMINATING`	A cancellation request has been issued; run is winding down.
`SUCCEEDED`	All steps completed without error.
`PARTIALLY_SUCCEEDED`	At least one step failed after another step completed successfully.
`FAILED`	One or more steps failed. Check dbt logs in the Fivetran dashboard for details.
`CANCELED`	The run was canceled before completion.
`BLOCKED`	The transformation cannot run due to a prerequisite condition (for example, a project configuration error).

(Troubleshooting) Handle 409 — transformation already running

HTTP 409 from POST /v1/transformations/{transformationId}/run can indicate that the transformation is paused or that another run is already in progress. Apply the following retry pattern only when the response message states that the transformation is already running.

Do not retry POST /run in a tight loop on 409. If another run is in progress, poll GET /v1/transformations/{transformationId} until the run reaches a known terminal state, then retry POST /run exactly once. If the transformation is paused, resume it instead of retrying.

This example addresses the specific case where your API client repeatedly retries POST /run because it interprets 409 as a request failure rather than an active-run conflict. It demonstrates safe retry control flow; it does not prescribe how your workflow should interpret the outcome of the previous run.

run_transformation_with_409_handling() {
  local TRANSFORMATION_ID="$1"
  local RUN_RESPONSE
  local RUN_RESPONSE_BODY
  local HTTP_STATUS
  local ERROR_MESSAGE
  local STATUS_RESPONSE
  local TRANSFORMATION_STATUS
  local PREVIOUS_RUN_FINISHED=false
  local RETRY_RESPONSE
  local RETRY_RESPONSE_BODY
  local RETRY_HTTP_STATUS

  # Attempt to trigger the run and preserve both the response body and HTTP status.
  if ! RUN_RESPONSE=$(curl --silent --show-error \
    --write-out "\n%{http_code}" \
    --request POST \
    --url "$FIVETRAN_BASE_URL/transformations/$TRANSFORMATION_ID/run" \
    --header "Accept: application/json" \
    --header "Authorization: Basic $FIVETRAN_AUTH_HEADER" \
    --header "Content-Type: application/json"); then
    echo "Request failed before the API returned a response."
    return 1
  fi

  HTTP_STATUS=$(echo "$RUN_RESPONSE" | tail -n 1)
  RUN_RESPONSE_BODY=$(echo "$RUN_RESPONSE" | sed '$d')

  if [ "$HTTP_STATUS" -eq 409 ]; then
    ERROR_MESSAGE=$(echo "$RUN_RESPONSE_BODY" | jq -r '.message // empty')

    if [[ "$ERROR_MESSAGE" != *"already running"* ]]; then
      echo "Run not started: ${ERROR_MESSAGE:-Unexpected 409 response.}"
      return 1
    fi

    echo "409: A run is already in progress. Waiting for it to finish..."

    # Poll until the current run reaches a known terminal status.
    for i in {1..60}; do
      if ! STATUS_RESPONSE=$(curl --silent --show-error \
        --request GET \
        --url "$FIVETRAN_BASE_URL/transformations/$TRANSFORMATION_ID" \
        --header "Accept: application/json" \
        --header "Authorization: Basic $FIVETRAN_AUTH_HEADER"); then
        echo "Failed to retrieve the transformation status."
        return 1
      fi

      if ! TRANSFORMATION_STATUS=$(echo "$STATUS_RESPONSE" | jq -er '.data.status'); then
        echo "The status response did not contain a valid transformation status."
        return 1
      fi

      echo "Polling attempt $i: status=$TRANSFORMATION_STATUS"

      case "$TRANSFORMATION_STATUS" in
        RUNNING|SCHEDULING|NEW|TERMINATING)
          sleep 10
          ;;
        SUCCEEDED|PARTIALLY_SUCCEEDED|FAILED|CANCELED|BLOCKED)
          echo "Previous run finished with status: $TRANSFORMATION_STATUS"
          PREVIOUS_RUN_FINISHED=true
          break
          ;;
        *)
          echo "Unexpected status: $TRANSFORMATION_STATUS"
          return 1
          ;;
      esac
    done

    if [ "$PREVIOUS_RUN_FINISHED" != "true" ]; then
      echo "Timed out waiting for the previous run to finish. Last status: $TRANSFORMATION_STATUS"
      return 1
    fi

    # Retry the run once and verify that the API accepted it.
    echo "Retrying POST /run..."
    if ! RETRY_RESPONSE=$(curl --silent --show-error \
      --write-out "\n%{http_code}" \
      --request POST \
      --url "$FIVETRAN_BASE_URL/transformations/$TRANSFORMATION_ID/run" \
      --header "Accept: application/json" \
      --header "Authorization: Basic $FIVETRAN_AUTH_HEADER" \
      --header "Content-Type: application/json"); then
      echo "Retry failed before the API returned a response."
      return 1
    fi

    RETRY_HTTP_STATUS=$(echo "$RETRY_RESPONSE" | tail -n 1)
    RETRY_RESPONSE_BODY=$(echo "$RETRY_RESPONSE" | sed '$d')

    if [ "$RETRY_HTTP_STATUS" -ne 200 ]; then
      echo "Retry failed with HTTP status $RETRY_HTTP_STATUS."
      echo "$RETRY_RESPONSE_BODY" | jq
      return 1
    fi

    echo "Run triggered successfully."
    echo "$RETRY_RESPONSE_BODY" | jq
  elif [ "$HTTP_STATUS" -eq 200 ]; then
    echo "Run triggered successfully."
    echo "$RUN_RESPONSE_BODY" | jq
  else
    echo "Run failed with HTTP status $HTTP_STATUS."
    echo "$RUN_RESPONSE_BODY" | jq
    return 1
  fi
}

run_transformation_with_409_handling "$TRANSFORMATION_ID"