Transformations for dbt Core Scheduling
Learn how scheduling works for Fivetran Transformations.
Pipeline overview
Fivetran transformation pipelines use the following elements:
- The start is the connector sync interval that initiates the pipeline.
- A connector updates source tables in the destination.
- A junction waits for multiple connectors to finish syncing before it triggers a transformation.
- A transformation is a model or a collection of models that updates downstream tables in the destination.
- An output model generates an analytics-ready table. It is usually the final element in a pipeline.
- A test is an assertion that you make about the models in your dbt project. A test may succeed or fail independently of model execution.
Integrated scheduling
Your output model, churn, uses an integrated schedule. The model references tables written by three connectors, each on an hourly schedule. The churn, customers, and revenue models are all refreshed every hour, after all three connectors have successfully synced.
Integrated scheduling with overlapping connector schedules
You have three connectors referenced by the same output model, churn. The churn output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite and salesforce connectors sync every hour, but the oracle connector syncs every 15 minutes.
The churn, revenue, and customers models run every 15 minutes after the oracle connector syncs and every hour after the salesforce, netsuite, and oracle connectors sync.
Integrated scheduling with sometimes overlapping connector schedules
You have three connectors feeding into the same output model, churn. The churn output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite and salesforce connectors sync every 2 hours, but the oracle connector syncs every 3 hours.
The churn, revenue, and customers models run every 2 hours after the salesforce and netsuite connectors sync, every 3 hours after the oracle connector syncs, and every 6 hours the salesforce, netsuite, and oracle connectors sync.
Custom scheduling
Custom scheduling allows you to select particular days of the week to run your transformation, a run frequency within a day, and, when the run frequency is set to 24 hours, a particular run time for your transformation. It also provides the Smart syncing option that allows you to further adjust the schedule:
- With the Smart syncing option enabled, the models run on the set schedule, but only when all upstream connectors have finished syncing
- With the Smart syncing option disabled, the models run on the set schedule regardless of the upstream connectors' sync schedule and state.
The following example demonstrates how custom scheduling with the Smart syncing option enabled works. You have three connectors feeding into the same output model, churn. The oracle connector runs every 15 minutes, the netsuite connector runs every hour, and the salesforce connector runs every 24 hours. You set the churn output model to update once a day to save on costs, but you don't want to run it until your destination data has been updated.
At the 24-hour mark, the churn, revenue, and customers models execute as soon as the three connectors successfully finish syncing.
