Transformations for dbt Core Integrated Schedulinglink
Learn how integrated scheduling works for Fivetran Transformations.
Pipeline overviewlink
Fivetran pipelines use the following elements:
- The start is the interval that initiates the pipeline.
- A connector updates source tables in the destination.
- A junction waits for multiple connectors to finish syncing before it triggers a dbt transformation.
- A transformation is a model or a collection of models that updates downstream tables in the destination.
- An output model generates an analytics-ready table. It is typically a leaf node on your data lineage graph.
- A test is an assertion that you make about the models in your dbt project. A test may succeed or fail independently of model execution.
Fully integrated schedulinglink
Your output model, churn
, uses an integrated schedule. The model references tables written by three connectors, each on an hourly schedule. The churn
, customers
, and revenue
models are all refreshed every hour, after all three connectors have successfully synced.
Fully integrated scheduling with overlapping connector scheduleslink
You have three connectors referenced by the same output model, churn
. The churn
output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite
and salesforce
connectors sync every hour, but the oracle
connector syncs every 15 minutes.
The churn
, revenue
, and customers
models run every 15 minutes after the oracle
connector syncs and every hour after the salesforce
, netsuite
, and oracle
connectors sync.
Fully integrated scheduling with sometimes overlapping connector scheduleslink
You have three connectors feeding into the same output model, churn
. The churn
output model uses an integrated schedule and runs whenever there is new data in any source. The netsuite
and salesforce
connectors sync every 2 hours, but the oracle
connector syncs every 3 hours.
The churn
, revenue
, and customers
models run every 2 hours after the salesforce
and netsuite
connectors sync, every 3 hours after the oracle
connector syncs, and every 6 hours the salesforce
, netsuite
, and oracle
connectors sync.
Fully integrated scheduling with downstream independent scheduleslink
You have three connectors feeding into the same output model, customers
. You set the customers
model to a fully integrated schedule, so it runs every 15 minutes. The two downstream models run on independent schedules - the revenue
model runs every hour and the churn
model runs once every 24 hours.
Partially integrated schedulinglink
You have three connectors feeding into the same output model, churn
. The oracle
connector runs every 15 minutes, the netsuite
connector runs every hour, and the salesforce
connector runs every 24 hours. You set the churn
output model to update once a day to save on costs, but you don't want to run it until your destination data has been updated.
At the 24-hour mark, the churn
, revenue
, and customers
models execute as soon as the three connectors successfully finish syncing.
Independent scheduling with upstream connectorslink
Your connectors run every hour, but you set the churn
output model to update once a day to save on costs.
At the 24-hour mark, the churn
, revenue
, and customers
models execute independently of the connectors, even if the connectors are currently running.
Independent scheduling with no upstream connectorslink
The churn
output model is not related to any Fivetran connectors, but it lives in your dbt Core project and references an externally populated source table. You can still schedule churn
and its upstream models in the Fivetran dashboard and run on an independent pipeline.