Sync Overview

Learn about how Fivetran syncs your data from your source into your destination.

Sync types and statuses

You must run a historical sync to begin using your connector. The first historical sync that Fivetran does for a connector is called the initial sync. During the historical sync, we extract and process all the historical data from the selected tables in the source. We periodically load extracted and processed data into the destination simultaneously.

After a successful initial sync, the connector runs in incremental sync mode. In this mode, only data that has been modified or added - also known as incremental changes - is extracted, processed, and loaded on schedule. We reimport tables when we aren't able to fetch only incremental changes. We use cursors to record the history of the syncs.

In some cases, you may need to re-run a historical sync to fix a data integrity error. This is called a re-sync. We sync all historical data in the source tables and their columns according to your connector configuration.

For some connectors, we perform a rollback sync once a day. During a rollback sync, we sync current data as well as more past report data than we do in an incremental sync. This approach ensures that we capture all changes, even if those changes occur outside the incremental sync time frame. The rollback sync's window size - or how many days' worth of data that we request in each sync - varies by connector. Some connectors let you choose the window size, while others have fixed window sizes. Note that increasing the window size also increases your Monthly Active Rows (MAR). We perform rollback syncs for the following connectors:

• Adobe Analytics
• AdRoll
• Amazon Ads
• Apple Search Ads
• Criteo
• Facebook Ads
• Facebook Pages
• Google Ad Manager
• Google Ads
• Google Analytics 4
• Google Campaign Manager 360
• Google Display & Video 360
• Google Search Console
• Instagram Business
• LinkedIn Ad Analytics
• LinkedIn Company Pages
• Marin Software
• Microsoft Advertising
• Outbrain
• Pinterest Ads
• Snapchat Ads
• Taboola
• The Trade Desk
• TikTok Ads
• Twitter Ads
• Twitter Organic

Sync History chart

The Sync History chart visualizes the status of your syncs, how long they took, and how much data was synced each time. View your Sync History chart on the Status page of your Fivetran dashboard.

The chart breaks down your syncs into Extract, Process, and Load steps. You can view the past hour, day, or week of sync history. The default display is one day.

Each colored bar represents a sync. Hover over a sync bar to view the following sync metrics:

• Sync type
• Duration
• Start and end time
• Status
• Data volume

You can also view sync logs.

The bar's color reflects the sync status:

TIP: Learn how to troubleshoot failed syncs in our sync troubleshooting documentation.

For most of our database connectors, we provide more detailed data volume measurements for each sync phase in the Sync History chart. Hover over each sync phase in the Sync History chart to view the following sync measurements:

• Start time
• End time
• Duration
• Status

These measurements help you understand exactly how much data we move through our pipeline in each phase. We define the data volume for each phase as follows:

• Extract Volume: Total volume of data we read from your source. This includes any data we filter out or don't sync but read as part of our queries.
• Process Volume: Total volume of data we process before our deduplication process filters out data. Process volume also includes internal Fivetran metadata that we use during this phase.
• Load Volume: Total volume of data we upload to your destination after normalization.

User actions and errors are tracked at the top of the Sync History chart. A round icon indicates the time of a particular user action or error. For errors, a solid line also shows how long the problem has persisted. Hover over the icon to see details of each event:

TIP: To learn more about an error, click View error. A pop-up window appears on the right to tell you why the error occurred and how to resolve it.

For connectors that support priority-first sync, the forward and backward syncs are displayed as a single sync bar. When you hover over the sync bar, Priority-first sync is listed as the sync type in the pop-up. Click View in the pop-up to see the historical sync progress for each table. The details include the date up to which the data has been fetched and the total number of days' worth of available data history to be fetched.

Sync frequency and scheduling

By default, Fivetran syncs your data from your source to your destination every six hours. If you want Fivetran to sync your data more or less often, you can adjust the sync frequency. The sync frequency defines the time period between the start times of two subsequent syncs. For example, if your sync frequency is 30 minutes, your next sync will start 30 minutes after the start of your last sync provided the latter successfully completed within 30 minutes from its start. The baseline sync start time for the first sync within the 24-hour cycle is midnight in the destination time zone.

IMPORTANT: When you add a new destination, Fivetran assigns it a fixed time offset. The offset can be any random value in minutes ranging from 0 to 60. It is derived from the destination ID hash. This offset is shared by every connector in the destination. The offset value remains the same regardless of the set sync frequency.

Depending on your plan level, you can choose from the following sync frequencies:

• 1 minute*
• 5 minutes
• 15 minutes
• 30 minutes
• 1 hour
• 2 hours
• 3 hours
• 6 hours (default)
• 8 hours
• 12 hours
• 24 hours

* You must have an Enterprise or Business Critical plan to use the 1-minute sync frequency. We support this sync frequency for all standard connectors except those listed below. We do not support it for any Lite connectors.

On the Setup tab of your connector's details page, select your Sync frequency.

Alternatively, you can create a custom sync schedule using the Fivetran REST API (Free, Standard, Enterprise and Business Critical accounts only). Learn how in our How to Set a Custom Sync Frequency Troubleshooting page.

If a sync is already running at the time of the scheduled sync start, Fivetran postpones the scheduled sync to the next scheduled sync start time. Here's an example for a connector with a 15-minute sync frequency and an offset value of 0 minutes:

Fivetran supports custom sync scheduling for connectors that sync once a day. You can choose what time your daily syncs start if you don't want to use Fivetran's default midnight starting time. If you don't know how often your connector syncs, check the sync frequency on the Setup tab of the connector details page in your Fivetran dashboard.

Sync start times and offsets

Fivetran uses the following formula to calculate the actual start times:

actual sync start time = scheduled sync start time + remainder of (offset value / sync frequency)

For example, we calculate the actual start time of the second sync in a 24-hour cycle with the following values:

• the sync frequency is 15 minutes
• the offset value is 44 minutes

00:29 = 00:15 + 00:14 where 00:14 is the remainder of 00:44 / 00:15.

Thus, the actual sync start time is 00:29.

Here is another example where we calculate the actual start time of the second sync in a 24-hour cycle with different values:

• the sync frequency is 15 minutes
• the offset value is 2 minutes

00:17 = 00:15 + 00:02 where 00:02 is the remainder of 00:02 / 00:15, which equals the offset value in this case.

In this example, the actual sync start time is 00:17.

The following table provides further examples of sync start times, which depend on the relationship between the sync frequency and the offset value. For the sync start times presented in the table, we assume that all syncs are successfully completed by the start of the next sync.

NOTE: When you change the sync frequency, the next sync will follow the new sync schedule corresponding to the set sync frequency regardless of the last sync's start time. For example, the sync frequency is set to 3 hours, the offset is 6 minutes, the last sync started at 15:06 and completed at 16:50. At 17:00 you set the sync frequency to 12 hours. The next sync will start at 00:06 rather than at 03:06.

Failed sync retries during the first 24 hours

For failed syncs, Fivetran calculates the start time for the next sync using the following formula:

time of the sync failure + retry interval

where the retry interval varies depending on the set sync frequency as listed in the following table:

For example, if the sync failed at 13:38, and the sync frequency is 2 hours, then the start time of the next sync will be:

13:38 + 40 min = 14:18

When a connection keeps failing

• If a connection fails with a task, for the first 24 hours following the first failure, it retries syncs as described in the Failed Sync Retries During the First 24 Hours section.
• After 24 hours of failed sync retries since the first failure, the connection sync retry frequency is switched to the sync frequency you set for the connection until either a sync retry succeeds, or the time since the initial sync failure reaches 72 hours.
• After 72 hours of failed sync retries since the first failure, the sync retry is run once a day until either a sync retry succeeds, or the time since the first failure reaches 14 days.
• If the sync retry does not succeed after 14 days since the first failure, Fivetran automatically pauses the connection.
• If you unpause the connection that was automatically paused after 14 days of failed retries, and the connection's sync still fails, Fivetran will retry syncing once a day for the next 14 days before automatically pausing it again.
• Once a sync retry succeeds at any point within 14 days since the first failure, the connection reverts to the sync frequency you set for the connection.

For rescheduled syncs, the rules and formulas used to define the sync start time depend on the connector type. Although rescheduling is most commonly caused by Fivetran hitting API quotas in the source, causes of sync rescheduling vary depending on the connector type.

Checking the sync start time offset value

You can use the sync frequency slider to check the sync start time offset value.

1. On the Setup tab of the connector details page, set the sync frequency to 24 hours.
2. Click Change.
3. Open the Daily Run Time menu. The offset value is the minute value. For example, if the sync start times are 00:48, 01:48, 02:48, and so on, then the offset value is 48 minutes.
4. Using the slider, set the sync frequency to the value you had before.

Syncing empty tables and columns

Most Fivetran connectors do not sync empty tables and columns (tables and columns that have no data), and don't create them in your destination. However, Fivetran can create empty tables and columns in your destination for some connectors. See our Features documentation for a list of supported connectors.

NOTE: An empty schema is a schema that doesn't contain any tables. A schema that contains empty tables is not an empty schema. Fivetran doesn't sync empty schemas.