Why Is My Sync Taking Longer Than Normal?

Question

Why is my sync taking longer than normal?

Environment

All Connectors

Answer

You may experience longer syncs due to the following reasons:

Increase in data volume

Increasing the volume of data we need to sync may increase your sync time. To check whether an increase in data volume is impacting your sync time, do the following:

1. In your connector dashboard, view the Sync History chart.
2. Hover over a sync that took a long time.
3. Note the sync's data volume.
4. Compare the data volume to a shorter sync.

Manual schema change

Your sync time may increase if you have included an additional column or table. You can use the Fivetran Platform Connector to check whether a new column or table was included in your sync. See our track schema changes documentation for more information.

Automatic schema change

If your connector's schema change settings are set to Allow all, we will automatically sync all newly detected columns and tables. If we detect and sync new, large tables, you may experience longer sync times. To view each connector's new tables and columns, do the following:

1. In your Fivetran dashboard, select Account Settings and then click Billing & Usage.
2. Click the Usage tab.
3. Go to the Usage breakdown section and click the MAR type dropdown.
4. Select the Initial Sync (Free) checkbox.

Significant change in source data

Significant changes in your source data, such as mass deletes or mass record updates, may increase the number of changes we need to sync and, thus, the sync time. You can use the Fivetran Platform Connector to check for significant changes that may impact your sync time. Fivetran generates records_modified log events that include a count field, representing the number of operations modified.

You can also check how many records were updated in your destination by querying the updated_at column with a WHERE clause. However, note that child tables may not include an updated_at column if they update automatically in conjunction with their associated parent record.

Network latency

Intermittent network latency may impact your sync speed. This may occur when we connect to your source database connector or your destination database. Check with your network team to verify if there are known or potential network latency issues.

API rate limits (application connectors only)

If you have reached the source's API rate limit, we may delay your sync until we can query the API again. You can use the Fivetran Platform Connector to check whether an API rate limit may have impacted your sync time. Fivetran generates warning log events that provide detailed information about the warning and affected tables.

Re-import tables are enabled

If a table contains a column that we can't sync incrementally, we must re-import the table in full to sync updated values. The frequency with which we update these re-import tables is determined by how fast we can ingest the data from the source. See the applicable connector's documentation to find out whether we sync re-import tables.

Nested API calls (application connectors only)

We rely on nested API calls to sync data for some tables, which may impact the sync speed. To avoid slow or delayed syncs, we display the following warning when you include these tables in your connector's schema:

Not recommended: Syncing <table> table might slow down your sync.