How to Optimize Sync Frequency for Row-Based Database Destinations
Use Case
Fivetran's syncs to your row-based database destination are slow. Databases sync slower than data warehouses do, but the syncs could also be slow because backups sync at the same time as Fivetran or because all of your connectors sync at the same intervals. You want to change your sync frequency to optimize your destination database loads.
Environment
Destination: One of the following row-based databases:
Recommendation
If you have maxed out your CPU and memory capabilities and are still experiencing slow syncs, try the following steps:
Make your syncs more frequent. When you sync more frequently, your syncs consist smaller batches of data. If your data set is not too large, we can load smaller batches of data into your destination faster than larger batches.
Set different sync frequencies for your connectors so that they're not syncing at the same time.
- To trigger syncs according to your own custom schedule, do the following:
i. Change theschedule_typetomanualusing the Modify a Connector endpoint.
ii. Trigger syncs manually using the Sync Connector Data endpoint. - If you only need data once a day, select the exact time that you want the sync to occur. Ideally, that time is not during busy operating hours for your destination. Learn how in this daily sync time use case.
- Set up a custom sync schedule. Learn how in our How to Set a Custom Sync Frequency Troubleshooting page.
- To trigger syncs according to your own custom schedule, do the following:
Increase the timeout settings on your row-based database destination. You can learn more about this process in our documentation for your destination or in community boards.
Considerations
We strongly suggest that you upgrade your destination to a horizontally scalable column store like Snowflake, Redshift, BigQuery, etc.