How Can I Incrementally Sync Data?

Question

By default, Fivetran performs a full data refresh for AWS Lambda connectors during each sync, updating the date in the _fivetran_synced column for all rows. How can I incrementally sync data?

Environment

Connector: AWS Lambda

Answer

The AWS Lambda connector provides a state object that you can use to maintain a record of the last data point we synced. See our Function request and response documentation for more information.

To retrieve only new or modified records, modify your Lambda function to ensure that API calls include a filter that compares the updated_at value with the cursor state.