Error: Temporary File Size Exceeds temp_file_limit

Issue

While processing a table containing a large amount of data, syncs fail. The following error appears:

org.postgresql.util.PSQLException: ERROR: temporary file size exceeds temp_file_limit

Environment

Connector: Amazon RDS PostgreSQL

Resolution

To resolve this issue, do the following:

1. In Fivetran, pause your connector.

   i. Go to your Amazon RDS PostgreSQL connector page.

   ii. In the top right, set the ENABLED toggle to OFF.

2. In your database, increase the temp_file_limit parameter to a value that can accommodate the data you want to process. We recommend setting it to at least twice the size of the largest table in your database. Use the following SQL query, replacing <new_value> with the applicable value.

   ALTER SYSTEM SET temp_file_limit = '<new_value>';
   SELECT pg_reload_conf();
   

3. In Fivetran, resume your connector.

   i. Go back to your Amazon RDS PostgreSQL connector page.

   ii. In the top right, set the ENABLED toggle to ON.

4. Monitor the temporary file usage and adjust the temp_file_limit as needed.

If the issue persists, consider excluding non-essential tables from your connector schema to reduce the temporary space requirements.

Cause

This issue occurs when the operations performed during our sync require more temporary space than your database allows, as determined by the temp_file_limit parameter.