Excessive WAL Build Up

Issue

The connector is not releasing the write-ahead log (WAL) files accumulating in the pg_wal directory.

Environment

Connector: PostgreSQL

Resolution

Option 1: Identify and address ongoing transactions

1. In your database, check for ongoing transactions by executing the following queries:

   SELECT pg_xact_changes(<STALLED_TXID>);
   SELECT * FROM pg_replication_slots;
   SELECT * FROM pg_stat_activity;
   SELECT * FROM pg_prepared_xacts;
   SELECT * FROM pg_prepared_statements;
   

2. Terminate or resolve the ongoing transactions.

Option 2: Drop and re-create the replication slot

1. In Fivetran, go to your PostgreSQL connector page.
2. In the top right corner, pause your connector by setting the ENABLED toggle to OFF.
3. In your database, drop the existing replication slot and wait for checkpoints to complete.
4. Create a new replication slot.
5. On your Fivetran connector page, select the Setup tab.
6. Next to Re-sync All Historical Data, select Click to re-sync all historical data from your source.
7. In the confirmation pop-up window, click Re-sync Connector.
8. Unpause your connector by setting the ENABLED toggle to ON.

Option 3: Allow the sync to continue

1. Check how much disk space you have available.
2. Add more disk space as needed.
3. Allow the sync to continue.

Cause

This issue occurs when incremental syncs takes longer than expected, causing the WAL to build up and consume a lot of disk space. This may happen for one of the following reasons:

• There are a lot of tables included in the connector schema.
• There is a high rate of change in the source database.
• The WAL slot is not advancing due to ongoing transactions.