Why Are My Managed Data Lake Queries Slower After an Initial Sync or Re-Sync?

Question

Why do queries against my managed data lake run more slowly and scan more data after an initial sync or re-sync?

Environment

• Destination: Managed Data Lake Service
• Table format: Fivetran-managed Apache Iceberg tables
• Query engines: Amazon Athena and other engines that read Fivetran-managed Iceberg tables

Answer

After an initial sync or re-sync, some queries may temporarily scan more data and run more slowly.

We write large volumes of historical data in bulk during these syncs. Broad time-range queries may not align efficiently with the resulting file layout, causing the query engine to scan more data. For more information, see Metadata Catalogs.

As regular incremental syncs resume, we write new data in smaller batches. Queries that focus on recent data may become more efficient over time. To determine whether the bulk write caused the slower performance, compare:

• A broad historical query, such as one covering the last 7 or 30 days
• A narrow recent-window query, such as one covering the last 12 or 24 hours

If the narrow query performs better, the file layout created during the initial sync or re-sync is likely the cause.