Can I Store Data From Multiple Connectors With the Same Schema and Table Name in a Single AWS Glue Database?

Question

Can I store data from multiple connectors with the same schema and table name in a single AWS Glue database?

Environment

Destination: S3 Data Lake

Answer

We do not recommend configuring multiple connectors with the same schema and table name in a single AWS Glue database.

Suppose you use the same AWS account ID to create multiple groups with different role Amazon Resource Names (ARNs) and S3 buckets in the same region. In this scenario, all groups share the same AWS Glue database.

If each group has a Fivetran connector with the same schema and table name, either of the following scenarios may occur:

• Only the first group to complete a sync will succeed and the subsequent connectors from different groups will fail. This will happen because the subsequent connectors will attempt to sync into the same bucket as the first group, using different role ARNs.
• If the role has access to all configured S3 paths, subsequent syncs for all connectors may succeed. However, this will introduce data integrity issues in your destination and will corrupt the tables.

Therefore, to avoid conflicts or sync failures, create connectors with unique schema and table name combinations.