How Can I Have More Than One Connector Writing to the Same Schema?
Question
How can I write multiple sources of data to the same schema in my destination?
Environment
All connectors
Answer
Approach 1: Combine schemas post-load
- Use a single destination.
- Create connectors with unique schema names.
- Transform data post-load using a command that best fits your use case or requirements.
Approach 2: Create multiple destinations in Fivetran
- Create a new destination for every sources of data you require to write to the same schema. Ensure that this destination uses the same destination credentials each time.
- Create a connector in each destination with the same schema name.
This solution can lead to potential data integrity issues from overwrites or longer sync times due to multiple writes queueing within the destination.
Context
Fivetran enforces a unique schema name within a single destination.
Considerations
The comparison between the approaches is:
- Approach 1: The burden of management.
- Approach 2: Latency to land data within a single schema.
Your understanding of the effect on your business should determine which approach is best for you.