MongoDB Sharded Cluster Setup Guide link
Follow these instructions to connect your MongoDB sharded cluster to your destination using Fivetran.
Setup instructionslink
Find host identifiers link
You need the MongoDB sharded cluster's host identifier to configure Fivetran. The sharded cluster's host identifier has one of the following formats:
Format | Example |
---|---|
SRV host identifier | mongodb+srv://server.example.com |
Connection string | mongodb://query-router-00.example.com:27016,query-router-01.example.com:27016,query-router-02.example.com:27016 |
Domain and port of the mongos query router | query-router-00.example.com:27016 |
IP address and port of the mongos query router | 1.2.3.4:27016 |
NOTE:
For SRV host identifier and Connection string, we support connection tags
Optionally, you can use Analytics nodes to isolate queries from your operational workload and improve cluster reliability and performance, by specifying a connection string similar to the following example:
mongodb+srv://example.mongodb.net/?readPreference=secondary&readPreferenceTags=nodeType:ANALYTICS
Optionally, you can specify the read preference in the connection string based on your priority as shown below:
Read Preference Priority secondaryPreferred
(default)Optimal performance primaryPreferred
orsecondaryPreferred
High availability nearest
Optimal performance for geographically distributed data Using the
primary
/primaryPreferred
read preference might affect operational performance of your cluster
Find your host identifiers using either MongoDB Atlas or the MongoDB shell.
MongoDB Atlaslink
Log in to the MongoDB Atlas dashboard.
In the Cluster Overview tab, click Connect.
Select Connect your application.
Copy the SRV host identifier.
MongoDB shelllink
- Connect to your sharded cluster using the MongoDB shell.
- Execute the
sh.status(true)
command. - Copy the host identifier and optionally the alternative host identifiers from
active mongoses
.
Allow database accesslink
Create a database user for Fivetran using either MongoDB Atlas or the MongoDB shell.
MongoDB Atlaslink
- Log in to the MongoDB Atlas dashboard.
- In the left-hand navigation menu, go to Security > Database Access.
- Click New Database User.
- Choose the password authentication method.
- Enter the username and password for the new Fivetran user.
- In the Database User Privileges drop-down menu, select Grant Specific User Privileges.
- Under Specific Privileges, add the following roles/privileges:
readAnyDatabase
clusterMonitor
(Only for MongoDB version below 4.0)
- Click Add User.
TIP: To learn how to restrict the Fivetran user's access to a subset of the databases, see the Excluding source data section.
For more information, see MongoDB Atlas' Configure Database Users documentation.
MongoDB shelllink
Create an identical Fivetran user in every primary shard node and the mongos
query router. Replace <username>
and <password>
with a username and password of your choice.
use admin
db.createUser({
user: "<username>",
pwd: "<password>",
roles: [ "readAnyDatabase", "clusterMonitor" ]
})
You can pick any username and password, but it must be consistent across all primary shard nodes and the mongos
query router.
NOTE: The Mongo shell command to create a user is
db.AddUser()
for versions 2.x.
If you want to find out which shards are connected to the mongos
query router, use the command db.adminCommand({ listShards: 1 })
. For more information, see MongoDB's Built-In Roles documentation.
Choose connection methodlink
Decide whether to connect Fivetran to your MongoDB cluster directly, using an SSH tunnel, or using a private link. For more information about the connection options, see How to Choose the Right Database Connection Option.
TIP: To enable TLS on your replica set, follow MongoDB's TLS Configuration instructions.
Connect directly (TLS required)link
Fivetran connects directly to your MongoDB cluster. This is the simplest method. To connect directly, configure your firewall and/or other access control systems to allow incoming connections to your MongoDB cluster from Fivetran's IPs for your cluster region. For more information, see MongoDB's Security documentation.
Use either MongoDB Atlas or the MongoDB shell.
MongoDB Atlaslink
- Log in to the MongoDB Atlas dashboard.
- In the Cluster Overview tab, make a note of MongoDB cluster cloud service provider and region.
- In the left-hand navigation menu, go to Security > Network Access.
- Click Add IP address.
- Allow Fivetran's IP for your MongoDB cluster's cloud service provider and region.
- In the Access List Entry field, enter the Fivetran IP and click Confirm.
MongoDB shelllink
For steps to safelist Fivetran's IPs, see MongoDB's Security Considerations documentation.
Connect using SSH (TLS optional)link
Fivetran connects to a separate server in your network that provides an SSH tunnel to your cluster. You must connect through SSH if your database is in an inaccessible subnet.
To connect using SSH, configure your firewall and/or other access control systems to allow incoming connections to your MongoDB port (usually 27017
) from your SSH tunnel server's IP.
Before you proceed to the next step, you must follow our SSH connection instructions. If you want Fivetran to tunnel SSH over TLS, follow MongoDB's TLS Configuration instructions to enable TLS on your cluster.
(Optional) Connect using AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect Betalink
IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect.
MongoDB Atlaslink
You can connect Fivetran to your MongoDB Atlas database using either AWS PrivateLink or Azure Private Link. Fivetran uses your chosen service to move your data securely between our system and your MongoDB Atlas database.
Configure a private endpoint service using either the Atlas CLI or Atlas UI depending on your service and cloud provider in the destination.
Atlas CLI
- Make sure that the status of the private endpoint service is Available.
NOTE: It can take several minutes for the service to be available. If the status is not Available, wait for some time till the endpoint service is Available.
- Run the
atlas privateEndpoints
command.describe - Send the response to your Fivetran account manager.
Atlas UI
- During configuration using the Atlas UI, a Creating Atlas Endpoint Service message is displayed on the setup dialogue box. Close the dialog box.
- Go to the Network Access > Private Endpoint page.
- Copy the value of the Atlas Endpoint Service and send it to your Fivetran account manager.
- Make sure that the status of the private endpoint service is Available.
We create an interface endpoint inside the Fivetran network. We send you the ID and IP address of the endpoint.
NOTE: We don't send the IP address for AWS PrivateLink interface endpoints.
Finalize the private endpoint configuration on MongoDB Atlas using the Atlas CLI or Atlas UI:
Atlas CLI
AWS PrivateLink
- Use the
atlas privateEndpoints aws interfaces create <endpointServiceId> --privateEndpointId <endpointInterfaceId>
command. - Replace the values marked with
<>
.endpointServiceId
is the ID of the private endpoint service created above andendpointInterfaceId
is the interface endpoint ID provided by Fivetran.
NOTE: For more information, see MongoDB documentation.
Azure Private Link
- Use the
atlas privateEndpoints azure interfaces create <endpointServiceId> --privateEndpointId <endpointInterfaceId> --privateEndpointIpAddress <endpointInterfaceIPAddress>
command. - Replace the values marked with
<>
.endpointServiceId
is the ID of the private endpoint service created above,endpointInterfaceId
is the interface endpoint ID provided by Fivetran, andendpointInterfaceIPAddress
is the interface endpoint IP address provided by Fivetran.
NOTE: For more information, see MongoDB documentation.
Atlas UI
- Go to the Network Access > Private Endpoint page.
- In the Actions column, click the Edit button.
- Fill out the fields in a dialog with values provided by Fivetran and click Create.
- On the Network Access / Private Endpoint page, make sure that the Endpoint status and Atlas Endpoint Service Status is Available.
- Use the
NOTE: Use either a DNS seed list connection or standard connection string to configure the MongoDB Fivetran connector.
AWS-, Azure-, or GCP-hosted MongoDBlink
You can also connect Fivetran to a MongoDB database hosted on AWS, Azure or GCP VMs using Private Link. Learn how in our AWS Private Link setup guide, Azure Private Link setup guide, or Google Cloud Private Service Connect setup guide.
(Optional) Set oplog sizelink
Set the oplog size so that it can retain at least 24 hours' worth of changes. We recommend increasing the size to retain seven days' worth of data.
Adjust your oplog size using either MongoDB Atlas or the MongoDB shell:
- MongoDB Atlas: Follow MongoDB Atlas' Set Oplog Size tutorial.
- MongoDB shell: Follow MongoDB's Change the Size of the Oplog tutorial.
Choose pack modelink
Choose between packed mode or unpacked mode. By default unpacked
mode is selected, where Fivetran unpacks one layer of nested fields and infer types. In packed mode, we write the data without unpacking.
For more information, see our Pack mode documentation.
Finish Fivetran configurationlink
In your connector setup form, enter a Destination schema prefix. This prefix cannot be changed once your connector is created.
In the Host and port fields, enter the mongos host and port.
Enter the Fivetran-specific User that you created in Step 2.
Enter the Password for the Fivetran-specific user that you created in Step 2.
Choose your Connection Method. If you selected Connect via an SSH tunnel, provide the following information:
- SSH hostname (do not use a load balancer's IP address/hostname)
- SSH port
- SSH user
- If you enabled SSL/TLS on your database in Step 3, set the Require TLS through tunnel toggle to ON.
Click Save & Test. Fivetran tests and validates our connection to your MongoDB sharded cluster. Upon successful completion of the setup tests, you can sync your data using Fivetran.
Setup testslink
Fivetran performs the following tests to ensure that we can connect to your MongoDB cluster and that it is properly configured:
- The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It then checks that we can connect to your database using the SSH Tunnel. (We skip this test if you aren't connecting using SSH.)
- The Validate Host Test validates the database credentials you provided in the setup form. It then verifies that the database host is not private and checks that we can connect to the host.
- The Validating Certificate Test generates a pop-up window where you must choose which certificate you want Fivetran to use. Select the root certificate to trust all the configured hosts. It then validates that certificate and checks that we can connect to your database using TLS. (We skip this test if you aren't connecting directly.)
- The Connecting to Host Test connects to your database instance and checks if we can access the schemas in your database.
- The Database Access Test verifies if we have the permissions to query at least one collection.
- The Change Streams/Oplog Access Test checks if we can access the change streams. If we can't access the change streams, it then verifies if we can access the oplog and if the oplog contains at least 24 hours' worth of changes.
NOTE: The tests may take a few minutes to finish running.
Related articleslink
description Connector Overview
account_tree Schema Information
settings API Connector Configuration