Destination Connection Options
There are several ways to connect Fivetran to your destination:
- Directly by safelisting Fivetran's IP
- Using an SSH tunnel
- Using a reverse SSH tunnel
- Using AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect (Business Critical only)
- Using Private Google Access
- Using a VPN tunnel
Safelist Fivetran's IP
The fastest and easiest way to connect is to allow Fivetran's IP direct access to your destination port. For more information about how to do this, visit the setup guide for your destination.
SSH Tunnel
If it's not possible to provide direct access to the destination port, you can connect to Fivetran using an SSH tunnel. You can also choose this connection method for an added layer of security. To connect using an SSH tunnel, you need to:
- Set up an SSH tunnel server that has access to your destination port. The tunnel server's SSH port needs to be accessible from Fivetran's IP.
- Create an SSH user for Fivetran.
IMPORTANT: Fivetran generates a unique public SSH key for each destination. We support multiple connectors with a single SSH tunnel depending on the data volume and network bandwidth.
Follow the SSH tunnel setup instructions for your operating system.
- Linux
- Windows
Linux
Expand for instructions
Create SSH user
To create an SSH user, do the following:
Log in to your SSH tunnel host.
Create group
fivetran
. Execute:sudo groupadd fivetran
Create user
fivetran
. Execute:sudo useradd -m -g fivetran fivetran
Switch to the
fivetran
user. Execute:sudo su - fivetran
Create the
.ssh
directory. Execute:mkdir ~/.ssh
Set permissions. Execute:
chmod 700 ~/.ssh
Change to the
.ssh
directory. Execute:cd ~/.ssh
Create the
authorized_keys
file. Execute:touch authorized_keys
Set permissions. Execute:
chmod 600 authorized_keys
Add the public SSH key from the Fivetran destination setup form to the
authorized_keys
file, using your favorite text editor. The key must be all on one line. Make sure that you don’t introduce any line breaks when cutting and pasting. The public SSH key is generated uniquely for each Fivetran destination.
IMPORTANT: If you use OpenSSH with version 8.8 or higher, the use of RSA keys are disabled by default. To enable the use of RSA keys, you must modify your
sshd_config
file (in/etc/ssh
) and add the following lines:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com HostKeyAlgorithms=ssh-rsa,ssh-rsa-cert-v01@openssh.com
As an extra layer of security, Fivetran enables TLS on your SSH connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS through tunnel toggle to OFF.
IMPORTANT: If you set the Require TLS through tunnel toggle to OFF, Fivetran first attempts to connect over TLS inside the SSH tunnel. If this fails, Fivetran automatically retries the connection in clear text inside the SSH tunnel. You are responsible for configuring this option as per your corporate security policies.
Once the user is created, you'll need to allow port access.
Allow port access
Make sure that port access is allowed from:
- Fivetran's IP to your tunnel server's SSH port
- Your SSH tunnel server to your destination port
If your SSH server and destination happen to be in AWS, you can follow the instructions below to configure port access.
AWS
To configure an SSH server in AWS, open the EC2 console and select Running Instances.
Select the instance you intend to use as an SSH tunnel.
Select the Security groups and then select default.
Select the Inbound tab.
Click Edit.
Fill in Fivetran's IP and your SSH port (do not use a load balancer).
For VPC or EC2 classic, add a security rule.
Select SSH, enter Fivetran's IP, and click Save.
To complete setting up your destination connector, follow the setup instructions for your specific destination. You can confirm your server's SSH key by comparing the SHA 256 displayed when running the setup tests.
OpenSSH
Expand for instructions
Install OpenSSH
TIP: Learn more in Microsoft's OpenSSH for Windows overview documentation.
Install the sshd server.
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Verify that the OpenSSH server is installed.
Get-WindowsCapability -Online | ? Name -like 'OpenSSH.Server*'
NOTE: If OpenSSH is installed, you'll see the following message:
Name : OpenSSH.Server~~~~0.0.1.0 State : Installed
Set the firewall to allow inbound TCP connections on port 22.
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
Start both the sshd service and the ssh-agent.
Start-Service sshd Set-Service -Name sshd -StartupType 'Automatic' Start-Service ‘ssh-agent’ Set-Service -Name ‘ssh-agent’ -StartupType 'Automatic'
Create Fivetran user and group
IMPORTANT: Windows does not allow a user and a group to have the same name.
Add a local Fivetran user.
net user fivetran <password> /add /comment:"User for establishing SSH connection to Fivetran service." /passwordchg:no /passwordreq:no /logonpasswordchg:no
Add a Fivetran group.
net localgroup fivetran-group /comment:"Group for establishing SSH connection to Fivetran service." /add
In Windows command prompt, switch to the SSH server directory.
cd C:\ProgramData\ssh\ && start notepad .\sshd_config
Allow password authentication for the Fivetran user.
PasswordAuthentication yes
Allow the Fivetran user to connect to the SSH server. Add the following line to the
sshd_config
file.AllowUsers fivetran
If your Windows build is 1809 or later, comment out the following lines in the
sshd_config
file:# Match Group administrators # AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
Save the
sshd_config
file.Restart the agent and the sshd service. If the following command fails, you can restart from the Task Manager (
Alt
+Ctrl
+Delete
).Restart-Service ssh-agent; Restart-Service sshd
IMPORTANT: If you use OpenSSH with version 8.8 or higher, the use of RSA keys are disabled by default. To enable the use of RSA keys, you must modify your
sshd_config
file and add the following lines:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com HostKeyAlgorithms=ssh-rsa,ssh-rsa-cert-v01@openssh.com
Set up client
In your command line, go into the SSH server using Windows VM.
ssh fivetran@<Public IPv4 DNS>
Enter your password.
Create an
.ssh
folder in your home directory.mkdir .ssh
Add an
authorized_keys
file.If your client is Windows PowerShell:
type nul > authorized_keys echo <fivetran-ui-public-key> >> authorized_keys icacls C:\Users\fivetran\.ssh\authorized_keys /inheritance:r
If your client is Linux:
touch authorized_keys chmod 600 authorized_keys echo <fivetran-ui-public-key> >> authorized_keys
If you're running PowerShell in elevated mode, your setup is complete. If you're not running PowerShell in elevated mode, follow the instructions below.
TIP: An elevated PowerShell prompt displays
Administrator: Windows PowerShell
on the top of the prompt's border.
Finish setup (non-elevated PowerShell only)
Allow public key authentication.
PubkeyAuthentication yes
Remove password authentication.
PasswordAuthentication no PermitEmptyPasswords no
Save the
sshd_config
file.Verify that inheritance has been disabled and remove
Administrator
.
a. Right click on theauthorized_keys
file.
b. Select Properties.
c. Select the Security tab.
d. Select Advanced.
e. Verify that the bottom left reads Enable Inheritance, which means that inheritance is disabled.
f. RemoveAdministrator
from the file security permissions.Restart the agent and the sshd service.
Restart-Service ssh-agent; Restart-Service sshd
PuTTY/KiTTY
Expand for instructions
Generate private key
Launch PuTTYgen.
Under Actions, click Generate to generate a public/private key combination. The OpenSSH public key appears in the top box.
Make a note of your OpenSSH public key. You will need this later.
Click Save private key. When you're prompted to save a .ppk file, save the file on your desktop.
TIP: You do not need to save the key with a passphrase.
Configure connection
Launch KiTTY (run kitty-0.74.4.7.exe on your desktop).
Configure your session.
- Host Name: Enter
fivetran@<GCP VM IP>
- Port: 22
- Host Name: Enter
In the left menu, click Auth.
In the Private key file for authentication field, enter the private key you created in the previous step.
Click Tunnels.
In the Port forwarding section, make sure that the following checkboxes are selected:
- Local ports accept connections from other hosts
- Remote ports do the same
In the Source port field, add the high port (
55432
).In the Destination field, add your destination address and its corresponding port (for example,
localhost:5432
for a server hosted on the same VM).Make sure that Remote is selected.
Click Add to add a port forwarding rule.
In the left menu, go to Connection.
Set the seconds between keepalives value to 5.
Select both Reconnect options.
Return to the Tunnels page.
In the Saved Sessions/New Folder field, enter a name for this connection.
Hit Save. Another session populates in the bottom box.
Click Open, then leave the session open.
Reverse SSH tunnel
You can also connect Fivetran to your destination using a reverse SSH tunnel if you are unable to provide direct port access to your destination instance.
Follow the Reverse SSH tunnel setup instructions for your operating system.
Linux
Expand for instructions
To set up a reverse SSH tunnel to connect to Fivetran:
Contact Fivetran's Sales Engineers and provide the following SSH keys:
Your SSH public key. To generate your SSH public key, do the following on your SSH host:
i. Generate an SSH key pair. Execute:
ssh-keygen
.ii. View the contents of the public key. Execute:
cat ~/.ssh/id_rsa.pub
.
Copy the public key and send it to Fivetran's sales engineers along with the Fivetran user's SSH public key.
Collect the following information to complete your setup:
- SSH tunnel username (contact Fivetran's Support to get this)
- Reverse SSH port (contact Fivetran's Support to get this)
- Internal IP address or name of the local destination host machine
- Internal open port for communication with the destination host
- File path to the private key on the SSH host machine (this is normally
id_rsa.pem
or simplyid_rsa
)
Use the values above to replace the placeholder variables in the following script, then run it on the SSH host in a single line:
autossh -M 0 -f -N -R <SSH_HIGH_PORT>:<LOCAL_DB_MACHINE_NAME_OR_IP>:<LOCAL_DB_MACHINE_PORT> <FIVETRAN_SSH_USERNAME>@<FIVETRAN_SUPPLIED_IP> -g -i <PATH_TO_PRIVATE_KEY> -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o ExitOnForwardFailure=yes
TIP: To track the progress of this script, remove the
-f
flag and add the-v
flag to enable verbose logging. Without the flag, you will not see confirmation when the script finishes running successfully.NOTE: If you use this
autossh
script again later for the same SSH high port, you need to terminate your originalautossh
script before proceeding.After establishing a successful Reverse SSH connection, enter the following into the Fivetran setup form for your destination. Replace the fields in { brackets } with your own values.
Field Value Description Host localhost Allows your SSH host to handle port routing Port { SSH high port } e.g., 13306. The port that your SSH host will translate User { Destination user } Password { Destination user's password } Database { Database name } The database name you want to replicate to Connection Method Connect using SSH Tunnel SSH Host { IP Address } Supplied by Fivetran SSH Port 22 SSH User fivetran
PuTTY
Expand for instructions
Start a new PuTTY session.
Configure your session.
- Host Name: Enter the Fivetran server
- Port: 22
Go to Connection > Data, then enter
fivetran
as the tunnel username.In the left menu, go to Connection > SSH.
Select the Don’t start a shell or command at all checkbox.
Go to Connection > SSH > Auth.
Click Browse to find your PPK private key.
Under Auth, go to TTY.
Select the Don’t allocate a pseudo terminal checkbox.
Go to Connection > SSH > Tunnel.
In the Source port field, enter the connection port assigned to your connection (for example,
53359
).In the hostname:port field, enter your destination's host and port (for example,
127.0.0.1:5432
).Make sure that Remote is selected.
Click Add.
To save the connection, click Session in the left menu.
In the Saved Sessions section, enter a name for your stored session, then click Save.
Double-click the session name to initiate the connection.
AWS PrivateLink
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method.
Fivetran uses PrivateLink to connect and move your data securely between our system and your AWS-hosted destination.
By default, Fivetran enables TLS on your PrivateLink connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using PrivateLink toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Architecture
The following graphic illustrates how Fivetran connects to a destination using AWS PrivateLink:
Supported destinations
We support AWS PrivateLink for the following AWS-hosted destinations:
- Aurora PostgreSQL
- Amazon RDS for PostgreSQL
- Aurora MySQL
- AWS MSK
- Databricks
- Generic MySQL
- Generic PostgreSQL
- Generic SQL Server
- MariaDB
- Amazon RDS for MariaDB
- Amazon RDS for MySQL
- Redshift
- S3 Data Lake
- Snowflake
- Amazon RDS for SQL Server
Prerequisites
To set up AWS PrivateLink, you need the following:
- An Amazon-hosted destination in one of our supported regions
- An AWS endpoint service configured for your destination
- If your destination is hosted on one of the Amazon services listed below, you must create a Network Load Balancer (NLB) in your VPC. The NLB receives requests from Fivetran and routes it to your destination.
- Amazon Aurora
- Amazon EC2
- Amazon RDS
Setup instructions for AWS PrivateLink
Create an NLB
On a single IP service (EC2, non-RDS destination, etc.)
To create an NLB on a single IP service, follow the instructions in AWS’ documentation.
IMPORTANT: While creating the NLB, you must either ensure that the availability zones of the NLB match the target availability zones or enable cross-zone load balancing.
On RDS
NLB can only route traffic to an EC2 instance, an IP address, or a Lambda function through target groups. Since RDS does not have a dedicated IP address or EC2 instance ID, you can configure the NLB in one of the following ways to route the traffic to your RDS destination:
- Using a port forwarding instance
- Using an RDS IP address
Using a port forwarding instance
Deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS destination). You can use the following sample script to set up the EC2 port forwarding instance:
#!/bin/bash PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart. if test -f "$PREVLOGFILE"; then truncate -s 0 $PREVLOGFILE echo "State file $PREVLOGFILE has been emptied" fi python -m SimpleHTTPServer 801 & # NOTE: USE PORT 801 FOR <HEALTH_CHECKS> PARAMETER BELOW echo 1 -> /proc/sys/net/ipv4/ip_forward export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. iptables -t nat -A POSTROUTING -j MASQUERADE while true do LOGFILE=/root/ip.txt Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked. if [ $LOGFILE = "" ] ; then iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT echo $Current_IP > $LOGFILE else Old_IP=$(cat $LOGFILE) if [ "$Current_IP" = "$Old_IP" ] ; then echo "IP address has not changed ($Old_IP -> $Current_IP)" else iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT sysctl net.ipv4.ip_forward=1 iptables-save echo $Current_IP > $LOGFILE echo "IP address has changed ($Old_IP -> $Current_IP)" fi fi sleep 5 done
NOTE: If you are using Amazon Linux 2023, replace
python -m SimpleHTTPServer 801
withpython3 -m http.server 801
in line 6 of the script.Configure the NLB listener and target group to route traffic to the port forwarder EC2 instance.
Using an RDS IP address
When you set up an RDS destination, RDS provides an endpoint to access your destination. This endpoint resolves to an IP address. AWS does not recommend using this IP address, since it can change without notice. To work around this limitation, you can deploy a lambda function to periodically check the IP address and update the NLB target group when it changes.
To use the RDS IP address in your NLB target group, do the following:
Run the
nslookup
ordig
command with the domain name of your RDS endpoint as the input to find the IP address:dig +short YOUR_RDS_DNS_ENDPOINT
Set up your NLB target group with the IP address.
Deploy a lambda function to periodically perform
nslookup
on the RDS endpoint to check if the IP address has changed and update the target group with the new IP address.
Set up AWS PrivateLink
IMPORTANT: These setup instructions are not applicable to Databricks, Redshift, and Snowflake destinations. To connect these destinations using AWS PrivateLink, see their setup guides.
Endpoint service configurations are not in Fivetran’s control. Therefore, we recommend that you contact your AWS representatives to set up PrivateLink. However, we provide the following high-level instructions based on how customers typically configure their destinations.
Set up AWS Private Link with Fivetran’s assistance
Expand for instructions
Configure the NLB for each subnet (availability zone) where you want the service to be available.
Create a VPC endpoint service configuration and specify your NLB.
IMPORTANT: Make sure that you are familiar with the endpoint service considerations and you have met its prerequisites.
Safelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service.TIP: To learn how to safelist the Fivetran account ID, see AWS' documentation.
Open a Fivetran support ticket with the following details and we will complete the setup:
- Service name (VPCe) (for example,
com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
) - Host name of the service/destination
- Service name (VPCe) (for example,
Activate the connection by accepting the interface endpoint connection request from Fivetran.
NOTE: By default, you must manually accept the connection request. However, you can configure the acceptance settings of your endpoint service so that the connection request is automatically accepted.
Set up AWS Private Link without Fivetran’s assistance (self-service) Private Preview
Expand for instructions
The self-service functionality is not available by default. Contact our support team to enable this functionality for you.
Find connection service name
Log in to your AWS account.
Configure the NLB for each subnet (availability zone) where you want the service to be available.
Create a VPC endpoint service configuration and specify your NLB.
IMPORTANT: Make sure that you are familiar with the endpoint service considerations and you have met its prerequisites.
Safelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service.TIP: To learn how to safelist the Fivetran account ID, see AWS' documentation.
Make a note of your service name (VPCe) (for example,
com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
). You will need it to configure Fivetran.Activate the connection by accepting the interface endpoint connection request from Fivetran.
NOTE: By default, you must manually accept the connection request. However, you can configure the acceptance settings of your endpoint service so that the connection request is automatically accepted.
Configure PrivateLink connection
Log in to the Fivetran dashboard and go to the destination setup form.
In the Connection Method drop-down menu, select Connect via Private Networking.
Click + Configure a new PrivateLink connection.
Enter a Name for your PrivateLink connection.
In the PrivateLink connection service name field, enter the service name you found in Step 1.
Click Create and save. Fivetran will raise a connection request in AWS.
NOTE: It might take up to 10 minutes for us to raise the connection request.
Accept connection request
Go to your AWS account.
In the top left corner, click Services, and then select VPC.
In the VPC dashboard, click Endpoint services.
Refresh the Endpoint services page to see the new connection request from Fivetran. The new connection request is in Pending acceptance state.
NOTE: It might take some time for the new connection request to appear on the list.
Select the new connection request.
Go the Endpoint connections tab.
In the Actions drop-down menu, select Accept endpoint connection request.
In the Accept endpoint connection request pop-up window, enter
accept
in the text box, and then click Accept. The status of the request will change to Pending. After a few minutes, the status will change to Accepted and a Connection created message will appear in your destination setup form.
Postrequisites
To use AWS PrivateLink, you must select AWS as the Cloud service provider in the destination setup form.
Azure Private Link
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted services to communicate with one another without exposing traffic to the public internet.
Fivetran uses Private Link to move your data securely between our system and your Azure-hosted or Azure Virtual Machine-hosted destination.
By default, Fivetran enables TLS on your Private Link connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using Private Link toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Link toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Architecture
Depending on your destination type, Private Link works in the following ways:
- If your destination is hosted on Azure, Fivetran sets up the Private Link connection to customer's private-link resource based on the fully-qualified resource ID. For more information. see Azure's documentation.
- If your destination is hosted on an Azure Virtual Machine, you can use Azure Private Link Service to connect Fivetran to your destination. To use Azure Private Link Service, you must create an Azure Private Link Service with port forwarding VMs. The VMs then relay the network traffic towards your destination.
The following graphic illustrates how Fivetran connects to a destination using Azure Private Link:
Supported destinations
We support Azure Private Link for the following destinations:
- Azure Data Lake Storage
- Azure PostgreSQL Database
- Azure SQL Database
- Azure SQL Managed Instance
- Azure Synapse
- Databricks
- Generic MySQL
- Generic PostgreSQL
- Generic SQL Server
- MariaDB
- Snowflake
Prerequisites
To set up Azure Private Link, you need an Azure-hosted or Azure Virtual Machine-hosted destination in one of our supported regions.
Setup instructions for Azure-hosted destinations
Expand for instructions
IMPORTANT: These setup instructions are not applicable to Azure Data Lake Storage, Azure Synapse, Databricks, and Snowflake destinations. To connect these destinations using Azure Private Link, see their setup guides.
Instructions for destinations
- Verify that your destination supports Private Endpoint. For more information, see Microsoft’s documentation.
- Open a Fivetran support ticket and provide your Azure-hosted service’s fully-qualified resource ID, including the resource name and resource type of your destination.
- Wait to receive Private Endpoint request details from Fivetran. We create a Private Endpoint using your resource ID, type, and subresource. We then initiate a Private Link connection request as part of the Private Endpoint setup and share the details of that request with you.
- In the Azure Portal or CLI, verify and approve the Private Link connection request from Fivetran. Once you approve the request, Fivetran completes the Private Link setup for your Azure-hosted service.
Self-service instructions for destinations Private Preview
The self-service functionality is not available by default. Contact our support team to enable this functionality for you.
Find resource ID
- Verify that your destination supports Private Endpoint. For more information, see Microsoft’s documentation.
- Log in to the Azure portal and then go to your Azure-hosted service.
- Select your workspace and then go to Properties.
- Make a note of the Resource ID. You will need it to create the Private Link connection.
Configure Private Link connection
Log in to the Fivetran dashboard and go to the destination setup form.
In the Connection Method drop-down menu, select Connect via Private Networking.
Click + Configure a new PrivateLink connection.
Enter a Name for your Private Link connection.
In the PrivateLink Resource ID field, enter the resource ID you found in Step 1.
In the PrivateLink Subresource name drop-down menu, select the subresource corresponding to your resource type. The following table lists the subresources corresponding to each resource type:
Resource Type Subresource Azure SQL Managed Instance managedInstance
Azure SQL Database sqlServer
Azure Blob Storage (Azure Storage) blob
,dfs
Azure Service Bus namespace
Azure PostgreSQL Database postgresqlServer
Azure Cosmos DB sql
Azure Mongo DB MongoDB
Click Create and save. Fivetran will raise a connection request in Azure.
NOTE: It might take up to 10 minutes for us to raise the connection request.
Accept connection request
- Go to your Azure account.
- Go to Private Link Center > Pending connections.
- Select the connection request from Fivetran, and click Approve.
- In the pop-up window, confirm that you want to approve the connection request. Once Azure processes the request, a Connection created will appear in your destination setup form.
Setup instructions for Azure Virtual Machine-hosted destinations
Expand for instructions
IMPORTANT: These setup instructions are not applicable to Azure Data Lake Storage, Azure Synapse, Databricks, and Snowflake destinations. To connect these destinations using Azure Private Link, see their setup guides.
Depending on whether or not your destination already has an Azure Private Link Service, do one of the following:
- If your destination is already configured with a Private Link Service, follow the setup instructions for Azure-hosted destinations.
- If your destination does not have a Private Link Service, set up the service and then follow the setup instructions for Azure-hosted destinations. For more information about setting up a Private Link Service, see the Setup instructions for Private Link Service section of this documentation.
NOTE: A single Azure Private Link Service can support one or multiple destinations at the same time with different ports mapped to corresponding destination IP addresses.
Setup instructions for Private Link Service
Prerequisites
To set up Private Link Service, you must do the following:
- Configure a VNet with access to your destination.
- Ensure that your destination is accessible from the VNet where the Private Link Service Load Balancer will run.
Setup instructions
In the VNet, create a Standard Load Balancer with a NIC-based backend pool that has access to your destination.
Configure a health probe and a load balancer rule with the ports that your destination will use. For more information, see Azure's documentation.
Create a Private Link Service associated with the Standard Load Balancer you created in step 1. For more information, see Azure's documentation.
Create one or more Virtual Machines and place them in the backend pool of the load balancer you created in step 1.
Enable IP forwarding on each of the backend Virtual Machine network interfaces. For more information about how to enable IP forwarding, see Azure's documentation.
For security reasons, ensure that no public IP addresses are configured for the Virtual Machines. Private Link connections use only private IP addresses.
Log in into each backend Virtual Machine of the Load Balancer and run the script below. The script configures Network Address Translation (NAT), which forwards network packets from SLB-backend Virtual Machines to your destination. The script also configures a single port forwarding. However, you can add additional sets of iptables
PREROUTING
andPOSTROUTING
rules for additional ports.#!/bin/bash # This script configures Network Address Translation to forward incoming packets # from the Azure Standard Load Balancer to IP-based destinations and route them # back. In this script, "destination" means a destination server that Fivetran connects to. # local port where Load Balancer sends traffic to SOURCE_PORT=<local_port> # destination server inside internal network DESTINATION_IP=<destination_server_ip_address> DESTINATION_PORT=<destination_server_port> # enable IP forwarding on host echo 1 > /proc/sys/net/ipv4/ip_forward # clear existing iptables rules and chains iptables -F iptables -t nat -F iptables -X # change the packet recipient from local to destination socket (host & port) iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT} # change the source IP address from the LB NAT IP address to the IP of this LB-backend host iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
NOTE: For additional help, read Azure’s Private Link Service documentation or contact our support team.
Postrequisites
To use Azure Private Link, you must do the following in the destination setup form:
- Select Azure as the Cloud service provider.
- Enter all the Azure Private Link Service ports mapped to destination IP addresses.
Google Cloud Private Service Connect Beta
Google Cloud Private Service Connect allows VPCs and Google-hosted or on-premises services to communicate with one another without exposing traffic to the public internet.
Fivetran uses Private Service Connect to move your data securely between our system and your Google Cloud-hosted destination.
By default, Fivetran enables TLS on your Private Service Connect connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using Private Service Connect toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Service Connect toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Architecture
The following graphic illustrates how Fivetran connects to a destination using Google Cloud Private Service Connect:
Supported destinations
We support Private Service Connect for the following destinations:
- Databricks
- Generic MySQLBeta
- Generic PostgreSQL
- Generic SQL Server
- Google Cloud PostgreSQL
- MariaDBBeta
- Snowflake
IMPORTANT: The setup instructions below are not applicable to Snowflake destinations. To connect a Snowflake destination using Google Cloud Private Service Connect, see our Snowflake setup guide.
Prerequisites
To set up Google Cloud Private Service Connect, you need a GCP-hosted destination and a Fivetran instance running in one of our supported regions.
Setup instructions for Google Cloud Private Service Connect
Expand for instructions
We support connecting to any GCP-hosted resource as long as it is supported by Fivetran and exposed through Private Service Connect producer.
In the following example, we publish a service that runs on a specific instance. However, there are several other ways to expose your service. To learn more, contact your administrators or read Google's documentation.
You need the following as inputs for the commands used in the steps below:
<NETWORK>
- VPC network in which the exposed resource exists<SUBNET>
- Subnetwork where the exposed resource exists<ILB_SUBNET>
- Subnetwork used for allocation of internal load balancer addresses (forwarding rules)<PSC_NAT_SUBNET>
- Subnetwork used for allocation IPs for each customer endpoint address<REGION>
- Region where the exposed resource exists<ZONE>
- Zone where the exposed resource exists<VM_NAME>
- VM on which the exposed resource runs<VM_IP>
- Private IP on which the exposed resource is available<RESOURCE_PORT>
- Port on which the exposed resource is available in a VM<NETWORK_ENDPOINT_GROUP>
- Network endpoint group<HEALTH_CHECKS>
- Port health checks<BACKEND_SERVICES>
- Backend services<FORWARDING_RULE>
- Forwarding rule<SERVICE_ATTACHMENT>
- Service attachment
IMPORTANT: If you already have a regional internal load balancer for your resource, skip to Step 8 of this example.
Create a network endpoint group.
gcloud compute network-endpoint-groups create --network <NETWORK> --subnet <SUBNET> \ --network-endpoint-type gce-vm-ip --zone <ZONE> <NETWORK_ENDPOINT_GROUP>
Add an instance with the running resource as an endpoint to the network endpoint group.
gcloud compute network-endpoint-groups update --zone <ZONE> <NETWORK_ENDPOINT_GROUP> \ --add-endpoint='instance=<VM_NAME>
Create health checks to automatically enable and disable the instance. In this example, we use port checks.
gcloud compute health-checks create tcp --region <REGION> --check-interval=60s --port=<RESOURCE_PORT> <HEALTH_CHECKS>
Create the backend services.
gcloud compute backend-services create --region=<REGION> --health-checks=<HEALTH_CHECKS> \ --health-checks-region=<REGION> --load-balancing-scheme=INTERNAL <BACKEND_SERVICES>
Assign the network endpoint group you created at the backend.
gcloud compute backend-services add-backend <BACKEND_SERVICES> --network-endpoint-group=<NETWORK_ENDPOINT_GROUP> \ --network-endpoint-group-zone=<ZONE>
(Optional) Allocate a subnetwork dedicated to forwarding rules.
cloud compute networks subnets create --network <_NETWORK_> --region <REGION> --range=10.0.X.0/24 <ILB_SUBNET>
Create a forwarding rule for the internal load balancer. It points to
<BACKEND_SERVICES>
and allocates the IP address from<ILB_SUBNET>
in<NETWORK>
.gcloud compute forwarding-rules create --backend-service=<BACKEND_SERVICES> --region <REGION> \ --load-balancing-scheme=INTERNAL --ports=ALL --subnet=<ILB_SUBNET> --network=<NETWORK> <FORWARDING_RULE>
Allocate a NAT subnetwork dedicated to creating endpoints for each connected customer.
cloud compute networks subnets create --network <NETWORK> --region <REGION> --range=10.0.X.0/24 \ --purpose=PRIVATE_SERVICE_CONNECT <_PSC_NAT_SUBNET_>
Open traffic from the ILB and NAT networks to a VM or port using the corresponding ranges specified in previous steps.
gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-psc-nats gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-ilb-nats
Create a service attachment and point it to the internal load balancer you created above (
FORWARDING_RULE
) with manual accepting mode.gcloud compute service-attachments create --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \ --region=<REGION> --description='Producer for my resource in region' --nat-subnets <_PSC_NAT_SUBNET_> <SERVICE_ATTACHMENT>
Contact our support team to set up a Private Service Connect link in Fivetran. Fivetran will provide a
PSC_CONNECTION_ID
(Private Service Connect ID), which allows you to identify the connections that come from Fivetran before you approve them. Make a note of thePSC_CONNECTION_ID
. You will need it to set up your destination.
NOTE:
- If you want to auto-approve the Fivetran project, use the
--consumer-accept-list=fivetran_donkeys=2
parameter for gcloud.- The instructions above use network endpoint groups as it is easy to attach them to existing VMs. However, if a VM is already part of the instance groups, you can use it directly as a backend services target.
- You can test a newly-created producer in another VPC by allocating an IP and creating a forwarding rule, as described in Google's documentation.
Setup instructions for Google Cloud SQL resources exposed through private service access
Expand for instructions
To set up access for Google Cloud SQL destinations through private IP, you need an additional VM to act as a forwarding proxy. You can use solutions like HAProxy or iptables for this.
NOTE: If you use the following vm script, make sure that you persist iptables rules and consider using instance groups instead of standalone VMs.
#!/bin/bash
# This script configures Network Address Translation to forward incoming packets
# from the Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a destination server that Fivetran connects to.
# local port where the Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>
# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X
# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
Once you set up the VMs, follow the standard setup instructions for Google Cloud Private Service Connect.
Postrequisites
To use Google Cloud Private Service Connect, you must select GCP as the Cloud service provider in the destination setup form.
Private Google Access
Private Google Access allows Fivetran to communicate with your destination without exposing traffic to the public internet. For more information about Private Google Access, see Google's documentation.
Fivetran's Google network is configured with Private Google Access, which means traffic between Fivetran and your destination is always private when Fivetran runs on Google Cloud (GCP).
Supported destination
We support Private Google Access for BigQuery destinations.
Setup instructions for Private Google Access
You can configure a Private Google Access connection by configuring your Fivetran destination to run on GCP.
VPN Tunnel
To connect to Fivetran using a VPN tunnel, contact Fivetran's Support for help setting up the tunnel.