Destination Connection Optionslink
Updated 7 days ago
There are several ways to connect Fivetran to your destination:
- Directly by safelisting Fivetran's IP
- Using an SSH tunnel
- Using a reverse SSH tunnel
- Using AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect (Business Critical only)
- Using Private Google Access
- Using a VPN tunnel
Safelist Fivetran's IPlink
The fastest and easiest way to connect is to allow Fivetran's IP direct access to your destination port. For more information about how to do this, visit the setup guide for your destination.
SSH Tunnellink
If it's not possible to provide direct access to the destination port, you can connect to Fivetran using an SSH tunnel. You can also choose this connection method for an added layer of security. To connect using an SSH tunnel, you need to:
- Set up an SSH tunnel server that has access to your destination port. The tunnel server's SSH port needs to be accessible from Fivetran's IP.
- Create an SSH user for Fivetran.
IMPORTANT: Fivetran generates a unique public SSH key for each destination. We support multiple connectors with a single SSH tunnel depending on the data volume and network bandwidth.
Follow the SSH tunnel setup instructions for your operating system.
- Linux
- Windows
Linuxlink
Create SSH userlink
To create an SSH user, do the following:
Log in to your SSH tunnel host.
Create group
fivetran
. Execute:sudo groupadd fivetran
content_copyCreate user
fivetran
. Execute:sudo useradd -m -g fivetran fivetran
content_copySwitch to the
fivetran
user. Execute:sudo su - fivetran
content_copyCreate the
.ssh
directory. Execute:mkdir ~/.ssh
content_copySet permissions. Execute:
chmod 700 ~/.ssh
content_copyChange to the
.ssh
directory. Execute:cd ~/.ssh
content_copyCreate the
authorized_keys
file. Execute:touch authorized_keys
content_copySet permissions. Execute:
chmod 600 authorized_keys
content_copyAdd the public SSH key from the Fivetran destination setup form to the
authorized_keys
file, using your favorite text editor. The key must be all on one line. Make sure that you don’t introduce any line breaks when cutting and pasting. The public SSH key is generated uniquely for each Fivetran destination.
IMPORTANT: If you use OpenSSH with version 8.8 or higher, the use of RSA keys are disabled by default. To enable the use of RSA keys, you must modify your
sshd_config
file (in/etc/ssh
) and add the following lines:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com HostKeyAlgorithms=ssh-rsa,ssh-rsa-cert-v01@openssh.com
content_copy
As an extra layer of security, Fivetran enables TLS on your SSH connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS through tunnel toggle to OFF.
IMPORTANT: If you set the Require TLS through tunnel toggle to OFF, Fivetran first attempts to connect over TLS inside the SSH tunnel. If this fails, Fivetran automatically retries the connection in clear text inside the SSH tunnel. You are responsible for configuring this option as per your corporate security policies.
Once the user is created, you'll need to allow port access.
Allow port accesslink
Make sure that port access is allowed from:
- Fivetran's IP to your tunnel server's SSH port
- Your SSH tunnel server to your destination port
If your SSH server and destination happen to be in AWS, you can follow the instructions below to configure port access.
AWSlink
To configure an SSH server in AWS, open the EC2 console and select Running Instances.
Select the instance you intend to use as an SSH tunnel.
Select the Security groups and then select default.
Select the Inbound tab.
Click Edit.
Fill in Fivetran's IP and your SSH port (do not use a load balancer).
For VPC or EC2 classic, add a security rule.
Select SSH, enter Fivetran's IP, and click Save.
To complete setting up your destination connector, follow the setup instructions for your specific destination. You can confirm your server's SSH key by comparing the SHA 256 displayed when running the setup tests.
OpenSSHlink
Install OpenSSHlink
TIP: Learn more in Microsoft's OpenSSH for Windows overview documentation.
Install the sshd server.
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
content_copyVerify that the OpenSSH server is installed.
Get-WindowsCapability -Online | ? Name -like 'OpenSSH.Server*'
content_copyNOTE: If OpenSSH is installed, you'll see the following message:
Name : OpenSSH.Server~~~~0.0.1.0 State : Installed
content_copySet the firewall to allow inbound TCP connections on port 22.
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
content_copyStart both the sshd service and the ssh-agent.
Start-Service sshd Set-Service -Name sshd -StartupType 'Automatic' Start-Service ‘ssh-agent’ Set-Service -Name ‘ssh-agent’ -StartupType 'Automatic'
content_copy
Create Fivetran user and grouplink
IMPORTANT: Windows does not allow a user and a group to have the same name.
Add a local Fivetran user.
net user fivetran <password> /add /comment:"User for establishing SSH connection to Fivetran service." /passwordchg:no /passwordreq:no /logonpasswordchg:no
content_copyAdd a Fivetran group.
net localgroup fivetran-group /comment:"Group for establishing SSH connection to Fivetran service." /add
content_copyIn Windows command prompt, switch to the SSH server directory.
cd C:\ProgramData\ssh\ && start notepad .\sshd_config
content_copyAllow password authentication for the Fivetran user.
PasswordAuthentication yes
content_copyAllow the Fivetran user to connect to the SSH server. Add the following line to the
sshd_config
file.AllowUsers fivetran
content_copyIf your Windows build is 1809 or later, comment out the following lines in the
sshd_config
file:# Match Group administrators # AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
content_copySave the
sshd_config
file.Restart the agent and the sshd service. If the following command fails, you can restart from the Task Manager (
Alt
+Ctrl
+Delete
).Restart-Service ssh-agent; Restart-Service sshd
content_copy
IMPORTANT: If you use OpenSSH with version 8.8 or higher, the use of RSA keys are disabled by default. To enable the use of RSA keys, you must modify your
sshd_config
file and add the following lines:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com HostKeyAlgorithms=ssh-rsa,ssh-rsa-cert-v01@openssh.com
content_copy
Set up clientlink
In your command line, go into the SSH server using Windows VM.
ssh fivetran@<Public IPv4 DNS>
content_copyEnter your password.
Create an
.ssh
folder in your home directory.mkdir .ssh
content_copyAdd an
authorized_keys
file.If your client is Windows PowerShell:
type nul > authorized_keys echo <fivetran-ui-public-key> >> authorized_keys icacls C:\Users\fivetran\.ssh\authorized_keys /inheritance:r
content_copyIf your client is Linux:
touch authorized_keys chmod 600 authorized_keys echo <fivetran-ui-public-key> >> authorized_keys
content_copy
If you're running PowerShell in elevated mode, your setup is complete. If you're not running PowerShell in elevated mode, follow the instructions below.
TIP: An elevated PowerShell prompt displays
Administrator: Windows PowerShell
on the top of the prompt's border.
Finish setup (non-elevated PowerShell only)link
Allow public key authentication.
PubkeyAuthentication yes
content_copyRemove password authentication.
PasswordAuthentication no PermitEmptyPasswords no
content_copySave the
sshd_config
file.Verify that inheritance has been disabled and remove
Administrator
.
a. Right click on theauthorized_keys
file.
b. Select Properties.
c. Select the Security tab.
d. Select Advanced.
e. Verify that the bottom left reads Enable Inheritance, which means that inheritance is disabled.
f. RemoveAdministrator
from the file security permissions.Restart the agent and the sshd service.
Restart-Service ssh-agent; Restart-Service sshd
content_copy
PuTTY/KiTTYlink
Generate private keylink
Launch PuTTYgen.
Under Actions, click Generate to generate a public/private key combination. The OpenSSH public key appears in the top box.
Make a note of your OpenSSH public key. You will need this later.
Click Save private key. When you're prompted to save a .ppk file, save the file on your desktop.
TIP: You do not need to save the key with a passphrase.
Configure connectionlink
Launch KiTTY (run kitty-0.74.4.7.exe on your desktop).
Configure your session.
- Host Name: Enter
fivetran@<GCP VM IP>
- Port: 22
- Host Name: Enter
In the left menu, click Auth.
In the Private key file for authentication field, enter the private key you created in the previous step.
Click Tunnels.
In the Port forwarding section, make sure that the following boxes are checked:
- Local ports accept connections from other hosts
- Remote ports do the same
In the Source port field, add the high port (
55432
).In the Destination field, add your destination address and its corresponding port (for example,
localhost:5432
for a server hosted on the same VM).Make sure that Remote is selected.
Click Add to add a port forwarding rule.
In the left menu, go to Connection.
Set the seconds between keepalives value to 5.
Select both Reconnect options.
Return to the Tunnels page.
In the Saved Sessions/New Folder field, enter a name for this connection.
Hit Save. Another session populates in the bottom box.
Click Open, then leave the session open.
Reverse SSH tunnellink
You can also connect Fivetran to your destination using a reverse SSH tunnel if you are unable to provide direct port access to your destination instance.
Follow the Reverse SSH tunnel setup instructions for your operating system.
Linuxlink
To set up a reverse SSH tunnel to connect to Fivetran:
Contact Fivetran's Sales Engineers and provide the following SSH keys:
Your SSH public key. To generate your SSH public key, do the following on your SSH host:
i. Generate an SSH key pair. Execute:
ssh-keygen
.ii. View the contents of the public key. Execute:
cat ~/.ssh/id_rsa.pub
.
Copy the public key and send it to Fivetran's sales engineers along with the Fivetran user's SSH public key.
Collect the following information to complete your setup:
- SSH tunnel username (contact Fivetran Support to get this)
- Reverse SSH port (contact Fivetran Support to get this)
- Internal IP address or name of the local destination host machine
- Internal open port for communication with the destination host
- File path to the private key on the SSH host machine (this is normally
id_rsa.pem
or simplyid_rsa
)
Use the values above to replace the placeholder variables in the following script, then run it on the SSH host in a single line:
autossh -M 0 -f -N -R <SSH_HIGH_PORT>:<LOCAL_DB_MACHINE_NAME_OR_IP>:<LOCAL_DB_MACHINE_PORT> <FIVETRAN_SSH_USERNAME>@<FIVETRAN_SUPPLIED_IP> -g -i <PATH_TO_PRIVATE_KEY> -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o ExitOnForwardFailure=yes
content_copyTIP: To track the progress of this script, remove the
-f
flag and add the-v
flag to enable verbose logging. Without the flag, you will not see confirmation when the script finishes running successfully.NOTE: If you use this
autossh
script again later for the same SSH high port, you need to terminate your originalautossh
script before proceeding.After establishing a successful Reverse SSH connection, enter the following into the Fivetran setup form for your destination. Replace the fields in { brackets } with your own values.
Field Value Description Host localhost Allows your SSH host to handle port routing Port { SSH high port } e.g., 13306. The port that your SSH host will translate User { Destination user } Password { Destination user's password } Database { Database name } The database name you want to replicate to Connection Method Connect using SSH Tunnel SSH Host { IP Address } Supplied by Fivetran SSH Port 22 SSH User fivetran
PuTTYlink
Start a new PuTTY session.
Configure your session.
- Host Name: Enter the Fivetran server
- Port: 22
Go to Connection > Data, then enter
fivetran
as the tunnel username.In the left menu, go to Connection > SSH.
Check the Don’t start a shell or command at all box.
Go to Connection > SSH > Auth.
Click Browse to find your PPK private key.
Under Auth, go to TTY.
Check the Don’t allocate a pseudo terminal box.
Go to Connection > SSH > Tunnel.
In the Source port field, enter the connection port assigned to your connection (for example,
53359
).In the hostname:port field, enter your destination's host and port (for example,
127.0.0.1:5432
).Make sure that Remote is selected.
Click Add.
To save the connection, click Session in the left menu.
In the Saved Sessions section, enter a name for your stored session, then click Save.
Double-click the session name to initiate the connection.
AWS PrivateLink Betalink
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method.
Fivetran uses PrivateLink to connect and move your data securely between our system and your AWS-hosted destination.
By default, Fivetran enables TLS on your PrivateLink connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using PrivateLink toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Supported destinationslink
We support AWS PrivateLink for the following AWS-hosted destinations:
- Amazon Aurora PostgreSQL
- Amazon RDS PostgreSQL
- Aurora MySQLBeta
- AWS MSK Beta
- Databricks
- Generic MySQLBeta
- Generic PostgreSQL
- Generic SQL Server
- MariaDB MySQLBeta
- MariaDB RDS MySQLBeta
- MySQL RDSBeta
- Redshift
- S3 Data Lake
- Snowflake
- SQL Server RDS
Architecturelink
The following graphic illustrates how Fivetran connects to a destination using AWS PrivateLink:
Prerequisiteslink
To set up AWS PrivateLink, you need the following:
- An Amazon-hosted destination in one of our supported regions
- An AWS endpoint service configured for your destination
- If your destination is hosted on one of the Amazon services listed below, you must create a Network Load Balancer (NLB) in your VPC. The NLB receives requests from Fivetran and routes it to your destination.
- Amazon Aurora
- Amazon EC2
- Amazon RDS
Create an NLBlink
On a single IP service (EC2, non-RDS destination, etc.)link
To create an NLB on a single IP service, follow the instructions in AWS’ documentation.
IMPORTANT: While creating the NLB, you must either ensure that the availability zones of the NLB match the target availability zones or enable cross-zone load balancing.
On RDSlink
NLB can only route traffic to an EC2 instance, an IP address, or a Lambda function through target groups. Since RDS does not have a dedicated IP address or EC2 instance ID, you can configure the NLB in one of the following ways to route the traffic to your RDS destination:
- Using a port forwarding instance
- Using an RDS IP address
Using a port forwarding instance
Deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS destination). You can use the following sample script to set up the EC2 port forwarding instance:
#!/bin/bash PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart. if test -f "$PREVLOGFILE"; then truncate -s 0 $PREVLOGFILE echo "State file $PREVLOGFILE has been emptied" fi python -m SimpleHTTPServer 801 & # NOTE: USE PORT 801 FOR <HEALTH_CHECKS> PARAMETER BELOW echo 1 -> /proc/sys/net/ipv4/ip_forward export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. iptables -t nat -A POSTROUTING -j MASQUERADE while true do LOGFILE=/root/ip.txt Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked. if [ $LOGFILE = "" ] ; then iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT echo $Current_IP > $LOGFILE else Old_IP=$(cat $LOGFILE) if [ "$Current_IP" = "$Old_IP" ] ; then echo "IP address has not changed ($Old_IP -> $Current_IP)" else iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT sysctl net.ipv4.ip_forward=1 iptables-save echo $Current_IP > $LOGFILE echo "IP address has changed ($Old_IP -> $Current_IP)" fi fi sleep 5 done
content_copyNOTE: If you are using Amazon Linux 2023, replace
python -m SimpleHTTPServer 801
withpython3 -m http.server 801
in line 6 of the script.Configure the NLB listener and target group to route traffic to the port forwarder EC2 instance.
Using an RDS IP address
When you set up an RDS destination, RDS provides an endpoint to access your destination. This endpoint resolves to an IP address. AWS does not recommend using this IP address, since it can change without notice. To work around this limitation, you can deploy a lambda function to periodically check the IP address and update the NLB target group when it changes.
To use the RDS IP address in your NLB target group, do the following:
Run the
nslookup
ordig
command with the domain name of your RDS endpoint as the input to find the IP address:dig +short YOUR_RDS_DNS_ENDPOINT
Set up your NLB target group with the IP address.
Deploy a lambda function to periodically perform
nslookup
on the RDS endpoint to check if the IP address has changed and update the target group with the new IP address.
Setup instructions for AWS PrivateLinklink
IMPORTANT: These setup instruction are not applicable to Databricks, Redshift, and Snowflake destinations. To connect these destinations using AWS PrivateLink, see their setup guides.
Endpoint service configurations are not in Fivetran’s control. Therefore, we recommend that you contact your AWS representatives to set up PrivateLink. However, we provide the following high-level instructions based on how customers typically configure their destinations:
Configure the NLB for each subnet (availability zone) in which the service should be available.
Create a VPC endpoint service configuration and specify your NLB.
IMPORTANT: Make sure that you are familiar with the endpoint service considerations and you have met its prerequisites.
Safelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service.TIP: To learn how to safelist the Fivetran account ID, see AWS' documentation.
Send the following details to your Fivetran account manager and we will complete the setup:
- Service name (VPCe) (for example,
com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
) - Host name of the service/destination
- Service name (VPCe) (for example,
Activate the connection by accepting the interface endpoint connection request from Fivetran.
NOTE: By default, you must manually accept the connection request. However, you can configure the acceptance settings of your endpoint service so that the connection request is automatically accepted.
Postrequisiteslink
To use AWS PrivateLink, you must select AWS as the Cloud service provider in the destination setup form.
Azure Private Link Betalink
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted services to communicate with one another without exposing traffic to the public internet.
Fivetran uses Private Link to move your data securely between our system and your Azure-hosted or Azure Virtual Machine-hosted destination.
By default, Fivetran enables TLS on your Private Link connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using Private Link toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Link toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Supported destinationslink
We support Azure Private Link for the following destinations:
- Azure Data Lake Storage
- Azure PostgreSQL
- Azure SQL Database
- Azure SQL Managed Instance
- Azure SynapseBeta
- Databricks
- Generic MySQLBeta
- Generic PostgreSQL
- Generic SQL Server
- MariaDB MySQLBeta
- Snowflake
Architecturelink
Depending on your destination type, Private Link works in the following ways:
- If your destination is hosted on Azure, Fivetran sets up the Private Link connection to customer's private-link resource based on the fully-qualified resource ID. For more information. see Azure's documentation.
- If your destination is hosted on an Azure Virtual Machine, you can use Azure Private Link Service to connect Fivetran to your destination. To use Azure Private Link Service, you must create an Azure Private Link Service with port forwarding VMs. The VMs then relay the network traffic towards your destination.
The following graphic illustrates how Fivetran connects to a destination using Azure Private Link:
Prerequisiteslink
To set up Azure Private Link, you need an Azure-hosted or Azure Virtual Machine-hosted destination.
Setup instructions for Azure-hosted destinationslink
IMPORTANT: These setup instructions are not applicable to Azure Data Lake Storage, Azure Synapse, Databricks, and Snowflake destinations. To connect these destinations using Azure Private Link, see their setup guides.
- Verify that your destination supports Private Endpoint. For more information, see Microsoft’s documentation.
- Contact your Fivetran account manager and provide your Azure-hosted service’s fully-qualified resource ID, including the resource name and resource type of your destination.
- Wait to receive Private Endpoint request details from Fivetran. We create a Private Endpoint using your resource ID, type, and subresource. We then initiate a Private Link connection request as part of the Private Endpoint setup and share the details of that request with you.
- In the Azure Portal or CLI, verify and approve the Private Link connection request from Fivetran. Fivetran then completes the Private Link setup for your Azure-hosted service.
Setup instructions for Azure Virtual Machine-hosted destinationslink
IMPORTANT: These setup instructions are not applicable to Azure Data Lake Storage, Azure Synapse, Databricks, and Snowflake destinations. To connect these destinations using Azure Private Link, see their setup guides.
Depending on whether or not your destination already has an Azure Private Link Service, do one of the following:
- If your destination is already configured with a Private Link Service, follow the setup instructions for Azure-hosted destinations.
- If your destination does not have a Private Link Service, set up the service and then follow the setup instructions for Azure-hosted destinations. For more information about setting up a Private Link Service, see the Setup instructions for Private Link Service section of this documentation.
NOTE: A single Azure Private Link Service can support one or multiple destinations at the same time with different ports mapped to corresponding destination IP addresses.
Setup instructions for Private Link Servicelink
Prerequisiteslink
To set up Private Link Service, you must do the following:
- Configure a VNet with access to your destination.
- Ensure that your destination is accessible from the VNet where the Private Link Service Load Balancer will run.
Setup instructionslink
In the VNet, create a Standard Load Balancer with a NIC-based backend pool that has access to your destination.
Configure a health probe and a load balancer rule with the ports that your destination will use. For more information, see Azure's documentation.
Create a Private Link Service associated with the Standard Load Balancer you created in step 1. For more information, see Azure's documentation.
Create one or more Virtual Machines and place them in the backend pool of the load balancer you created in step 1.
Enable IP forwarding on each of the backend Virtual Machine network interfaces. For more information about how to enable IP forwarding, see Azure's documentation.
For security reasons, ensure that no public IP addresses are configured for the Virtual Machines. Private Link connections use only private IP addresses.
Log in into each backend Virtual Machine of the Load Balancer and run the script below. The script configures Network Address Translation (NAT), which forwards network packets from SLB-backend Virtual Machines to your destination. The script also configures a single port forwarding. However, you can add additional sets of iptables
PREROUTING
andPOSTROUTING
rules for additional ports.#!/bin/bash # This script configures Network Address Translation to forward incoming packets # from the Azure Standard Load Balancer to IP-based destinations and route them # back. In this script, "destination" means a destination server that Fivetran connects to. # local port where Load Balancer sends traffic to SOURCE_PORT=<local_port> # destination server inside internal network DESTINATION_IP=<destination_server_ip_address> DESTINATION_PORT=<destination_server_port> # enable IP forwarding on host echo 1 > /proc/sys/net/ipv4/ip_forward # clear existing iptables rules and chains iptables -F iptables -t nat -F iptables -X # change the packet recipient from local to destination socket (host & port) iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT} # change the source IP address from the LB NAT IP address to the IP of this LB-backend host iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
content_copy
NOTE: For additional help, read Azure’s Private Link Service documentation or contact your Fivetran account manager.
Postrequisiteslink
To use Azure Private Link, you must do the following in the destination setup form:
- Select Azure as the Cloud service provider.
- Enter all the Azure Private Link Service ports mapped to destination IP addresses.
Google Cloud Private Service Connect Betalink
Google Cloud Private Service Connect allows VPCs and Google-hosted or on-premises services to communicate with one another without exposing traffic to the public internet.
Fivetran uses Private Service Connect to move your data securely between our system and your Google Cloud-hosted destination.
By default, Fivetran enables TLS on your Private Service Connect connection for an extra layer of security. We recommend that you keep TLS enabled unless you know it is safe to disable it. To disable TLS, set the Require TLS when using Private Service Connect toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Service Connect toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Supported destinationslink
We support Private Service Connect for the following destinations:
- Generic MySQLBeta
- Generic PostgreSQL
- Generic SQL Server
- Google Cloud PostgreSQL
- MariaDB MySQLBeta
- Snowflake
- BigQuery
IMPORTANT: The setup instructions below are not applicable to Snowflake destinations. To connect a Snowflake destination using Google Cloud Private Service Connect, see our Snowflake setup guide.
Architecturelink
The following graphic illustrates how Fivetran connects to a destination using Google Cloud Private Service Connect:
Prerequisiteslink
To set up Google Cloud Private Service Connect, you need a GCP-hosted destination and a Fivetran instance running in one of our supported regions.
Setup instructions for Google Cloud Private Service Connectlink
We support connecting to any GCP-hosted resource as long as it is supported by Fivetran and exposed through Private Service Connect producer.
In the following example, we publish a service that runs on a specific instance. However, there are several other ways to expose your service. To learn more, contact your administrators or read Google's documentation.
You need the following as inputs for the commands used in the steps below:
<NETWORK>
- VPC network in which the exposed resource exists<SUBNET>
- Subnetwork where the exposed resource exists<ILB_SUBNET>
- Subnetwork used for allocation of internal load balancer addresses (forwarding rules)<PSC_NAT_SUBNET>
- Subnetwork used for allocation IPs for each customer endpoint address<REGION>
- Region where the exposed resource exists<ZONE>
- Zone where the exposed resource exists<VM_NAME>
- VM on which the exposed resource runs<VM_IP>
- Private IP on which the exposed resource is available<RESOURCE_PORT>
- Port on which the exposed resource is available in a VM<NETWORK_ENDPOINT_GROUP>
- Network endpoint group<HEALTH_CHECKS>
- Port health checks<BACKEND_SERVICES>
- Backend services<FORWARDING_RULE>
- Forwarding rule<SERVICE_ATTACHMENT>
- Service attachment
IMPORTANT: If you already have a regional internal load balancer for your resource, skip to Step 8 of this example.
Create a network endpoint group.
gcloud compute network-endpoint-groups create --network <NETWORK> --subnet <SUBNET> \ --network-endpoint-type gce-vm-ip --zone <ZONE> <NETWORK_ENDPOINT_GROUP>
content_copyAdd an instance with the running resource as an endpoint to the network endpoint group.
gcloud compute network-endpoint-groups update --zone <ZONE> <NETWORK_ENDPOINT_GROUP> \ --add-endpoint='instance=<VM_NAME>
content_copyCreate health checks to automatically enable and disable the instance. In this example, we use port checks.
gcloud compute health-checks create tcp --region <REGION> --check-interval=60s --port=<RESOURCE_PORT> <HEALTH_CHECKS>
content_copyCreate the backend services.
gcloud compute backend-services create --region=<REGION> --health-checks=<HEALTH_CHECKS> \ --health-checks-region=<REGION> --load-balancing-scheme=INTERNAL <BACKEND_SERVICES>
content_copyAssign the network endpoint group you created at the backend.
gcloud compute backend-services add-backend <BACKEND_SERVICES> --network-endpoint-group=<NETWORK_ENDPOINT_GROUP> \ --network-endpoint-group-zone=<ZONE>
content_copy(Optional) Allocate a subnetwork dedicated to forwarding rules.
cloud compute networks subnets create --network <_NETWORK_> --region <REGION> --range=10.0.X.0/24 <ILB_SUBNET>
content_copyCreate a forwarding rule for the internal load balancer. It points to
<BACKEND_SERVICES>
and allocates the IP address from<ILB_SUBNET>
in<NETWORK>
.gcloud compute forwarding-rules create --backend-service=<BACKEND_SERVICES> --region <REGION> \ --load-balancing-scheme=INTERNAL --ports=ALL --subnet=<ILB_SUBNET> --network=<NETWORK> <FORWARDING_RULE>
content_copyAllocate a NAT subnetwork dedicated to creating endpoints for each connected customer.
cloud compute networks subnets create --network <NETWORK> --region <REGION> --range=10.0.X.0/24 \ --purpose=PRIVATE_SERVICE_CONNECT <_PSC_NAT_SUBNET_>
content_copyOpen traffic from the ILB and NAT networks to a VM or port using the corresponding ranges specified in previous steps.
gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-psc-nats gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-ilb-nats
content_copyCreate a service attachment and point it to the internal load balancer you created above (
FORWARDING_RULE
) with manual accepting mode.gcloud compute service-attachments create --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \ --region=<REGION> --description='Producer for my resource in region' --nat-subnets <_PSC_NAT_SUBNET_> <SERVICE_ATTACHMENT>
content_copyContact Fivetran to set up a Private Service Connect link in Fivetran. Fivetran will provide a
PSC_CONNECTION_ID
, which allows you to identify the connections that come from Fivetran before you approve them.
NOTE:
- If you want to auto-approve the Fivetran project, use the
--consumer-accept-list=fivetran_donkeys=2
parameter for gcloud.- The instructions above use network endpoint groups as it is easy to attach them to existing VMs. However, if a VM is already part of the instance groups, you can use it directly as a backend services target.
- You can test a newly-created producer in another VPC by allocating an IP and creating a forwarding rule, as described in Google's documentation.
Setup instructions for Google Cloud SQL resources exposed through private service accesslink
To set up access for Google Cloud SQL destinations through private IP, you need an additional VM to act as a forwarding proxy. You can use solutions like HAProxy or iptables for this.
NOTE: If you use the following vm script, make sure that you persist iptables rules and consider using instance groups instead of standalone VMs.
#!/bin/bash
# This script configures Network Address Translation to forward incoming packets
# from the Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a destination server that Fivetran connects to.
# local port where the Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>
# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X
# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
content_copy
Once you set up the VMs, follow the standard setup instructions for Google Cloud Private Service Connect.
Postrequisiteslink
To use Google Cloud Private Service Connect, you must select GCP as the Cloud service provider in the destination setup form.
Private Google Accesslink
Private Google Access allows Fivetran to communicate with your destination without exposing traffic to the public internet. For more information about Private Google Access, see Google's documentation.
Fivetran's Google network is configured with Private Google Access, which means traffic between Fivetran and your destination is always private when Fivetran runs on Google Cloud (GCP).
Supported destinationlink
We support Private Google Access for BigQuery destinations.
Setup instructions for Private Google Accesslink
You can configure a Private Google Access connection by configuring your Fivetran destination to run on GCP.
VPN Tunnellink
To connect to Fivetran using a VPN tunnel, contact Fivetran's Technical Sales team for help setting up the tunnel.