Database Connection Options
There are several ways to connect Fivetran to your database:
- Directly by safelisting Fivetran's IP
- Using an SSH tunnel
- Using a reverse SSH tunnel
- Using AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect (Business Critical only)
- Using a VPN tunnel
- Using Proxy Agent
Safelist Fivetran's IP
The fastest and easiest way to connect is to allow Fivetran's IP direct access to your database port. For more information about how to do this, visit the setup guide for your database.
SSH Tunnel
If it's not possible to provide direct access to your database port, you can connect to Fivetran using an SSH tunnel. You can also choose this connection method for an added layer of security. To connect using an SSH tunnel, you need to set up an SSH tunnel server that has access to your database port. The tunnel server's SSH port needs to be accessible from Fivetran's IP. You’ll also need to create an SSH user for Fivetran.
IMPORTANT: Fivetran generates a unique public SSH key for each destination. We support multiple connectors on a single SSH tunnel depending on the data volume and network bandwidth.
Follow the SSH tunnel setup instructions for your operating system:
Linux
Expand for instructions
Create SSH user
Log in to your SSH tunnel host and run the following commands:
Create group
fivetran
:sudo groupadd fivetran
Create user
fivetran
:sudo useradd -m -g fivetran fivetran
Switch to the
fivetran
user:sudo su - fivetran
Create the
.ssh
directory:mkdir ~/.ssh
Set permissions:
chmod 700 ~/.ssh
Change to the
.ssh
directory:cd ~/.ssh
Create the
authorized_keys
file:touch authorized_keys
Set permissions:
chmod 600 authorized_keys
Using your favorite text editor, add the public SSH key from the database setup page in your Fivetran dashboard to the
authorized_keys
file. The key must be all on one line. Make sure that you don’t introduce any line breaks when cutting and pasting. The public SSH key is generated uniquely for each Fivetran destination.
IMPORTANT: The SHA-1 signature scheme has been deprecated from OpenSSH. As a result, Fivetran will attempt to connect using an SHA-2 RSA public key signature by default. If that does not work, then we will attempt to connect using an SHA-1 RSA public key signature. Fivetran supports SHA-2 connectivity with all OpenSSH server versions from 7.2 and later. For more details, refer to the OpenSSH documentation.
NOTE: If you encounter SSH connectivity errors even though your configuration is correct, it may be because your version of OpenSSH requires a connection using the SHA-1 signature scheme. You can use the following options to force enable the SHA-1 signature scheme:
To force an SHA-1 public key signature, you must modify your
sshd_config
file (in/etc/ssh
) and add the following line:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com
To force an SHA-1 host key, you must modify your
sshd_config
file (in/etc/ssh
) and add the following line:
HostKeyAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com
Changing the host key will require you to validate the new fingerprint in your connector and destination after running a setup test. You should expect downtime for connectors within this group during the migration process.
RedHat version 9+ and AWS-hosted SSH servers may need this command if they continue to get an
Auth Failed
error:sudo update-crypto-policies --set LEGACY
.
As an extra layer of security, Fivetran enables TLS on your SSH connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS through tunnel toggle to OFF.
IMPORTANT: If you set the Require TLS through tunnel toggle to OFF, Fivetran first attempts to connect over TLS inside the SSH tunnel. If this fails, Fivetran automatically retries the connection in clear text inside the SSH tunnel. You are responsible for configuring this option as per your corporate security policies.
Once the user is created, you'll need to allow port access.
Allow port access
Make sure that port access is allowed from:
- Fivetran's IP to your tunnel server's SSH port
- Your SSH tunnel server to your source database port
If your SSH server and database happen to be in AWS, you can follow the instructions below to configure port access.
AWS
To configure an SSH server in AWS, open the EC2 console and select Running Instances:
Select the instance you intend to use as an SSH tunnel:
Select the Security groups and then select default:
Select the Inbound tab.
Click Edit.
Fill in Fivetran's IP and your SSH port (do not use a load balancer).
For VPC or EC2 classic, add a security rule:
Select SSH, enter Fivetran's IP, and click Save:
To complete setting up your database connector, follow the setup instructions for your specific database. You can confirm your server's SSH key by comparing the SHA 256 displayed when running the setup tests.
Windows: OpenSSH
Expand for instructions
Install OpenSSH
TIP: Learn more in Microsoft's OpenSSH for Windows overview documentation.
Install the sshd server.
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Verify that the OpenSSH server is installed.
Get-WindowsCapability -Online | ? Name -like 'OpenSSH.Server*'
NOTE: If OpenSSH is installed, you'll see the following message:
Name : OpenSSH.Server~~~~0.0.1.0 State : Installed
Set the firewall to allow inbound TCP connections on port 22.
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
Start both the sshd service and the ssh-agent.
Start-Service sshd Set-Service -Name sshd -StartupType 'Automatic' Start-Service ‘ssh-agent’ Set-Service -Name ‘ssh-agent’ -StartupType 'Automatic'
Create Fivetran user and group
IMPORTANT: Windows does not allow a user and a group to have the same name.
Add a local Fivetran user.
net user fivetran <password> /add /comment:"User for establishing SSH connection to Fivetran service." /passwordchg:no /passwordreq:no /logonpasswordchg:no
Add a Fivetran group.
net localgroup fivetran-group /comment:"Group for establishing SSH connection to Fivetran service." /add
In Windows command prompt, switch to the SSH server directory.
cd C:\ProgramData\ssh\ && start notepad .\sshd_config
Allow password authentication for the Fivetran user.
PasswordAuthentication yes
Allow the Fivetran user to connect to the SSH server. Add the following line to the
sshd_config
file.AllowUsers fivetran
If your Windows build is 1809 or later, comment out the following lines in the
sshd_config
file:# Match Group administrators # AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
Save the
sshd_config
file.Restart the agent and the sshd service. If the following command fails, you can restart from the Task Manager (
Alt
+Ctrl
+Delete
).Restart-Service ssh-agent; Restart-Service sshd
IMPORTANT: The SHA-1 signature scheme has been deprecated from OpenSSH. As a result, Fivetran will attempt to connect using an SHA-2 RSA public key signature by default. If that does not work, then we will attempt to connect using an SHA-1 RSA public key signature. Fivetran supports SHA-2 connectivity with all OpenSSH server versions from 7.2 and later. For more details, refer to the OpenSSH documentation.
NOTE: If you encounter SSH connectivity errors even though your configuration is correct, it may be because your version of OpenSSH requires a connection using the SHA-1 signature scheme. You can use the following options to force enable the SHA-1 signature scheme:
To force an SHA-1 public key signature, you must modify your
sshd_config
file (in/etc/ssh
) and add the following line:
PubkeyAcceptedAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com
To force an SHA-1 host key, you must modify your
sshd_config
file (in/etc/ssh
) and add the following line:
HostKeyAlgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com
Changing the host key will require you to validate the new fingerprint in your connector and destination after running a setup test. You should expect downtime for connectors within this group during the migration process.
Set up client
In your command line, go into the SSH server using Windows VM.
ssh fivetran@<Public IPv4 DNS>
Enter your password.
Create an
.ssh
folder in your home directory.mkdir .ssh
Add an
authorized_keys
file.If your client is Windows PowerShell:
type nul > authorized_keys echo <fivetran-ui-public-key> >> authorized_keys icacls C:\Users\fivetran\.ssh\authorized_keys /inheritance:r
If your client is Linux:
touch authorized_keys chmod 600 authorized_keys echo <fivetran-ui-public-key> >> authorized_keys
If you're running PowerShell in elevated mode, your setup is complete. If you're not running PowerShell in elevated mode, follow the instructions below.
TIP: An elevated PowerShell prompt displays
Administrator: Windows PowerShell
on the top of the prompt's border.
Finish setup (non-elevated PowerShell only)
Allow public key authentication.
PubkeyAuthentication yes
Remove password authentication.
PasswordAuthentication no PermitEmptyPasswords no
Save the
sshd_config
file.Verify that inheritance has been disabled and remove
Administrator
.
a. Right click on theauthorized_keys
file.
b. Select Properties.
c. Select the Security tab.
d. Select Advanced.
e. Verify that the bottom left reads Enable Inheritance, which means that inheritance is disabled.
f. RemoveAdministrator
from the file security permissions.Restart the agent and the sshd service.
Restart-Service ssh-agent; Restart-Service sshd
Reverse SSH Tunnel
You can connect Fivetran to your database using a reverse SSH tunnel if you are unable to provide direct port access to your instance. Reverse SSH tunneling allows you to securely connect to a device located behind a firewall or NAT (Network Address Translation) remotely. With this method, the remote machine initiates the connection to the local machine, enabling remote access to services on the local machine.
To set up a reverse SSH tunnel to connect to Fivetran, contact Fivetran Support and provide the following:
- The name of the Fivetran destination(s) where you want your reverse SSH tunnel to be set up
- Your SSH public key (see instructions below to learn how to generate)
- The public IP address (CIDR notation) of your SSH bastion host or proxy server
Follow the reverse SSH tunnel setup instructions specific to your operating system.
Linux
Expand for instructions
Perform the following steps to set up the reverse SSH tunnel on Linux:
Create an SSH user and generate key pair
Log in to your SSH tunnel host and run the following commands:
Create group
fivetran
:sudo groupadd fivetran
Create user
fivetran
:sudo useradd -m -g fivetran fivetran
Switch to the
fivetran
user:sudo su - fivetran
Create the
.ssh
directory:mkdir ~/.ssh
Set permissions:
chmod 700 ~/.ssh
Generate an SSH key pair. Make a note of the key path - you will need it to complete your setup.
ssh-keygen
IMPORTANT: Do not enter a password for the key file. When prompted to provide a password, hit Enter instead.
View the contents of the public key.
cat ~/.ssh/id_rsa.pub
Send the public key along with the SSH server's IP address and SSH username (
fivetran
) to Fivetran Support.
Once we create the Reverse SSH host and complete the setup on our side, we will provide you with the Reverse SSH host IP address and username. Then, proceed to the next step to initiate the SSH connection between your SSH server and Fivetran's Reverse SSH host.
Initiate SSH connection
To set up a secure tunnel between your local database and Fivetran's server, run the autossh
command on your SSH server.
autossh -M 0 -f -N -R <SSH_HIGH_PORT>:<LOCAL_DB_MACHINE_NAME_OR_IP>:<LOCAL_DB_MACHINE_PORT> <FIVETRAN_SSH_USERNAME>@<FIVETRAN_SUPPLIED_IP> -g -i <PATH_TO_PRIVATE_KEY> -o ServerAliveInterval=10 -o ServerAliveCountMax=1 -o ExitOnForwardFailure=yes
Replace the placeholder variables with the actual values:
SSH_HIGH_PORT
: It should be unique per connector instance and should not be a reserved port number (for instance, port22
is reserved for SSH connections and port443
is reserved for HTTPS).TIP: For the SSH high port number, we recommend adding a single digit - usually
1
- as a prefix to the source database port. For example, if you connect SQL Server, your database's default port is1433
. Therefore, we recommend using port11433
as the SSH high port for your first SQL Server connector, port11434
for your second connector, and so on.LOCAL_DB_MACHINE_NAME_OR_IP
: Internal IP address or name of the local database host machineLOCAL_DB_MACHINE_PORT
: Internal port for communication with the database host. For High-Volume Agent connectors, this is the Agent Listener Port (for example,4343
).FIVETRAN_SSH_USERNAME
: SSH username for Fivetran's Reverse SSH host -fivetran
FIVETRAN_SUPPLIED_IP
: Reverse SSH host IP address provided by Fivetran (contact Fivetran Support or your Fivetran Technical Team to get this)PATH_TO_PRIVATE_KEY
: File path to the private key on the SSH host machine (this is normallyid_rsa.pem
or simplyid_rsa
)
TIP: To track the progress of the
autossh
connection process, replace the-f
flag with the-v
flag to enable verbose logging. Without the-v
flag, you will not see detailed logs of the connection process or confirmation when the script finishes running successfully. However,autossh
will still run in the background if the-f
flag is used.
NOTE: If you want to rerun the
autossh
command using the same SSH high port number that you used previously, you must first stop or terminate the existingautossh
process that is currently using that port. This is necessary to avoid conflicts and ensure that the port is free for the new connection.
Enter values into connector setup form
After establishing a successful reverse SSH tunnel connection, enter the following values in the Fivetran connector setup form for your database:
Field | Value | Description |
---|---|---|
Host | localhost | Allows your SSH host to handle port routing |
Port | { SSH high port } | The port that your SSH host will translate, e.g., 13306 . |
User | { Database user } | |
Password | { Database user's password } | |
Database | { Database name } | The name of the database you want to replicate |
Connection Method | Connect via an SSH tunnel | |
SSH Host | { IP Address } | Supplied by Fivetran |
SSH Port | 22 | |
SSH User | fivetran |
All fields in { brackets } must be replaced with your actual values.
Windows
Expand for instructions
To set up a reverse SSH tunnel on Windows, you need to install PuTTY (an SSH and Telnet client) and PuTTYgen (a key generation tool) on your Windows machine. You can download the standalone binaries (EXE) or the installer package (MSI) for these tools from the official PuTTY download page.
TIP: If you choose to install using the MSI installer, it typically includes the entire PuTTY suite, which consists of PuTTY, PuTTYgen, and other tools.
Perform the following steps to set up a reverse SSH tunnel on Windows:
Generate key pair
Launch PuTTYgen.
Under Parameters, in the Type of key to generate field, select RSA.
In the Number of bits in a generated key field, enter
2048
.Under Actions, click Generate to create a public/private key pair.
Click Save public key to save the public key file to your local computer.
Click Save private key to save the private key file (.ppk) to your local computer.
TIP: Do not save the key with a passphrase.
Send the public key, along with the SSH server's (Windows machine) IP address and SSH username (local Windows user running PuTTY) to Fivetran Support.
Once we create the Reverse SSH host and complete the setup on our side, we will provide you with the Reverse SSH host IP address and user name. Then proceed to the next step to configure and initiate the SSH connection between your SSH server and Fivetran's Reverse SSH Host.
Configure and initiate SSH connection
NOTE: These instructions use PuTTY v0.78.
Launch PuTTY.
In the Session category, configure your session:
- In the Host Name field, enter the user name and IP address of the Reverse SSH Host provided by the Fivetran team in the format
{ssh_user}@{ssh_host_ip}
(e.g.,fivetran@35.230.100.17
). - In the Port field, enter
22
. - Under Connection type, select the
SSH
option.
- In the Host Name field, enter the user name and IP address of the Reverse SSH Host provided by the Fivetran team in the format
In the Connection category, set the Seconds between keepalives to
5
.In the Connection > SSH category, select the Don’t start a shell or command at all checkbox.
In the Connection > SSH > Auth > Credentials category, go to the Private key file for authentication field. Click Browse and select the private key file (.ppk) you created in the previous step (Generate key pair).
In the Connection > SSH > TTY category, select the Don’t allocate a pseudo-terminal checkbox.
In the Connection > SSH > Tunnels category, configure port forwarding:
- Select the Local ports accept connections from other hosts checkbox.
- Select the Remote ports do the same checkbox.
- Under Add new forwarded port:
- In the Source port field, enter a high port number (e.g.,
55432
). The high port can be any non-reserved and unused port. - In the Destination field, enter your destination IP address and the database's port (e.g.,
localhost:5432
for a server hosted on the same computer). - Select the Remote option.
- Select the Auto option.
- Click Add to add this entry to the list of Forwarded ports.
- In the Source port field, enter a high port number (e.g.,
In the Session category, save the configuration and initiate the SSH session:
- In the Saved Sessions field, enter a name for this configuration (e.g.,
fivetran-outbound
). - Click Save to save your configuration.
- In the saved sessions list, select the newly saved session (e.g.,
fivetran-outbound
). - Click Open to start the SSH session. This will open a terminal window and establish the SSH connection based on your configuration.
- In the Saved Sessions field, enter a name for this configuration (e.g.,
AWS PrivateLink
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in AWS’ PrivateLink documentation.
Fivetran uses PrivateLink to move your data securely between our system and your AWS-hosted or on-premises source. PrivateLink works differently depending on your source type:
- If your data source is hosted in AWS, Fivetran can connect to your source using a PrivateLink connection. We query and process the data from the source into our system.
- If your data source is hosted on-premises, Fivetran can use AWS Direct Connect to access your source data. AWS Direct Connect establishes a private network connection between your premises and an AWS VPC. We connect to that AWS VPC using a PrivateLink connection, then query and process the data from the source into our system. Learn more in AWS’ Direct Connect documentation.
You can also use AWS Private Link with multiple destinations. See a complete list in our Supported destinations documentation.
As an extra layer of security, Fivetran enables TLS on your PrivateLink connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using PrivateLink toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Prerequisites
To set up AWS PrivateLink, you need:
- An AWS-hosted (EC2, RDS, or S3 only) or on-premises* source in one of our supported regions
* Your on-premises source must be one of our supported databases. See a complete list in our Databases documentation.
How you connect AWS PrivateLink to your source depends on whether your source is hosted in AWS or on-premises.
Postrequisites
To use AWS PrivateLink, you must select AWS as a Cloud service provider in the Finish Fivetran configuration step of the relevant destination setup guide.
Setup instructions for AWS-hosted source
Expand for instructions
We support connecting to the following AWS-hosted sources using PrivateLink:
NOTE: Third-party managed database services, such as MongoDB Atlas, may not be supported. Contact your Fivetran account manager to confirm if your managed database service is supported.
AWS Host | Supported Connectors |
---|---|
Amazon Aurora | MySQL PostgreSQL |
Amazon EC2 | Amazon DynamoDB DocumentDB MariaDB MongoDB MySQL Oracle PostgreSQL SQL Server |
Amazon RDS | Amazon RDS for MariaDB Amazon RDS for MySQL Amazon RDS for Oracle Amazon RDS for PostgreSQL Amazon RDS for SQL Server |
Other | Amazon S3 AWS Lambda Salesforce |
NOTE: Amazon S3 does not require any configuration. If your S3 bucket is in the same region as your Fivetran account, your network traffic does not traverse the public Internet. The Amazon S3 Gateway Endpoints ensures that regional traffic stays within the AWS network.
You must have an AWS endpoint service configured for your source before you set up a PrivateLink connection with Fivetran. AWS endpoint services only work with network load balancers (NLB), so you must create an NLB inside your VPC if you do not already have one. The NLB receives requests from Fivetran and routes it to your source.
The following graphic illustrates how Fivetran connects to the customer database using AWS Private Link:
Since endpoint service configurations are out of Fivetran’s control, we recommend that you contact your AWS representatives for help setting up PrivateLink. However, we do provide the following high-level instructions based on how customers typically configure their data sources:
In your VPC, create an NLB for your data source and configure it for each subnet (availability zone) in which the service should be available. For help, see the Create an NLB section.
IMPORTANT: Skip this step if your data source is already running behind an NLB.
If you chose to use security groups with your NLB, you must allow the following Fivetran's internal CIDR range on the security groups:
10.0.0.0/8
.Create a VPC endpoint service configuration and specify your NLB.
IMPORTANT: Make sure that you are familiar with the endpoint service considerations and have met its prerequisites.
Safelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service. Send the service name (VPCe) to your Fivetran account manager. For example,com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
.TIP: To learn how to safelist the Fivetran account ID, see AWS’ endpoint service permission documentation.
Send the host name of the service/source to your Fivetran account manager. Fivetran will finish the setup on our side.
To activate the connection, accept the interface endpoint connection request from Fivetran. By default, connection requests must be manually accepted. However, you can configure the acceptance settings for your endpoint service so that any connection requests are automatically accepted.
Create an NLB
On a single static IP service (EC2, non-RDS database, etc.)
To create an NLB on a single static IP service, follow the instructions in AWS’ creating a network load balancer documentation. Either ensure that the NLB availability zones match the target availability zones or enable cross-zone load balancing.
On a dynamic IP service (Amazon Aurora database, RDS database, etc.)
NLB can only route traffic to an EC2 instance, an IP address, or a Lambda function through target groups. Since Aurora and RDS databases don’t have a dedicated IP address or EC2 instance ID, there are two different ways to configure an NLB to route traffic to these databases - using a port forwarding instance or using the IP address (dynamic IP address) of the database. Follow the instructions below for your chosen method.
Using a port forwarding instance
You must deploy an EC2 instance that is configured to do port forwarding (accepting requests from the NLB and forwarding those requests to the RDS database). Here is a sample script that you can use to set up the EC2 port forwarding instance:
#!/bin/bash PREVLOGFILE=/root/ip.txt # Note the below section of the code is important in the event of a server restart. if test -f "$PREVLOGFILE"; then truncate -s 0 $PREVLOGFILE echo "State file $PREVLOGFILE has been emptied" fi python -m SimpleHTTPServer 801 & # NOTE: USE PORT 801 FOR <HEALTH_CHECKS> PARAMETER BELOW echo 1 -> /proc/sys/net/ipv4/ip_forward export RDS_ENDPOINT=<<PROSPECT RDS INSTANCE ENDPOINT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. export RDS_PORT=<<PROSPECTS RDS INSTANCE PORT>> #NOTE: DO NOT INCLUDE THE <<>> CHARACTERS, NO QUOTATION MARKS. iptables -t nat -A POSTROUTING -j MASQUERADE while true do LOGFILE=/root/ip.txt Current_IP=$(dig +short $RDS_ENDPOINT | tail -n1) #NOTE: THE "/ TAIL -n1" piece is critical to ensure only the IP address of the RDS instnce is picked. if [ $LOGFILE = "" ] ; then iptables -I INPUT -i eth1 -s $Current_IP -j ACCEPT echo $Current_IP > $LOGFILE else Old_IP=$(cat $LOGFILE) if [ "$Current_IP" = "$Old_IP" ] ; then echo "IP address has not changed ($Old_IP -> $Current_IP)" else iptables -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Old_IP:$RDS_PORT iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $Current_IP:$RDS_PORT sysctl net.ipv4.ip_forward=1 iptables-save echo $Current_IP > $LOGFILE echo "IP address has changed ($Old_IP -> $Current_IP)" fi fi sleep 5 done
NOTE: If you are using Amazon Linux 2023, this doesn't support python2. You need to change the following part of Line 6 of the script above from:
python -m SimpleHTTPServer 801
to:
python3 -m http.server 801
Once you’ve finished setting up the port forwarding instance, configure the NLB listener and target group to route traffic to the portforwarder EC2 instance.
Using the dynamic IP address
Aurora and RDS databases provide an endpoint to access your database when you set them up. This endpoint resolves to an IP address. AWS doesn’t recommend using this IP address, since it can change without notice. To work around this limitation, you can deploy a lambda function to periodically check the IP address and update the NLB target group when it changes.
To use the IP address of the RDS or Aurora database in your NLB target group, do the following:
Run the
nslookup
ordig
command with the domain name of RDS endpoint as the input to find the IP address:dig +short <YOUR_RDS_DNS_ENDPOINT>
Set up your NLB target group with the IP address.
Deploy a lambda function to periodically perform
nslookup
on the RDS endpoint to see if the IP address has changed and update the target group with the new IP address.
Setup instructions for on-premises source
Expand for instructions
Contact your account manager for help setting up PrivateLink for your on-premises source. Our team will help you set up AWS Direct Connect to an AWS VPC, which Fivetran can connect to using PrivateLink.
Setup instructions for self-service Fivetran accounts Private Preview
Expand for instructions
The self-service functionality is not available by default. Contact your Fivetran account manager to enable this functionality for you.
With self-service, you can enable AWS PrivateLink on the following connectors:
Find connection service name
Log in to your AWS account.
Configure the NLB for each subnet (availability zone) where you want the service to be available.
Create a VPC endpoint service configuration and specify your NLB.
IMPORTANT: Make sure that you are familiar with the endpoint service considerations and you have met its prerequisites.
Safelist Fivetran’s AWS VPC Account ID (
arn:aws:iam::834469178297:root
) to allow access to your VPC endpoint service.TIP: To learn how to safelist the Fivetran account ID, see AWS' documentation.
Make a note of your service name (VPCe) (for example,
com.amazonaws.vpce.<region_id>.vpce-svc-xxxxxxxxxxxxxxxxx
). You will need it to configure Fivetran.
Configure PrivateLink connection
In your connector setup form, select Connect via Private Networking in the Connection Method drop-down menu.
Click + Configure a new PrivateLink connection.
Enter a Name for your PrivateLink connection.
In the PrivateLink connection service name field, enter the service name you found in Step 1.
Click Create and save. Fivetran will raise a connection request in AWS.
NOTE: It might take up to 10 minutes for us to raise the connection request.
Accept connection request
Go to your AWS account.
In the top left corner, click Services, then select VPC.
In the VPC dashboard, click Endpoint services.
Refresh the Endpoint services page to see the new connection request from Fivetran. The new connection request is in Pending acceptance state.
NOTE: It might take some time for the new connection request to appear on the list.
Select the new connection request.
Go the Endpoint connections tab.
In the Actions drop-down menu, select Accept endpoint connection request.
In the Accept endpoint connection request pop-up window, enter
accept
in the text box, then click Accept. The status of the request will change to Pending. After a few minutes, the status will change to Accepted and a Connection created message will appear in your source setup form.
Azure Private Link
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft's Azure Private Link documentation.
Fivetran uses Private Link to move your data securely between our system and your Azure-hosted, Azure Virtual Machine-hosted, or on-premises source. Private Link works differently depending on your source type:
- If your data source is hosted in Azure, Fivetran can connect to your source using a Private Link connection. We query and process the data from the source into our system.
- If your data source is hosted in an Azure Virtual Machine or on-premises, you can use Azure Private Link Service to connect Fivetran to your source. To use Azure Private Link Service, you must create an Azure Private Link Service with port forwarding VMs. The VMs relay the network traffic into your Azure VNet, which then connects to your source through Azure ExpressRoute or Azure VPN Gateway.
You can also use Azure Private Link with multiple destinations. See a complete list in our Supported destinations documentation.
As an extra layer of security, Fivetran enables TLS on your Private Link connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS when using PrivateLink toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Link toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Prerequisites
To set up Azure Private Link, you need an Azure-hosted, Azure Virtual Machine-hosted, or on-premises source in one of our supported regions.
NOTE: Your on-premises source must be one of our supported databases. See a complete list in our Databases documentation.
How you connect Azure Private Link to your source depends on whether your source is hosted in Azure, in an Azure Virtual Machine, or on-premises.
Postrequisites
To use Azure Private Link, you must select Azure as a Cloud service provider in the Finish Fivetran configuration step of the relevant destination setup guide.
Setup instructions for Azure-hosted sources
Expand for instructions
We support connecting to the following Azure-hosted sources using Private Link:
- Azure Blob Storage
- Azure Cosmos DB
- Azure Functions
- Azure Database for MariaDB
- Azure Database for MySQL
- Azure Database for PostgreSQL
- Azure SQL Database
- Azure SQL Managed Instance
The following graphic illustrates how Azure Private Link establishes a connection between the Fivetran network and the customer network. Fivetran uses Private Endpoints to establish a Private Link with the Azure Database for MySQL and Snowflake.
To establish a Private Link connection, do the following:
- Verify that your Azure-hosted source supports Private Endpoint in Microsoft’s Private Endpoint documentation.
- Contact your Fivetran account manager and provide your Azure-hosted service’s fully-qualified resource ID, including the resource name and resource type of your Azure-hosted source.
- Wait to receive Private Endpoint request details from Fivetran. We create a Private Endpoint using your resource ID, type, and subresource. We then initiate a Private Link connection request as part of the Private Endpoint setup and share the details of that request with you.
- In the Azure Portal or CLI, verify and approve the Private Link connection request from Fivetran. Fivetran then finishes setting up Private Link for your Azure-hosted service on our side.
Setup instructions for Azure Virtual Machine-hosted or on-premises sources
Expand for instructions
How you set up Private Link for your Azure Virtual Machine-hosted or on-premises source depends on whether or not your source already has an Azure Private Link Service.
If your source is already configured with a Private Link Service, follow the setup instructions for Azure-hosted sources to complete your Fivetran setup.
If you do not have a Private Link Service yet, you will need to configure one before establishing a Private Link connection by following the steps below:
NOTE:
- You must have a VNet with access to your sources.
- If you are configuring your on-premises sources and they aren't connected to Azure yet, either follow the instructions in the Azure ExpressRoute documentation (recommended) or Azure VPN Gateway documentation to connect them to Azure.
- Azure ExpressRoute provides a dedicated, high-bandwidth connection between the on-premise network and Azure, bypassing the public internet.
- Azure VPN Gateway establishes a secure, encrypted VPN tunnel over the public internet between the on-premise network and Azure.
- If you are configuring your Azure Virtual Machine-based sources, ensure that they are accessible from the VNet where the Private Link Service Load Balancer will be running.
- A single Azure Private Link Service can support one or multiple sources at the same time with different ports mapped to corresponding data source IP addresses. You will need to enter these ports in the Fivetran connector setup form.
- Inside the VNet, create a Standard Load Balancer with a NIC-based backend pool that has access to your data sources. Configure a health probe and a load balancer rule with the ports that your connectors will use. Learn how in Azure's Create a load balancer guide.
- Create a Private Link Service associated with the load balancer that you created in step 1. Learn how in Azure's Create a private link service guide.
- Create one or more Virtual Machines and place them in the backend pool of the load balancer you created in step 1.
- Enable IP forwarding on each of the backend Virtual Machine network interfaces. Learn how to enable IP forwarding in Azure's Enable IP forwarding documentation.
- For security purposes, ensure that no public IP addresses are configured for the Virtual Machines. Private Link connections use only private IP addresses.
- Log in into each Load Balancer-backend Virtual Machine and run the script below. The script configures Network Address Translation (NAT), which forwards network packets from SLB-backend Virtual Machines to your data sources. The script below configures a single port forwarding, but feel free to add additional sets of iptables
PREROUTING
andPOSTROUTING
rules for additional ports:
#!/bin/bash
# This script configures Network Address Translation to forward incoming packets
# from the Azure Standard Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a data source server that Fivetran connects to.
# local port where Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>
# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X
# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
For additional help, read Microsoft’s Private Link Service documentation or contact your Fivetran account manager.
Setup instructions for self-service Fivetran accounts Private Preview
Expand for instructions
The self-service functionality is not available by default. Contact your Fivetran account manager to enable this functionality for you.
With self-service, you can enable Azure Private Link on the following sources:
Find resource ID
- Verify that your source supports Private Endpoint. For more information, see Microsoft’s documentation.
- Log in to the Azure portal and then go to your Azure-hosted service.
- Select your workspace and then go to Properties.
- Make a note of the Resource ID. You will need it to create the Private Link connection.
Configure Private Link connection
In your connector setup form, select Connect via Private Networking in the Connection Method drop-down menu.
Click + Configure a new PrivateLink connection.
Enter a Name for your Private Link connection.
In the PrivateLink Resource ID field, enter the resource ID you found in Step 1.
In the PrivateLink Subresource name drop-down menu, select the subresource corresponding to your resource type. The following table lists the subresources corresponding to each resource type:
Resource Type Subresource Azure SQL Database sqlServer
Azure Cosmos DB sql
Click Create and save. Fivetran will raise a connection request in Azure.
NOTE: It might take up to 10 minutes for us to raise the connection request.
Accept connection request
- Go to your Azure account.
- Go to Private Link Center > Pending connections.
- Select the connection request from Fivetran, then click Approve.
- In the pop-up window, confirm that you want to approve the connection request. Once Azure processes the request, a Connection created will appear in your source setup form.
- Update your username to
<username>@dbname
.
Google Cloud Private Service Connect Beta
Google Cloud Private Service Connect allows VPCs and Google-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Google Cloud's Private Service Connect documentation.
Fivetran uses Private Service Connect to move your data securely between our system and your Google Cloud-hosted sources.
You can also use Private Service Connect with multiple destinations. See a complete list in our Supported destinations documentation.
As an extra layer of security, Fivetran enables TLS on your Private Service Connect connection by default. We recommend that you keep TLS enabled unless you know it is safe to forgo it. To disable TLS, set the Require TLS when using Private Service Connect toggle to OFF.
IMPORTANT: If you set the Require TLS when using Private Service Connect toggle to OFF, Fivetran first attempts to connect over TLS. If this fails, Fivetran automatically retries the connection in clear text. You are responsible for configuring this option according to your corporate security policies.
Prerequisites
To set up Google Cloud Private Service Connect, you need an GCP-hosted source* and Fivetran instance running in the same supported regions.
* Your source must be one of our supported databases. See a complete list in our Databases documentation.
Postrequisites
To use Google Cloud Private Service Connect, you must select GCP as a Cloud service provider in the Finish Fivetran configuration step of the relevant destination setup guide.
Setup instructions for Google Cloud Private Service Connect
Expand for instructions
We support connecting to any GCP-hosted resource as long as it's supported by Fivetran and exposed through a Private Service Connect producer.
In the following example, we publish a service that runs on a specific instance. However, there are several other ways to expose your service. To learn more, ask your administrators for help or read Google's Publish managed services using Private Service Connect documentation.
You'll need the following in Inputs for commands used in steps below:
<NETWORK>
- the VPC network in which the exposed resource exists<SUBNET>
- the subnetwork where the exposed resource exists<ILB_SUBNET>
- the subnetwork used for allocation of internal load balancers addresses (forwarding rules)<PSC_NAT_SUBNET>
- the subnetwork used for allocation IPs for each customer endpoint address<REGION>
- the region where the exposed resource exists<ZONE>
- the zone where the exposed resource exists<VM_NAME>
- the VM on which the exposed resource runs<VM_IP>
- the private IP on which the exposed resource is available<RESOURCE_PORT>
- the port on which the exposed resource is available in a VM<NETWORK_ENDPOINT_GROUP>
- the network endpoint group<HEALTH_CHECKS>
- the port health checks<BACKEND_SERVICES>
- the backend services<FORWARDING_RULE>
- the forwarding rule<SERVICE_ATTACHMENT>
- the service attachment
IMPORTANT: If you already have a regional internal load balancer for your resource, proceed to step 8 of this section.
Create a network endpoint group.
gcloud compute network-endpoint-groups create --network <NETWORK> --subnet <SUBNET> \ --network-endpoint-type gce-vm-ip --zone <ZONE> <NETWORK_ENDPOINT_GROUP>
Add an instance with the running resource as an endpoint to the network endpoint group.
gcloud compute network-endpoint-groups update --zone <ZONE> <NETWORK_ENDPOINT_GROUP> \ --add-endpoint='instance=<VM_NAME>
Create health checks to automatically enable and disable the instance. In this example, we use port checks.
gcloud compute health-checks create tcp --region <REGION> --check-interval=60s --port=<RESOURCE_PORT> <HEALTH_CHECKS>
Create backend services.
gcloud compute backend-services create --region=<REGION> --health-checks=<HEALTH_CHECKS> \ --health-checks-region=<REGION> --load-balancing-scheme=INTERNAL <BACKEND_SERVICES>
Assign the network endpoint group you created as a backend.
gcloud compute backend-services add-backend <BACKEND_SERVICES> --network-endpoint-group=<NETWORK_ENDPOINT_GROUP> \ --network-endpoint-group-zone=<ZONE>
(Optional) Allocate a subnetwork dedicated to forwarding rules.
cloud compute networks subnets create --network <_NETWORK_> --region <REGION> --range=10.0.X.0/24 <ILB_SUBNET>
Create a forwarding rule for the internal load balancer. It points to
<BACKEND_SERVICES>
and allocates the IP address from<ILB_SUBNET>
in<NETWORK>
.gcloud compute forwarding-rules create --backend-service=<BACKEND_SERVICES> --region <REGION> \ --load-balancing-scheme=INTERNAL --ports=ALL --subnet=<ILB_SUBNET> --network=<NETWORK> <FORWARDING_RULE>
Allocate a NAT subnetwork dedicated to creating endpoints for each connected customer.
cloud compute networks subnets create --network <NETWORK> --region <REGION> --range=10.0.X.0/24 \ --purpose=PRIVATE_SERVICE_CONNECT <_PSC_NAT_SUBNET_>
Open traffic from the ILB and NAT networks to a VM or port using the corresponding ranges specified in previous steps.
gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-psc-nats gcloud compute firewall-rules create --direction=INGRESS --priority=1000 --network=<_NETWORK_> --action=ALLOW \ --rules=tcp:_RESOURCE_PORT_ --source-ranges=10.0.X.0/24 --destination-ranges=<_VM_IP_>/32 db-demo-allow-ilb-nats
Create a service attachment and point it to the internal load balancer you created above (
FORWARDING_RULE
) with manual accepting mode.gcloud compute service-attachments create --producer-forwarding-rule=<FORWARDING_RULE> --connection-preference=ACCEPT_MANUAL \ --region=<REGION> --description='Producer for my resource in region' --nat-subnets <_PSC_NAT_SUBNET_> <SERVICE_ATTACHMENT>
Contact Fivetran to set up a Private Service Connect link on the Fivetran side. Fivetran will provide a
PSC_CONNECTION_ID
, which allows you to identify which connections come from Fivetran before you approve them.
NOTE:
- If you want to auto-approve the Fivetran project, use the
--consumer-accept-list=fivetran_donkeys=2
parameter for gcloud.- The instructions above use network endpoint groups, since it's easy to attach them to existing VMs. However, if a VM is already a part of the instance groups, you can use it directly as a backend services target.
- You can test a newly-created producer in another VPC by allocating an IP and creating a forwarding rule, as described in Google Cloud's Configure Private Service Connect Services documentation.
Setup instructions for CloudSQL resources exposed using private service access
Expand for instructions
When setting up access for CloudSQL databases using private IP, you need an additional VM to act as a forwarding proxy. You can use your solution like HAProxy or iptables.
NOTE: If you use the following vm script, make sure to persist iptables rules and consider using instance groups instead of standalone VMs.
#!/bin/bash
# This script configures Network Address Translation to forward incoming packets
# from the Load Balancer to IP-based destinations and route them
# back. In this script, "destination" means a data source server that Fivetran connects to.
# local port where the Load Balancer sends traffic to
SOURCE_PORT=<local_port>
# destination server inside internal network
DESTINATION_IP=<destination_server_ip_address>
DESTINATION_PORT=<destination_server_port>
# enable IP forwarding on host
echo 1 > /proc/sys/net/ipv4/ip_forward
# clear existing iptables rules and chains
iptables -F
iptables -t nat -F
iptables -X
# change the packet recipient from local to destination socket (host & port)
iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT}
# change the source IP address from the LB NAT IP address to the IP of this LB-backend host
iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
Once you have the VMs properly set up, follow the standard setup instructions for Google Cloud Private Service Connect.
Setup instructions for on-premises source
Expand for instructions
To connect your on-premises source to Fivetran, you must first use Dedicated Interconnect to connect a VPC to your on-premises source, then use Private Service Connect to connect that VPC to Fivetran.
Follow the instructions in Google's Dedicated Interconnect provisioning overview to set up Dedicated Interconnect and use it to connect a VPC to your on-premises source.
When setting up access for on-premises databases using private IP, you need an additional VM to act as a forwarding proxy. You can use your solution like HAProxy or iptables.
NOTE: If you use the following vm script, make sure to persist iptables rules and consider using instance groups instead of standalone VMs.
#!/bin/bash # This script configures Network Address Translation to forward incoming packets # from the Load Balancer to IP-based destinations and route them # back. In this script, "destination" means a data source server that Fivetran connects to. # local port where the Load Balancer sends traffic to SOURCE_PORT=<local_port> # destination server inside internal network DESTINATION_IP=<destination_server_ip_address> DESTINATION_PORT=<destination_server_port> # enable IP forwarding on host echo 1 > /proc/sys/net/ipv4/ip_forward # clear existing iptables rules and chains iptables -F iptables -t nat -F iptables -X # change the packet recipient from local to destination socket (host & port) iptables -t nat -A PREROUTING -p tcp --dport ${SOURCE_PORT} -j DNAT --to-destination ${DESTINATION_IP}:${DESTINATION_PORT} # change the source IP address from the LB NAT IP address to the IP of this LB-backend host iptables -t nat -A POSTROUTING -p tcp -d ${DESTINATION_IP} --dport ${DESTINATION_PORT} -j SNAT --to-source $(hostname -i)
Once you have the VMs properly set up, follow the standard setup instructions for Google Cloud Private Service Connect.
VPN Tunnel
A VPN (Virtual Private Network) tunnel is used primarily to establish a secure connection between your device and a remote server. It's often used to access the internet securely, hide your IP address, and encrypt your online activities.
A VPN can be used to secure all internet traffic from your device, meaning everything you do online goes through the VPN tunnel.
You need to subscribe to a VPN service and install their software or app. Once connected, all your internet traffic is routed through their servers.
To connect to Fivetran using a VPN tunnel, contact Fivetran's Support team for help setting up the tunnel.
NOTE: Fivetran connects from a Dynamic port to the one you specified in the setup form. When creating your inbound firewall rules for the VPN Tunnel, you have to specify only the destination port.
Proxy Agent
NOTE: To use this connection method, you need a Fivetran account on a Standard, Enterprise, or Business Critical plan.
The Fivetran Proxy Agent eliminates the need for other complex networking options. Installed within a customer's network, it creates an outbound network connection to the Fivetran Managed SaaS (see the list of IP addresses in our documentation). This allows for secure communication between Fivetran processes and your database without opening an inbound port in your firewall and/or other access control systems. The Proxy Agent creates and maintains an outbound WebSocket connection to the Proxy Server in Fivetran’s environment using TLS up to and including version 1.3 and communicates with the server over port 443
.
The Proxy Agent is configured by a destination region, so it can only communicate with connectors running within the same region. For example, if the Proxy Agent is in the us-east4
region, and both BigQuery and Snowflake are hosted there as data destinations, the Proxy Agent can work with both.
System requirements
Proxy Agent requires the following system resources:
- CPU: Minimum 4 vCPUs with x86-64 processors
- Memory: Minimum 5 GB of RAM
- Storage: Minimum 2 GB allocated disk space for the executables and logs
- Java: The Proxy Agent includes a bundled Java Runtime Environment (JRE) based on open-source Azul Zulu JDK. You do not need to install or purchase any additional Java licenses.
Supported connectors
You can use Proxy Agent with the following connectors:
- HVA Db2 for i
- HVA Oracle
- HVA SAP ECC on Oracle
- HVA SAP ECC on SQL Server
- HVA SQL Server
- MariaDB
- MySQL
- Oracle
- PostgreSQL
- SQL Server
For the supported High-Volume Agent (HVA) connectors, this connection method requires the Proxy Agent to be installed on the HVA's host or on a host that has access to HVA. See the sample system architecture with Proxy Agent and HVA below.
For non-HVA connectors, the source communicates directly with the Proxy Agent.
NOTE:
- A Proxy Agent can support multiple connections. However, we recommend that you use a maximum of 10 connections per Proxy Agent.
- A Proxy Agent can only support connections from a single cloud region. For example, a Proxy Agent in Azure East US 2 cannot service connections in GCP US East 4.
Configure Proxy Agent
Follow the instructions below to configure a connection through the Proxy Agent.
Generate Proxy Agent settings
In your connector setup form, do the following:
In the Connection Method drop-down menu, select Connect via proxy agent.
Click Configure a new proxy agent.
Proceed to the Configure a new proxy agent dialog.
Download High-Volume Agent if you have not already. Then, select the I've downloaded the agent checkbox and click Next.
Enter a name for your proxy agent and click Generate proxy agent config to generate a proxy agent configuration file.
Download the generated proxy agent configuration file (
config.json
) and save it in a location that is easy to access. You will need this file for the installation of the proxy agent.Select the I have downloaded the file checkbox and click Save to finalize the proxy agent configuration.
Install Proxy Agent
Before installing the Proxy Agent, you need to ensure that your network configuration permits connections to Fivetran's Proxy Server. To authenticate Fivetran Proxy connection through IP, safelist the appropriate hostnames or IP addresses in your firewall. After configuring your firewall, verify connectivity to Fivetran's Proxy Server by running one of the following commands on your local system, replacing <proxy server IP>
with the correct IP address:
- Using Telnet:
telnet <proxy server IP> 443
- Using Netcat:
nc <proxy server IP> 443 -v
Follow the Proxy Agent installation instructions for your operating system.
Install Proxy Agent on Windows using EXE file (Installer)
When installing the Proxy Agent as a service, the user who installs the Proxy Agent must have permission to manage Windows services. We recommend that you install the Proxy Agent as an Administrator user.
Run the downloaded
.exe
file (e.g.fivetran-6.1.0_23-hub_and_agent-windows-x64-64bit_ga_patch-setup.exe
).In the installation wizard dialog, click Next.
Read the License Agreement, select I accept the agreement and click Next.
Specify the installation directories and click Next.
NOTE: If this is a first-time installation, the High-Volume Agent (HVA) and Proxy Agent can be installed in the same directory. Otherwise, you can either install the Proxy Agent in a different directory than the HVA or manually install the Proxy Agent using a ZIP file.
Specify the name for the program folder and click Next.
Select the role of the installation:
- Proxy Agent to install only the Proxy Agent or
- High-Volume Agent (HVA) and Proxy Agent to install both the Proxy Agent and HVA.
If you selected High-Volume Agent (HVA) and Proxy Agent in the previous step, enter value for the High-Volume Agent Listener Port and click Next.
Paste the Proxy Agent settings generated in step 6 and click Next.
Select the user account for running the Proxy Agent service.
If Specified user is selected, enter values for User, Password, Confirm Password and click Next.
If Local System account is selected, click Next.
Select Add HVR_HOME, HVR_CONFIG, and HVR_TMP (if required) and click Next. If you are only installing the Proxy Agent, we recommend that you skip this step.
This is to set the environment variables
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
in your operating system. These variables point to the corresponding installation directories created in step 4 above.Click Next to initiate the installation.
NOTE: If you selected High-Volume Agent (HVA) and Proxy Agent in step 6, this will install both the Proxy Agent and High-Volume Agent under the same installation directory.
Click Finish to start the Proxy Agent. Once installed, the Proxy Agent service appears running in Windows Services.
NOTE: If you selected High-Volume Agent (HVA) and Proxy Agent in step 6, this will start both the Proxy Agent and High-Volume Agent services.
Install Proxy Agent on Windows using ZIP file
Perform the following steps in the user account that will be used for operating the Proxy Agent:
Configure the environment variables
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
for your operating system using commandsetx
orset
. Each of these environment variables should be pointed to the installation directories -hvr_home
,hvr_config
, andhvr_tmp
:Environment variables set using
setx
command are available in the future command windows only and the environment variables set usingset
command are available in the current command window only.setx HVR_HOME C:\fivetran\hvr_home setx HVR_CONFIG C:\fivetran\hvr_config setx HVR_TMP C:\fivetran\hvr_tmp
set HVR_HOME=C:\fivetran\hvr_home set HVR_CONFIG=C:\fivetran\hvr_config set HVR_TMP=C:\fivetran\hvr_tmp
Also, add the executable directory path (e.g
C:\fivetran\hvr_home\bin
) to the environment variablePATH
.setx PATH "%PATH%C:\fivetran\hvr_home\bin"
set PATH=%PATH%;C:\fivetran\hvr_home\bin
Alternatively, environment variables can be configured using Windows GUI.
Navigate to Control Panel ▶ System and Security ▶ System ▶ Advanced system settings
Alternatively, use the command
sysdm.cpl
to open System Properties.In the Advanced tab, click Environment Variables...
In section System variables or User Variables for user_name, click New.
- Enter Variable name (e.g,
HVR_HOME
) and Variable value (e.g,C:\fivetran\hvr_home
). - Click OK.
- Repeated the above steps for each environment variable.
- Enter Variable name (e.g,
Add the executable directory path to the environment variable
Path
.- In section System variables or User Variables for user_name, from the list of variables, select Path and click Edit....
- Click New and enter the path for the Proxy Agent executable.
- Click OK.
Create the installation directory -
hvr_home
(e.g.C:\fivetran\hvr_home
):md %HVR_HOME%
- Other directories (
hvr_config
andhvr_tmp
) will be created automatically as needed. hvr_home
is regarded a read-only directory.
- Other directories (
Uncompress (unzip) the installation file (e.g. fivetran-6.1.0_23-hub_and_agent-windows-x64-64bit_ga_patch.zip) into the
hvr_home
directory:cd %HVR_HOME% C:\fivetran\hvr_home>tar -xf C:\Users\Admin\Downloads\fivetran-6.1.0_23-hub_and_agent-windows-x64-64bit_ga_patch.zip
Alternatively, files can be uncompressed using the 'Extract All' option in Windows GUI.
Paste the Proxy Agent settings generated in Step 6 to the
proxyagent.conf
file located in theHVR_CONFIG/proxy
directory.NOTE: If missing, create the file manually.
Run the following command to validate the Proxy Agent settings:
%HVR_HOME%/jre/bin/java -jar %HVR_HOME%/proxy/proxyagent.jar -v %HVR_CONFIG%/proxy/proxyagent.conf
Create a
.bat
file (e.g.install_and_run_proxy_service.bat
) with the following contents.FOR /F "delims=" %%i IN ('CALL "%HVR_HOME%/bin/hvr" hvrhubserversvc -s') DO SET HvrHubServerSvcOutput=%%i REM We use a normalized HVR_CONFIG path hash to guarantee a unique service name SET HubConfigHash=%HvrHubServerSvcOutput:~13,8% REM In the CLI below use "DomainName\UserName" for --ServiceUser and specify --ServicePassword parameter if needed REM Adjust the other parameters as necessary REM Ensure to have unique service name (in //IS//<ServiceName>) and display name %HVR_HOME%/proxy/prunsrv.exe //IS//FivetranProxy_%HubConfigHash% ^ --StartParams "%HVR_CONFIG%/proxy/proxyagent.conf" ^ --ServiceUser "LocalSystem" ^ --DisplayName "Fivetran Proxy Agent [%HubConfigHash%]" ^ --Description "Fivetran Proxy Agent installed into %HVR_HOME%\proxy" ^ --Jvm "%HVR_HOME%/jre/bin/server/jvm.dll" ^ --JavaHome "%HVR_HOME%/jre" ^ --Classpath "%HVR_HOME%/proxy/proxyagent.jar" ^ --StartPath "%HVR_HOME%/proxy/" ^ --Startup=auto ^ --StartMode=jvm ^ --StartClass=com.fivetran.proxy.agent.ProxyAgent ^ --StartMethod=main ^ --StopMode=jvm ^ --StopClass=com.fivetran.proxy.agent.ProxyAgent ^ --StopMethod=stop ^ --JvmOptions "-XX:+HeapDumpOnOutOfMemoryError" ^ --StdOutput "%HVR_CONFIG%/proxy/logs/agent-out.log" ^ ++Environment "HVR_CONFIG=%HVR_CONFIG%" ^ ++Environment "HVR_HOME=%HVR_HOME%" ^ ++Environment "HVR_TMP=%HVR_TMP%" %HVR_HOME%/proxy/prunsrv.exe //ES//FivetranProxy_%HubConfigHash%
Run the
.bat
file to configure and start the Proxy Agent:install_and_run_proxy_service.bat
Install Proxy Agent on Linux
Perform the following steps as the user that will be used for operating Proxy Agent:
The commands to set the environment variables depend on the shell you use to interface with the operating system. This procedure lists examples that can be used in Bourne Shell (sh) and KornShell (ksh).
Configure the environment variables
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
for your operating system. Each of these environment variables should be pointed to the installation directories -hvr_home
,hvr_config
, andhvr_tmp
.export HVR_HOME=/home/fivetran/hvr_home export HVR_CONFIG=/home/fivetran/hvr_config export HVR_TMP=/home/fivetran/hvr_tmp
Also, add the executable directory path to the environment variable
PATH
.PATH=$PATH:$HVR_HOME/bin
Add the environment and the executable directory path into the startup file (e.g.
.profile
).export HVR_HOME=/home/fivetran/hvr_home export HVR_CONFIG=/home/fivetran/hvr_config export HVR_TMP=/home/fivetran/hvr_tmp export PATH=$PATH:$HVR_HOME/bin
Create the installation directory -
hvr_home
using the following commands:umask 022
mkdir $HVR_HOME
TIP:
umask 022
is used so that the files and directories created in the following commands are readable by everyone (other Linux users and groups), but only writable by the owner. Other directories (HVR_CONFIG
andHVR_TMP
) will be created automatically as needed. TheHVR_HOME
directory is regarded as read-only.Uncompress the installation file (e.g.
fivetran-6.1.0_23-hub_and_agent-linux_glibc2.17-x64-64bit_ga.tar.gz
) into theHVR_HOME
directory:cd $HVR_HOME
tar xzf /tmp/hvr-6.1.0_23-hub_and_agent-linux_glibc2.17-x64-64bit_ga.tar.gz
Once installed, the
jre
andproxy
folders are created in yourHVR_HOME
directory.Create a new directory for the
proxyagent.conf
file inhvr_config
(/home/fivetran/hvr_config/proxy
). Paste the Proxy Agent settings generated in step 6 to theproxyagent.conf
file located in the newly created directory.Run the following command to verify if the Linux machine is ready for the setup. The output of the command should be
systemd
.ps -p 1 -o comm=
Update the
fivetran_proxy.service
file in the/etc/systemd/system
directory with the correct path tojava.jar
,proxyagent.jar
, andproxyagent.conf
files.If missing, create the
fivetran_proxy.service
file manually. The contents of the file should be as follows:[unit] Description=Fivetran Proxy Agent #<agent_id> [Service] Type=simple ExecStart=/home/fivetran/hvr_home/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/fivetran/hvr_config/proxy/ -jar /home/fivetran/hvr_home/proxy/proxyagent.jar /home/fivetran/hvr_config/proxy/proxyagent.conf # Restart this service to after a crash Restart=always # The number of seconds to wait before attempting a restart RestartSec=5s [Install] WantedBy=multi-user.target
Execute the following commands as the
root
user to start the Proxy Agent service.a. Enable the service:
systemctl enable fivetran_proxy.service
b. Start the service:
systemctl start fivetran_proxy.service
c. Verify the status of the service:
systemctl status fivetran_proxy.service
The following directories are created during the installation of the Proxy Agent. Each directory may contain files and subdirectories:
hvr_home
: Contains executables and files essential for running the Proxy Agent.Expand to see the directories and files in
hvr_home
📁 hvr_home │ ├─ 📁 api -- REST API documentation │ ├─ 📁 bin -- Executable files and shared libraries (dynamic-link │ libraries on Windows) for running the Proxy Agent │ ├─ 📁 dbms -- Database-specific SQL files and templates │ ├─ 📁 etc -- Configuration files and other miscellaneous files │ │ │ ├─ 📁 cert -- Bundled certificates, such as root CAs │ │ │ ├─ 📁 snmp -- SNMP MIB file │ │ │ ├─ 📁 xml -- DTD of XML-serialized data streams │ │ │ ├─ 📄 constsqlexpr.pat -- SQL expressions treated as constants to │ │ optimize replication performance │ │ │ ├─ 📄 hvrosaccess_example.conf -- Sample configuration for allowing HVR to │ │ run plugins from non-standard directories │ │ │ └─ 📄 purge.manifest -- Manifest file (see hvrstrip -m) │ ├─ 📁 examples -- Sample channel definitions │ ├─ 📁 lib -- Shared libraries and database drivers │ ├─ 📁 plugin -- Plugins shipped (installed) with the Proxy Agent │ │ │ ├─ 📁 agent │ │ │ └─ 📁 transform │ ├─ 📁 plugin_examples -- Sample plugins. To use a sample plugin, save │ │ it in the 'hvr_config/plugin' directory │ ├─ 📁 agent │ │ │ ├─ 📁 authentication │ │ │ ├─ 📁 rewrite │ │ │ └─ 📁 transform │ ├─ 📁 sbin -- Manually created trusted executables │ ├─ 📁 script -- Internal Proxy Agent script files │ ├─ 📁 www -- HVR UI-related files │ ├─ 📄 hvr.3rdparty -- License agreements, copyrights, versions, and notices │ for third-party software used by the Proxy Agent │ ├─ 📄 hvr.rel -- Proxy Agent Release Notes │ └─ 📄 hvr.ver -- Proxy Agent version number
hvr_config
: Contains directories and files associated with the Proxy Agent configuration.Expand to see the directories and files in
hvr_config
📁 hvr_config -- Directories and files for the Proxy Agent configuration │ ├─ 📁 etc -- Configuration files for the Proxy Agent │ ├─ 📁 intermediate -- Temporary files for Compare/Refresh jobs │ ├─ 📁 logs -- Proxy Agent-level log files │ ├─ 📁 plugin -- User-installed plugins (see 'hvr_home/plugin_examples/') │ │ │ ├─ 📁 agent │ │ │ └─ 📁 transform │ ├─ 📁 public -- Log file retention information │ ├─ 📁 run -- Runtime state, such as `.pid` files │ └─ 📁 tmp -- Temporary files (default, if HVR_TMP is not defined)
hvr_tmp
(optional): Contains temporary files associated with the Proxy Agent.
Making any changes to the
hvr_home
directory or its subdirectories is strictly prohibited. This directory must remain unchanged to ensure security and integrity during updates or upgrades. User-specific configurations and runtime data is stored in thehvr_config
directory.
(Optional) Configure Proxy Agent recovery on Windows
Expand for instructions
If your Proxy Agent is installed on a Windows system, you can set up a recovery configuration for the agent by performing the following steps:
NOTE: This process is not required on Linux systems.
Go to Control Panel > Administrative Tools > Computer Management > Services and Applications > Services or use the
services.msc
command to open the Services console.In the Services console, locate the Fivetran Proxy Agent service. Ensure the service description matches the installation directory.
Right-click the service and select Properties.
Go to the Recovery tab.
Set your preferred recovery options for the failure scenario.
TIP: The SC failure command-line utility is also available for more advanced configurations. This tool allows you to automate recovery actions directly through the command line.
Uninstall Proxy Agent
Follow the instructions below to uninstall the Proxy Agent.
Uninstall Proxy Agent from Windows
The steps to uninstall the Proxy Agent depend on whether you used an EXE file (Installer) or a ZIP file to install it.
IMPORTANT: Ensure that no Windows Services management console applications (opened via
services.msc
) are running.
Uninstall Proxy Agent that was installed using an EXE file
Navigate to the
uninstall
folder in the Proxy Agent installation directory%HVR_HOME%
(for example,C:\fivetran\hvr_home\uninstall
).Double-click the
uninstall.exe
file to initiate the uninstallation process.
Uninstall Proxy Agent that was installed using a ZIP file
Create a
.bat
file (for example,delete_proxy_agent_service.bat
) with the following contents:FOR /F "delims=" %%i IN ('CALL "%HVR_HOME%/bin/hvr" hvrhubserversvc -s') DO SET HvrHubServerSvcOutput=%%i REM We use a normalized HVR_CONFIG path hash to guarantee a unique service name SET HubConfigHash=%HvrHubServerSvcOutput:~13,8% %HVR_HOME%/proxy/prunsrv.exe //DS//FivetranProxy_%HubConfigHash%
Run the
.bat
file as an Administrator to remove all Proxy Agent service entries.
Uninstall Proxy Agent from Linux
Stop the Proxy Agent service:
sudo systemctl stop fivetran_proxy.service
Disable the service from auto-starting at boot:
sudo systemctl disable fivetran_proxy.service
Remove the service configuration file:
sudo rm /etc/systemd/system/fivetran_proxy.service
Reload the systemd manager configuration:
sudo systemctl daemon-reload
Clear the systemd state for any failed services:
sudo systemctl reset-failed
High-Volume Agent connection options
There are several ways to connect Fivetran to your database using High-Volume Agent (HVA) connector. Select the configuration that best aligns with your specific networking environment and preferences. Detailed instructions for each connection option can be found in our HVA connection options documentation:
- Direct connection
- Private Link connection - AWS PrivateLink, Azure Private Link, or Google Cloud Private Service Connect
- SSH tunnel connection
- Reverse SSH tunnel connection
- VPN tunnel connection
- Proxy Agent connection