Self-Hosted Elasticsearch Setup Guide Beta

Follow our setup guide to connect self-hosted Elasticsearch to Fivetran.

Prerequisites

To connect your self-hosted Elasticsearch account to Fivetran, you need:

An Administrator account
Elasticsearch version 7.10 or above

Setup instructions

Choose connection method

First, decide whether to connect Fivetran to your self-hosted Elasticsearch instance directly or by using an SSH tunnel.

Connect directly

Fivetran connects directly to your Elasticsearch cluster. This is the simplest method.

Connect using SSH

Fivetran connects to a separate server in your network that provides an SSH tunnel to your Elasticsearch cluster.

If you connect using SSH, you will configure your tunnel server's security group to allow Fivetran access and configure your database's security to allow access from the tunnel.

Before you proceed to the next step, you must follow our SSH connection instructions.

Choose authentication method

Fivetran uses two authentication mechanisms to reach your Elasticsearch cluster, basic authentication and API Key.

Basic authentication

For a basic access authentication, you need to provide a user name and password to connect your Elasticsearch cluster to Fivetran.

Create role

In your self-hosted Elasticsearch instance, execute the following command to create a role for Fivetran:

curl -u username:user-password -X POST "domain-endpoint/_security/role/fivetranRole?pretty" -H 'Content-Type: application/json' -d'{"cluster": ["monitor"],"indices": [{"names": ["*"], "privileges": ["view_index_metadata", "read", "monitor"]}]}'

In the command above, replace fivetranRole with memorable role name of your choice.
Replace username and user-password with the username and password of a user that has full permissions to the cluster and can manage roles.
Replace domain-endpoint with the endpoint URL of your Elasticsearch instance. The endpoint URL must include the port.
In the indices field, enter the indices that you want Fivetran to have access to. You can either enter * to allow Fivetran to access all indices or list specific index names.

Create user

In your self-hosted Elasticsearch instance, execute the following command to create a user for Fivetran's exclusive use:

curl -u username:user-password -X POST "domain-endpoint/_security/user/fivetranUser?pretty" -H 'Content-Type: application/json' -d'{"password": "your-password","roles": ["fivetranRole"], "full_name": "fivetran", "email": "fivetran@fivetran.com"}'

In the command above, replace fivetranUser with a memorable username of your choice.
Replace username and user-password with the username and password of a user that has at least manage_security cluster privileges.
Replace domain-endpoint with the endpoint URL of your Elasticsearch instance. The endpoint URL must include the port.
Replace your-password with a password of your choice.
Replace fivetranRole with the role you created previously.

API key-based authentication

For an authentication based on an API key, you need to create and provide your API key to connect your self-hosted Elasticsearch instance to Fivetran.

Create API key

In your self-hosted Elasticsearch instance, execute the following command to create an API key for Fivetran's exclusive use:

curl -u username:user-password -X POST "domain-endpoint/_security/api_key?pretty" -H 'Content-Type: application/json' -d'{"name": "fivetran-apikey","role_descriptors": {"fivetran-role": {"cluster": ["monitor"],"indices": [{"names": ["*"], "privileges": ["view_index_metadata", "read", "monitor"]}]}}}'

In the command above, replace username and user-password with the username and password of a user that has at least manage_security cluster privilege. The API key will be created on behalf of this user, so the resultant permissions would be an intersection of the API key permissions and authenticated user's permissions. Make sure the user has the required permissions.
Replace domain-endpoint with the endpoint URL of your Elasticsearch instance. The endpoint URL must include the port.
Replace fivetran-apikey with a name of your choice.
Replace fivetran-role with a name of your choice.

Finish Fivetran configuration

In the connector setup form, enter a destination schema prefix. This prefix applies to each replicated schema and cannot be changed once your connector is created.
In the Host field, enter your Elasticsearch instance host's IP (e.g., 1.2.3.4) or domain (e.g., your.server.com).
Enter your Elasticsearch instance's port number.
Choose your authentication method. If you selected Use API key, provide the API key created in Step 2. If you selected Grant user access, provide the following information:
- Fivetran-specific user that you created in Step 2.
- Password for the Fivetran-specific user that you created in Step 2.
Choose your connection method. If you selected Connect via an SSH tunnel, provide the following information:
- SSH hostname (do not use a load balancer's IP address/hostname)
- SSH port
- SSH user
Click Save & Test. Fivetran tests and validates our connection to your self-hosted Elasticsearch instance. Upon successful completion of the setup tests, you can sync your data using Fivetran.

Setup tests

Fivetran performs the following tests to ensure that we can connect to your self-hosted Elasticsearch and that it is properly configured:

The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It generates a pop-up window where you must verify the SSH fingerprint. It then checks that we can connect to your database using the SSH Tunnel. (We skip this test if you are connecting directly.)
The Validating Elasticsearch Connection Test validates the database credentials you provided in the setup form and checks that we can connect to the host.
The Validating Certificate Test validates the certificate you want Fivetran to use. It generates a pop-up window where you must choose the certificate. It then checks that we can connect to your database using TLS. (We skip this test if you are connecting using an SSH tunnel and did not choose to require TLS.)
The Validating Database Type Test checks that your database type matches the connector type. This test will throw a warning if we are unable to validate the database type.
The Validating Elasticsearch Version Test verifies that your Elasticsearch version is one of the versions that we support. This test will fail if we don't support the version.
The Validating User Privileges Test validates that the Fivetran user has the necessary privileges to perform a sync.

NOTE: The tests may take a few minutes to finish running.