OpenSearch Setup Guide Beta

Follow our setup guide to connect OpenSearch to Fivetran.

Prerequisites

To connect your OpenSearch account to Fivetran, you need:

An Administrator account
OpenSearch version 1.0.0 or above

Setup instructions

Choose connection method

First, decide whether to connect Fivetran to your OpenSearch instance directly or by using an SSH tunnel.

Connect directly

Fivetran connects directly to your OpenSearch cluster. This is the simplest method.

Connect using SSH

Fivetran connects to a separate server in your network that provides an SSH tunnel to your OpenSearch cluster.

If you connect using SSH, you will configure your tunnel server's security group to allow Fivetran access and configure your database's security to allow access from the tunnel.

Before you proceed to the next step, you must follow our SSH connection instructions.

Create role

Fivetran uses the basic authentication mechanism to reach your OpenSearch cluster. You must provide a username and password to connect your OpenSearch cluster to Fivetran.

Create a role for Fivetran based on the level of access you want to grant us:

Create a role for Fivetran with access to all indices

In your OpenSearch instance, execute the following command to create a role for Fivetran with access to all indices:

curl -u username:user-password -X PUT "domain-endpoint/_plugins/_security/api/roles/fivetranRole?pretty" -H 'Content-Type: application/json' -d'{"cluster_permissions":["cluster_monitor","indices:data/read/scroll","indices:data/read/scroll/clear"],"index_permissions":[{"index_patterns":["*"],"allowed_actions": ["indices:admin/stats", "indices:admin/get", "read", "indices:admin/mappings/get", "indices:monitor/stats"]}]}'

In the command above, replace fivetranRole with memorable role name of your choice.
Replace username and user-password with the username and password of a user that has full permissions to the cluster and can manage roles.
Replace domain-endpoint with the endpoint URL of your OpenSearch instance. The endpoint URL must include the port.

Create a role for Fivetran with access to a limited set of indices

Execute the following command to only grant Fivetran access to a limited set of indices:

curl -u username:user-password -X PUT "domain-endpoint/_plugins/_security/api/roles/fivetranRole?pretty" -H 'Content-Type: application/json' -d'{"cluster_permissions":["cluster_monitor","indices:data/read/scroll","indices:data/read/scroll/clear"],"index_permissions":[{"index_patterns":["*"],"allowed_actions":["indices:admin/get"]},{"index_patterns":["index_name"],"allowed_actions": ["indices:admin/stats", "read", "indices:admin/mappings/get"]}]}'

In the command above, replace fivetranRole with memorable role name of your choice.
Replace username and user-password with the username and password of a user that has full permissions to the cluster and can manage roles.
Replace domain-endpoint with the endpoint URL of your OpenSearch instance. The endpoint URL must include the port.
In the index_name field, enter the indices that you want Fivetran to have access to.

Create user

In your OpenSearch instance, execute the following command to create a user for Fivetran's exclusive use:

curl -u username:user-password -X PUT "domain-endpoint/_plugins/_security/api/internalusers/fivetranUser?pretty" -H 'Content-Type: application/json' -d'{"password": "your-password"}'

In the command above, replace fivetranUser with a memorable username of your choice.
Replace username and user-password with the username and password of a user that has at least manage_security cluster privileges.
Replace domain-endpoint with the endpoint URL of your OpenSearch instance. The endpoint URL must include the port.
Replace your-password with a password of your choice.

Map role

Execute the following command to map the role directly to the Fivetran user.

curl -u username:user-password -X PUT "domain-endpoint/_plugins/_security/api/rolesmapping/fivetranRole?pretty" -H 'Content-Type: application/json' -d'{"users": ["fivetranUser"]}'

Replace username and user-password with the username and password of a user that has at least manage_security cluster privileges.
Replace domain-endpoint with the endpoint URL of your OpenSearch instance. The endpoint URL must include the port.
Replace fivetranRole with the Fivetran role you created in Step 2.
Replace fivetranUser with the Fivetran user you created in Step 3.

Finish Fivetran configuration

In the connector setup form, enter a destination schema prefix. This prefix applies to each replicated schema and cannot be changed once your connector is created.
In the Host field, enter your OpenSearch instance host's IP (e.g., 1.2.3.4) or domain (e.g., your.server.com).
Enter your OpenSearch instance's port number.
Enter your Fivetran-specific user that you created in Step 3.
Enter your Password for the Fivetran-specific user that you created in Step 3.
Choose your connection method. If you selected Connect via an SSH tunnel, copy or make a note of the Public Key and add it to the authorized_keys file while configuring the SSH tunnel, and provide the following information:
- SSH hostname (do not use a load balancer's IP address/hostname)
- SSH port
- SSH user
Click Save & Test. Fivetran tests and validates our connection to your OpenSearch instance. Upon successful completion of the setup tests, you can sync your data using Fivetran.

Setup tests

Fivetran performs the following tests to ensure that we can connect to your OpenSearch and that it is properly configured:

The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It generates a pop-up window where you must verify the SSH fingerprint. It then checks that we can connect to your database using the SSH Tunnel. (We skip this test if you are connecting directly.)
The Validating Elasticsearch Connection Test validates the database credentials you provided in the setup form and checks that we can connect to the host.
The Validating Certificate Test validates the certificate you want Fivetran to use. It generates a pop-up window where you must choose the certificate. It then checks that we can connect to your database using TLS. (We skip this test if you are connecting using an SSH tunnel and did not choose to require TLS.)
The Validating Database Type Test checks that your database type matches the connector type. This test will throw a warning if we are unable to validate the database type.
The Validating Elasticsearch Version Test verifies that your OpenSearch version is one of the versions that we support. This test will fail if we don't support the version.
The Validating User Privileges Test validates that the Fivetran user has the necessary privileges to perform a sync.

NOTE: The tests may take a few minutes to finish running.