Confluent Cloud Destination Setup Guide

Follow our setup guide to connect Confluent Cloud to Fivetran.

Prerequisites

To connect Confluent Cloud to Fivetran, you need the following:

• A Confluent Cloud account with an active Kafka server
• Permissions to generate access keys for the broker and schema registry

Setup instructions

Find Stream Governance API Endpoint

1. Log in to Confluent Cloud.

2. Click View environments.

3. Select the environment that contains your Kafka cluster.

4. Make a note of the Stream Governance API Endpoint. You will need it to configure Fivetran.

   

Find Bootstrap server names

1. In the Clusters tab, select your cluster.

2. On the navigation menu, click Cluster Settings.

   

3. Make a note of the Bootstrap server names that appear in the Endpoints section. You will need them to configure Fivetran.

   

Complete Fivetran configuration

1. Log in to your Fivetran account.

2. Go to the Destinations page and click Add destination.

3. Enter a Destination name of your choice and then click Add.

4. Select Confluent Cloud as the destination type.

5. In the destination setup form, enter the names of the Bootstrap Servers you found in Step 2. The server names must be in <host>:<port> format.

   NOTE: Click + to add more than one server name.

6. Enter your SASL API Key and API Secret.

7. Enter the number of Partitions you want to create in your topic. Partitions allow us to split the data of a topic across multiple brokers and balance the load between them.

   NOTE: A consumer group can consume data from only one partition. Therefore, the number of consumer groups that consume data in parallel depend on the number of partitions in the topic. For more information about the partition limits applicable to your cluster type, see Confluent Cloud's documentation.

8. Enter a Replication Factor to specify the number of replicas you want to create for each of your topic partitions.

   NOTE: The partition replicas increase the reliability and fault tolerance of the connection. The minimum and maximum supported values are 2 and 4, respectively. However, for best performance, we recommend that you set the Replication Factor to 3.

9. Select your Data Format: JSON or AVRO.

   NOTE: You cannot change the data format after you set up the connection.

10. In the Schema Registry URL field, enter the Stream Governance API Endpoint you found in Step 1.

11. Enter the Schema API Registry Key and Schema Registry API Secret.

12. Choose your Data processing location.

13. Choose your Cloud service provider and its region as described in our Destinations documentation.

14. Choose your Time zone.

15. (Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.

16. Click Save & Test.

Fivetran tests and validates the Confluent Cloud connection. On successful completion of the setup test, you can sync your data using Fivetran connectors to the Confluent Cloud destination.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup test

Fivetran performs the Connecting to Kafka test to validate the broker credentials and to check if we can access your topic schema through the schema registry.

NOTE: The test may take a couple of minutes to complete.

Related articles

description Destination Overview

assignment Release Notes

settings API Destination Configuration

home Documentation Home