Confluent Cloud Setup Guide
Follow our setup guide to connect Confluent Cloud to Fivetran.
Prerequisites
To connect Confluent Cloud to Fivetran, you need a Confluent Cloud account with Kafka Server running.
Setup instructions
Find server name
Open the Confluent Cloud Clusters.
Select your cluster.
On the CLIENT CONFIG tab, select the Java configuration.
Copy the server name from the
bootstrap.serversconfiguration. You will need it to configure Fivetran.
Finish Fivetran configuration
In the connection setup form, enter the Destination schema name of your choice.
In the Destination schema names field, choose the naming convention you want Fivetran to use for the schemas, tables, and columns in your destination:
- Fivetran naming: Standardizes the schema, table, and column names in your destination according to the Fivetran naming conventions.
- Source naming: Preserves the original schema, table, and column names from the source system in your destination.
If you want to modify your selection, make sure you do it before you start the initial sync.
Enter a Consumer Group name. The consumer group name should be unique to a connection to keep data integrity.
In the Servers section, click + Add. Enter the server name that you found in Step 1 in
<host>:<port>format.Select your message type: Json, Avro, Protobuf, or Text.
If you choose Json, select a sync type: Packed or Unpacked.
If you choose Avro or Protobuf, in the Schema Registry Urls section, click + Add and then enter the following schema registry credentials:
- Schema Registry Urls
- Schema Registry Key
- Schema Registry Secret
(Optional) Select a security protocol: PLAINTEXT or SASL. If you choose SASL, select an Authentication Method:
We do not recommend selecting PLAINTEXT unless you operate in a trusted and isolated network. This protocol transmits all data unencrypted, which can expose sensitive information to interception.
API Key: Enter your API Key and API Secret.
OAuth 2.0: Configure your OAuth 2.0 identity provider in Confluent Cloud, then enter the following credentials:
- Client ID: The client ID of your OAuth 2.0 application.
- Client Secret: The client secret of your OAuth 2.0 application.
- Token Endpoint URL: The URL of your identity provider's token endpoint.
- Scope: (Optional) The OAuth 2.0 scope to request. Leave it blank to use the Identity Provider's (IdP) default scope.
- Logical Cluster ID: The Confluent Cloud cluster ID (format:
lkc-xxxxxxx). Find it under Cluster settings in the Confluent Cloud Console. - Identity Pool ID: (Optional) The identity pool ID if your IdP uses multiple identity pools.
(Hybrid Deployment only) If your destination is configured for Hybrid Deployment, the Hybrid Deployment Agent associated with your destination is pre-selected for the connection. To assign a different agent, click Replace agent, select the agent you want to use, and click Use Agent.
Click Save & Test. Fivetran will take it from here and sync your Confluent Cloud data.
Fivetran tests and validates the Confluent Cloud connection. On successful completion of the setup tests, you can sync your Confluent Cloud data to your destination.
Setup tests
Fivetran performs the following Confluent Cloud connection tests:
- The Connecting to Kafka test validates the connection and checks the accessibility of your Kafka topics.
- The Unpacking JSON Messages test checks whether the connection can successfully fetch and unpack messages from your Kafka topics. We perform this test only if you select Json as the Message Type and Unpacked as the Sync Type.
- The Validate Schema Registry Protocol test checks if the Schema Registry URLs that you specified start with
http. We perform this test only if you select Avro or Protobuf as the Message Type.