Aiven for Apache Kafka Destination Setup Guidelink
Updated February 5, 2024
Follow our setup guide to connect Aiven for Apache Kafka to Fivetran.
To connect Aiven for Apache Kafka to Fivetran, you need the following:
- An Aiven account
- An Aiven for Apache Kafka cluster
Set up schema registrylink
Fivetran supports the following Aiven for Apache Kafka implementations:
- Aiven for Apache Kafka with Karapace Schema Registry
- Aiven for Apache Kafka with Confluent Cloud Schema Registry
Set up Karapace Schema Registrylink
Set up your Karapace Schema Registry by following the instructions in Karapace's documentation.
Set up Confluent Cloud Schema Registrylink
Set up your Confluent Cloud Schema Registry by following the instructions in Confluent Cloud's documentation.
Find Kafka cluster detailslink
Log in to your Aiven account.
Open your Kafka cluster.
In the Overview tab, go to Apache Kafka.
Choose SASL as the Authentication Method.
Make a note of the values of the following fields:
- Service URI
You will need them to configure Fivetran.
Create truststore certificatelink
Click Quick Connect.
In the Connect with drop-down menu, select Java.
Choose SASL as the authentication method.
Create the truststore certificate by following the instructions that appear in the window.
Make a note of the password you specified for the truststore. You will need it to configure Fivetran.
Complete Fivetran configurationlink
Log in to your Fivetran account.
Go to the Manage Account page.
In the Destinations tab, click Add Destination.
In the Add Destination To Fivetran page, enter a Destination name of your choice.
Select Aiven for Apache Kafka as the destination type.
In the Bootstrap Servers field, click + Add and enter the service URI you found in Step 2. The server name must be in the
TIP: Click + Add to add more than one server name.
In the API Key field, enter the username you found in the Step 2.
In the API Secret field, enter the password you found in the Step 2.
Upload the Truststore Certificate you created in Step 3 and enter the Truststore Password.
Enter the number of Partitions you'd like to create in your topic. Partitions allow us to split the data of a topic across multiple brokers and balance the load between them.
NOTE: A consumer group can consume data from only one partition. Therefore, the number of consumer groups that consume data in parallel depend on the number of partitions in the topic.
Enter a Replication Factor to specify the number of replicas you'd like to create for each of your topic partitions.
NOTE: The partition replicas increase the reliability and fault tolerance of the connection. The minimum and maximum supported values are
4, respectively. However, for best performance, we recommend that you set the Replication Factor to
Select your Data Format: JSON or AVRO.
NOTE: You cannot change the data format after you set up the connection.
Select your Schema Registry.
Enter the following schema registry credentials:
- Schema Registry URL in the
- Schema Registry API Key
- Schema Registry API Secret
- Schema Registry URL in the
Choose your Data processing location.
Choose your Cloud service provider and its region as described in our Destinations documentation.
Choose your Time zone.
(Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make note of the IP addresses of the secondary region and safelist these addresses in your firewall.
Click Save & Test.
Fivetran tests and validates the Aiven for Apache Kafka connection. On successful completion of the setup test, you can sync your data using Fivetran connectors to the Aiven for Apache Kafka destination.
Fivetran performs the Connecting to Kafka test to validate the broker credentials and to check if we can access your topic schema through your schema registry.
NOTE: The test may take a couple of minutes to complete.