Aiven for Apache Kafka Destination Setup Guide

Follow our setup guide to connect Aiven for Apache Kafka to Fivetran.

Prerequisites

To connect Aiven for Apache Kafka to Fivetran, you need the following:

An Aiven account
An Aiven for Apache Kafka cluster

Setup instructions

Set up schema registry

Fivetran supports the following Aiven for Apache Kafka implementations:

Aiven for Apache Kafka with Karapace Schema Registry
Aiven for Apache Kafka with Confluent Cloud Schema Registry

Set up Karapace Schema Registry

Set up your Karapace Schema Registry by following the instructions in Karapace's documentation.

Set up Confluent Cloud Schema Registry

Set up your Confluent Cloud Schema Registry by following the instructions in Confluent Cloud's documentation.

Find Kafka cluster details

Log in to your Aiven account.
Open your Kafka cluster.
In the Overview tab, go to Apache Kafka.
Choose SASL as the Authentication Method.
Make a note of the values of the following fields:
- Service URI
- User
- Password
You will need them to configure Fivetran.

Create truststore certificate

Click Quick Connect.
In the Connect with drop-down menu, select Java.
Choose SASL as the authentication method.
Create the truststore certificate by following the instructions that appear in the window.
Make a note of the password you specified for the truststore. You will need it to configure Fivetran.

Complete Fivetran configuration

Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice and then click Add.
Select Aiven for Apache Kafka as the destination type.
In the Bootstrap Servers field, click + Add and enter the service URI you found in Step 2. The server name must be in the <host>:<port> format.
TIP: Click + Add to add more than one server name.
In the API Key field, enter the username you found in the Step 2.
In the API Secret field, enter the password you found in the Step 2.
Upload the Truststore Certificate you created in Step 3 and enter the Truststore Password.
Enter the number of Partitions you want to create in your topic. Partitions allow us to split the data of a topic across multiple brokers and balance the load between them.
NOTE: A consumer group can consume data from only one partition. Therefore, the number of consumer groups that consume data in parallel depend on the number of partitions in the topic.
Enter a Replication Factor to specify the number of replicas you want to create for each of your topic partitions.
NOTE: The partition replicas increase the reliability and fault tolerance of the connection. The minimum and maximum supported values are 2 and 4, respectively. However, for best performance, we recommend that you set the Replication Factor to 3.
Select your Data Format: JSON or AVRO.
NOTE: You cannot change the data format after you set up the connection.
Select your Schema Registry.
Enter the following schema registry credentials:
- Schema Registry URL in the https://<host>:port format
- Schema Registry API Key
- Schema Registry API Secret
Choose your Data processing location.
Choose your Cloud service provider and its region as described in our Destinations documentation.
Choose your Time zone.
(Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.
Click Save & Test.

Fivetran tests and validates the Aiven for Apache Kafka connection. On successful completion of the setup test, you can sync your data using Fivetran connectors to the Aiven for Apache Kafka destination.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup test

Fivetran performs the Connecting to Kafka test to validate the broker credentials and to check if we can access your topic schema through your schema registry.

NOTE: The test may take a couple of minutes to complete.

Destination Overview

Release Notes

Documentation Home