Apache Kafka Setup Guide

Follow our setup guide to connect Apache Kafka to Fivetran.

Prerequisites

To connect Apache Kafka to Fivetran, you need an Apache Kafka server.

Setup instructions

Choose connection method

Decide whether to connect Fivetran to your Kafka servers directly or using an SSH tunnel.

Connect directly

Fivetran connects directly to your Kafka servers. This is the simplest connection method.

To connect directly, create a firewall rule to allow access to Fivetran's IPs.

Connect using SSH

Fivetran connects to a separate server in your network that provides an SSH tunnel to your Kafka servers. You must connect through SSH if your servers are in an inaccessible subnet on a virtual network.

To connect using SSH, do the following:

1. In the connector setup form, select Connect via SSH Tunnel to expose Fivetran's public SSH key.

2. Copy the Public Key and paste it into the .ssh/authorized_keys file inside the home folder on your SSH servers.

   NOTE: The key must be all on one line, so make sure that you don't introduce any line breaks when copying and pasting.

3. Follow our SSH tunnel connection instructions.

Generate and import client certificate to server truststore

1. Create a key and keystore for the consumer client. Execute the following command in your Apache Kafka server:

   keytool -keystore <client_keystore_name>.jks -alias <alias_name> -validity <VALIDITY> -genkey -keyalg RSA

   NOTE: You will be asked to enter a few details. For What is your first and last name?, enter your Apache Kafka server hostname. Remember the key and keystore password you provide.

2. Export the client certificate from the keystore. Execute the following command:

   keytool -exportcert -rfc -file <client_certificate_name>.pem -alias <alias_name> -keystore <client_keystore_name>.jks

   NOTE: The <alias_name> must be the same as in the previous command.

3. Import the client certificate into the truststore of your Apache Kafka server. Execute the following command:

   keytool -keystore <server_truststore_name>.jks -alias <alias_name> -import -file <client_certificate_name>.pem

Get client certificate key from client keystore

1. Convert the client keystore from JKS to PKCS12 format. Execute the following command:

   keytool -v -importkeystore -srckeystore <client_keystore_name>.jks -srcalias <alias_name> -destkeystore <client_keystore_name>.p12 -deststoretype PKCS12

2. Extract the client certificate key into a .pem file. Execute the following command:

   openssl pkcs12 -in <client_keystore_name>.p12 -nocerts -nodes > <client_certificate_key>.pem

Download required certificates and key

Download the following files from your Apache Kafka server:

• Client certificate (<client_certificate_name>.pem)
• Client certificate key (<client_certificate_key>.pem)
• CA certificate (Trusted Certificate). This is a public-private key pair and certificate in your Apache Kafka server which is used to sign other certificates

Finish Fivetran configuration

1. In the connector setup form, enter the Destination schema name of your choice.

2. Enter a Consumer Group name. The consumer group should be unique to a connector to keep data integrity.

3. In the Servers section, click + Add. Enter your server details in the <server_host_name>:<port> format.

4. Select your message type: Json, Avro, Protobuf, or Text.

   • If you choose Json, select a sync type: Packed or Unpacked.

   • If you choose Avro or Protobuf, in the Schema Registry Urls section, click + Add and then enter the following schema registry credentials:

     • Schema Registry Urls
     • Schema Registry Key
     • Schema Registry Secret

5. (Optional) Select your sync type: Packed or Unpacked.

6. (Optional) Select a security protocol: PLAINTEXT, TLS, or SASL.

   • If you choose TLS, upload the Trusted Certificate, Client Certificate, and Client Certificate Key files.

   • If you choose SASL, select the SASL Mechanism, and then enter your API Key and API Secret.

7. (Hybrid Deployment only) If your destination is configured for Hybrid Deployment, the Hybrid Deployment Agent associated with your destination is pre-selected in the Select an existing agent drop-down menu. To use a different agent, select the agent of your choice, and then select the same agent for your destination.

8. (Optional)(Not applicable to Hybrid Deployment) Set the Enable Advanced Options toggle to ON.

   • Select your Connection Method: Direct or Connect via SSH Tunnel.

   • If you chose Connect via SSH Tunnel, do the following:

     • Enter the IP Address of host tunnel machine
     • Enter the username of account in host tunnel machine
     • Enter the Port number

     IMPORTANT: In the Servers section, enter the DNS address instead of IP Address. You also must have IP address to DNS mapping for each Kafka server in the /etc/hosts file of the tunnel host or the name should resolve to an IP address using an internal DNS server. NOTE : For Hybrid Deployment, by default Direct Connection Method is used.

9. Click Save & Test. Fivetran will take it from here and sync your Apache Kafka data.

Fivetran tests and validates the Apache Kafka connection. On successful completion of the setup tests, you can sync your Apache Kafka data to your destination.

Setup tests

Fivetran performs the following Apache Kafka connection tests:

• The Validating TLS Requirements test validates if the files you specified for the Trusted Certificate, Client Certificate, and Client Certificate Key fields are in the .pem format. We perform this test only if you select TLS as the Security Protocol.
• The Verify Broker Address test verifies that the broker address you specified is not an IP address. We perform this test only if you select Connect via SSH Tunnel as the Connection Method.
• The Connecting to Kafka test validates the connection and checks the accessibility of your Kafka topics.
• The Unpacking JSON Messages test checks whether the connector can successfully fetch and unpack messages from your Kafka topics. We perform this test only if you select Unpacked as the Sync Type.
• The Validate Schema Registry Protocol test checks if the Schema Registry URLs that you specified start with http. We perform this test only if you select Avro or Protobuf as the Message Type.

Related articles

description Connector Overview

assignment Release Notes

settings API Connector Configuration

home Documentation Home