Apache Kafka Setup Guide link
Follow our setup guide to connect Apache Kafka to Fivetran.
Prerequisiteslink
To connect Apache Kafka to Fivetran, you need an Apache Kafka server.
Setup instructionslink
Run the following commands in your Apache Kafka server.
Generate and import client certificate to server truststorelink
Create a key and keystore for the consumer client.
keytool -keystore <client_keystore_name>.jks -alias <alias_name> -validity <VALIDITY> -genkey -keyalg RSA
You will be asked to enter a few details. For
What is your first and last name?
, enter your Apache Kafka server hostname. Remember the key and keystore password you provide.Export the client certificate from the keystore.
keytool -exportcert -rfc -file <client_certificate_name>.pem -alias <alias_name> -keystore <client_keystore_name>.jks
The
<alias_name>
must be the same as in step 1.Import the client certificate into the truststore of your Apache Kafka server.
keytool -keystore <server_truststore_name>.jks -alias <alias_name> -import -file <client_certificate_name>.pem
Get client certificate key from client keystorelink
Convert the client keystore from
JKS
toPKCS12
format.keytool -v -importkeystore -srckeystore <client_keystore_name>.jks -srcalias <alias_name> -destkeystore <client_keystore_name>.p12 -deststoretype PKCS12
Extract client certificate key into a
.pem
file.openssl pkcs12 -in <client_keystore_name>.p12 -nocerts -nodes > <client_certificate_key>.pem
Download required certificates and keylink
Download the following files from your Apache Kafka server:
- Client certificate(
<client_certificate_name>.pem
) - Client certificate key (
<client_certificate_key>.pem
) - CA certificate (Trusted Certificate): It is a public-private key pair and certificate in your Apache Kafka server which was used to sign other certificates
Finish Fivetran configuration link
In the connector setup form, enter the Destination schema name of your choice.
Enter a Consumer Group name. The consumer group should be unique to a connector to keep data integrity.
In the Servers section, click + Add. Enter your server details in
<server_host_name>:<port>
format.Select your message type: Json, Avro, Protobuf, or Text.
If you choose Json, select a sync type: Packed or Unpacked.
If you choose Avro or Protobuf, in the Schema Registry Urls section, click + Add and then enter the following schema registry credentials:
- Schema Registry Urls
- Schema Registry Key
- Schema Registry Secret
(Optional) Select your sync type: Packed or Unpacked.
Select a security protocol: PLAINTEXT, TLS, or SASL.
If you choose TLS, upload the Trusted Certificate, Client Certificate, and Client Certificate Key files.
If you choose SASL, select the SASL Mechanism, and then enter your API Key and API Secret.
Click Save & Test. Fivetran will take it from here and sync your Apache Kafka data.
Related articleslink
description Connector Overview
settings API Connector Configuration