Heroku Kafka Setup Guide

Follow our setup guide to connect Heroku Kafka to Fivetran.

Prerequisites

To connect Heroku Kafka to Fivetran, you need a Heroku Kafka server.

Setup instructions

Create consumer group

1. Log in to the Heroku CLI.

2. Create a unique consumer group.

   heroku kafka:consumer-groups:create <group_id> --app <your-app-name>

   NOTE: If you have a dedicated plan, skip to Step 3.

Get Kafka prefix value

IMPORTANT: Perform this step only if you do not have a dedicated plan.

1. Get the KAFKA_PREFIX value.

   heroku config:get KAFKA_PREFIX --app <your-app-name>

2. Save the consumer group ID as <KAFKA_PREFIX><group_id>.

   For example, if KAFKA_PREFIX is northcanadian-1000. and the consumer group you created is fivetran-group, the consumer group ID will be northcanadian-1000.fivetran-group.

Get Heroku Kafka credentials

1. Get Kafka URLs.

   heroku config:get KAFKA_URL --app <your-app-name>

2. Get Kafka trusted certificate, client certificate, and private key and save them in separate .pem files.

   heroku config:get KAFKA_TRUSTED_CERT --app <your-app-name>
   heroku config:get KAFKA_CLIENT_CERT --app <your-app-name>
   heroku config:get KAFKA_CLIENT_CERT_KEY --app <your-app-name>
   

Finish Fivetran configuration

1. In the connector setup form, enter your Destination schema name.

2. Enter the Consumer Group you created Step 1. The consumer group should be unique to a connector to keep data integrity.

3. Enter your exact server details . (For example, if the server name is kafka+ssl://ec2-10-100-10-10.compute-1.amazonaws.com:9096, then enter the server name as kafka+ssl://ec2-10-100-10-10.compute-1.amazonaws.com:9096.)

4. Select a message type: Json or Text. If you selected Json, select a sync type: Packed or Unpacked.

5. Select a security protocol: PLAINTEXT or TLS. If you selected TLS, upload the certificates and private key files that you found in Step 3, then enter trust store type and key store type (default value is PKCS12).

6. (Hybrid Deployment only) If your destination is configured for Hybrid Deployment, the Hybrid Deployment Agent associated with your destination is pre-selected in the Select an existing agent drop-down menu. To use a different agent, select the agent of your choice, and then select the same agent for your destination.

7. Click Save & Test. Fivetran will take it from here and sync your Heroku Kafka data.

Fivetran tests and validates the Heroku Kafka connection. On successful completion of the setup tests, you can sync your Heroku Kafka data to your destination.

Setup tests

Fivetran performs the following Heroku Kafka connection tests:

• The Validating TLS Requirements test validates if the files you specified for the Trusted Certificate, Client Certificate, and Client Certificate Key fields are in the .pem format. We perform this test only if you select TLS as the Security Protocol.
• The Connecting to Kafka test validates the connection and checks the accessibility of your Kafka topics.
• The Unpacking JSON Messages test checks whether the connector can successfully fetch and unpack messages from your Kafka topics. We perform this test only if you select Unpacked as the Sync Type.

Related articles

description Connector Overview

assignment Release Notes

settings API Connector Configuration

home Documentation Home