Heroku Kafka Setup Guide link
Follow our setup guide to connect Heroku Kafka to Fivetran.
Prerequisiteslink
To connect Heroku Kafka to Fivetran, you need a Heroku Kafka server.
Setup instructionslink
Create consumer grouplink
Log in to the Heroku CLI.
Create a unique consumer group.
heroku kafka:consumer-groups:create <group_id> --app <your-app-name>
NOTE: If you have a dedicated plan, skip to Step 3.
Get Kafka prefix valuelink
IMPORTANT: Perform this step only if you do not have a dedicated plan.
Get the
KAFKA_PREFIX
value.heroku config:get KAFKA_PREFIX --app <your-app-name>
Save the consumer group ID as
<KAFKA_PREFIX><group_id>
.For example, if
KAFKA_PREFIX
isnorthcanadian-1000.
and the consumer group you created isfivetran-group
, the consumer group ID will benorthcanadian-1000.fivetran-group
.
Get Heroku Kafka credentialslink
Get Kafka URLs.
heroku config:get KAFKA_URL --app <your-app-name>
Get Kafka trusted certificate, client certificate, and private key and save them in separate
.pem
files.heroku config:get KAFKA_TRUSTED_CERT --app <your-app-name> heroku config:get KAFKA_CLIENT_CERT --app <your-app-name> heroku config:get KAFKA_CLIENT_CERT_KEY --app <your-app-name>
Finish Fivetran configurationlink
In the connector setup form, enter your Destination schema name.
Enter the Consumer Group you created Step 1. The consumer group should be unique to a connector to keep data integrity.
Enter your server details
<host>:<port>
format. (For example, if the server name iskafka+ssl://ec2-10-100-10-10.compute-1.amazonaws.com:9096
, then enter the server name asec2-10-100-10-10.compute-1.amazonaws.com:9096
.)Select a message type: Json or Text. If you selected Json, select a sync type: Packed or Unpacked.
Select a security protocol: PLAINTEXT or TLS. If you selected TLS, upload the certificates and private key files that you found in Step 3, then enter trust store type and key store type (default value is
PKCS12
).Click Save & Test. Fivetran will take it from here and sync your Heroku Kafka data.
Related articleslink
description Connector Overview
settings API Connector Configuration