AWS MSK Setup Guide link
Follow our setup guide to connect AWS MSK to Fivetran.
Prerequisiteslink
To connect AWS MSK to Fivetran, you need an AWS MSK cluster.
Setup instructionslink
IMPORTANT: If you have enabled public access to the brokers of MSK clusters, skip to the Finish Fivetran configuration step.
(Optional) Configure AWS PrivateLink link
IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.
AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in AWS’ PrivateLink documentation.
Follow our AWS PrivateLink setup instructions to configure PrivateLink for your AWS MSK platform.
For more information about configuring MSK clusters with AWS PrivateLink, see AWS' documentation.
IMPORTANT: In PrivateLink connections, you don't need to expose brokers to public IP addresses. If you are using a PrivateLink connection, skip ahead to the Finish Fivetran configuration step.
Find ENI IDlink
Open the AWS MSK console and go to your cluster.
In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).
Associate addresslink
Open the EC2 dashboard and select Network Interfaces.
For every broker ENI ID you found in Step 2, select Actions > Associate address to associate an Elastic IP address.
Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.
Configure security groupslink
On the EC2 dashboard, select Security Groups.
In the Security Groups of the ENI IDs you found in Step 2, safelist Fivetran's IP addresses.
Set propertylink
Set the advertised.listeners
property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use dynamic configuration to configure the broker-level properties.
(Optional - TLS) Download required certificates and passwordslink
Download the
kafka.client.truststore.jks
andkafka.client.keystore.jks
files you created while configuring TLS. You will need them to configure Fivetran.Make a note of the
Your-Store-Pass
andYour-Key-Pass
passwords. You will need them to configure Fivetran.TIP: We recommend using the same value for
Your-Store-Pass
andYour-Key-Pass
while configuring TLS.
(Optional - IAM role-based authentication) Create IAM policy and IAM rolelink
Note: This is required if you select security protocol as SASL and SASL Mechanism as IAM role-based authentication.
Create IAM policylink
Open your Amazon IAM console.
On the navigation menu, select Policies, and then click Create Policy.
Switch to the JSON tab.
Copy the following policy and paste it into the Policy editor. Replace
{your-cluster-region}
with your cluster region,{your-account-number}
with your account number,{your-cluster-name}
with your cluster name and{your-cluster-UUID}
with the UUID of your AWS MSK cluster. You can get this information from your cluster ARN.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kafka-cluster:Connect", "kafka-cluster:DescribeCluster" ], "Resource": [ "arn:aws:kafka:{your-cluster-region}:{your-account-number}:cluster/{your-cluster-name}/{your-cluster-UUID}" ] }, { "Effect": "Allow", "Action": [ "kafka-cluster:DescribeTopic", "kafka-cluster:ReadData" ], "Resource": [ "arn:aws:kafka:{your-cluster-region}:{your-account-number}:topic/{your-cluster-name}/*" ] }, { "Effect": "Allow", "Action": [ "kafka-cluster:AlterGroup", "kafka-cluster:DescribeGroup" ], "Resource": [ "arn:aws:kafka:{your-cluster-region}:{your-account-number}:group/{your-cluster-name}/*" ] } ] }
Click Next.
In the Policy details section, specify the Policy name, for example, "AWS_MSK_Access".
Click Create policy.
Create IAM rolelink
In the connector setup form, find the automatically-generated External ID and make a note of it.
NOTE: The automatically-generated External ID is tied to your account. If you close and re-open the setup form, the ID will remain the same. You can keep the tab open in the background while you configure your source for convenience.
On the navigation menu, select Roles, and then click Create role.
Select AWS account, and then select Another AWS account and enter Fivetran’s AWS VPC Account ID,
834469178297
, in the Account ID field.Select the Require external ID checkbox and enter the External ID you found above, then click Next.
In the Add permissions step, select the policy you created and click Next.
In the Name, review, and create step, specify the role name, for example "Fivetran", and click Create role at the bottom of the page.
Click the Fivetran role you created.
On the Summary page for the role, make a note of the ARN. You will need it to configure Fivetran.
Finish Fivetran configurationlink
In the connector setup form, enter the Destination schema name of your choice.
Enter a Consumer Group name. The consumer group should be unique to a connector to keep data integrity.
Enter your Servers details using the
<host>:<port>
format. Based on your connectivity type, do either of the following:Public access: Enter the public broker endpoints from the AWS MSK Management Console. In the Cluster summary section, go to the View client information tab. Copy the Public endpoint information.
TIP: Click + to add the host:port pairs. For example, if the endpoint is
b-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092,b-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092
, enterb-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092
andb-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092
.Non-public access: Use the Elastic IPs you found in Step 3 as
<host>
. For example, if the Elastic IP for the broker is10.100.10.10
and the authentication isSASL/SCRAM
, then enter the server name as10.100.10.10:9096
.NOTE: Click + to add more than one set of server details.
Private Link access: Enter the private endpoints from the AWS MSK Management Console. In the Cluster summary section, go to the View client information tab. Copy the Private endpoint (multi-VPC) information.
TIP: Click + to add the host:port pairs. For example, if the endpoint is
b-1.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14001,b-2.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14002
, enterb-1.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14001
andb-2.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14002
.
Select your message type: Json, Avro, Protobuf, or Text.
If you choose Json, select a sync type: Packed or Unpacked.
If you choose Avro or Protobuf, enter the following schema registry credentials:
- Schema Registry URLs
- Schema Registry Key
- Schema Registry Secret
Select a security protocol: PLAINTEXT, SASL, or TLS.
If you choose SASL, you must select one of the following as the SASL Mechanism and provide details:
- SASL/SCRAM authentication: Enter your secret's Username and Password.
- IAM role-based authentication: Enter the Role ARN that you created.
If you choose TLS, provide the following details you found in Step 6:
- In the Truststore File field, upload the
kafka.client.truststore.jks
file. - In the Keystore File field, upload the
kafka.client.keystore.jks
file. - Enter the Keystore Password.
- Enter the Key Password.
- In the Truststore File field, upload the
Select your connection method: Connect directly or Connect via PrivateLink. If you choose Connect via PrivateLink, Fivetran connects to your message brokers using AWS PrivateLink.
Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.
Related articleslink
description Connector Overview
settings API Connector Configuration