AWS MSK Setup Guide

Follow our setup guide to connect AWS MSK to Fivetran.

Prerequisites

To connect AWS MSK to Fivetran, you need an AWS MSK cluster.

Setup instructions

IMPORTANT: If you have enabled public access to the brokers of MSK clusters, skip to the Finish Fivetran configuration step.

(Optional) Configure AWS PrivateLink

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in AWS’ PrivateLink documentation.

Follow our AWS PrivateLink setup instructions to configure PrivateLink for your AWS MSK platform.

For more information about configuring MSK clusters with AWS PrivateLink, see AWS' documentation.

IMPORTANT: In PrivateLink connections, you don't need to expose brokers to public IP addresses. If you are using a PrivateLink connection, skip ahead to the Finish Fivetran configuration step.

Find ENI ID

1. Open the AWS MSK console and go to your cluster.

2. In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).

   

Associate address

1. Open the EC2 dashboard and select Network Interfaces.

2. For every broker ENI ID you found in Step 2, select Actions > Associate address to associate an Elastic IP address.

   

3. Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.

Configure security groups

1. On the EC2 dashboard, select Security Groups.

2. In the Security Groups of the ENI IDs you found in Step 2, safelist Fivetran's IP addresses.

Set property

Set the advertised.listeners property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use dynamic configuration to configure the broker-level properties.

(Optional - TLS) Download required certificates and passwords

1. Download the kafka.client.truststore.jks and kafka.client.keystore.jks files you created while configuring TLS. You will need them to configure Fivetran.

2. Make a note of the Your-Store-Pass and Your-Key-Pass passwords. You will need them to configure Fivetran.

   TIP: We recommend using the same value for Your-Store-Pass and Your-Key-Pass while configuring TLS.

(Optional - IAM role-based authentication) Create IAM policy and IAM role

Note: This is required if you select security protocol as SASL and SASL Mechanism as IAM role-based authentication.

Create IAM policy

1. Open your Amazon IAM console.

2. On the navigation menu, select Policies, and then click Create Policy.

   

3. Switch to the JSON tab.

   

4. Copy the following policy and paste it into the Policy editor. Replace {your-cluster-region} with your cluster region, {your-account-number} with your account number, {your-cluster-name} with your cluster name and {your-cluster-UUID} with the UUID of your AWS MSK cluster. You can get this information from your cluster ARN.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "kafka-cluster:Connect",
                   "kafka-cluster:DescribeCluster"
               ],
               "Resource": [
                   "arn:aws:kafka:{your-cluster-region}:{your-account-number}:cluster/{your-cluster-name}/{your-cluster-UUID}"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "kafka-cluster:DescribeTopic",
                   "kafka-cluster:ReadData"
               ],
               "Resource": [
                   "arn:aws:kafka:{your-cluster-region}:{your-account-number}:topic/{your-cluster-name}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "kafka-cluster:AlterGroup",
                   "kafka-cluster:DescribeGroup"
               ],
               "Resource": [
                   "arn:aws:kafka:{your-cluster-region}:{your-account-number}:group/{your-cluster-name}/*"
               ]
           }
       ]
   }
   

5. Click Next.

   

6. In the Policy details section, specify the Policy name, for example, "AWS_MSK_Access".

   

7. Click Create policy.

   

Create IAM role

1. In the connector setup form, find the automatically-generated External ID and make a note of it.

   NOTE: The automatically-generated External ID is tied to your account. If you close and re-open the setup form, the ID will remain the same. You can keep the tab open in the background while you configure your source for convenience.

2. On the navigation menu, select Roles, and then click Create role.

   

3. Select AWS account, and then select Another AWS account and enter Fivetran’s AWS VPC Account ID, 834469178297, in the Account ID field.

   

4. Select the Require external ID checkbox and enter the External ID you found above, then click Next.

   

5. In the Add permissions step, select the policy you created and click Next.

   

6. In the Name, review, and create step, specify the role name, for example "Fivetran", and click Create role at the bottom of the page.

   

7. Click the Fivetran role you created.

   

8. On the Summary page for the role, make a note of the ARN. You will need it to configure Fivetran.

   

Finish Fivetran configuration

1. In the connector setup form, enter the Destination schema name of your choice.

2. Enter a Consumer Group name. The consumer group should be unique to a connector to keep data integrity.

3. Enter your Servers details using the <host>:<port> format. Based on your connectivity type, do either of the following:

   • Public access: Enter the public broker endpoints from the AWS MSK Management Console. In the Cluster summary section, go to the View client information tab. Copy the Public endpoint information.

     TIP: Click + to add the host:port pairs. For example, if the endpoint is b-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092,b-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092, enter b-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092 and b-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092.

   • Non-public access: Use the Elastic IPs you found in Step 3 as <host>. For example, if the Elastic IP for the broker is 10.100.10.10 and the authentication is SASL/SCRAM, then enter the server name as 10.100.10.10:9096.

     NOTE: Click + to add more than one set of server details.

   • Private Link access: Enter the private endpoints from the AWS MSK Management Console. In the Cluster summary section, go to the View client information tab. Copy the Private endpoint (multi-VPC) information.

     TIP: Click + to add the host:port pairs. For example, if the endpoint is b-1.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14001,b-2.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14002, enter b-1.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14001 and b-2.iam.dummyclustername.abc123.c12.kafka.us-east-1.amazonaws.com:14002.

4. Select your message type: Json, Avro, Protobuf, or Text.

   • If you choose Json, select a sync type: Packed or Unpacked.

   • If you choose Avro or Protobuf, enter the following schema registry credentials:

     • Schema Registry URLs
     • Schema Registry Key
     • Schema Registry Secret

5. Select a security protocol: PLAINTEXT, SASL, or TLS.

   • If you choose SASL, you must select one of the following as the SASL Mechanism and provide details:

     • SASL/SCRAM authentication: Enter your secret's Username and Password.
     • IAM role-based authentication: Enter the Role ARN that you created.

   • If you choose TLS, provide the following details you found in Step 6:

     • In the Truststore File field, upload the kafka.client.truststore.jks file.
     • In the Keystore File field, upload the kafka.client.keystore.jks file.
     • Enter the Keystore Password.
     • Enter the Key Password.

6. Select your connection method: Connect directly or Connect via PrivateLink. If you choose Connect via PrivateLink, Fivetran connects to your message brokers using AWS PrivateLink.

7. Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.

Fivetran tests and validates the AWS MSK connection. On successful completion of the setup tests, you can sync your AWS MSK data to your destination.

Setup tests

Fivetran performs the following AWS MSK connection tests:

• The Verifying connection using Private Link test validates the connection and checks the accessibility of your cluster using AWS PrivateLink. We perform this test only if you select Connect via Private link as the Connection Method.
• The Connecting to AWS MSK test checks the connection and the cluster accessibility.
• The Unpacking JSON messages test checks whether the connector can successfully fetch and unpack messages from your AWS cluster. We perform this test only if you select Json as the Message Type and Unpacked as the Sync Type.
• The Validate Schema Registry protocol test checks if the Schema Registry URLs that you specified start with http. We perform this test only if you select Avro or Protobuf as the Message Type.

Related articles

description Connector Overview

assignment Release Notes

settings API Connector Configuration

home Documentation Home