AWS MSK Destination Setup Guide

Create schema registry

Create your AWS Glue Schema Registry by following the instructions in AWS' documentation.

NOTE: Use the GlueSchemaRegistryKafkaDeserializer class to deserialize the values while consuming data from the topic.

Find External ID

1. In the destination setup form, in the Schema Registry drop-down menu, select AWS Glue.
2. Find the automatically-generated External ID and make a note of it. You will need it to create an IAM role.

NOTE: The automatically-generated External ID is tied to your account. The ID does not change even if you close and re-open the setup form. For your convenience, you can keep the browser tab open in the background while you configure your destination.

Create IAM policy for AWS Glue

1. Open the Amazon IAM console.

2. Go to Policies, and then click Create Policy.

   

3. Go to the JSON tab.

   

4. Copy the following policy and paste it in the JSON editor.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "glue:ListSchemaVersions",
                   "glue:RegisterSchemaVersion",
                   "glue:GetSchemaVersionsDiff",
                   "glue:RemoveSchemaVersionMetadata",
                   "glue:UpdateSchema",
                   "glue:GetSchema",
                   "glue:PutSchemaVersionMetadata",
                   "glue:DeleteSchema",
                   "glue:QuerySchemaVersionMetadata",
                   "glue:ListSchemas",
                   "glue:CreateSchema",
                   "glue:DeleteSchemaVersions",
                   "glue:GetSchemaByDefinition"
               ],
               "Resource": [
                   "arn:aws:glue:{region}:{your-account-id}:schema/*",
                   "arn:aws:glue:{region}:{your-account-id}:registry/{your-registry-name}"
               ]
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": "glue:CheckSchemaVersionValidity",
               "Resource": "*"
           },
           {
               "Sid": "VisualEditor2",
               "Effect": "Allow",
               "Action": "glue:GetSchemaVersion",
               "Resource": "*"
           }
       ]
   }
   
   

5. In the policy, replace {your-account-id} with your AWS account ID, {region} with the region of your Glue registry, and {your-registry-name} with the name of your registry.

6. Click Next.

7. Enter a Policy name, and then click Create policy.

Create Glue IAM role

1. On the Amazon IAM console, go to Roles, and then click Create role.

   

2. Select AWS account, and then select Another AWS account.

   

3. In the Account ID field, enter Fivetran's account ID, 834469178297.

4. Select the Require external ID checkbox, and then enter the External ID you found.

   

5. Click Next.

6. Select the IAM policy you created.

   

7. Click Next.

8. In the Role name field, enter a name for the role, and then click Create role.

   

9. In the Roles page, select the role you created.

10. Make a note of the ARN. You will need it to configure Fivetran.

NOTE: After completing this step, skip to the Configure AWS PrivateLink step.

Set up your Confluent Cloud Schema Registry by following the instructions in Confluent Cloud's documentation.

If you want to use SASL/SCRAM authentication, provide read and write access to your Kafka topics by following the instructions in AWS' documentation.

NOTE: After completing this step, skip to the Find Bootstrap server names step.

Find MSK External ID

1. In the destination setup form, in the SASL Mechanism drop-down menu, select AWS_MSK_IAM.
2. Find the automatically-generated External ID and make a note of it. You will need it to create an IAM role.

Create IAM policy for AWS MSK

1. Open the Amazon IAM console.

2. Go to Policies, and then click Create Policy.

3. Go to the JSON tab.

4. Copy the following policy and paste it in the JSON editor.

   {
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
               "kafka-cluster:DescribeTopicDynamicConfiguration",
               "kafka-cluster:AlterTopicDynamicConfiguration",
               "kafka-cluster:WriteDataIdempotently",
               "kafka-cluster:AlterClusterDynamicConfiguration",
               "kafka-cluster:CreateTopic",
               "kafka-cluster:AlterTopic",
               "kafka-cluster:ReadData",
               "kafka-cluster:AlterCluster",
               "kafka-cluster:DescribeTopic",
               "kafka-cluster:Connect",
               "kafka-cluster:DeleteTopic",
               "kafka-cluster:WriteData"
           ],
           "Resource": [
               "arn:aws:kafka:{region}:{your-account-id}:topic/*/*/*",
               "{msk-cluster-arn}"
           ]
       }
   ] 
   }
   

5. In the policy, replace {your-account-id} with your AWS account ID, {region} with the region of your MSK Cluster, and {msk-cluster-arn} with your MSK Cluster ARN.

6. Click Next.

7. Enter a Policy name, and then click Create policy.

Create MSK IAM role

1. On the Amazon IAM console, go to Roles, and then click Create role.
2. Select AWS account, and then select Another AWS account.
3. In the Account ID field, enter Fivetran's account ID, 834469178297.
4. Select the Require external ID checkbox, and then enter the External ID you found.
5. Click Next.
6. Select the IAM policy you created.
7. Click Next.
8. In the Role name field, enter a name for the role, and then click Create role.
9. In the Roles page, select the role you created.
10. Make a note of the ARN. You will need it to configure Fivetran.