Bring Your Own S3 Bucket

After following the steps below, you may enter the following information in the Activation Source creation wizard:

1. Bucket Name
2. Role ARN

Initial setup

This guide will take you through the steps to prepare a bucket for use with an Activation Source.

We will use the aws CLI to create and configure your bucket - you should install this tool and configure it to use your credentials before starting. Similar configuration is also possible using the AWS Console UI but those instructions are not provided here.

In order to proceed, you will also need to add the Activations AWS Account IDs to your bucket's trust policy and your customer-owned bucket External ID. Your trust policy must include all of the following AWS principals:

• 341876425553 (historically used by Fivetran)
• 834469178297

An External ID is provided by the Activation Source creation wizard.

1. Choose a bucket name

Choose a name for your bucket that is memorable and clear. For example, if your company name is "FooCorp", you might call your bucket "foocorp-census-sync-files".

2. Set up shell variables

In a shell, set up a few variables we'll use throughout the rest of this guide:

The CENSUS_CUSTOMER_BUCKET_EXTERNAL_ID is provided by the Activation Source creation wizard.

export BUCKET_NAME=<your bucket name>
export CENSUS_CUSTOMER_BUCKET_EXTERNAL_ID=<provided by Fivetran>

3. Create and configure the bucket

Create the S3 bucket. Note that it's okay if your data warehouse is located in a different region than your bucket.

aws s3api create-bucket --acl private --bucket $BUCKET_NAME --region=us-east-1

Remove all public access from the bucket:

aws s3api put-public-access-block --bucket $BUCKET_NAME \
  --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Encrypt all data in the bucket:

aws s3api put-bucket-encryption --bucket $BUCKET_NAME \
  --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'

Choose an automatic retention period that is longer than your expected longest sync duration (at least 3 days) and sufficient for how long you want to retain records for detailed tracking and debugging. Avoid setting the retention period too short to prevent premature deletion of important data, and not too long to avoid unnecessary storage costs. A 14-day retention period is recommended, as it balances these needs effectively. In this example, we set the retention period to 14 days, the same value Activations uses by default.

aws s3api put-bucket-lifecycle-configuration --bucket $BUCKET_NAME --lifecycle-configuration \
  '{"Rules":[{"Expiration":{"Days":14},"ID":"expire-census-temp-files","Prefix":"","Status":"Enabled","AbortIncompleteMultipartUpload":{"DaysAfterInitiation":14}}]}'

4. Create an IAM role granting Activations access

Activations needs to be able to list, read, write, and delete items from the S3 bucket. In order to do so, you'll create an IAM role defining the allowed permissions and give the Activations AWS account the ability to assume that role.

Feel free to adjust the names of the role and the inline policy to match your organization's naming scheme if you have one.

First, create an empty role with a policy that allows Activations to assume it via AWS Account IDs and External ID. Include all Fivetran AWS principals:

aws iam create-role --role-name census-data-warehouse-client --assume-role-policy-document \
  '{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"sts:AssumeRole","Principal":{"AWS":["arn:aws:iam::341876425553:root","arn:aws:iam::834469178297:root"]},"Condition":{"StringEquals":{"sts:ExternalId":"'$CENSUS_CUSTOMER_BUCKET_EXTERNAL_ID'"}}}}'

This command should print out a role document that looks something like this:

{
  "Role": {
    "Path": "/",
    "RoleName": "census-data-warehouse-client",
    "RoleId": "...",
    "Arn": "arn:aws:iam::12345678:role/census-data-warehouse-client",
    "CreateDate": "2020-09-22T20:48:21+00:00",
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Principal": {
          "AWS": [
            "arn:aws:iam::341876425553:root",
            "arn:aws:iam::834469178297:root"
          ]
        },
        "Condition": {
          "StringEquals": {
            "sts:ExternalId": "CensusCustomS3Bucket/your-external-id"
          }
        }
      }
    }
  }
}

Important: Take note of the Arn from the role document - you'll need this value to finish the configuration.

Then add a policy to the role granting limited access to the S3 bucket:

aws iam put-role-policy --role-name census-data-warehouse-client \
  --policy-name census-data-warehouse-client --policy-document \
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetBucketLocation","s3:GetObject","s3:DeleteObject","s3:ListBucket","s3:PutObject"],"Resource":["arn:aws:s3:::'$BUCKET_NAME'/*","arn:aws:s3:::'$BUCKET_NAME'"]}]}'

5. Finish up

Provide the bucket name you chose and your Role ARN (it should look like arn:aws:iam::12345678:role/census-data-warehouse-client) to the Activation Source creation wizard. Confirm that your role's trust policy includes both Fivetran AWS principals as shown above. Press "Connect" to validate the connection. The wizard will continue only if the provided bucket and role are valid and configured correctly.

Ongoing maintenance

After you set up your bucket, you don't need to take further action. Fivetran will automatically create and remove files as needed, and the S3 retention policies you defined in the previous step will automatically remove any stale data in the unlikely event that we fail to do so.

We strongly recommend that once set up, you do not modify the bucket or any of the keys it contains in any way. Adding, removing, modifying, or renaming data in your Activations bucket is not supported and will likely cause activation syncs to fail. In addition, the Activations bucket should not be simultaneously employed for any other purposes.

If you need to rename your bucket or the IAM roles you have granted to the bucket, please contact Fivetran Support.