Webhooks Setup Guide

Follow our setup guide to connect Webhooks to Fivetran.

Prerequisites

To connect Webhooks to Fivetran, you need an HTTP client setup on your website, application, or any other place.

IMPORTANT: Only use an HTTPS URL with TLS version 1.2 or above.

Setup instructions

Begin Fivetran configuration

1. In the connector setup form, enter the Destination schema name of your choice.

2. Enter the Destination table name.

3. Select the Sync format in which form you'd like your data to be loaded, Packed or Unpacked. For more information, see the Sync Overview section.

4. Find the Webhook URL and make a note of it. The URL is in the format https://webhooks.fivetran.com/webhooks/<endpoint_ID>. You will need it to configure Webhooks to connect with Fivetran.

5. Select the container service you want to use:

   • Fivetran: We will store the events in our own container service.

   • AWS S3: We will store the events in your S3 bucket.

     NOTE: If you choose AWS S3, find the automatically-generated External ID and make a note of it. You will need it to configure AWS to connect with Fivetran.

   • Azure Blob Storage: We will store the events in your Azure Blob Storage container.

   • Google Cloud Storage: We will store the events in your Google Cloud Storage bucket.

   NOTE: Make sure you don't have any Life Cycle Management Policies configured in your custom storage container.

6. (Optional) Select the Authentication mechanism you want to use to authenticate incoming events:

   IMPORTANT: If you choose HMAC or Token Based authentication, a Secrets field will be displayed that contains your authentication secrets. Make sure to copy and store the secrets securely. This field is not displayed after you click Save & Test in the setup form.

   • None: Fivetran doesn't authenticate incoming events.

   • HMAC: Fivetran uses an HMAC (Hash-based Message Authentication Code) signature to authenticate.

     NOTE: You must pass the signature in the header field, X-Fivetran-Hmac-Sign, in each webhook event.

     NOTE: Pass your preferred HMAC algorithm in the header field, X-Fivetran-Hmac-Algo. For requests without an algorithm header, we use HMAC_SHA_256 as the default algorithm. We support the following algorithms:

     • HMAC_MD5
     • HMAC_SHA_1
     • HMAC_SHA_224
     • HMAC_SHA_256
     • HMAC_SHA_384
     • HMAC_SHA_512

   • Token Based: Fivetran uses a secret token to authenticate.

     NOTE: You must pass the secrets in the header field, X-Fivetran-Secret-Token, in each webhook event.

   NOTE: Fivetran validates the secrets from the header and accepts the event only if the values match the secrets displayed initially in the setup form.

Create IAM policy

To allow Fivetran access to your S3 bucket, follow the instructions below.

1. Open the Amazon IAM console.

2. Go to Policies, then select Create Policy.

   

3. Go to the JSON tab.

   

5. Copy the following policy, replacing {your-bucket-name} with the name of your S3 bucket.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:Get*",
                   "s3:List*",
                   "s3:PutObject"
               ],
               "Resource": "arn:aws:s3:::{your-bucket-name}*"
           }
       ]
   }
   

6. Click Review Policy.

   

7. Name the policy "Fivetran-S3-Access".

   

8. Click Create Policy.

Create IAM role

1. Go to Roles, then select Create New Role.

   

2. Select Another AWS Account, then enter Fivetran's account ID, 834469178297, in the Account ID field.

   

3. Select the Require external ID checkbox.

4. Enter the External ID you found in Step 1.

   

5. Click Next: Permissions.

6. Select the 'Fivetran-S3-Access' policy that you created in Step 2.

   

7. Click Next: Tags, which is optional.

8. Click Next: Review.

9. Name your new role 'Fivetran', then click Create Role.

   

10. Select the Fivetran role you just created.

    

11. Find the Role ARN and make a note of it. You will need to enter this value in the connector setup form.

    

(Optional) Generate SAS URL

If you have selected Azure Blob Storage as custom external bucket, you need to generate a SAS URL as follows:

1. Inside your Azure Blob Storage, click Shared Access Token and select the Read, Write and List permissions.

   

2. Click Generate SAS token and URL and copy the value displayed in the Blob SAS URL field.

   

Finish Fivetran configuration

1. If you have selected a custom external bucket:

   • For AWS S3 - Enter the S3 bucket name and the Role ARN that you created in AWS.
   • For Azure Blob Storage - Enter the Azure Blob SAS URL.
   • For Google Cloud Storage - Enter the GCS Bucket.

2. Click Save & Test. Fivetran will load events into your destination.

IMPORTANT: It takes us around 10 to 15 minutes to load the first event into your destination.

Fivetran tests and validates the Webhooks connection. On successful completion of the setup tests, you can sync your Webhooks data to your destination.

Setup tests

Fivetran performs the following Webhooks connection tests:

• The Validating read and write access test for Azure checks whether the connector can successfully read and write data to your Azure Blob Storage container. We perform this test only if you choose Azure Blob Storage as the container service.
• The Validating read and write access test for GCS checks whether the connector can successfully read and write data to your GCS bucket. We perform this test only if you choose Google Cloud Storage as the container service.
• The Validating read and write access tests for AWS S3 checks whether the connector can successfully read and write data to your S3 bucket. We perform this test only if you choose AWS S3 as the container service.

Related articles

description Connector Overview

account_tree Schema Information

assignment Release Notes

settings API Connector Configuration

home Documentation Home