AWS Lambda Setup Guide

Prerequisites

To connect AWS Lambda functions to Fivetran, you need:

An AWS Lambda function
An AWS account with Administrator privileges

Setup instructions

Begin Fivetran configuration

In the connector setup form, enter the Destination schema name of your choice.
Make a note of the External ID. You will need it to configure AWS to connect with Fivetran.
NOTE: The automatically-generated External ID is tied to your account. If you close and re-open the setup form, the ID will remain the same.
Decide your sync method. Whether to sync directly or using a S3 bucket.

Create IAM policy

Open the Amazon IAM console.
Go to Access management > Policies.
Click Create policy.
In the Create policy window, go to the JSON tab.

Copy the following policy and paste it in the JSON tab, and then click Review policy.

NOTE: If you want to sync directly, remove Sid:AccessS3bucket from the following policy. If you want to sync through S3 bucket, replace \<bucket-name\> with your bucket name in the following policy.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "InvokePermission",
           "Effect": "Allow",
           "Action": "lambda:InvokeFunction",
           "Resource": "*"
       },
       {
           "Sid": "AccessS3bucket",
           "Effect": "Allow",
           "Action": [
               "s3:Put*",
               "s3:Get*",
               "s3:Delete*"
           ],
           "Resource": [
               "arn:aws:s3:::<bucket-name>",
               "arn:aws:s3:::<bucket-name>/*"
           ]
       }
   ]
}

In the Review policy window, enter the policy name and add a description. For example, 'Fivetran-Lambda-Invoke'.
Click Create policy.

Create IAM role

Go to Access management > Roles, and then select Create role.
In the Create role window, select Another AWS account, and then in the Account ID field, enter Fivetran's account ID, 834469178297.
In Options, select the Require external ID checkbox.
In the External ID field, enter the External ID you got from Step 1.
Click Next: Permissions.
Select the 'Fivetran-Lambda-Invoke' policy that you created in Step 2.
Click Next: Tags. Entering tags is optional, but you must click through the step.
Click Next: Review.
Name your new role 'Fivetran' and then click Create role.
Select the Fivetran role that you just created.
In the Summary section, make a note of the Role ARN value. You will need to enter this value in Step 6.

copy role arn

Create Lambda function

Open AWS Lambda console.
Click Create function.
Select Use a blueprint if you want to create the function using an existing template, or select Author from scratch and enter the following information:
- Function Name
- Runtime (C#, Go, Java, Node.js, Python)
From the Execution role options, select Use an existing role and then choose the role you created above. Click Create function.
NOTE: If your IAM role does not appear in the Existing role drop-down menu, you'll need to edit the trust relationship through the AWS IAM portal. Do the following to add Lambda services to your trusted relationships in addition to the Fivetran entity:
a. Select the designated Fivetran role.
b. In the Summary section, click the Trust Relationships tab, and then click Edit trust relationships.
c. On the next page, add the following snippet to your JSON code:
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::834469178297:user/gcp_donkey"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "your_fivetran_externalID"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```
Your JSON policy document should look like the one illustrated below.
d. Make sure that the External ID field has your External ID from Step 1. Click Update Trust Policy.
Add your code for the Lambda function. You can upload the code or write the code in the editor.
Click Deploy.
NOTE: For more information about Lambda function implementation, see our AWS Lambda Sample Functions.
Click Test > Configure test event to add a new test event.
Enter the Event name and select the Template.
Enter the JSON and click Save.
Click Test to verify the Lambda function response.

verify response

Configure execution time

AWS Lambda terminates the execution of a function when the Timeout value is reached. The default execution time value for an AWS Lambda function is three seconds, and the maximum value is 15 minutes.

As your function involves bulk data transfer, configure the function to run for a longer time. We recommend that you set the timeout value based on your expected execution time. To change the default timeout value:

Sign in to the AWS Management Console.
Navigate to the AWS Lambda console.
Select your function.
Click the Configuration tab and then go to the Basic settings section.
Set the Timeout value.

set execution time

NOTE: If you don't increase the execution time of your function, after setting up the connector, you may receive a Function Execution error on your dashboard.

Finish Fivetran configuration

In the connector setup form, enter the Role ARN you created in Step 3.
Enter the Lambda Function name you created in Step 4.
Choose your Region.
(Optional) Click + Add secrets to specify your secrets as key-value pairs.
- Enter your Secret Name. The secret name must be unique.
- Enter your Secret Key associated with the secret name.
For example, if you want to pass the secrets as {'apiKey': 'yourApiKey', 'consumerKey': 'test'}, add a key-value pair for each entry in the JSON structure. Make sure that the key names are unique in the key-value pairs.
TIP: For Lambda connectors created before October 4, 2022, enter your Secrets using the following JSON format:
```
 {
  "consumerKey": "",
  "consumerSecret": "",
  "apiKey": "yourApiKey"
  }
```
NOTE: For more information on secrets, see our Function request and response documentation.
(Optional) Click + Add payload(s) to specify your custom payload(s) as key-value pair(s). Make sure that the key names are unique in the key-value pairs.
- Enter your Payload Name. The payload name must be unique.
- Enter your Payload Value associated with the payload name.
Choose the Sync Method: Sync directly or Sync through S3 bucket. If you select Sync through S3 bucket, enter the S3 Bucket name you want to use to push data from your function.
(Optional) To always connect using AWS PrivateLink, set the Require PrivateLink toggle to ON.
NOTE: By default, we use PrivateLink to connect if your AWS Lambda function and destination are in the same region. Enabling this option ensures that we always use PrivateLink to connect. If the regions are different, Fivetran cannot connect using PrivateLink.
Click Save & Test.

Fivetran tests and validates the AWS Lambda connection. On successful completion of the setup tests, you can sync the data returned by the Lambda function to your destination.

Setup tests

Fivetran performs the following AWS Lambda connection tests:

The Validate Secrets test checks if you have entered the secrets in a valid JSON format.
The Validate Payloads test checks if you have entered the payloads in a valid JSON format.
The Function Connection test checks if we can connect to your Lambda Function and if the function’s response format is correct.

IMPORTANT: The setup tests may timeout if your function’s response size is more than 500 megabytes (MB). In the setup tests request, Fivetran passes the setup_test field as true; and your function must return a lightweight JSON object with the hasMore field as false, not exceeding 500 MB in size. The 500 MB size limitation is only applicable to the setup tests and not for your syncs.

Connector Overview

Release Notes

API Connector Configuration

Documentation Home