03 Jul 2019 | Article

Four Options When Fivetran Doesn't Support Your Source

Jeff Tietz
Jeff Tietz
Four Options When Fivetran Doesn't Support Your Source
It's possible to load data from unsupported sources into your warehouse, and we're always adding connectors based on customer demand.

If Fivetran hasn’t built a connector for your data source, you can use one of the four data loading methods outlined below — but we’re continually building new connectors, so be sure to ask your Fivetran account manager whether the one you need is already under construction. If not, please let us know which connectors you need by reaching out to your account manager, contacting sales@fivetran.com or logging in to your account and filling out our connector request form:

connectorrequest

If your data source isn’t on our roadmap yet, here are your options for loading data into a warehouse:

  • File replication. Convert your data into files that can connect to your warehouse using one of the Fivetran file replication services.
  • Webhooks. Using the Fivetran webhooks connector, send data to your warehouse from applications that support webhooks.
  • Database applications. If your application sits on top of an underlying database, send data to Fivetran from applications using one of the many Fivetran database connectors.
  • Cloud functions. If you have access to your source’s API, you can write a custom cloud function allowing Fivetran to read the data from the application and write it to your warehouse.

File Replication

Fivetran can ingest data through the following file storage and transfer services:

  • Amazon Cloudfront
  • Amazon Kinesis
  • Amazon S3
  • Azure Blob Storage
  • Fivetran native browser uploader
  • Dropbox
  • Fivetran native email ingestor
  • FTP, FTPS, SFTP
  • Google Cloud Storage
  • Google Sheets

Just drop the files into any of these services and we'll bring the data into your warehouse.

Many Fivetran users are able to automate loading data into a file storage service, which is often easier and more robust than trying to automate loading directly into a warehouse. Fivetran file connectors load data into your warehouse and continuously update and normalize it.

To learn more about specific file replication connectors, visit our documentation.

Webhooks

A webhook is an HTTP callback triggered by an event on your website or application. While webhooks cannot pull historical data, they’re a great way to gather new data. Most SaaS applications support webhooks, but you will need to confirm that capability with the specific application.

To get set up, Fivetran will give you a URL that you put inside your SaaS service. As new events occur in your applications, the SaaS service will post data to that URL, which Fivetran will collect and ingest into your data warehouse. Please note that webhooks are developer tools; to implement them correctly, users need a minimum level of expertise.

To learn more, visit our webhooks documentation.

Database Applications

Fivetran supports most popular databases as data connectors. Many applications are built on top of these databases. If Fivetran supports the underlying database, we can connect to it and collect data from your application. To find out if your application can be supported in this way, reach out to your application contact or do a simple internet search to identify the underlying database of your source. When researching the underlying database, be sure to check pre-configuration documents to confirm permission requirements for connecting to your software program through the database.

Fivetran supports connectors for the following databases:

  • PostgreSQL
  • Azure
  • Oracle
  • MongoDB
  • SQL Server
  • MySQL
  • MariaDB

To learn more about database connectors, visit our documentation page.

Cloud Functions

A functions connector allows you to code a custom data connector as an extension of Fivetran. Fivetran will execute custom code in your account on a cloud platform and sync the returned data to your data warehouse. For example, if you have a custom-built application with a private REST API, and want to use Fivetan to perform analytics on the data, you can write a cloud function to read data from the application and use the Fivetran connector to write data to your warehouse. Fivetran supports cloud functions from AWS Lambda, Azure Functions and Google Cloud Functions.

To learn more about cloud functions connectors, visit our documentation page.

Additional Resources

If you don’t want to utilize internal resources, Fivetran has a global network of partners with the expertise to help you deploy any of these methods. Just get in touch with your account manager and ask about the right consultant for your use case and needs, or reach out to sales@fivetran.com.

Are You A Data Expert?

Get started with a free trial today.

Discover the smartest solution for data-driven results.