October 2020

Database Historical Sync Progress

You can now monitor your database historical sync progress in the status tab of your Fivetran dashboard. The status tab displays the table that is currently being synced as well as the number of tables that have finished syncing. This feature is available for all our database connectors.

Sync progress UI

Fivetran UK and Canada Servers

We have added new servers in Canada and the United Kingdom (UK).

When configuring your destination, you can now select these regions as the data processing location. The connectors configured in a destination run in their designated regions, meaning that your data doesn't leave our region-specific servers during processing. For example, if you configure your connectors to use our UK servers, your connector data will not leave the UK during processing.

For more information, see our data residency documentation.

History Mode

Now we calculate the _fivetran_end column value as the time when the record was deleted (the delete timestamp) minus one millisecond. Previously we calculated the column value as the time when the record was deleted (the delete timestamp).

New connectors

Box

Box is an online file sharing and cloud content management service for businesses.

Read our Box connector documentation.

Schema changes

Braze

We have added two new tables, USER and USER_ALIAS. See Braze's Export User endpoint documentation for more information.

We have also created foreign key relationships to the following tables that reference the primary key column user_id of the USER table:

ATTRIBUTION_EVENT
CAMPAIGN_CONVERSION_EVENT
CAMPAIGN_ENROLLMENT_EVENT
CAMPAIGN_CONVERSION_EVENT
CANVAS_ENTRY_EVENT
CONTENT_CARD_EVENT
CUSTOM_EVENT
EMAIL_EVENT
IN_APP_MESSAGE_EVENT
LOCATION_EVENT
NEWSFEED_CARD_EVENT
NEWSFEED_IMPRESSION_EVENT
PURCHASE_EVENT
PUSH_NOTIFICATION_EVENT
SESSION_EVENT
SMS_EVENT
SUBSCRIPTION_EVENT
UNINSTALL_EVENT

Jira

We now support the Jira Service Management (formerly Jira Service Desk) Organization fields in Jira issues. To sync fields of this type for all issues, re-sync your connector.

Snowplow

We have added a new column, mac_address, to the _EVENT table.

We have added two new fields, mkt_clickid and mkt_network, to the _EVENT table to support Campaign attribution enrichment 1-0-1. The two fields store the click ID and the network responsible for the click respectively.

Improvements

BigQuery

We now support clustering for partitioned and non-partitioned tables. See our clustered tables documentation for more information.

We now support partitioning for update queries. Now, during an update operation, if your partition key is based on the destination primary key, we scan only the corresponding partitions instead of scanning the full table. We are gradually rolling out this new feature to all existing destinations. If you'd like to enable this feature on your destination, contact our support team.

Files

We have a fixed a bug that caused Int32 and Int64 values to sync incorrectly from Parquet files. Previously, our syncs did not include the scale of these values (for example, a value of 85.00 was loaded into the destination as 8500000). We now scale these values correctly. If any of your Int32 or Int64 values were incorrectly synced prior to this fix, you will need to initiate a re-sync to correct them.

Google Ads

Now we perform a table re-sync whenever a field is added to or removed from a custom table. Previously, we only used incremental and rollback syncs, which may have caused incorrect metrics values being displayed in the table and unwanted report segmentation inconsistencies when the field set was changed.

Google Campaign Manager 360

We have upgraded our connector from version 3.3 to 3.4 of the Campaign Manager API.

Read Google's documentation to learn about recent Campaign Manager API changes.

We now select the mandatory dimensions just after connector creation. Previously, setup tests sometimes failed because mandatory dimensions weren't selected. Now, we handle the selection automatically. The only exception is the advertiser dimension for the REACH report type - you need to select it manually.

Google Search Console

The Google Search Console connector now supports the priority-first sync. The Google Search Console's priority-first sync fetches the most recent 30 days' worth of data from the reporting tables. To learn more, see our priority-first sync documentation.

Greenhouse

We have made our syncs more resilient to server failures from Greenhouse. If we can't connect to Greenhouse's servers during our syncs, we now retry the connection three times before the sync fails. Previously, our syncs failed immediately.

HubSpot

We now support future timestamp values in the ENGAGEMENT_MEETING table. If you previously synced future timestamp values, re-sync the table to ensure data integrity.

Jira

We have improved how we sync historical changes for the standard issue array fields components, fixVersions, and versions. We now write the actual values for each historical change, making the table much easier to use. The Jira API delivers empty values if part of the value is removed, and previously we just wrote that empty value.

Mixpanel

We have released a pre-built, dbt Core-compatible data model for Mixpanel. Find the model in Fivetran's dbt hub or data models documentation. Learn more about our dbt Core integration in our Transformations for dbt Core documentation*.

* dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.

MongoDB

We have improved incremental sync performance for databases which have many changes in collections that are not included in the sync.

PostgreSQL

We now sync unparsable DATERANGE values (such as 10000-01-01) to the destination as null values.

Recurly

We now mask the characters you enter in the API Key field of the connector setup form. We have changed the API Key field type from TextField to PasswordField to improve security.

SendGrid

We have added support for marketing.read API permission for our SendGrid Legacy accounts. You can now retrieve data from both legacy and new SendGrid marketing campaigns. We have also enabled backward compatibility. You can execute the queries designed for the old schema without any modifications on the new data.

We have added support for marketing_campaign.read API permission for our SendGrid New accounts. You can now retrieve data from both new and legacy SendGrid marketing campaigns.

Snowplow

We now support two new fields, derived_tstamp and true_tstamp. See Snowplow Analytics' documentation for more information about timestamps.

We have replaced the dvce_tstamp field with the dvce_created_tstamp field. We will discontinue the replaced field after January 10, 2021. Be sure to update your queries with the new field name before we remove the old field.

We now support the IP anonymization enrichment. See our Snowplow documentation for more information.

We now support Snowplow JavaScript Tracker 2.15.0.

Twitter Ads

We have released pre-built, dbt Core-compatible data models for Twitter Ads. Find the models in Fivetran's dbt hub or data models documentation. Learn more about our dbt Core integration in our Transformations for dbt Core documentation*.

* dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.

REST API Improvements

Displaying patch settings in standard config response

We have made changes to all public API endpoints that return the table and column configs as a response. Now you can see patch settings information for tables and columns in the response. It may help you understand whether you are allowed to enable or disable sync for them in the connector standard config. See the documentation for the following endpoints to learn more about the patch settings:

Support external OAuth credentials to authorize connectors in Public API

We have expanded the public API endpoints Create a Connector and Modify a Connector. Now you can authorize the following connectors by adding the auth section to payloads: