July 2019link
Manage Transformations Permissionlink
You can now assign the Manage Transformations permission for your custom roles. Manage custom roles on the account page.
Transformationslink
We have added a new feature to the Fivetran pipeline – transformations. Transformations are SQL scripts that are executed in the destination based on specific events or conditions. The primary purpose of transformations is to map data into a specific shape that is easier or faster to use in the next pipeline elements.
Read our Transformations documentation.
New connectorslink
Fivetran Platform Connectorlink
Fivetran Platform Connector syncs logs and metadata about your connectors.
Read our Fivetran Platform Connector documentation.
Schema changeslink
Facebook Ad Accountlink
We have removed the following fields from the AD_ACCOUNT_HISTORY
table.
business_manager_name
business_manager_timezone_id
business_manager_primary_page
business_manager_updated_by
business_manager_created_by
business_manager_update_time
business_manager_created_time
Facebook recently changed the permissions required to fetch these fields. We now require business_management
permissions to sync these fields. We've submitted the Fivetran App to Facebook App Review so that Facebook will grant business_management
permissions for it. After approval, we will sync these fields again.
LinkedIn Ad Analyticslink
We have dropped some existing fields from the CREATIVE_HISTORY
table and added new fields. The new fields sync all the data that the previous fields contained. The table now supports separate fields for each creative type with added prefix.
You will need to drop the CREATIVE_HISTORY
and CREATIVE_SERVING_STATUS_HISTORY
table. The tables will get populated in next sync.
Fields dropped from the
CREATIVE_HISTORY
table:text
title
activity
direct_sponsored_content
share
content
media_asset
- some custom fields
Fields added to the
CREATIVE_HISTORY
table:text_ad_text
text_ad_title
sponsored_update_activity
sponsored_update_direct_sponsored_content
sponsored_update_share
sponsored_in_mail_content
sponsored_video_media_asset
sponsored_videuser_generated_content_post
sponsored_update_carousel_activity
sponsored_update_carousel_direct_sponsored_content
sponsored_update_carousel_share
follow_company_call_to_action
follow_company_organization_logo
follow_company_organization_name
follow_company_description
follow_company_headline
follow_company_organization
spotlight_call_to_action
spotlight_custom_background
spotlight_description
spotlight_forum_name
spotlight_headline
spotlight_logo
spotlight_show_member_profile_photo
jobs_company_name
jobs_logo
jobs_organization
jobs_button_label
jobs_headline
Microsoft Advertisinglink
The Bing Ads API recently removed support for the click_type_id
field with impression_*
fields in the PRODUCT_DIMENSION_PERFORMANCE_REPORT
table.
We removed the column click_type_id
from the following tables:
PRODUCT_DIMENSION_IMPRESSION_PERFORMANCE_HOURLY_REPORT
PRODUCT_DIMENSION_IMPRESSION_PERFORMANCE_DAILY_REPORT
Salesforcelink
We have upgraded our Salesforce connector from Salesforce API version 41.0 to 45.0. The upgrade supports new objects available in Salesforce API 45.0. We will roll out the new version of our connector to all customers gradually over 3 months starting on July 12th.
New objects:
- AccountContactRoleChangeEvent
- AppDefinition
- AppTabMember
- BatchApexErrorEvent
- Calendar
- ColorDefinition
- ContactPointTypeConsent
- ContactPointTypeConsentHistory
- ContactRequest
- ContactRequestShare
- ContentDocumentSubscription
- ContentNotification
- ContentTagSubscription
- ContentUserSubscription
- ContentVersionComment
- ContentVersionRating
- ContentWorkspaceSubscription
- CustomHelpMenuItem
- CustomHelpMenuSection
- CustomHttpHeader
- DataIntegrationRecordPurchasePermission
- DataUseLegalBasis
- DataUseLegalBasisHistory
- DataUseLegalBasisShare
- DataUsePurpose
- DataUsePurposeHistory
- DataUsePurposeShare
- EmailDomainFilter
- EmailRelay
- EmbeddedServiceLabel
- EventChangeEvent
- EventRelationChangeEvent
- FlowRecordRelation
- FlowStageRelation
- IconDefinition
- IframeWhiteListUrl
- Image
- ImageShare
- Individual
- IndividualHistory
- IndividualShare
- LightningExitByPageMetrics
- LightningExperienceTheme
- LightningToggleMetrics
- LightningUsageByAppTypeMetrics
- LightningUsageByBrowserMetrics
- LightningUsageByFlexiPageMetrics
- LightningUsageByPageMetrics
- ListEmailChangeEvent
- ListEmailIndividualRecipient
- MyDomainDiscoverableLogin
- OnboardingMetrics
- OpportunityContactRoleChangeEvent
- OrderStatus
- PermissionSetTabSetting
- PlatformStatusAlertEvent
- Recommendation
- RecordAction
- RecordActionHistory
- SiteIframeWhiteListUrl
- TabDefinition
- TaskChangeEvent
- TaskRelationChangeEvent
- TopicUserEvent
- UserChangeEvent
- UserEmailPreferredPerson
- UserEmailPreferredPersonShare
Shopifylink
We have added the new table
FULFILLMENT_EVENT
which represents tracking events that belong to a fulfillment of one or more items in an order. It contains the following fields:id
order_id
fulfillment_id
address1
city
country
created_at
estimated_delivery_at
happened_at
latitude
longitude
message
province
shop_id
status
updated_at
zip
Added the
quantity
column to theORDER_LINE_REFUND
table.
Twitter Adslink
If we encounter insufficient permissions when we try to sync endpoints for a specific account, then we skip that endpoint for that account and give a warning in the dashboard.
The warning includes endpoints with their respective accounts as follows.
Account ID | Skipped Endpoints |
---|---|
account_1 | endpoint_1 |
endpoint_2 | |
account_2 | endpoint_1 |
If you receive this warning, give Account Administrator permissions for that account to the user which you used to create your Twitter Ads App.
Zendesklink
We have stopped syncing data to the TICKET_METRIC_EVENT
table.
Improvementslink
BigQuerylink
We have added all the regional locations allowed in BigQuery to be data location. Now you can choose any location among them to be your data location. The data location you choose will be the same location where the query processing occurs.
The new locations allowed are:
Americaslink
Region Name | Region Description |
---|---|
us-west2 | Los Angeles |
northamerica-northeast1 | Montréal |
us-east4 | Northern Virginia |
southamerica-east1 | São Paulo |
Europelink
Region Name | Region Description |
---|---|
europe-north1 | Finland |
europe-west2 | London |
europe-west6 | Zürich |
Asia Pacificlink
Region Name | Region Description |
---|---|
asia-east2 | Hong Kong |
asia-south1 | Mumbai |
asia-northeast2 | Osaka |
asia-east1 | Taiwan |
asia-northeast1 | Tokyo |
asia-southeast1 | Singapore |
australia-southeast1 | Sydney |
BigQuery service accountslink
Newly created warehouses will use a unique service account to access data instead of a fixed service account. Visit updated documentation to see the changes in the setup process.
For existing warehouses we provide a way to perform migration to unique service accounts. Whenever you save a transformation or destination configuration, a popup window will appear that will guide you through the migration process.
Brazelink
Fixed an out of memory error that occurs when the number of files in the current Braze bucket is extremely large.
Facebook Ad Insightslink
We will sunset service versions 3, 4, and 5 of our Facebook Ad Insights on October 1, 2019.
You need to upgrade to our newest version, service version 6, by October 1, 2019. To upgrade, create a new Facebook Ad Insights connector and sync into a new schema.
The newest version of the Facebook Ad Insights connector supports daily, weekly and monthly aggregation with prebuilt reports. This new implementation generates a new schema. You can review the schema information in our docs.
HubSpotlink
We have added the table blocking feature to all HubSpot tables. You can now choose which tables you'd like to sync or omit from syncing. To select or deselect tables, go to the schema page of your Fivetran dashboard.
If you wish to sync a child table, you must also sync its parent table.
Jiralink
We now support capturing deleted issues by using webhooks. We are gradually rolling out this feature to all our customers. When it's available for your account, you will see a switch on the Jira connector setup form labeled "Capture deletes". To enable capturing deletes, toggle the switch.
Oracle Table-Level Supplemental Logging Supportlink
We have added support for table-level supplemental logging. This will reduce overhead when syncing with Fivetran if you only want to replicate some of your tables in an Oracle database.
Pinterest Adslink
We released a new version of our Pinterest Ads connector, service version 1. You can now select how many months' worth of data you'd like to include in your initial sync. Choose from the following options:
- 3 months
- 6 months
- 12 months
- 24 months
- All time
PostgreSQLlink
We have fixed a bug in our PostgreSQL connectors which was preventing some syncs from making progress.
Previously, PostgreSQL connectors using WAL syncs could not make progress through the WAL if all of the log events were related to tables Fivetran did not sync. For example, if you selected to sync the table PUBLIC.MY_TABLE
, and all log events were on PUBLIC.YOUR_TABLE
, Fivetran could not make progress.
Now Fivetran clears the WAL slot to the most recent LSN we examined if no rows were added, deleted, or updated in Fivetran-delivered tables.
Recurlylink
We have changed the way we perform the initial sync for our Recurly connectors. Previously during the initial sync, our Recurly connector scanned for records from EPOCH to the time when the initial sync started.
Now Fivetran scans in reverse and completes the initial sync in small batches. Every sync fetches records created in the past between two dates that differ by 7 days. This process continues until we reach the beginning of time.
Redshift VACUUM removallink
- Fivetran no longer performs
VACUUM DELETE
for your Redshift warehouses on the weekends, because Redshift now does it automatically. For more information on how Amazon Redshift schedules theVACUUM DELETE
operation, check out Amazon's changelog.
SFTPlink
- Fixed an out of memory error that occurs when the number of files in the folder is extremely large.
- Fivetran now reconnects if the SFTP session is disconnected by the SFTP server.
Stripe authentication changelink
- We have deprecated the OAuth 2.0 authentication mechanism for creating a new connector. Going forward, you need to use the restricted API key to create a new connector. We will continue to run existing connectors set up using the OAuth 2.0 mechanism. We encourage you to move to the new authentication for greater security.
Twitter Adslink
You can now configure your Twitter Ads connector through the Fivetran REST API. This feature is in BETA and available only for Standard and Enterprise accounts.