October 2018link
Faster BigQuery Loadinglink
We’ve made two major updates to the way we load data into BigQuery. Because of these updates, the data writing process is more than five times faster than it was before.
We are now using the interactive priority method for our queries instead of the batch priority method. Despite being designed for Fivetran’s batch update use-case, the BigQuery team has informed us that the batch priority method simply queued up our jobs without any actual improvement in the write performance.
We have also re-enabled BigQuery's SQL tool to use the ‘MERGE’ function. This reduces the data manipulation language (DML) statements per load from two to one.
User Facing Logslink
Bug Fixlink
We have fixed a bug that caused incomplete failure records in our user-facing logs. Previously, when connection tests failed because of a failed sync, our logs only showed the connection test failure. Now both failures are accurately recorded.
Format Changeslink
Our user-facing logs now include a UUID-format incident identifier, exception_id
, that gives our support and developer teams access to more detailed information about connector issues.
Log eventslink
Fivetran now supports flushing dashboard activity event data to supported log services.
Each log service has a unique path for this data:
Log Service | Path Pattern |
---|---|
AWS CloudWatch | {ACCOUNT_NAME}/{GROUP_NAME}/dashboard/all |
Azure Monitor | {ACCOUNT_NAME}_{GROUP_NAME}_dashboard_all_CL |
Google Cloud Logging | {ACCOUNT_NAME}-{GROUP_NAME}-dashboard-all |
Warehouse primary key changeslink
For the following warehouses, Fivetran now drops the primary_key
field from existing tables in your warehouse if there is a primary_key
coming in from your data source:
- MySQL
- PostgreSQL
- SQL Server
- Redshift
New connectorslink
Adobe Analyticslink
Adobe Analytics lets you analyze data about customer engagement with your website or application.
Link to Adobe Analytics Documentation
Iterablelink
Iterable is a growth marketing and customer engagement platform. It enables brands to create, execute and optimize campaigns across email, push, SMS, in-app and more with unparalleled data flexibility.
Link to Iterable Documentation
Schema changeslink
Facebook Ad Accountlink
We have added two new tables,
CAROUSEL_MEDIA
andCAROUSEL_MEDIA_URL_TAG
.We have added six new fields to the
CREATIVE_HISTORY
table:carousel_ad_link
page_link
template_page_link
page_message
template_description
link_description
Greenhouselink
The timestamp
field now supports two new datetime formats, UTC (ex. '2018-06-27 23:51:40 UTC') and Java TimeZone (ex. '2018-07-03T22:49:22.689Z').
We have changed the id
column in the CANDIDATE_TAG
table to tag_id
.
We have created a number of new primary keys:
candidate_id
andindex
are the new primary keys for the following tables:PHONE_NUMBER
EMAIL_ADDRESS
WEBSITE_ADDRESS
ATTACHMENT
SOCIAL_MEDIA_ADDRESS
ADDRESS
scorecard_id
andindex
are the new primary keys for the following tables:SCORE_CARD_ATTRIBUTE
SCORE_CARD_QNA
application_id
andindex
are the new primary keys for theAPPLICATION_QNA
table.job_post_id
andindex
are the new primary keys for theJOB_POST_QUESTION
table.
Snapchat Adslink
We have added a new field, operation
, to the AD_SQUAD_SEGMENT_HISTORY
table.
Xerolink
We have added two new fields, edition
and class
, to the ORGANIZATION
table.
Improvementslink
Google Analyticslink
- Fivetran has increased the stability of the Google Analytics connector.
- We have upgraded the Google Analytics Reporting API to version 4.
- Our Google Analytics reports now support the fields
segments
andfilter_expression
.
LinkedIn Ad Analyticslink
You can now sync all campaigns for a specific ad account without having to select them manually in our setup form.
MongoDBlink
We have optimized the way we do incremental updates from the oplog of nested updates. Previously, when you updated deeply nested fields, we had to fetch the entire document using collection.find({_id:?})
. This process can be extremely slow and delay syncs for hours, especially when deeply nested fields are frequently updated. We can now fetch these documents in batches, making our MongoDB syncs ten times faster.
NetSuite SuiteAnalyticslink
We have released service version 3 of the NetSuite SuiteAnalytics connector, which includes the following new features:
- Tables that can't be incrementally updated can now be re-imported. The frequency with which we update these re-import tables is based on how fast we can ingest them from NetSuite. For more information, see our NetSuite SuiteAnalytics guide.
- The
ITEM_LOCATION_MAP
re-import table will update every sync. To see the duration of each table import, you can refer to the table warning field on the Schema tab of your Fivetran dashboard's Connector page. - We can now upload records for tables where the primary key column contains
null
values. For more information, see our NetSuite SuiteAnalytics guide.
Pinterest Adslink
We no longer process ITEM
reports for advertisers who are not running shopping campaigns, because Pinterest only allows advertisers who are running shopping campaigns to access these reports.
PostgreSQLlink
We have clarified the wording of the rescheduled sync notification message in your Sync History.
Segmentlink
The Segment connector now handles delete
event types. A new table called DELETE
will be created in your destination to track these events.