August 2019

SSH verification

Fivetran is rolling out SSH verification for connections to databases and destinations. SSH allows for both encryption and authentication of the connections. Select your trusted certificates in your database and destination setup tests. See details in our SSH Verification documentation.

Schema changes

Criteo

We have added the campaign_id column as an additional primary key to the CATEGORY table to differentiate records per campaign with same category ID.

Google Sheets

Fivetran no longer uses the _fivetran_deleted column. During each sync, we now delete all the data in each table and re-pull all its contents. This change means that deleted data will be absent from the destination instead of being marked as deleted in the _fivetran_deleted column.

HubSpot

We have added a new column, activityType, to the ENGAGEMENT table.

Jira

We now support SLA fields in Jira issues. We are gradually rolling out this feature to all our connectors.

When it's available for your connector, you will see an additional SLA table in your destination. You can see the SLA table on our Jira ERD. To sync all SLA fields, re-sync your connector.

Lightspeed Retail

We have added a new table, SHIP_TO_PHONE_HISTORY to support multiple phone numbers for a single SHIP_TO_HISTORY record.

The SHIP_TO_HISTORY table will no longer contain phone_number and phone_use_type columns. You need to join the two tables (SHIP_TO_PHONE_HISTORY and SHIP_TO_HISTORY) for this information.

Pardot

We have deprecated the VISITOR_REFERRER table from our schema because the Pardot API does not provide us with information to populate this table.

Salesforce Marketing Cloud

We have added a new column, subscriber_key, to the SUBSCRIBER table.

Square

We have added the following new columns to the ORDER table:

• order_source_name
• return_amount
• net_amount
• rounding_adjustment
• created_at
• updated_at
• closed_at
• state
• total_service_charge_money

We have added the uid column to the following tables:

• ORDER_LINE_ITEM
• ORDER_LINE_ITEM_DISCOUNT
• ORDER_LINE_ITEM_MODIFIER
• ORDER_LINE_ITEM_TAX

We have added the order_id column to the TENDER and REFUND tables.

We have added the following new tables:

• ORDER_RETURN
• ORDER_RETURN_DISCOUNT
• ORDER_RETURN_LINE_ITEM
• ORDER_RETURN_LINE_ITEM_MODIFIER
• ORDER_RETURN_SERVICE_CHARGE
• ORDER_RETURN_TAX
• ORDER_SERVICE_CHARGE

To support changes announced by Square, we will discontinue the TRANSACTION table in the future. All the data in the TRANSACTION table will be available in the ORDER table.

Stripe

We have made the subscription_id column a primary key.

The SUBSCRIPTION_ITEM table now holds only existing subscription items. We will delete subscription items which are not related to any subscription. For example, when a subscription item is removed from a subscription, it does not belong to any subscription, so we delete that subscription item from the table.

Taboola

We have added a new column, sub_categories, to the TARGETING_OS table. See our Taboola ERD for details.

Zuora

We have upgraded our Zuora connector from Zuora API WSDL version 91.0 to 96.0. We have made the following changes as a result of the upgrade:

• Added the sequence_set_id column to the ACCOUNT_FIELDS table.
• Added the balance column to the INVOICE_ITEM table.
• Added the following new columns to the REVENUE_EVENT, REVENUE_EVENT_INVOICE_ITEM, and REVENUE_EVENT_INVOICE_ITEM_ADJUSTMENT tables:
  • amount
  • currency
• Added the following new columns to the TAXATION_ITEM table:
  • balance
  • credit_amount
  • payment_amount
• Removed support for the CUSTOMER table.

Improvements

Adobe Analytics

Fivetran now supports all the available elements and metrics for all the report suites for your accounts.

Amazon DynamoDB

We have made Amazon DynamoDB alerts more informative. Alerts now show the names of the table that fell behind the schedule. You can select only those tables to perform re-sync instead of re-syncing all the tables.

AppsFlyer

We have deprecated service version 0 of our AppsFlyer connector. This connector will stop functioning after October 1, 2019.

Braze

We have added a new connection test to the Braze setup screen that helps you adopt security best practices. The test checks your bucket policy settings to make sure you have created your Role ARN using the given externalID when configuring your AWS S3 bucket. If you mistakenly create your Role ARN without using the given externalID, the configuration will now fail the test. You will have to correct your Role ARN configuration.

We now validate that the configured storage contains Braze Current files. You must configure Braze Current before you configure the Braze connector.

We treat the API key and role as passwords, so we don't display them on the screen to improve security.

Files

Fivetran's incremental file update mechanism now re-syncs files that were created at the same time as the last observed cursor position.

Front

We have changed the Front connector authentication method from API token to OAuth. All new Front connectors must authenticate with OAuth. Existing Front connectors which you set up with the API token can continue to use it until November 22, 2019. After November 22, 2019, we will no longer support API token authentication, and you will have to re-authenticate with OAuth. You may re-authenticate sooner if you prefer.

Google Ad Manager

We have upgraded our Google Ad Manager connector from version 201811 to 201908 of the Ad Manager API and handled deprecation and updates. Read more about the deprecations and changes in Google's Ad Manager API documentation.

• New dimensions:

  • programmatic_channel_id
  • programmatic_channel
  • dp_deal_id
  • dp_app_id
  • video_break_type
  • video_break_type_name
  • video_ad_request_duration_id
  • video_ad_request_duration
  • classified_advertiser_id
  • classified_advertiser_name
  • classified_brand_id
  • classified_brand_name
  • ad_exchange_optimization_type

• Deprecated dimensions:

  • buyer_id
  • buyer_name
  • content_hierarchy

• New columns:

  • dp_clicks
  • total_ad_requests
  • total_responses_served
  • total_unmatched_ad_requests
  • total_fill_rate
  • ad_server_responses_served
  • adsense_responses_served
  • ad_exchange_responses_served

• Deprecated columns:

  • ad_server_inventory_level_percent_impressions
  • ad_server_inventory_level_percent_clicks
  • ad_server_inventory_level_without_cpd_percent_revenue
  • dynamic_allocation_inventory_level_impressions
  • dynamic_allocation_inventory_level_clicks
  • dynamic_allocation_inventory_level_ctr
  • dynamic_allocation_inventory_level_revenue
  • dynamic_allocation_inventory_level_average_ecpm
  • dynamic_allocation_inventory_level_percent_impressions
  • dynamic_allocation_inventory_level_percent_clicks
  • dynamic_allocation_inventory_level_without_cpd_percent_revenue
  • total_inventory_level_impressions
  • total_inventory_level_clicks
  • total_inventory_level_ctr
  • total_inventory_level_cpm_and_cpc_revenue
  • total_inventory_level_without_cpd_average_ecpm

Google Ads

We have updated the Google Ads connector setup form. You can only add report fields that are compatible with each other. You no longer need to refer to Google Ads’ documentation to identify conflicting fields.

Google Campaign Manager 360

We have upgraded our connector from version 3.2 to 3.3 of the Campaign Manager API. Read Google's documentation to learn about recent API changes.

Google Sheets

Fivetran now fetches unformatted data from the Google Sheets API for new connectors. For example, if A1 is 1.23 and A2 is =A1 and formatted as currency, then A2 would return the number 1.23.

Iterable

We will now be using both webhooks and API connections to perform incremental syncs. This ensures that you will no longer experience missing event data due to webhook timeouts.

Marketo

Fivetran now allows you to exclude secondary attribute columns (for example, client_ip_address, query_parameters) from all activity tables. You can exclude these columns by navigating your schema tab in your dashboard and deselecting them for a particular table.

MongoDB

We fixed a bug that prevented our connector from syncing data of some of our MongoDB and MongoDB Sharded users who use multiple hosts with Fivetran. The bug prevented us from syncing using standard Mongo drivers. To work around it, we now connect to the first host you list in the setup form, and if it fails, we try the next one. If needed, we continue in this manner through all of your listed hosts.

You do not need to take any action to leverage this improvement. If your connector was unaffected by the original bug, we will continue to use standard Mongo drivers method.

MySQL

Fivetran has released version 4 of our MySQL connector, which offers improved sync reliability. We will add a number of upgrades to this version in the coming months:

• We will sync DATETIME values as TIMEZONE_NTZ values in your destination.

• We will support parsing for TRUNCATE DDL events.

• We will support the following new data types:

  • BINARY
  • VARBINARY
  • YEAR
  • BLOB
  • SMALLBLOB
  • MEDIUMBLOB
  • LONGBLOB
  • GEOMETRY
  • GEOMETRYCOLLECTION
  • LINESTRING
  • MULTILINESTRING
  • POINT
  • MULTIPOINT
  • POLYGON
  • MULTIPOLYGON

We have begun rolling out support for BINARY and VARBINARY types.

For more information about data types, see our MySQL documentation.

MySQL destination

We have changed the MySQL version requirements for Fivetran to store data as JSON in a MySQL destination and create tables with primary keys.

We now convert JSON to TEXT in versions below 5.7.8.

We now create primary keys in either of the following scenarios:

• MySQL version earlier than 5.7.7:

  • If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of STRING primary keys is less than 191.

• MySQL version 5.7.7 and later:

  • If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of STRING primary keys is less than 768.

Oracle

We have optimized our LogMiner query. The query now only selects for and processes tables which have changes inside the window for which LogMiner is active.

Incremental updates should now make progress even when the connection to the server is unstable. This will reduce the number of failed syncs due to temporary network issues.

Improvements to fast import strategy

Our quick import strategy now works for tables stored in BIGFILE tablespaces. However, this requires access to another view: dba_tablespaces. You will receive a one-time warning requesting access to dba_tablespaces in your dashboard the next time your Oracle connector runs.

If you have previously given us permission for dba_extents and choose not to provide access to dba_tablespaces, your connector's initial imports will still benefit from the quicker imports but may not sync data from tables stored in BIGFILE tablespaces.

The Systems Views Permission setup test will now include information about dba_tablespaces, in addition to dba_extents, if you choose not to provide access to them.

Changes to connection method requirements for Oracle connectors

Going forward, all connectors for Oracle databases versions 12.1 and earlier will be required to connect via SSH tunnels. This connection method ensures that all connections to your databases are secure and encrypted. If you do not currently connect via SSH tunnel, you will be required to update your connection method to continue syncing data. To set up an SSH tunnel, reach out to your network administrator.

Pendo

Our Pendo connector now supports projects in the EU region. To let a connector fetch data from Pendo's EU data center, you must change the integration key for the connector to one that is compatible with Pendo's EU endpoint.  Generate a compatible integration key from app.eu.pendo.io.

PostgreSQL

We fixed a bug that caused connectors to break when syncing Postgis Geometry or Geography objects.

QuickBooks

We now capture deletes for the following tables via the _fivetran_deleted column:

• BILL
• BILL_PAYMENT
• CREDIT_MEMO
• DEPOSIT
• ESTIMATE
• INVOICE
• JOURNAL_ENTRY
• PAYMENT
• PURCHASE
• PURCHASE_ORDER
• REFUND_RECEIPT
• SALES_RECEIPT
• TIME_ACTIVITY
• TRANSFER
• VENDOR_CREDIT

You can obtain deletes in child tables by joining with the parent table.

Shopify

Fivetran now deletes all the records in child tables when their parent table gets an update that indicates the parent table no longer has child records.

Twitter Ads

We released a new version of our Twitter Ads connector, service version 2. You can now select how many months' worth of data you'd like to include in your initial sync. Choose from the following options:

• 3 months
• 6 months
• 12 months
• 24 months
• All time

Twitter Ads connectors running on service version 0 and 1 will sunset on November 21, 2019. If you are using either of these service versions, upgrade to the newest version by creating a new Twitter Ads connector and syncing into a new schema.

Zuora

We now support the Zuora Performance Test environment. To connect, add the Zuora Sandbox connector in your Fivetran dashboard and select the Performance Test environment.