August 2019link
SSH Verificationlink
Fivetran is rolling out SSH verification for connections to databases and warehouses. SSH allows for both encryption and authentication of the connections. Select your trusted certificates in your database and destination setup tests. See details in our SSH Verification docs.
Schema changeslink
Criteolink
- We have added the column
campaign_id
as additional primary key to theCATEGORY
table to differentiate records per campaign with same category ID.
Facebook Ad Accountlink
We have added a new table, APP_LINK
.
If we encounter an error when we try to sync a creative, then we skip that creative for that account and give a warning in the dashboard.
The warning includes the creative and the error with their respective accounts as follows.
Account ID | Creative ID | Error |
---|---|---|
account_1 | creative_1 | The creative is invalid |
account_2 | creative_2 | The creative might be deleted or you lack permissions to see it |
We have added back the following fields in the AD_ACCOUNT_HISTORY
table.
business_manager_name
business_manager_timezone_id
business_manager_primary_page
business_manager_updated_by
business_manager_created_by
business_manager_update_time
business_manager_created_time
Facebook have granted us business_management
permissions so once again we are able to sync these fields. You have to reconnect so that we can get new access_token with business_management
permission.
Google Sheetslink
Fivetran no longer uses the _fivetran_deleted
column. During each sync, we now delete all the data in each table and re-pull all its contents. This change means that deleted data will be absent from the destination instead of being marked as deleted in the _fivetran_deleted
column.
HubSpotlink
- We have added the column
activityType
to theENGAGEMENT
table.
Jiralink
We now support SLA fields in Jira issues. We are gradually rolling out this feature to all our customers. When it's available for your connector, you will see an additional sla
table in your destination. You can see the SLA
table on our Jira ERD. To sync all SLA fields, resync your connector.
Lightspeed Retaillink
We have added a new table, SHIP_TO_PHONE_HISTORY
to support multiple phone numbers for a single SHIP_TO_HISTORY
record.
The SHIP_TO_HISTORY
table will no longer contain phone_number
and phone_use_type
columns.
You need to join the two tables (SHIP_TO_PHONE_HISTORY
& SHIP_TO_HISTORY
) for this information.
Pardotlink
We have deprecated the VISITOR_REFERRER
table from our schema because the Pardot API does not provide us with information to populate this table.
Salesforce Marketing Cloudlink
- We have added the column
subscriber_key
to theSUBSCRIBER
table.
Squarelink
We have added the following fields to the
ORDER
table:order_source_name
return_amount
net_amount
rounding_adjustment
created_at
updated_at
closed_at
state
total_service_charge_money
We have added the
uid
field to the following tables:ORDER_LINE_ITEM
ORDER_LINE_ITEM_DISCOUNT
ORDER_LINE_ITEM_MODIFIER
ORDER_LINE_ITEM_TAX
We have added the
order_id
field to theTENDER
andREFUND
tables.We have added the following tables to the Square schema:
ORDER_RETURN
ORDER_RETURN_DISCOUNT
ORDER_RETURN_LINE_ITEM
ORDER_RETURN_LINE_ITEM_MODIFIER
ORDER_RETURN_SERVICE_CHARGE
ORDER_RETURN_TAX
ORDER_SERVICE_CHARGE
In order to incorporate changes announced by Square, we will deprecate the
TRANSACTION
table in the future. All the data in theTRANSACTION
table will be available in theORDER
table.
Stripelink
- We have made the
subscription_id
column a primary key. - The
SUBSCRIPTION_ITEM
table now holds only existing subscription items. We will delete subscription items which are not related to any subscription. - For example, when a subscription item is removed from a subscription, it does not belong to any subscription, so we delete that subscription item from the table.
Taboolalink
We have added the new column sub_categories
to the TARGETING_OS
table. Check our Taboola ERD for details.
Zuoralink
We have upgraded our Zuora connector from Zuora API WSDL version 91.0 to 96.0. We have made the following changes as a result of the upgrade:
- Added the
sequence_set_id
column to theACCOUNT_FIELDS
table. - Added the
balance
column to theINVOICE_ITEM
table. - Added the following new columns to the
REVENUE_EVENT
,REVENUE_EVENT_INVOICE_ITEM
, andREVENUE_EVENT_INVOICE_ITEM_ADJUSTMENT
tables:amount
currency
- Added the following new columns to the
TAXATION_ITEM
table:balance
credit_amount
payment_amount
- Removed support for the
CUSTOMER
table.
Improvementslink
Adobe Analyticslink
- Fivetran now supports all the available elements and metrics for all the report suites for your accounts.
AppsFlyerlink
We have deprecated service version 0 of our AppsFlyer connector. This connector will stop functioning after October 1, 2019.
Brazelink
We have added a new connection test to the Braze setup screen that helps you adopt security best practices. The test checks your bucket policy settings to make sure you have created your Role ARN using the given externalID when configuring your AWS S3 bucket. If you mistakenly create your Role ARN without using the given externalID, the configuration will now fail the test. You will have to correct your Role ARN configuration.
We now validate that the configured storage contains Braze Current files. You must configure Braze Current before you configure the Braze connector.
We treat the API key and role as passwords, so we don't display them on the screen to improve security.
Campaign Managerlink
We have upgraded our Campaign Manager connector from version 3.2 to 3.3 of the Campaign Manager API.
Read Google's documentation to learn about recent API changes.
DynamoDBlink
We have made DynamoDB alerts more informative. Alerts now show the names of the table that fell behind the schedule. You can select only those tables to perform re-sync instead of re-syncing all the tables.
Fileslink
Fivetran's incremental file update mechanism now re-syncs files that were created at the same time as the last observed cursor position.
Frontlink
We have changed the Front connector authentication method from API token to OAuth. All new Front connectors must authenticate with OAuth. Existing Front connectors which you set up with the API token can continue to use it until November 22, 2019. After November 22, 2019, we will no longer support API token authentication, and you will have to re-authenticate with OAuth. You may re-authenticate sooner if you prefer.
Google Ad Managerlink
We have upgraded our Google Ad Manager connector from version 201811 to 201908 of the Ad Manager API and handled deprecation and updates.
Read more about the deprecations and changes in Google's Ad Manager API documentation.
New dimensions:
programmatic_channel_id
programmatic_channel
dp_deal_id
dp_app_id
video_break_type
video_break_type_name
video_ad_request_duration_id
video_ad_request_duration
classified_advertiser_id
classified_advertiser_name
classified_brand_id
classified_brand_name
ad_exchange_optimization_type
Deprecated dimensions:
buyer_id
buyer_name
content_hierarchy
New columns:
dp_clicks
total_ad_requests
total_responses_served
total_unmatched_ad_requests
total_fill_rate
ad_server_responses_served
adsense_responses_served
ad_exchange_responses_served
Deprecated columns:
ad_server_inventory_level_percent_impressions
ad_server_inventory_level_percent_clicks
ad_server_inventory_level_without_cpd_percent_revenue
dynamic_allocation_inventory_level_impressions
dynamic_allocation_inventory_level_clicks
dynamic_allocation_inventory_level_ctr
dynamic_allocation_inventory_level_revenue
dynamic_allocation_inventory_level_average_ecpm
dynamic_allocation_inventory_level_percent_impressions
dynamic_allocation_inventory_level_percent_clicks
dynamic_allocation_inventory_level_without_cpd_percent_revenue
total_inventory_level_impressions
total_inventory_level_clicks
total_inventory_level_ctr
total_inventory_level_cpm_and_cpc_revenue
total_inventory_level_without_cpd_average_ecpm
Google Adslink
We have updated the Google Ads connector setup form. You can only add report fields that are compatible with each other. You no longer need to refer to Google Ads’ documentation to identify conflicting fields.
Google Sheetslink
Fivetran now fetches unformatted data from the Google Sheets API for new connectors. For example, if A1
is 1.23
and A2
is =A1
and formatted as currency, then A2
would return the number 1.23
.
Iterablelink
We will now be using both webhooks and API connections to perform incremental syncs. This ensures that you will no longer experience missing event data due to webhook timeouts.
Marketolink
Fivetran now allows you to exclude secondary attribute columns (for example, client_ip_address
, query_parameters
) from all activity tables. You can exclude these columns by navigating your schema tab in your dashboard and deselecting them for a particular table.
MongoDBlink
We fixed a bug that prevented our connector from syncing some of our MongoDB and MongoDB Sharded customers who use multiple hosts with Fivetran. The bug prevented us from syncing using standard Mongo drivers. To work around it, we now connect to the first host you list in the setup form, and if it fails, we try the next one. If needed, we continue in this manner through all of your listed hosts.
You do not need to take any action to leverage this improvement. If your connector was unaffected by the original bug, we will continue to use standard Mongo drivers method.
MySQLlink
Fivetran has released version 4 of our MySQL connector, which offers improved sync reliability. We will add a number of upgrades to this version in the coming months:
We will sync DATETIME values as TIMEZONE_NTZ values in your destination.
We will support parsing for TRUNCATE DDL events.
We will support the following new data types:
- BINARY
- VARBINARY
- YEAR
- BLOB
- SMALLBLOB
- MEDIUMBLOB
- LONGBLOB
- GEOMETRY
- GEOMETRYCOLLECTION
- LINESTRING
- MULTILINESTRING
- POINT
- MULTIPOINT
- POLYGON
- MULTIPOLYGON
This month, we have begun rolling out support for BINARY and VARBINARY types.
For more information about data types, see our MySQL documentation.
MySQL Warehouselink
We have changed the MySQL version requirements for Fivetran to store data as JSON in a MySQL warehouse and create tables with primary keys.
We now convert JSON to text in versions below 5.7.8.
We now create primary keys in either of the following scenarios:
MySQL version earlier than 5.7.7:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 191.
MySQL version 5.7.7 and later:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 768.
Oraclelink
- Table import progress is now available in the logs. Table names are included in one to two lists: "complete" or "inProgress".
- We have optimized our log miner query. The query now only selects for and processes tables which have changes inside the window for which log miner is active.
- Incremental Updates should now make progress even when the connection to the server is unstable. This will reduce the number of failed syncs due to temporary network issues.
Improvement to fast import strategylink
Our quick import strategy now works for tables stored in BIGFILE
tablespaces. However, this requires access to another view: dba_tablespaces
. You will receive a one-time warning requesting access to dba_tablespaces
in your dashboard the next time your Oracle connector runs.
If you have previously given us permission for dba_extents
and choose not to provide access to dba_tablespaces
, your connector's initial imports will still benefit from the quicker imports but may not sync data from tables stored in BIGFILE
tablespaces.
The Systems Views Permission
setup test will now include information about dba_tablespaces
, in addition to dba_extents
, if you choose not to provide access to them.
Changes to connection method requirements for Oracle connectorslink
Going forward, all connectors for Oracle databases versions 12.1 and earlier will be required to connect via SSH tunnels. This connection method ensures that all connections to your databases are secure and encrypted. If you do not currently connect via SSH tunnel, you will be required to update your connection method to continue syncing data. To set up an SSH tunnel, please reach out to your network administrator.
Pendolink
Our Pendo connector now supports projects in the EU region. To let a connector fetch data from Pendo's EU data center, you must change the integration key for the connector to one that is compatible with Pendo's EU endpoint. Generate a compatible integration key from app.eu.pendo.io.
PostgreSQLlink
We fixed a bug that caused connectors to break when syncing Postgis Geometry/Geography objects.
QuickBookslink
The following tables in QuickBooks now capture deletes via the _fivetran_deleted
column:
BILL
BILL_PAYMENT
CREDIT_MEMO
DEPOSIT
ESTIMATE
INVOICE
JOURNAL_ENTRY
PAYMENT
PURCHASE
PURCHASE_ORDER
REFUND_RECEIPT
SALES_RECEIPT
TIME_ACTIVITY
TRANSFER
VENDOR_CREDIT
You can obtain deletes in child tables by joining with the parent table.
Shopifylink
- Fivetran now deletes all the records in child tables when their parent table gets an update that indicates the parent table no longer has child records.
Twitter Adslink
We released a new version of our Twitter Ads connector, service version 2. You can now select how many months' worth of data you'd like to include in your initial sync. Choose from the following options:
- 3 months
- 6 months
- 12 months
- 24 months
- All time
Twitter Ads connectors running on service version 0 and 1 will sunset on November 21, 2019. If you are using either of these service versions, please upgrade to the newest version by creating a new Twitter Ads connector and syncing into a new schema.
Zuoralink
We now support the Zuora Performance Test environment. To connect, add the Zuora Sandbox connector in your Fivetran dashboard and select the Performance Test environment.