Column hashing is a method for anonymizing data in your destination while preserving its analytical value. You can join across data sets without introducing sensitive data to your destination. Because column hashing lets you anonymize personally identifiable information (PII) and store it in your destination, it brings you into compliance with GDPR.
Hashing is a one-way operation, unlike encryption where data can be both encrypted and decrypted. Once data is hashed, it cannot be returned to its previous state.
How Column Hashing works
When you select Hash Column, the next time your data syncs, Fivetran ingests your data, hashes it, and then writes the hashed data to your destination. In order to add an extra layer of security, Fivetran uses a unique salt per destination to ensure that the data cannot be decoded based on knowledge of the default Fivetran algorithm.
The salt is per destination so that all identical fields, such as email addresses, are still joinable between all data Fivetran loads into that destination.
If you are loading data with your own process into the same destination and would like to match our hashing, an Owner-permissioned user of your Fivetran Account can request the salt and hashing method from Fivetran support.
Once you choose to hash a column, we apply column hashing to all future syncs. If you also wish to hash the historical data in that column, you need to re-sync.
Column Hashing is available for the following connectors:
- Dynamics 365
- SQL Server
How to use Column Hashing
Go to the Schema tab on your connector details page and select Hash Column next to the columns you'd like to hash.
Calculating your own hashes
To generate the same hash value that we use for columns, perform the following steps:
Contact our Support team for the salt value and the hashing algorithm used for your connector.
Add the salt value as a suffix to the original value of the column. For example, if the value of your columns is
foor_barand the salt value is
i_am_the_secret_salt, the concatenated value will be
Tip: If the original column value is
null, replace the value with an empty string.
Convert the value (from Step 2) to byte array (byte) format using UTF-8 encoding.
Tip: In Java, you can convert using the
Use the byte array (from Step 3) as an input to the hashing algorithm.
Convert the output byte array to a Base64 encoded string.
This encoded string is the final hash value.