Automatic PII Detection Private Previewlink
IMPORTANT: This feature is available only for Business Critical and Enterprise accounts.
Overviewlink
With Automatic Personally Identifiable Information (PII) Detection, Fivetran automatically detects and tags sensitive data so that you can block or hash the columns tagged as PII. This feature reduces your risk of accidentally exposing PII by screening for it within your data pipeline. Automatic PII Detection also helps you adhere to compliance regulations, such as GDPR and CCPA, that have stringent requirements about how companies store, tag, and delete PII data. You can also scale your data pipeline more easily, since you can apply the same PII data policy across all your sources.
How it workslink
Automatic PII Detection runs in parallel to your Fivetran data pipeline, which ensures that your connectors' performance isn't impacted and that your data remains in the environment or region you chose.
We analyze sample data from each source column as it moves through your pipeline. Our sampling process randomly selects a number of data points across your sync to screen for PII. We then identify whether the data fits into one of the following PII categories:
- Name
- Social Security number (SSN)
- Email address
- Phone number
- Credit card number
- Address
- Date of birth
When we detect any of these characteristics, we tag the columns as containing PII on the connector schema page in your dashboard. You can view which columns contain PII and edit your sync settings to block or hash those columns. If you do not block or hash columns with PII, we sync that data as usual and the PII is readable in your destination. However, as long as you haven't removed the PII tags we created, you can still trigger downstream policies about PII in the destination using the column's metadata.
Enable Automatic PII Detectionlink
How you enable Automatic PII Detection depends on whether your connector already exists or you are creating a new one:
Existing connectorlink
In your Fivetran dashboard, go to your connector's Schema tab. If we detect PII in your source, you'll see a PII Detected banner at the top of the page.
Click Filter by PII to see the tables where we detected PII. You can see how many columns have PII in each table in the gray tag to the right of the table name.
Next to the table name, click the arrow to expand the column list. You can see which type of PII we detected for each column in the gray tag to the right of the column name.
Follow our Configure Blocking and Hashing for Existing Connectors documentation to block or hash the PII in these columns.
NOTE: If a column is a primary key or part of a composite primary key, you cannot block the column.
New connectorlink
In your Fivetran dashboard, create a new connector.
When given the option to sync all data or choose columns to block or hash, do the following based on your connector type:
- Applications, Files, and Events connectors: Select Block or hash sensitive data. We run our PII detection prior to your initial sync.
- Database connectors: Select Start initial sync. We run our PII detection as part of the initial sync and display the results once the sync is complete.
Once your schema is ready for review, go to your connector's Schema tab.
Click Filter by PII to see the tables where we detected PII. You can see how many columns have PII in each table in the gray tag to the right of the table name.
Next to the table name, click the arrow to expand the column list. You can see which type of PII we detected for each column in the gray tag to the right of the column name.
(Databases only) Follow our Configure Blocking and Hashing for Existing Connectors documentation to block or hash the PII in these columns.
NOTE: If a column is a primary key or part of a composite primary key, you cannot block the column.
Change PII taglink
In your Fivetran dashboard, go to your connector's Schema tab and locate the column whose PII type you want to change.
Click the drop-down arrow in the gray PII tag, then click the new PII type you want to use instead. You can only choose from the listed options.
Click Save changes.
Remove PII taglink
In your Fivetran dashboard, go to your connector's Schema tab and locate the column whose PII tag you want to remove.
Click the drop-down arrow in the gray PII tag, then clear the This column contains PII checkbox.
Click Save changes.
View PII column metadatalink
You can use the Fivetran Platform connector to view metadata on columns tagged as PII. The SOURCE_PII_TAGS
table stores only PII metadata for either your destination or entire account, depending on whether the Fivetran Platform Connector was added at the destination level or account level. Learn more in our Fivetran Platform connector documentation.
The SOURCE_PII_TAGS
table has the following format:
Column Name | Type | Notes |
---|---|---|
id | BIGSERIAL PRIMARY KEY | The auto-generated PII tag ID. This ID is unique to each record. |
column_id | LONG NOT NULL | The ID of the source column containing the PII. |
validated_pii_type | TEXT | The PII type we automatically assign through Automatic PII Detection. If you change the column's PII tag, we override this field.* |
created_at | TIMESTAMPTZ NOT NULL | The time when Automatic PII Detection was run. |
updated_at | TIMESTAMPTZ NOT NULL | The time when you changed or removed the column's PII tag.* |
*NOTE: If you change the column's PII tag multiple times, this field reflects only the most recent change.