Snowplow

Snowplow Analytics is an event tracking platform that allows you to track events across all channels and action the data in real-time. Snowplow uses trackers to track your information in whichever language your project is built.

Features

Supported trackers

We pull information about your events from the code that tracks them. We officially support tracking in JavaScript, but we can also integrate tracking events from other languages, such as Ruby and Python.

Supported enrichments

We support the following Snowplow enrichments:

NOTE: Your JSON configuration file must follow the Snowplow schema format. You must use the enrichment-specific name (see the Notes section of the table above) for your configuration file.

TIP: To disable an enrichment, in the JSON configuration file, set the enabled field to false.

Setup guide

Follow our step-by-step Snowplow setup guide to connect Snowplow with your destination using Fivetran connectors.

Sync overview

After instrumenting the Snowplow tracking code on your website, server, or mobile application, the Fivetran event pipeline collects, enriches, and loads all of this data into your destination in near real-time:

1. Snowplow events are sent to webhooks.fivetran.com.

2. We collect the events.

3. We queue the events.

4. We process and enrich the events.

5. We prepare (transform and clean) the events and write them to a CSV file.

6. We load the events into your destination.

Snowplow does not allow Fivetran to sync your historical data. We can only capture data from your connection date forward. We re-import all tables during every sync.

Connection mapping

Each connection that is created within Fivetran (illustrated below in the connection icon circle) is given a specific URL to post data to. You can use Snowplow tracking services and send all of the events to the specific Fivetran URL. Any data posted to that URL is automatically normalized into a set of destination tables within the same schema.

Naming

You can name the destination schema in the Fivetran dashboard while creating the connector. Within your destination schema, we create a set of default tables for you in your destination.

Schema information

Snowplow has standard and custom fields. Each integration creates a single schema, shown below. If you choose to use custom events, each custom event type has its own table.

EVENT

Snowplow has a primary standard table, EVENT.

Fivetran de-duplicates events based on the event_id and collector_tstamp columns. If two events have the same event_id and collector_tstamp, we overwrite the first event.

The EVENT table contains the following columns:

UA_PARSER_CONTEXT

Fivetran follows the Snowplow UA parser enrichment to parse the user agents into a separate context. The user agent parser contexts (UA_PARSER_CONTEXT table) contains the following columns:

BAD_EVENT

If Fivetran detects Snowplow events that don't conform to the Snowplow Canonical Event format, we add the BAD_EVENT table to your destination schema to store these events.

The BAD_EVENT table contains the following columns:

Load custom fields

If you implement the Snowplow custom click feature, all the custom clicks are loaded into the EVENT table in your destination Snowplow schema.

Fivetran supports unstructured events. You can track your custom objects.

The custom event contexts (CUSTOM_CONTEXT table) contains the following columns:

To load custom fields into your destination, do the following:

1. Write a JSON schema describing your custom attributes known as an iglu file in the Snowplow documentation. Iglu files are JSON schemas, and the Snowplow integration is configured to work with JSON schemas. For more information, you can see an explanation of how JSON schemas work.

2. Validate your schema before you start sending your data to us.

3. We don't have a repository analogous to Snowplow "iglu", so put your schema at a regular URL such as a public GitHub repository, such as: https://github.com/fivetran/snowplow-schemas/blob/master/hello_world.json or https://github.com/fivetran/snowplow-schemas/blob/master/test_table.json.

4. Use the URL that you established in Step 3 as the schema URL when you call Snowplow:

   window[snowplowName]('trackPageView', null, [{
   schema: 'https://raw.githubusercontent.com/fivetran/snowplow-schemas/master/hello_world.json',
   data: {
   hello: 'Hello world!',
   hello_array: ['Hello', 'world!']
   }
   }]);
   

5. Make sure to have a "title" attribute for the schema, as the title attribute becomes the name of the table where the data gets stored. For example, the following creates a table named example (id int):

   {
   "title": "example",
   "properties": {
   "id": { 
       "type": "integer"
       }
   }
   }
   

6. If you want to create a decimal type (for example, for storing currency values), just use the "multipleOf" property:

   {
   "title": "example",
   "properties": {
       "value": { 
       "type": "number",
       "multipleOf": 0.01
       }
   }
   }
   

   This creates a destination table example (value decimal (18, 2)).

7. If you want to store an enum value, add a data type associated with it. All the enum values should be of the same data type:

   {
   "properties": {
       "value": { 
       "type": "string",
       "enum": ["value1", "value2"]
       }
   }
   }
   

NOTE: We only support values and homogenous arrays. Nested objects and tuple-style arrays are stored in but not extracted into your destination since they don't have an obvious representation.

Exclude source data

You cannot exclude source data directly in Fivetran. If you want to prevent specific data from being sent to Fivetran, you need to exclude it independently.

Data retention

We retain data from your connector so that it can be re-synced if needed. Data retention period: Persistent.

Sync limitations

We only support publicly-hosted schemas in Iglu or GitHub.