High-Volume Agent Oracle

Oracle is an object-relational database management system created by the Oracle Corporation. Fivetran replicates data from your Oracle source database and loads it into your destination using High-Volume Agent connector.

NOTE: You must have an Enterprise or Business Critical plan to use the High-Volume Agent Oracle connector.

Supported services

The Fivetran High-Volume Agent connector supports the following Oracle database services:

• Generic Oracle
• Oracle RAC

Supported configurations

Fivetran supports the following Oracle configurations:

Which Oracle instance types we can connect to depend on your database service.

Limitations

We do not support the following with the High-Volume Agent Oracle connector:

• Amazon RDS for Oracle
• Limited support for special* or lowercase characters in column and table names
• No support for special* or lowercase characters in schema names. For example, lowercase_schema, lowercase_SCHEMA, or lOwErCaSeScHeMa are all unsupported names.

* Special characters include (, &, @, *, $, |, %, ~, -, and ). Underscores are not special characters.

Features

Setup guide

Follow our step-by-step High-Volume Agent Oracle setup guide for specific instructions on how to set up your Oracle database with the High-Volume Agent connector.

Sync overview

Once connected to your database, the Fivetran connector runs an initial sync, pulling a full dump of selected data from your database and sending it to your destination. After a successful initial sync, the connector runs in an incremental sync mode. In this mode, Fivetran automatically detects new or updated data, such as new tables or data type changes, and persists these changes into your destination. We use log-based capture to extract your database's change data, then process and load these changes at regular intervals, ensuring a consistently updated synchronization between your database and destination.

NOTE: Choosing a 1-minute sync frequency does not guarantee that your sync completes within one minute.

Schema information

Fivetran tries to replicate the exact schema and tables from your Oracle source database to your destination according to our standard database update strategies. We create schemas in your destination that map directly to the schemas in your source Oracle database, ensuring that the data in your destination is in a familiar format to work with.

When you connect to Fivetran and specify a source database, you also select a schema prefix. We map the schemas we discover in your source database to your destination and prepend the destination schema name with the prefix you selected.

Fivetran-generated columns

Fivetran adds the following columns to every table in your destination:

• _fivetran_deleted (BOOLEAN) marks rows that were deleted in the source database.
• _fivetran_synced (UTC TIMESTAMP) indicates the time when Fivetran last successfully synced the row.
• _fivetran_id (STRING) is a unique ID that Fivetran uses to avoid duplicate rows in tables that do not have a primary key.

We add these columns to give you insight into the state of your data and the progress of your data syncs. For more information about these columns, see our System Columns and Tables documentation.

Type transformations and mapping

As we extract your data, we match Oracle data types to types that Fivetran supports. If we don't support a certain data type, we automatically change that type to the closest supported type or, in some cases, don't load that data at all. Our system automatically skips columns of data types that we don't accept or transform.

The following table illustrates how we transform your Oracle data types into Fivetran supported types:

* Learn more in our Long Value Truncation documentation.

If we are missing an important type that you need, reach out to support.

In some cases, when loading data into your destination, we may need to convert Fivetran data types into data types that are supported by the destination. For more information, see the individual data destination pages.

Supported encodings

We support the following character encodings:

• CESU-8
• CL8ISO8859P5MN
• GBK
• ISO-8859-1
• ISO-8859-5
• ISO 8859-6 Since v6.1.0/53
• ISO-8859-15
• JA16SJIS Since v6.1.0/54
• TH8TISASCII Since v6.1.0/32
• US-ASCII
• UTF-8
• UTF-16BE / UTF-16LE
• WINDOWS-949
• WINDOWS-1252

NOTE: Since <version> indicates the HVA version in which this encoding was first supported. To verify the version of HVA installed on your machine, use the command hvragenttest <agent_host> <agent_port>.

Excluding source data

If you do not want to sync all the data from your source database, you can exclude schemas, tables, or columns from your syncs on your Fivetran dashboard. To do so, go to your connector details page and uncheck the objects you would like to omit from syncing. For more information, see our Data Blocking documentation.

You can also use SELECT permissions if you want to exclude specific schemas and tables from syncing. The Fivetran user can only discover schemas and tables for which they have been given explicit SELECT permission. Note that permissions do not shield confidential data, such as PII (Personally Identifiable Information), from Fivetran because they do not apply to archived redo logs. Using the archived redo log files, Fivetran has access to the full contents of any changed rows of all tables that pass through your system. However, we filter out the prohibited data at the earliest possible stage of our syncs and do not load it into your destination.

Initial sync

Once connected to your database, the Fivetran connector copies all rows from every table in every schema for which a Fivetran user has SELECT permissions (except for those you have excluded in your Fivetran dashboard) and sends them to your destination. Additionally, we add Fivetran-generated columns to every table in your destination offering visibility into the state of your data during the syncs.

NOTE: While we import a maximum of four tables at a time, the process is designed to be efficient. As soon as one table import completes, we initiate the import of a new table. This approach ensures that we do not wait for all four tables to complete before starting the next one.

Updating data

Fivetran performs incremental updates by extracting new or modified data from your source database's log files using one of the following proprietary incremental sync/capture methods:

• Direct Capture: This method captures changes directly from Oracle's online and archived redo logs. By default, High-Volume Agent detects locations of the redo logs automatically.
• Archive Log Only: This method captures changes from Oracle's archived redo logs only. Online redo logs are not used at all. High-Volume Agent can either detect the location of the archived redo logs automatically or use pre-configured custom path and format. It is important to understand that during the capture process, High-Volume Agent is not connected to the source database. Therefore, using a custom path allows the High-Volume Agent to capture from a location other than Oracle's archives location(s).

INFO: The Archive Log Only capture method generally exhibits higher latency than the Direct Capture method because changes can only be captured when an archived redo log file is created. While this capture method enables high-performance data capture with minimal operating system and database privileges, it comes at the cost of higher capture latency.

Tables with a primary key

We merge changes to tables with primary keys into the corresponding tables in your destination:

• An INSERT in the source table generates a new row in the destination with _fivetran_deleted = FALSE.
• A DELETE in the source table updates the corresponding row in the destination with _fivetran_deleted = TRUE.
• An UPDATE in the source table updates the data in the corresponding row in the destination.

If we detect that your primary key has changed, we handle DELETEs and UPDATEs differently:

• If you have set SUPPLEMENTAL LOG DATA (ALL) on your source table, we update the row with the old primary key value with _fivetran_deleted = TRUE. We then insert a row with the new primary key value with _fivetran_deleted = FALSE.
• If you have not set SUPPLEMENTAL LOG DATA (ALL) on your source table, we do not update the row with the old primary key value because we can't identify it. We insert a row with the new primary key value with _fivetran_deleted = FALSE.

NOTE: If we detect that your primary key has changed, and you have not set SUPPLEMENTAL LOG DATA (ALL) on your source table, we raise a warning telling you to set SUPPLEMENTAL LOG DATA (ALL) on the table.

Tables without a primary key

For tables without a primary key, we designate our _fivetran_id column as the primary key. We populate this column with Oracle's ROWID pseudo column value.

NOTE: For a table without a primary key with row movement enabled, any update operation may incur additional MAR if it causes a row movement. When the row movement occurs, Oracle generates the following three log events, and Fivetran counts them as two MAR:

UPDATE (to an existing row)
DELETE (the row from the current partition)
INSERT (the row into a new partition)

• If the table is partitioned and row movement is enabled, we use a hash of the column's non-Fivetran values to populate our _fivetran_id column.
• Otherwise, we use Oracle's ROWID pseudo column to populate our _fivetran_id column.
  • If you enable row movement later, we automatically re-sync the table and use a hash of the column's non-Fivetran values to populate our _fivetran_id column.
  • If we detect any DDL operations (excluding GRANT), we automatically re-sync the table and re-import its contents with the newly assigned ROWIDs.

Deleted rows

We do not delete rows from your destination. When a row is deleted from the source table, we set the _fivetran_deleted column value of the corresponding row in the destination to TRUE.

Deleted columns

We do not delete columns from your destination. When a column is deleted from the source table, we replace the existing values in the corresponding destination column with NULL values.

Excluded tables

Fivetran does not sync the following tables:

• Temporary tables
• System-generated supplemental tables (for example, MLOG$ tables)
• System tables (for example, those in the SYSTEM tablespace or those owned by SYS user)