SAP ERP on HANA

SAP Enterprise Resource Planning (ERP) applications like SAP ECC and SAP S/4HANA manage customer finance, operations, supply chain, and HR processes. With the SAP ERP on HANA connector, Fivetran offers the possibility to replicate data from SAP ECC and S/4HANA systems running on a HANA platform.

SAP HANA is an in-memory, column-oriented, relational database management system. Fivetran replicates data from your SAP HANA source database and loads it into your destination.

Features

Supported deployment models

We support the SaaS and Hybrid deployment models for the connector.

Known limitations

• Table and view support:

  • We support column-store and row-store transparent tables, as well as views from the SAP data dictionaries. Avoid replicating base tables that point to proxy objects, such as compatibility views, because they may return incomplete or no data. Instead, sync the compatibility view directly. Learn more in Excluded objects.
  • No support for parameterized views and projection views. Learn more in Excluded objects.
  • No support for change data capture (CDC) in row store transparent tables and views. Learn more in Replication of views and row-store tables.
  • No support for SAP BW Advanced DataStore Object (ADSO) tables.

• Search and selection:

  When selecting tables to sync, the following limitations apply (see Selecting tables to sync for more details):

  • Each search term must include at least three alphanumeric characters.
  • You can use the following special characters in your search terms: *, ?, _, -, ., and /.

• Schema and naming:

  • Limited support for special* or lowercase characters in column and table names.

  • No support for special* or lowercase characters in schema names. For example, lowercase_schema , lowercase_SCHEMA, or lOwErCaSeScHeMa are all unsupported names.

    * Special characters include (, &, @, *, $, |, %, ~, -, and ). Underscores are not considered special characters.

• API and authorization:

  • Authorization through API is supported only when using SSH tunneling.

• Sync behavior:

  • No support for the following DDL statements: CREATE TABLE and most ALTER TABLE operations. If such operations occur, the respective table must be manually re-synced. However, ADD COLUMN and DROP COLUMN operations are supported.
  • No support for updating a primary key value. If a primary key changes in a source, we treat it as a new INSERT in a destination.
  • History mode is supported, but we only capture record updates at the time of sync. Updates that occur between two syncs are not captured.
  • No support for adding new tables through schema updates due to SAP API limitations.

• Version requirements:

  • We require SAP ABAP server version 7.4 or higher with Unicode support. Learn more in Prerequisites in SAP ERP on HANA Setup Guide.
  • Compatible with SAP HANA 2.0 SPS08 and supports the partitioning features available in this version.

Setup guide

Follow our step-by-step SAP ERP on HANA Setup Guide for specific instructions on how to set up your SAP ERP on HANA using the Fivetran connector.

Sync overview

The SAP ERP on HANA connector interfaces with the application layer of an SAP system. Within this application layer, you need to install the Fivetran Netweaver API, which then pulls a full dump of all selected data from your SAP system. We then use our proprietary tracking mechanism to retrieve modified data at regular intervals.

Syncing empty tables and columns

Fivetran can sync empty tables for your SAP ERP on HANA connection.

We can also sync empty columns. However, we require at least one row of data that includes this new column to be added. This is because we detect changes to a table's column cardinality when we see a row with a new or removed column during an update.

For more information, see our Features documentation.

Schema information

Fivetran replicates the exact schema and tables from your HANA source database to your destination according to our standard database update strategies. This ensures that the data in your destination is in a familiar format to work with.

When you connect to Fivetran and specify a source database, you also select a schema prefix. We map the SAP schema in your source database to your destination and prepend the destination schema name with the prefix you selected.

Fivetran-generated columns

Fivetran adds system columns to your destination tables to give you insight into your sync progress and row-level changes. The specific columns added depend on the sync mode and certain connector-specific features.

We add the following columns to every table in your destination, depending on the sync mode you use:

• Either sync mode: _fivetran_synced (UTC TIMESTAMP) indicates the time when Fivetran last successfully synced the row.
• Soft delete mode: _fivetran_deleted (BOOLEAN) marks rows that were deleted in the source database.
• History mode: _fivetran_active (BOOLEAN), _fivetran_start (UTC TIMESTAMP), and _fivetran_end (UTC TIMESTAMP). These are used to mark which rows are currently active. They keep a record of the previous record and their lifetime.

If archive deletes tracking is enabled in your SAP source, we will add the _fivetran_sap_archived column to indicate the archived records.

For certain packed tables such as STXL and MDTC, we may generate additional columns during unpacking. For more information, see the Unpack data section.

Type transformation and mapping

As we extract your data, we match SAP data types to the types that Fivetran supports. Our system attempts to infer the types of any columns with data types that we do not recognize.

The following table illustrates how we transform your SAP data types (based on the SAP Data Dictionaries) into Fivetran supported types:

If we are missing an important data type that you need, reach out to support.

In some cases, when loading data into your destination, we may need to convert Fivetran data types into data types that are supported by the destination. For more information, see the individual data destination pages.

Data mapping workflows

Depending on the values, the data mapping for the connection can be in one of the following situations:

• The value can be parsed to a Fivetran data type - Fivetran takes the value and sends it to the destination in the appropriate data type.
• The value cannot be parsed to a Fivetran data type, and the column is not a primary key - Fivetran coerces the value a null and displays a warning. The sync continues.
• The value cannot be parsed to a Fivetran data type, and the column is a primary key - The sync for the respective table ends with an error.

Schema changes

New tables are not automatically added. Use the Schema tab to add tables as needed.

Selecting data to sync

The SAP ERP on HANA connector does not auto-ingest your SAP catalog because SAP systems often contain hundreds of thousands of dictionary tables. You must explicitly select the tables or views you want to sync during the Select data to sync step when setting up your connection.

The following SAP object types are available for selection:

• Column-store transparent tables

• Row-store transparent tables

• Views

  For more information on how views and row-store transparent tables are handled during replication, see our Replication of views and row-store tables documentation.

New tables are not added automatically. You can add or remove tables or views at any time on your connection’s Schema tab; changes take effect on the next sync.

You can also restrict which tables and views appear for selection using the Source tables list parameter in Fivetran's SAP configuration screen. For more information, see our Limiting table availability documentation.

Limiting table availability

You can control which tables are available for selection in the Schema tab by defining the Source tables list in the Fivetran's SAP configuration screen accessible through the SAP transaction /N/FIVETRAN/CONFIG. For more information, see Configure Fivetran in SAP.

• If the list is empty (default), all supported tables and views are available for search and selection.
• If the list includes specific tables (for example, T000 and BSEG), only those tables will be visible in the Schema tab.

To stop syncing a previously added table that may already contain data in your destination, remove it from the Source tables list. We will disable the table in the Schema tab during the next sync and prevent further replication.

Example scenarios:

Here are some examples to illustrate how table filtering works:

• Scenario 1 - No Filtering: If no tables are added to the Source tables list in the Fivetran's SAP configuration screen, all supported tables and views will be available for replication.

• Scenario 2 - Adding new tables: The Schema tab contains tables T000 and T001. If a user adds tables T000, T001, T002, and BSEG to the Source tables list, the existing tables (T000 and T001) can still be replicated. However, only the newly added tables (T002 and BSEG) will be available for searching and adding for sync in the Schema tab.

• Scenario 3 - Replacing existing tables: The Schema tab contains tables T000 and T001. If a user adds only table BSEG to the Source tables list, the existing tables (T000 and T001) will be automatically removed from the connection's Schema tab. Only table BSEG will be available for searching and adding for sync in the Schema tab.

• Scenario 4 - Removing and adding tables: If the Schema tab contains tables T000 and T001, and these tables are no longer permitted because they are not specified in the Source tables list, Fivetran will gray them out in the Schema tab and will not sync them. If you re-add tables T000 and T001, Fivetran will sync them and gray them out during the next sync cycle.

Filtering rows based on column values

You can apply row-level filters to replicate only the data you need from a specific table. For more information on creating, editing, or deleting filters, including supported operators and data types, see our Row Filtering documentation.

Initial sync

The initial sync can be performed once the connection to the SAP ERP on HANA database is established. The NetWeaver Fivetran API manages all data transport during this process.

Updating data

Fivetran performs incremental updates of any new or modified data from your source database. New changes are fetched and then synchronized into the destination. The update process varies depending on your sync mode:

Soft delete mode

• An INSERT in the source table generates a new row in the destination with _fivetran_deleted = FALSE.
• A DELETE in the source table updates the corresponding row in the destination with _fivetran_deleted = TRUE.
• An UPDATE in the source table updates the corresponding row in the destination with _fivetran_deleted = FALSE.
• An UPDATE of the primary key in the source table generates is treated like a new row being added. We therefore add a new row in the destination with _fivetran_deleted = FALSE. The old row remains in the destination with _fivetran_deleted = FALSE until the corresponding table is re-synced.

History mode

• An INSERT in the source table generates a new row in the destination with _fivetran_active = TRUE, _fivetran_start = <CURRENT_TIMESTAMP>, and _fivetran_end = 9999-12-31 23:59:59.999+00.
• A DELETE in the source table marks the corresponding row in the destination inactive with _fivetran_ACTIVE = FALSE and _fivetran_end = <CURRENT_TIMESTAMP>.
• An UPDATE in the source table first marks the corresponding row in the destination inactive with _fivetran_ACTIVE = FALSE and _fivetran_end = <CURRENT_TIMESTAMP>. We then add a new row in the destination with the same data and _fivetran_active = TRUE, _fivetran_start = <CURRENT_TIMESTAMP>, and _fivetran_end = 9999-12-31 23:59:59.999+00.
• An UPDATE of the primary key in the source table is treated like a new row being added. We therefore add a new row in the destination with _fivetran_active = TRUE, _fivetran_start = <CURRENT_TIMESTAMP>, and _fivetran_end = 9999-12-31 23:59:59.999+00. The old active row remains in the destination with _fivetran_active = TRUE, _fivetran_start = <CURRENT_TIMESTAMP>, and _fivetran_end = 9999-12-31 23:59:59.999+00 until the corresponding table is re-synced.

Deleted columns

We do not delete columns from your destination.

Capturing deletes

Capturing deletes requires careful consideration because deletes can occur in data that was already imported in previous cycles and otherwise go unnoticed.

To detect deletes, we use triggers and shadow tables created by the Fivetran NetWeaver API for each imported standard table:

• A shadow (DELETES) table named /FIVETRAN/DELETES_<TABLE> (for example, /FIVETRAN/DELETES_T000).
• A DELETE trigger on the source table that writes the deleted row’s primary key(s) into the corresponding DELETES table.

During each sync, we read the DELETES table and apply the deletes in your destination.

Controlling where triggers are used

By default, Fivetran creates delete-capture triggers for all supported standard tables. To reduce overhead or gain finer control, you can limit deletes capture to specific tables in SAP using the Deletes detection using triggers parameter in transaction /N/FIVETRAN/CONFIG. When configured for selected tables, only those tables’ deletes are recorded and synced. Deletes on other tables are not captured. Inserts and updates continue to sync for all tables. For detailed steps, see Disable delete triggers for specific tables.

Re-sync partition and re-sync table scenarios

The DROP and TRUNCATE operations have certain side effects that can impact the reliability of your connection. Fivetran can detect and automate specific processes without disrupting the data replication:

• In the case of a TRUNCATE (partition or table) operation: Fivetran automatically detects truncations for each partition and initiates a re-sync during the next sync cycle.
• In the case of a DROP partition operation (typically done via ALTER TABLE): Automatic detection from Fivetran side is currently not supported. This capability is still under development. Therefore, you must manually re-sync your table in case one partition is dropped.
• In the case of a DROP TABLE operation: No further action is taken for that table, and it is treated as if there are no updates. The data remains intact on the destination.

Create partitions

Creating a new partitions for any table is automatically detected by Fivetran. This will be synced normally during the next sync cycle.

Fivetran also detects modified/restructured partitions, triggering a re-sync for those partitions.

Log replay delay tolerance during failover

In high-availability SAP HANA replication setups, a failover from the primary to the secondary instance may involve a short log replay delay.

To ensure data consistency without unnecessary re-sync, we support a log replay delay of up to 10 minutes between replicas during failover. If the delay is within this threshold, tables continue syncing without re-sync. If the delay exceeds 10 minutes, affected tables are re-synced individually based on the detected delay.

Unpack data

We support unpacking for two tables, STXL and MDTC.

STXL unpacking

We unpack long text STXL table into a readable and transparent format. We have modified the table structure to account for this. The following is the replicated STXL version:

We replaced the SRTF2, CLUSTR and CLUSTD columns by the new /FIVETRAN/STXL_TEXT column. The aggregation of these three columns contain the necessary information to decode the human-readable text for the new column.

MDTC unpacking

We unpack table MDTC table into a readable and transparent format. We have modified the table structure to account for this. The following is the replicated MDTC version:

We have replaced the CLUSTR and CLUSTD columns by adding 59 new columns:

• We expanded the primary key with two columns, DTNUM (PK) and DTPOS (PK).
• We added 57 non-primary key columns containing unpacked data.

Unpacking MDTC is a many-to-many type of unpacking - one or more records from the source could be unpacked into one or more records in your destination, in human-readable format.

Handling deleted data during SAP archiving process

You can mark records archived by a SAP archiving process as non-deleted in your destination by checking the Start tracking archive deletes box under SAP Archiving job users in the Fivetran's SAP configuration screen to enable this feature.

When the box is switched on, an input list labeled SAP Archiving job users appears under SAP Archiving job users. You can specify the user name(s) under which the SAP Archiving job is executed.

When you have enabled marking archived records as non-deleted for the SAP source, Fivetran does the following:

• A new column is added to all tables synced from the relevant source in your destination, _fivetran_sap_archived (BOOLEAN). The column value for the previously synced data becomes NULL.
• The _fivetran_sap_archived column value is set to FALSE for the records synced after the feature was enabled.
• The _fivetran_sap_archived column value is set to TRUE for the records archived on the SAP side if the user committing the delete is included in the input list of users. The specified username and the username of the SAP user related to the SAP archiving process must be an exact match. In all other cases, (for example, if the box was checked, but the input user list remains empty), the _fivetran_sap_archived column value remains FALSE.
• If this column is TRUE, then the corresponding record will not be marked as deleted. For example, depending on the sync mode, such record continues to have _fivetran_deleted=FALSE in case of soft deletes or _fivetran_active=TRUE in case of history mode.

See our Configure Fivetran in SAP documentation for more information about configuring SAP archiving deletes.

Monthly Active Rows calculation

The records archived by SAP archiving processes are seen in Fivetran as either updates (if this feature is switched on) or deletes (if the feature is switched off) and count towards paid MAR. The standard rules for Monthly Active Rows (MAR) still apply for these operations.

Replication of views and row-store transparent tables

Fivetran allows you to replicate data from:

• Row-store transparent tables
• Column-store transparent tables
• Views

Due to structural limitations in views and row-store transparent tables, we cannot incrementally sync them. Instead, we fully re-sync these objects at each sync cycle. Here’s how we handle records during the re-sync of views or row-store transparent tables:

• If a record doesn't exist in the destination, we insert it into the destination.
• If a record exists in the destination, we update the existing record.
• If a record exists in the destination but is not present in the latest sync cycle, we permanently delete it from the destination.

Handling deletes

Since views and row-store transparent tables are fully re-synced at each cycle, Fivetran does not use the delete-capturing mechanism. As a result:

• We do not create triggers or shadow DELETES tables
• We do not add the Fivetran-generated columns for tracking deletes:
  • _fivetran_deleted (soft delete mode)
  • _fivetran_active, _fivetran_start, and _fivetran_end (history mode)
  • _fivetran_sap_archived (archive deletes).

Primary keys and unique identifiers

Views don't have primary keys. To ensure uniqueness, we introduce the _fivetran_id (LONG) column, which auto-increments for each record.

Monthly Active Rows calculation

The standard rules for Monthly Active Rows (MAR) still apply to views and row-store transparent tables.

MANDT filtering

When logging into an SAP system, the login is client-specific, and data is automatically filtered by client (MANDT). Our connector has a broader data access and can replicate data for all clients, regardless of the logon client (client-independent data selection). You can enable MANDT filtering in the Fivetran's SAP configuration screen to limit the replicated data to only data relevant to the connected client.

For more information about enabling the MANDT filtering, see our Configure Fivetran in SAP documentation.

Excluded objects

Fivetran does not sync the following objects:

• Database-level system tables
• Tables and views not present in the SAP data dictionaries
• Temporary tables
• SAP functions
• Projection views
• Parameterized views
• Tables with underlying proxy objects (for example, compatibility views). However, it is possible to replicate compatibility views themselves, as they are treated as views. For more information, see Replication of views and row-store tables. Example:
  • In some SAP systems, table MBEW is based on compatibility view MBVMBEW. Do not replicate table MBEW, as it may be empty, contain partial information, or, in rare cases, include unexpected data. Instead, replicate the view MBVMBEW.