High-Volume Agent SAP ECC on Oracle with NetWeaver Setup Guide Beta

Follow the instructions listed here to replicate data from SAP ECC on Oracle with NetWevaer to your destination using Fivetran.

The Fivetran connector setup form lets you specify essential information for creating an Oracle connection, such as database credentials, connection method, High-Volume Agent (HVA) connection credentials, and other database-specific parameters. You need to set up your infrastructure, including network and database configurations, to obtain the parameter values required for this setup form.

Before entering details into the setup form, you must perform the following configurations:

• Ensure the prerequisites are in place
• Install and configure HVA
• Configure your network
• Configure database access and permissions:
  • Install NetWeaver RFC SDK libraries
  • Import Fivetran transports
  • Configuring Fivetran AppConnect in SAP
  • Create SAP NetWeaver user
  • Configure ARCHIVELOG mode
  • Enable supplemental logging
  • Configure Direct Capture

For detailed instructions, refer to the relevant sections below.

Prerequisites

To connect your Oracle database to Fivetran using the HVA, you need:

• A Fivetran account with an Enterprise or Business Critical plan
• Oracle 11g R2 or above
• SAP NetWeaver 7.5 or above
• Unicode compliance enabled for your SAP system
• The IP address of your database host
  • If using Oracle RAC, either an Application Virtual IP (VIP) or the public IP of a node
  • The IP address must be accessible from Fivetran directly, using an SSH tunnel, private networking, or Proxy Agent
• High-Volume Agent installed on your Oracle database host
  • The HVA IP address and port number (usually 4343)
• Supplemental logging enabled in your database
• ARCHIVELOG mode enabled in your database
• A SAP Communication user with the permissions and authorizations specified in the SAP NetWeaver user section

HVA installation and configuration

This section guides you through the process of installing the HVA on various operating systems, as well as configuring and starting the HVA. You will find information on compatibility checks, disk space requirements, and system permissions necessary for installation. Additionally, you will learn how to integrate the HVA into your system's startup to ensure it runs automatically upon boot.

HVA requirements

Before proceeding with the HVA installation, ensure that you have the following prerequisites in place:

• Compatibility: Verify that the HVA version is compatible with your operating system and DBMS. Refer to the COMPATIBILITY section in the relevant release notes on the Downloads page of your Fivetran dashboard.

• Sufficient disk space: Ensure that the machine where you want to install the HVA has ample disk space. We recommend a minimum of 10 GB of available disk space.

• HVA installation: Download the HVA installation file from the Downloads page of your Fivetran dashboard. Select the installation file for your database server's operating system.

  IMPORTANT: You must install HVA on the host where your Oracle database is running, or on a host that has access to the database.

• System permissions: HVA needs permission to read Oracle's redo and archive files at the operating system-level. This can be achieved in one of the following ways:

  • Install and run the agent under the database owner user account (for example, oracle).
  • Install and run the agent under a different user (for example, fivetran_user) that must be a member of Oracle's default operating system group. Typically, this is either oinstall or dba on Unix-based systems and ora_dba on Windows.

Install HVA

Follow the instructions below to install the HVA.

Configure HVA

Follow the instructions below to configure the HVA:

1. Create an HVA user to set up your Fivetran connector. Enter and confirm the password when prompted.

   NOTE: The password must be at least 10 characters long and must not contain any special characters.

   hvragentuserconfig -c <username>  
   

2. Make a note of the username and password. You will need them to configure your connector in the Fivetran dashboard.

3. Run the following command to disable the setup mode:

   hvragentconfig Setup_Mode_Timed_Until=
   

4. Run the following command to extract the HVA public certificate:

   hvragentconfig Agent_Server_Public_Certificate
   

5. Save the output value of the Agent_Server_Public_Certificate. You will need it to configure your connector in the Fivetran dashboard.

Start HVA

Perform the following steps as the user who runs HVA.

To start the agent, you need to run the hvragentlistener command with the -d flag followed by the port number that you want to run on. In the example, the default port is 4343.

hvragentlistener -d 4343

NOTE: For Oracle RAC, configuration using shared storage is highly recommended. Running the agent as a cluster resource and only on one node at a time is the preferred setup. If you do not want to follow this approach, you can run the agent only on one of the nodes in the cluster. If this node fails, you will need to start the agent on one of the other nodes in the cluster.

Add HVA to system autostart (Linux only)

After completing the initial HVA setup, you must add the agent to your system’s startup and shutdown procedures.

Network connection

This section outlines the methods available for connecting Fivetran to your SAP ECC on Oracle system.

Choose connection method

Decide on your preferred method for connecting Fivetran to your SAP ECC on Oracle system, and then configure the necessary settings for that method.

Database access and configuration

This section provides instructions for:

• Installing NetWeaver RFC SDK libraries
• Importing Fivetran transports
• Configuring Fivetran AppConnect in SAP
• Creating SAP NetWeaver user
• Configuring ARCHIVELOG mode
• Enabling supplemental logging
• Configuring Direct Capture

Install NetWeaver RFC SDK libraries

Fivetran requires NetWeaver RFC SDK version 7.50.8 or above installed on the machine from where HVA connects to the SAP NetWeaver.

1. Go to the SAP software download page.

2. Download and extract the Unicode SAP NetWeaver RFC SDK libraries that are specific to your operating system.

3. Copy the following files from the /lib directory (available in the extracted path) to a directory of your choice:

   OPERATING SYSTEM	LIBRARIES
   Linux 64 bit	libicudata.so.xx libicudecnumber.so libicui18n.so.xx libicuuc.so.xx libsapnwrfc.so libsapucum.so
   Windows 64 bit	icudtxx.dll icuinxx.dll icuucxx.dll libsapucum.dll libicudecnumber.dll sapnwrfc.dll

   NOTE: xx in the library names can vary depending on the library versions.

4. For Linux, set the read, write, and execute permissions for each NetWeaver RFC library file.

Import Fivetran transports

Import the transports containing Fivetran's ABAP functions. Use your company's default method to import the transports provided.

NOTE: While importing you may need to use the Ignore Invalid Component Version option to suppress import errors.

The transport files are available in the HVR_HOME/dbms/netweaver directory. For information about the latest transports files, refer to the readme.txt file in the same directory.

Configure Fivetran AppConnect in SAP

Fivetran's AppConnect has a number of configuration parameters that can be used to tune and optimize data extraction. You can access these parameters using the following SAP transaction: /N/HVR/CONFIG.

In most cases, you can retain the default values set for these parameters. When you want to make a change, trigger a change mode and save the new values into the ACTIVE variant.

Create SAP NetWeaver user

NOTE: HVA connects through the application layer and inherits the database authorizations of the standard SAP database user. You can manage table-level authorizations using the SAP authorization object S_TABU_SQL.

Fivetran's AppConnect requires a Communication user with the following permissions:

• RFC access to standard SAP functions to establish connection:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_RFC	RFC_TYPE	FUGR
  	RFC_NAME	BTCH RFC1 SDIFRUNTIME

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_RFC	RFC_TYPE	FUNC
  	RFC_NAME	RFCPING

• RFC access to the HVR function group:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_RFC	RFC_TYPE	FUGR
  	RFC_NAME	/HVR/SAPAPPCONNECT

• Authorization required for batch process administration to schedule background jobs:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_RFC	RFC_TYPE	FUGR
  	RFC_NAME	S_BTCH_JOB

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_BTCH_ADM	BTCADMIN	Y

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_BTCH_JOB	JOBACTION	RELE
  	JOBGROUP	*

• Data access authorizations:
  The S_TABU_SQL authorization object can be used to manage access rights for every data table in the SAP database.
  To grant full access, apply the following settings:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_TABU_SQL	ACTVT	33
  	DBSID	*
  	TABOWNER	standard DB schema(SAPABAP1)
  	TABLE	*

  Alternatively, to restrict access and grant permissions only to specific data tables that you want to synchronize, apply the following settings:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_TABU_SQL	ACTVT	33
  	DBSID	*
  	TABOWNER	standard DB schema(SAPABAP1)
  	TABLE	Target table name

  NOTE: When granting permissions only to specific data tables, the authorization object S_TABU_SQL must be added to user profile.

  Additionally, when granting permissions only to specific data tables, Fivetran requires access to certain standard SAP tables; you must grant access rights to these tables:

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_TABU_SQL	ACTVT	33
  	DBSID	*
  	TABOWNER	*
  	TABLE	CLIENTSIDE_ENCRYPTION_COLUMN_K CS_ALL_COLUMNS CS_COLUMNS_ CS_CONCAT_ATTRIBUTES_ CS_TABLES_ DUAL DUMMY HAS_NEEDED_SYSTEM_PRIV HAS_NEEDED_SYSTEM_PRIV_INCL_SY INDEX_COLUMNS M_CONNECTIONS M_DATABASE NUMA_NODE_PREFERENCE_ P_DATATYPES_ P_GRANTEDPRIVS_ P_INDEXES_ P_MASK_EXPRESSION_ P_OBJTYPES_ P_PRINCIPALS_ P_PROCEDURES_ P_SCHEMAS_ P_USERS_ RS_COLUMNS_ RS_TABLES_ SERIES_DATA_ TABLES TABLE_COLUMNS VIRTUAL_COLUMNS_ VIRTUAL_TABLES_ _SYS_CS_TABLE_COLUMNS _SYS_GRANTED_OBJECTS _SYS_GRANTEE_OIDS _SYS_RS_TABLE_COLUMNS _SYS_SCHEMAS_WITH_PRIVILEGES_O _SYS_VIRTUAL_TABLE_COLUMNS :BF0000 ALL_OBJECTS CDEF$ CON$ DBA_OBJECTS DEFERRED_STG$ DEPENDENCY$ IND$ I_COBJ# I_CON2 I_DEFERRED_STG1 I_DEPENDENCY1 I_FILE#_BLOCK# I_IND1 I_LINK1 I_OBJ# I_OBJ1 I_OBJ2 I_OBJ4 I_OBJ5 I_OBJAUTH1 I_OLAP_CUBES$ I_SUM$_1 I_TABCOMPART$ I_TABPART_OBJ$ I_TABSUBPART$_OBJ$ I_TRIGGER2 I_TS# I_USER# I_USER1 I_USER2 OBJ$ SEG$ SUM$ TAB$ TABCOMPART$ TABPART$ TABSUBPART$ TRIGGER$ TS$ USER$ USER_EDITIONING$ X$CON X$DNFS_FILES X$KCCAL X$KCCDI X$KCCDI2 X$KCCFN X$KCCIC X$KCCLE X$KCCRT* X$KGLCURSOR_CHILD X$KQLFXPL* X$KRSTDEST X$KSLED* X$KSLWT X$KSPPCV X$KSPPI X$KSPPSV X$KSUSE* X$KTADM X$KTCXB X$KZEKMENCWAL X$KZSPR X$KZSRO X$VERSION VW*

  AUTHORIZATION OBJECT	AUTHORIZATION FIELD	AUTHORIZATION VALUE
  S_TABU_SQL	ACTVT	33
  	DBSID	*
  	TABOWNER	standard DB schema(SAPABAP1)
  	TABLE	/HVR/BACKUP_CONFIGURATION /HVR/LOG_PARTITIONS /HVR/SERVICE_LOG_POSITIONS /HVR/TABLE_COLUMNS /HVR/TABLE_VIRTUAL_FILES DD02L* DD02T* DD03L* DD04L* DD06T* DD08L* DD09L* DD16S DDNTF DDNTT* HVR_IS_ARCHBLK HVR_STIN_ARCBLK TADIR*

Configure ARCHIVELOG mode

It is required to have your Oracle database in the ARCHIVELOG mode. This is normally the default for most production systems as it allows online backups and point-in-time recovery. However, to ensure Fivetran has access to all the required redo information, having the database in the ARCHIVELOG mode is required.

In addition, we also recommend that you keep at least 24 hours' worth of archive logs available.

The following section provides an overview of how the archive logging can be enabled. We also recommend that you review the Oracle documentation.

1. If the ARCHIVELOG mode is not enabled on your database, enable the ARCHIVELOG mode.

   IMPORTANT: If your database is not in the ARCHIVELOG mode, you must restart the database to place it in the ARCHIVELOG mode.

   • If using single instance database:

     NOTE: Enabling the ARCHIVELOG mode requires the Oracle instance to be briefly taken offline. To learn more, see Oracle's archived redo log file documentation.

     shutdown immediate;
     startup mount;
     alter database archivelog;
     alter database open;
     

2. Configure Oracle RMAN to retain backups and archive logs for at least 24 hours. We recommend you retain data for seven days.

   IMPORTANT: To sync your data, Fivetran must have a minimum of 3 hours' worth of log data to analyze. You cannot finish setting up your connector until 3 hours after you configure RMAN to retain archive logs.

   RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
   

3. (Recommended) Set the DB_RECOVERY_FILE_DEST_SIZE parameter to a value that matches your available disk space, because expired and obsolete log and backup files can quickly fill your disk. For more information, see Oracle's DB_RECOVERY_FILE_DEST_SIZE documentation.

Enable supplemental logging

The HVA requires the Oracle supplemental logging feature to be enabled on tables it replicates. Without this feature, Oracle logs only the columns that are changed. The agent, however, needs additional data (for example, key columns) to generate a full update statement for the target destination.

Oracle supplemental logging can be set at both the database level and for specific tables.

To enable supplemental logging, use the following SQL commands:

1. Enable database-level minimal supplemental logging:

    ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
   

2. Enable additional table-level supplemental logging depending on the table type:

   • For a table with a primary key, enable primary key supplemental logging:

      ALTER TABLE "<schema>"."<table>" ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
     

   • For a table without a primary key or unique index, or for a table with any type of compression enabled (regardless of whether they have a primary key and/or unique indexes), enable logging for all columns:

      ALTER TABLE "<schema>"."<table>" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
     

     NOTE: Enabling supplemental logging for all table columns ensures that any row change captures the values of all columns. This method allows the replication process to identify changes based on the entire row's data, which is especially beneficial for tables without a primary key.

     NOTE: For tables without a primary key, any update operation may incur additional MAR if it causes row movement. Oracle generates three log events for row movement — UPDATE (to an existing row), DELETE (of the row from its current partition), and INSERT (of the row into a new partition). Fivetran counts these as two MAR.

   • For a table without a primary key but with a unique index, enable conditional supplemental logging:

      ALTER TABLE "<schema>"."<table>" ADD SUPPLEMENTAL LOG GROUP "<group_name>" ("<column_name>", ...) ALWAYS;
     

     NOTE: This command creates a user-defined supplemental log group for specific column(s) within a table, offering a precise control over which column changes are logged. When using a unique index, ensure each table has its own unique log group.

SELECT supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all FROM v$database;

SELECT log_group_type FROM all_log_groups WHERE table_name='<schema>.<table>';

 ALTER DATABASE DROP SUPPLEMENTAL LOG DATA;

Configure Direct Capture

To enable Direct Capture, create a temporary table and grant read/write access to the SAP database user:

CREATE GLOBAL TEMPORARY TABLE HVR_SYS_TABLE (TABLE_NAME VARCHAR(128), TABLE_OWNER VARCHAR(128));
GRANT SELECT, INSERT, UPDATE, DELETE ON HVR_SYS_TABLE TO <database_user>;

NOTE: HVA uses the temporary table to make its queries more efficient during replication. Since this is a temporary table, there is no database logging on this table, and no rows remain at the end of the session. Storage for this table goes into the temporary tablespace.

Finish Fivetran configuration

This section outlines the steps to configure the HVA connector using the connector setup form within the Fivetran dashboard.

Configure connector in Fivetran dashboard

1. Enter a Destination schema prefix of your choice. This prefix applies to each replicated schema and cannot be changed once your connector is created.

2. In the Host field, enter the following information:

   • If you are using a single instance database, specify the hostname or IP address of your database. Set the value to localhost if the HVA and the database are on the same server.
   • If you are using Oracle RAC, specify an Application Virtual IP (VIP) address associated with the HVA.

     NOTE: The VIP address must be part of the cluster configuration and will move between nodes in case of node failure or restarts. This provides you with the option to run the agent on a specific node in the cluster. For this approach, we recommend using shared storage for the HVA.

3. Enter your Instance Number.

4. Enter the Client ID.

5. Enter the SAP NetWeaver User you created.

   TIP: In Oracle, usernames are case-sensitive. Make sure to enter the exact username.

6. Enter the Password for the SAP NetWeaver User.

7. Enter the RFC SDK library files directory.

8. Select your chosen Connection Method:

   • If you selected Connect via an SSH tunnel (this option is applicable for both standard SSH connections and reverse SSH connections), copy or make a note of the Public Key and add it to the authorized_keys file while configuring the SSH tunnel, and provide the following information:

     • SSH Host: Enter the hostname or IP address of your SSH server. Do not use a load balancer's IP address/hostname.
     • SSH Port: Enter the port number of your SSH server. The default port number is 22.
     • SSH User: Enter the username for SSH access.

   • If you selected Connect via proxy agent, choose the necessary proxy agent from the Proxy agents drop-down list (if available) or configure a new proxy agent.

9. (TDE-encrypted database only) If you want to use TDE encryption, set the Use TDE Encryption toggle to ON.

10. (Optional) Enter TDE Wallet Password if your database is configured with a Password-Based keystore. Leave the field blank if your database is configured with Auto-Login keystore.

11. In the Agent Host field, enter the HVA's hostname or IP address depending on the connection method you have chosen and your database configuration.

    • Connection method:
      • Direct: Enter the database hostname or IP address.
      • SSH: Enter the hostname or private IP address within your internal network, accessible by Fivetran through the SSH tunnel. For reverse SSH, set the value to localhost.
      • Private networking: Enter the hostname or private IP address within your VPC, accessible by Fivetran through a private endpoint.
      • Proxy Agent: Enter the hostname or private IP address within your internal network, accessible through the Proxy Agent. Use localhost if the Proxy Agent and HVA are installed on the same host.
    • Oracle RAC:
      • For Oracle RAC where the High-Volume Agent is configured on a node, the HVA's IP address is the Virtual IP (VIP) of the RAC node where the agent is running.
      • For Oracle RAC where the High-Volume Agent is configured as a cluster resource on a shared storage system, the HVA's IP address is the Virtual IP (VIP) created and dedicated to the agent.

12. In the Agent Port field, enter the HVA's port number. The default port number is 4343. For reverse SSH, use the value of <SSH_HIGH_PORT> (Linux) or Source port (Windows).

13. In the Agent User ID field, enter the name of the HVA user you created.

14. In the Agent User Password, enter the password of the HVA user you created.

15. In the Agent Public Cert, enter the HVA's Server Public Certificate you obtained.

16. Safelist the specified Fivetran's IPs for your selected cloud service provider and region in your firewall or security group.

17. Click Save & Test. Fivetran tests and validates our connection to your Oracle database. Upon successful completion of the setup tests, you can sync your data using Fivetran.

Setup tests

Fivetran performs the following tests to ensure that we can connect to your generic Oracle database and that it is properly configured:

• The Connecting to SSH Tunnel Test validates the SSH tunnel details you provided in the setup form. It then checks that we can connect to your database using the SSH tunnel. (We skip this test if you aren't connecting using an SSH tunnel.)
• The Validating Certificate Test generates a pop-up window where you must choose which certificate you want Fivetran to use. It then validates that certificate and checks that we can connect to your database using TLS. (We skip this test if you are connecting directly.)
• The Validating Agent Setup Test checks that we can connect to the high-volume agent and whether it operates correctly.
• The Validating Archive Log Access Test checks that we can access your archive log.

NOTE: The tests may take a few minutes to finish running.

Related articles

description Connector Overview

account_tree Schema Information

assignment Release Notes

settings API Connector Configuration

home Documentation Home