High-Volume Agent SAP S/4HANA with NetWeaver Setup Guide Private Previewlink
Follow these instructions to replicate your SAP HANA database to your destination using Fivetran High-Volume Agent (HVA) connector.
IMPORTANT:
- This connector only supports Snowflake as a destination in Private Preview.
- Fivetran supports SAP HANA 2.0 SPS 07.
Prerequisiteslink
To connect your HANA database to Fivetran using the High-Volume Agent connector, you need:
- A Fivetran account with an Enterprise or Business Critical plan
- SAP ABAP version 7.5 and above
- SAP system that must be Unicode
- SAP HANA from version 1.0 SPS 11 to version 2.0 SPS 06)
- Your server IP (for example,
1.2.3.4
) or domain (for example,your.server.com
) - Your SAP instance number (a two-digit number)
- Client id (a three-digit number)
- Your database name
- Installation of the agent on your SAP HANA database host
- The agent IP address and port number (usually
4343
)
- The agent IP address and port number (usually
- Table types need to be Column Store
- Installation of SAP NetWeaver and User Permissions
Fivetran requires a Communication user with the following permissions:
RFC access to standard SAP functions to establish connection:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC RFC_TYPE FUGR RFC_NAME BTCH
RFC1
SDIFRUNTIMEAUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC RFC_TYPE FUNC RFC_NAME RFCPING RFC access to the Local Data Processing function group:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC RFC_TYPE FUGR RFC_NAME /HVR/SAPAPPCONNECT Authorization required for batch process administration to schedule background jobs:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC RFC_TYPE FUGR RFC_NAME S_BTCH_JOB AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_BTCH_ADM BTCADMIN Y AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_BTCH_JOB JOBACTION RELE JOBGROUP * Data access authorizations:
The S_TABU_SQL authorization object can be used to manage access rights for every data table in the SAP database.
To grant full access, apply the following settings:AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE * Alternatively, to restrict access and grant permissions only to specific data tables that you want to synchronize, apply the following settings:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE Target table name NOTE: When granting permissions only to specific data tables, the authorization object S_TABU_SQL must be added to user profile.
Additionally, when granting permissions only to specific data tables, Fivetran requires access to certain standard SAP tables; you must grant access rights to these tables:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER * TABLE CLIENTSIDE_ENCRYPTION_COLUMN_K
CS_ALL_COLUMNS
CS_COLUMNS_
CS_CONCAT_ATTRIBUTES_
CS_TABLES_
DUMMY
HAS_NEEDED_SYSTEM_PRIV
HAS_NEEDED_SYSTEM_PRIV_INCL_SY
INDEX_COLUMNS
M_CONNECTIONS
M_DATABASE
NUMA_NODE_PREFERENCE_
P_DATATYPES_
P_GRANTEDPRIVS_
P_INDEXES_
P_MASK_EXPRESSION_
P_OBJTYPES_
P_PRINCIPALS_
P_PROCEDURES_
P_SCHEMAS_
P_USERS_
RS_COLUMNS_
RS_TABLES_
SERIES_DATA_
TABLES
TABLE_COLUMNS
VIRTUAL_COLUMNS_
VIRTUAL_TABLES_
_SYS_CS_TABLE_COLUMNS
_SYS_GRANTED_OBJECTS
_SYS_GRANTEE_OIDS
_SYS_RS_TABLE_COLUMNS
_SYS_SCHEMAS_WITH_PRIVILEGES_O
_SYS_VIRTUAL_TABLE_COLUMNS
:BF0000
ALL_OBJECTS
CDEF$
CON$
DBA_OBJECTS
DEFERRED_STG$
DEPENDENCY$
IND$
I_COBJ#
I_CON2
I_DEFERRED_STG1
I_DEPENDENCY1
I_FILE#_BLOCK#
I_IND1
I_LINK1
I_OBJ#
I_OBJ1
I_OBJ2
I_OBJ4
I_OBJ5
I_OBJAUTH1
I_OLAP_CUBES$
I_SUM$_1
I_TABCOMPART$
I_TABPART_OBJ$
I_TABSUBPART$_OBJ$
I_TRIGGER2
I_TS#
I_USER#
I_USER1
I_USER2
OBJ$
SEG$
SUM$
TAB$
TABCOMPART$
TABPART$
TABSUBPART$
TRIGGER$
TS$
USER$
USER_EDITIONING$
X$CON
X$DNFS_FILES
X$KCCAL
X$KCCDI
X$KCCDI2
X$KCCFN
X$KCCIC
X$KCCLE
X$KCCRT*
X$KGLCURSOR_CHILD
X$KQLFXPL*
X$KRSTDEST
X$KSLED*
X$KSLWT
X$KSPPCV
X$KSPPI
X$KSPPSV
X$KSUSE*
X$KTADM
X$KTCXB
X$KZEKMENCWAL
X$KZSPR
X$KZSRO
X$VERSION
VW*AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE /HVR/BACKUP_CONFIGURATION
/HVR/LOG_PARTITIONS
/HVR/SERVICE_LOG_POSITIONS
/HVR/TABLE_COLUMNS
/HVR/TABLE_VIRTUAL_FILES
DD02L*
DD02T*
DD03L*
DD04L*
DD06T*
DD08L*
DD09L*
DD16S
DDNTF
DDNTT*
HVR_IS_ARCHBLK
HVR_STIN_ARCBLK
TADIR*
Setup instructionslink
Choose connection methodlink
First, decide whether to connect Fivetran to your HANA database:
IMPORTANT: No matter which option you select, you must install the Fivetran High-Volume Agent on your database host.
Connect directlylink
Fivetran connects directly to the High-Volume Agent installed on the database host. All communication between Fivetran and the agent is encrypted. The agent has a direct connection to your HANA database.
Connect using SSHlink
Fivetran connects to a separate server in your network that provides an SSH tunnel to your database host. You must connect through SSH if your database is in an inaccessible subnet.
To connect using SSH, follow our SSH connection instructions.
NOTE: If Reverse SSH Tunnel is required, contact our support team as additional steps are required to set up the connector.
Connect using private networkinglink
IMPORTANT: You must have a Business Critical plan to use private networking.
Private networking enables communication between private networks and services without exposing traffic to the public internet. Private networking is the most secure connection method. We support the following providers:
- AWS PrivateLink – used for VPCs and either AWS-hosted or on-premises services. See our AWS PrivateLink setup guide for details.
- Azure PrivateLink – used for Virtual Networks (VNets) and either Azure-hosted or on-premises services. See our Azure PrivateLink setup guide for details.
- Google Cloud Private Service Connect – used for VPCs and either Google-hosted or on-premises services. See our Google Cloud Private Service Connect setup guide for details.
Configure log modelink
For Fivetran to capture changes from HANA, the automatic backup of transaction logs must be enabled in HANA. Fivetran reads changes from the 'online' transaction log file, but if interrupted (say for 2 hours) then it must be able to read from transaction log backup files to capture the older changes. Fivetran is not interested in full backups; it only reads the transaction log backup file.
To enable automatic log backup in HANA, the log mode must be set to normal
. The log mode can be changed using HANA Studio. Once the log mode is changed from overwrite
to normal
, a full data backup must be created.
For detailed steps, search for change log modes in the SAP HANA documentation.
Transaction log (archive) retentionlink
If a backup process has already moved the transaction log files to the tape and deleted them, you must initiate a full re-sync. The required 'retention' period, measured in hours or days, depends on organizational factors (such as the desired level of real-time data) and practical considerations (such as the duration of a data refresh, whether it takes 1 hour or 24 hours).
Configure NetWeaverlink
To use Fivetran with SAP NetWeaver for replication, install NetWeaver RFC SDK libraries.
Fivetran requires NetWeaver RFC SDK version 7.50.8 or above installed on the machine from where HVA connects to the SAP NetWeaver.
Go to the SAP software download page.
Download and extract the Unicode SAP NetWeaver RFC SDK libraries that are specific to the operating system.
Copy the below-mentioned files from the
/lib
directory (available in the extracted path) to any directory of your preference.Click here for the list of library files to be copied.
Operating System Libraries Linux 64 bit - libicudata.so.xx
- libicudecnumber.so
- libicui18n.so.xx
- libicuuc.so.xx
- libsapnwrfc.so
- libsapucum.so
Windows 64 bit - icudtxx.dll
- icuinxx.dll
- icuucxx.dll
- libsapucum.dll
- libicudecnumber.dll
- sapnwrfc.dll
For Linux, set the
read
,write
, andexecute
permissions for each NetWeaver RFC library file.
Configure capture from Backintlink
NOTE: Perform this step only when you need to capture data from log files stored in Backint for SAP HANA.
To retrieve data from backups created using Backint for SAP HANA, the following configuration steps are required:
Configure options Backint Executable Path and Backint Configuration Path. This can be done when setting up the connector in Step 13.
Add the Backint application as a trusted external application:
a. Copy the
hvrosaccess_example.conf
file fromHVR_HOME/etc
toHVR_CONFIG/etc
and rename it tohvrosaccess.conf
.b. Edit the
hvrosaccess.conf
file to add the full path of the Backint application underAllowed_Plugin_Paths
. Typically, the default path is/usr/sap/HDB/SYS/global/hdb/opt
.Example:
{ ## Following section 'safelists' directories for command execution. Allowed_Plugin_Paths: [ /usr/sap/HDB/SYS/global/hdb/opt ] }
Configure capture from HANA encrypted logs and log backupslink
NOTE: Perform this step only when you need to capture data from HANA encrypted logs and log backups.
To capture from HANA encrypted log files and log backups, the following must be configured:
In addition to the grants provided in section Configure capture, the Fivetran database user must be granted the following privilege to read the encrypted log files and log backups:
grant execute on _HVR."/HVR/ROOT_KEYS_EXTRACT" to <username>;
If using a multi-tenant configuration, the encryption configuration management should be delegated to tenants. Execute the following command on the system database to change control over to the tenant databases:
alter system encryption configuration controlled by local databases;
Set a password for the encryption root keys backup in the source database:
alter system set encryption root keys backup password <password>;
Make a note of the password set for the encryption root keys backup in the previous step. You will need it to configure the Root Keys backup Password option in the connector setup form (Step 13).
Create Receiver namespacelink
Fivetran-specific ABAP code is created in namespace /HVR/
. Before importing the ABAP code, the namespace must be created in SAP.
- Log in to SAP.
- Navigate to Transport Organizer (transaction shortcut is SE09).
- Click Goto ▶ Transport Organizer Tools ▶ Administration ▶ Display/Change Namespaces.
- Switch to Change Mode.
- Click New Entries.
- Create a namespace with the following details:
- Enter /HVR/ in Namespace.
- Enter C (Recipient) in Namespace role.
- Enter a short description (for example, Fivetran Software - data extraction objects) in Short Text.
- Enter a name (for example, Fivetran Software) in Owner.
- Save the namespace.
Import transportslink
Import the transports containing Fivetran’s ABAP functions. Use your company's default method to import the transports provided.
NOTE: While importing, you may need to use the Ignore Invalid Component Version option to suppress import errors.
The transport files are available in the HVR_HOME/dbms/netweaver
directory.
Transport Name | Description | Database | Annotation |
---|---|---|---|
HVRK900106 | SAPAPPCONNECT Main functions Workbench objects in this transport
| ||
HVRK900082 | SAPAPPCONNECT HANASYSVIEW Workbench objects in this transport
| HANA | This transport is required only if system views are to be created via the Application server import. |
Configure Fivetran in SAPlink
Fivetran has a number of configuration parameters that can be used to tune and optimize the data extraction. These parameters can be accessed using the SAP transaction - /N/HVR/CONFIG
.
In most cases, the parameters can be left with the default values.
Parameter | Description |
---|---|
Max wasted iterations | Stop SAP listener process after X failed attempts to read data. |
Max wait, microseconds | Delay for SAP listener process in between GET requests. |
Memory tunnel size, byte | How much shared memory buffer in SAP is used. |
Max wait background, microsec | Stop retrieval process after X microseconds. |
Max wait foreground, microsec | Stop retrieval process after X microseconds. |
Max size uncompressed, byte | Maximum data package size. |
Evaluate max size for strings | Calculate string length for dynamic data types. |
DB Connection Name | Secondary database connection name to be used in case you do not want to use the standard SAP database connection. This is not the same connection used to deploy HANA views. |
Application server | Name of the dedicated application server. It is mandatory to set this parameter. It must be set to the same server as where HVA logs on to allow in-memory data transfer. The current version does not support SAP load balancing. |
Configure capturelink
The following access configuration is required for capturing changes from NetWeaver on HANA.
Fivetran needs access to data from system dictionaries. This access is provided using views created as the HANA SYSTEM
user.
Log in to SAP.
Start transaction DBCO.
Create a secondary connection using the HANA
SYSTEM
user.NOTE: This connection should only be used for creating views, never for the standard replication process.
Run the program
/HVR/SAPAPPCONNECT_HANASYSVIEW
using transaction SA38.Populate the SYS DB Connection field with the secondary connection created earlier and select the Show Log option.
Execute the program.
NOTE: Views are created in the default SAP schema with a name beginning with
/HVR/<view_name>
and will not conflict with other existing SAP standard and custom objects.
Install HVAlink
This section provides detailed instructions on how to install HVA and outlines the necessary requirements.
HVA requirementslink
Before proceeding with the installation of HVA, ensure that you have the following prerequisites in place:
Compatibility: Verify that the HVA version is compatible with your operating system and DBMS. Refer to the COMPATIBILITY section in the relevant release notes on the Downloads page of your Fivetran dashboard.
Sufficient disk space: Ensure that the machine where you want to install HVA has ample disk space. We recommend a minimum of 10 GB of available disk space.
HVA installation file: Download the HVA installation file from the Downloads page of your Fivetran dashboard. Select the installation file suitable for your database server's operating system.
IMPORTANT: You must install HVA on the same host where your HANA database is running.
Install and run HVA under the database owner user account (for example,
hana
).
Follow the instructions below to install HVA.
Install HVA on Linuxlink
Expand for instructions
Create folder structure
i. Create the HVA installation directory (for example,
/opt/fivetran
) and set the appropriate owner (for example,hana
) and group (for example,oinstall
) for it.NOTE: This command must be executed by the
root
user.mkdir /opt/fivetran chown -R hana:oinstall /opt/fivetran
ii. In the
/opt/fivetran
directory, create three key subdirectorieshvr_home
,hvr_config
, andhvr_tmp
:mkdir -p /opt/fivetran/hvr_home mkdir -p -m 01775 /opt/fivetran/hvr_config /opt/fivetran/hvr_tmp
Configure environment
i. Configure the
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
environment variables to point to the relevant HVA installation subdirectories:export HVR_HOME=/home/myhvr/hvr_home export HVR_CONFIG=/home/myhvr/hvr_config export HVR_TMP=/home/myhvr/hvr_tmp
ii. Add the
$HVR_HOME/bin
executable directory path to the environment variablePATH
:export PATH=$PATH:$HVR_HOME/bin
iii. Add the environment variables and the executable directory path into the startup file (for example,
.profile
or.bash_profile
):export HVR_HOME=/opt/fivetran/hvr_home export HVR_CONFIG=/opt/fivetran/hvr_config export HVR_TMP=/opt/fivetran/hvr_tmp export PATH=$PATH:$HVR_HOME/bin
Install HVA
Navigate to the
$HVR_HOME
directory and extract the contents of the HVA installation file (for example,fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
).cd $HVR_HOME tar xzf /tmp/fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
Once completed, the HVA and all required components are installed into the
$HVR_HOME
directory (/opt/fivetran/hvr_home
).NOTE: The size of the installation is less than 150 MB. After installing (extracting), you can delete the installation file as it is no longer required.
Configure HVAlink
Follow the instructions below to configure HVA:
Create an HVA user to set up your Fivetran connector. In the following example, we use
hva_user
.NOTE: The minimum password length for the user must be 10 characters and not contain special characters.
hvragentuserconfig -c hva_user Password for 'hva_user': <enter a password> Retype password: <confirm the password>
Make a note of the username and password. You will need them to configure Fivetran.
Run the following command to disable the setup mode:
hvragentconfig Setup_Mode_Timed_Until=
Run the following command to extract the HVA public certificate:
hvragentconfig Agent_Server_Public_Certificate
Save the output value of the
Agent_Server_Public_Certificate
. You will need it to set up your connector.
Start HVAlink
To start the agent, run the hvragentlistener
command with the -d
flag followed by the port number that you want to use. The default port is 4343
.
cd $HVR_HOME/bin
./hvragentlistener -d 4343
After the initial setup, you must add the agent to the system auto startup/shutdown sequence to ensure the HVA starts and stops with the database host. For instructions, see section Autostart.
Autostartlink
The following steps should be performed as user root
to configure systemd.
Create the systemd unit files
hvr.socket
andhvr@.service
in the/etc/systemd/system
directory.The
hvr.socket
file should contain the following:[Unit] Description=Fivetran agent service socket [Socket] ListenStream=4343 Accept=true TriggerLimitIntervalSec=1s TriggerLimitBurst=10000 MaxConnectionsPerSource=100 MaxConnections=500 KeepAlive=true [Install] WantedBy=sockets.target
NOTE:
TriggerLimitIntervalSec
is supported since systemd version 230.TriggerLimitBurst
is supported since systemd version 230.MaxConnectionsPerSource
is supported since systemd version 232. Thehvr@.service
should contain the following:[Unit] Description=Fivetran Agent service [Service] Environment="HVR_HOME=/opt/fivetran/hvr_home" Environment="HVR_CONFIG=/opt/fivetran/hvr_config" Environment="HVR_TMP=/opt/fivetran/hvr_tmp" User=hana ExecStart=/opt/fivetran/hvr_home/bin/hvragent StandardInput=socket KillMode=process [Install] WantedBy=multi-user.target
TIP: Set
User
as the user for which HVA is installed/running.To enable and start the service, execute the following commands:
systemctl enable hvr.socket systemctl start hvr.socket
To verify whether the service is active, execute the following command:
systemctl status hvr.socket
A sample output:
hvr.socket - HVR service socket Loaded: loaded (/etc/systemd/system/hvr.socket; enabled; vendor preset: enabled) Active: active (listening) since Mon 2020-09-07 17:54:44 CEST; 5s ago Listen: [::]:4343 (Stream) Accepted: 0; Connected: 0
Finish Fivetran configurationlink
In your connector setup form, enter a Destination schema. This applies to the connector and cannot be changed once your connector is created.
Choose the NetWeaver Connection Type:
Application Server: a simple connection for a single server. If you choose this type, complete the following fields:
- Host: Hostname or IP-address of the server on which the SAP NetWeaver is running.
- Instance Number: Two-digit number (00-97) of the SAP instance within its host.
- Client ID: Three digit (000-999) identifier of the SAP client, which is sent to an AS ABAP upon logon.
Message Server: connect using load balancer, when more than one dialog instance is configured. If you choose this type, complete the following fields:
- Host: Hostname or IP-address of the server on which the SAP NetWeaver is running.
- Client Id: Three digit (000-999) identifier of the SAP client, which is sent to an AS ABAP upon logon.
- Choose the Remote Service Identification used for establishing connection to a message server:
- System ID. If you choose this option, enter the value in the System ID field below. The value should be the unique identifier
<sapsid>
of the SAP system. - Service. If you choose this option, enter the value in the Service field below. The value should be the port number or service name (like
sapms<SID>
) available in local/etc/services
file. Specify this parameter only if the message server does not listen on the standard servicesapms<SysID>
or if this service is not defined in the services file, and you need to specify the network port directly.
- System ID. If you choose this option, enter the value in the System ID field below. The value should be the unique identifier
- (optional) Enter the name of the SAP Logon Group. The default value is
PUBLIC
.
Enter the Fivetran-specific User.
Enter the Password for the Fivetran-specific user.
Enter the RFC SDK library files directory.
(Optional) To use secure network connection, set the SNC toggle to ON and provide the following information:
- Name. This is a token or identifier representing the external RFC program. Another term for SNC name is Client SNC name (DataStage Server SNC Name). It is also referred to as client Personal Security Environment (PSE) Name. An example of the SNC name is
p:CN=MYUSER_AGENT, O=FIVETRAN
. - Partner Name. This is a token or identifier representing the backend system or the communication partner’s SNC name. For example,
p:CN=EC3, O=Kerberos
. - SNC Library Path. This is a path to the external security product’s library. For example,
/home/ec2-user/sec/libsapcrypto.so
.
- Name. This is a token or identifier representing the external RFC program. Another term for SNC name is Client SNC name (DataStage Server SNC Name). It is also referred to as client Personal Security Environment (PSE) Name. An example of the SNC name is
Choose a Capture Method:
NOTE: We support DirectDBMS Log Reading and Archive Only capture methods when the HVA and HANA database are located on the same node. However, we only support the Archive Only capture method if they are on separate nodes.
- DirectDBMS Log Reading: Captures changes directly from HANA's log segments and log backups.
- Archive Only: Captures changes from HANA's log backups only. Selecting this method reveals the following fields:
- (Optional) Backup Directory: Enter the directory path containing log backups on the machine where HVA is installed. If this field is left empty, High-Volume Agent will detect the location of the log backups automatically.
- (Optional) FileName Format: Specify the filename format (template) of log backups stored in the directory specified in the Backup Directory field. Fivetran will only read the log backups that match the specified format. This field accepts the following format variables:
%v: log volume ID
%p: log partition ID
%s: start sequence number
%e: end sequence number
%t: start timestamp (in milliseconds since UNIX epoch)
%%: matches %
*: wildcard, matches zero or more characters
NOTE: The %s, %e, and %t format variables are mandatory. If the format is not defined, the default format is log_backup_%v_%p_%s_%e.%t.
(Optional) Backint Executable Path: Directory path for the Backint application installed on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(Optional) Backint Configuration Path: Directory path for the Backint configuration on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(For encrypted logs and log backups only) Enter your Root Keys backup Password created in Step 5.
Choose your Connection Method:
Connect directly
Connect via an SSH tunnel: Provide the following information:
- SSH Host (do not use a load balancer's IP address/hostname)
- SSH Port (the default port number is
22
) - SSH User
Connect via PrivateLink
Connect via proxy agent: Choose the necessary proxy agent from the Proxy agents drop-down list (if available) or configure a new proxy agent.
In the Agent Host field, enter the hostname or IP address of the HVA. In most cases, it should match the database hostname or IP address. However, it may vary depending on the connection method you have chosen. For more details, refer to High-Volume Agent network configuration options.
NOTE: Ensure that your Firewall/Security Group allows Fivetran's IPs for your database's region.
In the Agent Port field, enter the HVA's port number. The default port number is
4343
.In the Agent User ID field, enter the name of the HVA created in Step 11.
In the Agent User Password, enter the password of the HVA user created in Step 11.
In the Agent Public Cert, enter the HVA's Server Public Certificate you found in Step 11.