High-Volume Agent SAP ECC on HANA with NetWeaver Setup Guide Private Previewlink
Follow these instructions to replicate your SAP HANA database to your destination using Fivetran High-Volume Agent (HVA) connector.
IMPORTANT: This connector only supports Snowflake as a destination in Private Preview.
Prerequisiteslink
To connect your HANA database using the High-Volume Agent connector to Fivetran, you need:
- A Fivetran account with an Enterprise or Business Critical plan
- SAP ABAP version 7.5 and above
- SAP system that must be Unicode
- SAP HANA from version 1.0 SPS 11 to version 2.0 SPS 06)
- Your server IP (for example,
1.2.3.4
) or domain (your.server.com
) - Your SAP instance number (a two-digit number)
- Client id (a three-digit number)
- Your database name
- Installation of the agent on your SAP HANA database host.
- The agent IP address and port number (usually
4343
)
- The agent IP address and port number (usually
- Table types need to be Column Store
- Installation of SAP NetWeaver
- NetWeaver RFC SDK version 7.50.8 or above installed on the machine from where HVA connects to the SAP NetWeaver
- SAP Communication user
Setup instructionslink
Choose connection methodlink
First, decide whether to connect Fivetran to your HANA database:
IMPORTANT: No matter which option you select, you must install the Fivetran High-Volume Agent on your database host.
Connect directlylink
Fivetran connects directly to the High-Volume Agent installed on the database host. All communication between Fivetran and the Agent is encrypted. The agent connects directly to your HANA database.
Connect using SSHlink
Fivetran connects to a separate server in your network that provides an SSH tunnel to your database host. You must connect through SSH if your database is in an inaccessible subnet.
To connect using SSH, follow our SSH connection instructions.
NOTE: If Reverse SSH Tunnel is required, contact our support team as additional steps will be required in the setup of the connector.
Connect using private networkinglink
IMPORTANT: You must have a Business Critical plan to use private networking.
Private networking enables communication between private networks and services without exposing traffic to the public internet. Private networking is the most secure connection method. We support the following providers:
- AWS PrivateLink – used for VPCs and either AWS-hosted or on-premises services. See our AWS PrivateLink setup guide for details.
- Azure PrivateLink – used for Virtual Networks (VNets) and either Azure-hosted or on-premises services. See our Azure PrivateLink setup guide for details.
- Google Cloud Private Service Connect – used for VPCs and either Google-hosted or on-premises services. See our Google Cloud Private Service Connect setup guide for details.
Install NetWeaver RFC SDK librarieslink
Fivetran requires NetWeaver RFC SDK version 7.50.8 or above installed on the machine from where HVA connects to the SAP NetWeaver.
Go to the SAP's software download page.
Download and extract the Unicode SAP NetWeaver RFC SDK libraries that are specific to your operating system.
Copy the following files from the
/lib
directory (available in the extracted path) to a directory of your choice:OPERATING SYSTEM LIBRARIES Linux 64 bit libicudata.so.xx
libicudecnumber.so
libicui18n.so.xx
libicuuc.so.xx
libsapnwrfc.so
libsapucum.soWindows 64 bit icudtxx.dll
icuinxx.dll
icuucxx.dll
libsapucum.dll
libicudecnumber.dll
sapnwrfc.dllNOTE: xx in the library names can vary depending on the library versions.
For Linux, set the read, write, and execute permissions for each NetWeaver RFC library file.
Import Fivetran transportslink
Import the transports containing Fivetran's ABAP functions. Use your company's default method to import the transports provided.
NOTE: While importing, you may need to use the Ignore Invalid Component Version option to suppress import errors.
The transport files are available in the HVR_HOME/dbms/netweaver
directory. For information about the latest transport files, refer to the readme.txt
file in the same directory.
Configure Fivetran AppConnect in SAPlink
Fivetran's AppConnect has a number of configuration parameters that can be used to tune and optimize data extraction. You can access these parameters using the following SAP transaction: /N/HVR/CONFIG
.
In most cases, you can retain the default values set for these parameters. When you want to make a change, trigger a change mode and save the new values into the ACTIVE
variant.
PARAMETER | DESCRIPTION |
---|---|
Max wait background | Maximum idle runtime for background processes. |
Max wait foreground | Maximum idle runtime for foreground processes. |
DB read max wait active | Maximum time spent in active mode in the databse read process. |
DB read max wait passive | Maximum time spent in passive (wait) mode in the database read process. |
RFC read max wait passive | Maximum time spent in passive (wait) mode in the RFC read process. |
Memory tunnel size | Size of shared memory buffer used in SAP. |
Max size uncompressed | Maximum data package size. |
Application server | Name of the dedicated application server. You can set the value for this parameter to NONE . However, you should only use it when instructed by Fivetran support. |
DB connection name | Secondary database connection name to be used in case you do not want to use the standard SAP database connection. You can set the value for this parameter to DEFAULT . However, you should only use it when instructed by Fivetran support. |
Activate event log | Activate logging for database read process (job logs). |
Activate RFC trace | Activate logging for RFC read processes (application log). |
Assign access permissions to SAP NetWeaver userlink
Fivetran's AppConnect requires a Communication user with the following permissions:
RFC access to standard SAP functions to establish connection:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC ACTVT 16 RFC_TYPE FUGR RFC_NAME BTCH
RFC1
SDIFRUNTIMEAUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC ACTVT 16 RFC_TYPE FUNC RFC_NAME RFCPING RFC access to the HVR function group:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC ACTVT 16 RFC_TYPE FUGR RFC_NAME /HVR/SAPAPPCONNECT Authorization required for batch process administration to schedule background jobs:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_RFC ACTVT 16 RFC_TYPE FUGR RFC_NAME S_BTCH_JOB AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_BTCH_ADM BTCADMIN Y AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_BTCH_JOB JOBACTION RELE JOBGROUP * Data access authorizations:
The
S_TABU_SQL
authorization object can be used to manage access rights for every data table in the SAP database.To grant full access, apply the following settings:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE * Alternatively, to restrict access and grant permissions only to specific data tables that you want to synchronize, apply the following settings:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE Target table name NOTE: When granting permissions only to specific data tables, the authorization object
S_TABU_SQL
must be added to user profile.Additionally, when granting permissions only to specific data tables, Fivetran requires access to certain standard SAP tables; you must grant access rights to these tables:
AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER * TABLE CLIENTSIDE_ENCRYPTION_COLUMN_K
CS_ALL_COLUMNS
CS_COLUMNS_
CS_CONCAT_ATTRIBUTES_
CS_TABLES_
DUMMY
HAS_NEEDED_SYSTEM_PRIV
HAS_NEEDED_SYSTEM_PRIV_INCL_SY
INDEX_COLUMNS
M_CONNECTIONS
M_DATABASE
NUMA_NODE_PREFERENCE_
P_DATATYPES_
P_GRANTEDPRIVS_
P_INDEXES_
P_MASK_EXPRESSION_
P_OBJTYPES_
P_PRINCIPALS_
P_PROCEDURES_
P_SCHEMAS_
P_USERS_
RS_COLUMNS_
RS_TABLES_
SERIES_DATA_
TABLES
TABLE_COLUMNS
VIRTUAL_COLUMNS_
VIRTUAL_TABLES_
_SYS_CS_TABLE_COLUMNS
_SYS_GRANTED_OBJECTS
_SYS_GRANTEE_OIDS
_SYS_RS_TABLE_COLUMNS
_SYS_SCHEMAS_WITH_PRIVILEGES_O
_SYS_VIRTUAL_TABLE_COLUMNS
:BF0000
ALL_OBJECTS
CDEF$
CON$
DBA_OBJECTS
DEFERRED_STG$
DEPENDENCY$
IND$
I_COBJ#
I_CON2
I_DEFERRED_STG1
I_DEPENDENCY1
I_FILE#_BLOCK#
I_IND1
I_LINK1
I_OBJ#
I_OBJ1
I_OBJ2
I_OBJ4
I_OBJ5
I_OBJAUTH1
I_OLAP_CUBES$
I_SUM$_1
I_TABCOMPART$
I_TABPART_OBJ$
I_TABSUBPART$_OBJ$
I_TRIGGER2
I_TS#
I_USER#
I_USER1
I_USER2
OBJ$
SEG$
SUM$
TAB$
TABCOMPART$
TABPART$
TABSUBPART$
TRIGGER$
TS$
USER$
USER_EDITIONING$
X$CON
X$DNFS_FILES
X$KCCAL
X$KCCDI
X$KCCDI2
X$KCCFN
X$KCCIC
X$KCCLE
X$KCCRT*
X$KGLCURSOR_CHILD
X$KQLFXPL*
X$KRSTDEST
X$KSLED*
X$KSLWT
X$KSPPCV
X$KSPPI
X$KSPPSV
X$KSUSE*
X$KTADM
X$KTCXB
X$KZEKMENCWAL
X$KZSPR
X$KZSRO
X$VERSION
VW*AUTHORIZATION OBJECT AUTHORIZATION FIELD AUTHORIZATION VALUE S_TABU_SQL ACTVT 33 DBSID * TABOWNER standard DB schema(SAPABAP1) TABLE /HVR/BACKUP_CONFIGURATION
/HVR/LOG_PARTITIONS
/HVR/SERVICE_LOG_POSITIONS
/HVR/TABLE_COLUMNS
/HVR/TABLE_VIRTUAL_FILES
DD02L*
DD02T*
DD03L*
DD04L*
DD06T*
DD08L*
DD09L*
DD16S
DDNTF
DDNTT*
HVR_IS_ARCHBLK
HVR_STIN_ARCBLK
TADIR*
Configure log modelink
For Fivetran to capture changes from HANA, you must enable the automatic backup of HANA transaction logs. Fivetran reads changes from the 'online' transaction log file. However, if the capture process is interrupted (for example, for 2 hours), Fivetran needs to access the transaction log backup files to retrieve the older changes. Fivetran does not process full backups but only reads transaction log backup files.
To enable automatic log backup in HANA, set the log mode to normal
. You can change the log mode using HANA Studio. After changing the log mode from overwrite
to normal
, you must create a full data backup.
For detailed steps, search for change log modes in SAP HANA documentation.
Transaction log (archive) retentionlink
If a backup process has already moved the transaction log files to the tape and deleted them, you must initiate a full re-sync. The required 'retention' period, measured in hours or days, depends on organizational factors (such as the desired level of real-time data) and practical considerations (such as the duration of a data refresh, whether it takes 1 hour or 24 hours).
Configure capture from Backintlink
NOTE: Perform this step only when you need to capture data from log files stored in Backint for SAP HANA.
To retrieve data from backups created using Backint for SAP HANA, add the Backint application as a trusted external application:
a. Copy the hvrosaccess_example.conf
file from HVR_HOME/etc
to HVR_CONFIG/etc
and rename it to hvrosaccess.conf
.
b. Edit the hvrosaccess.conf
file to add the full path of the Backint application under Allowed_Plugin_Paths
. Typically, the default path is /usr/sap/HDB/SYS/global/hdb/opt
.
Example:
{
## Following section 'safelists' directories for command execution.
Allowed_Plugin_Paths: [
/usr/sap/HDB/SYS/global/hdb/opt
]
}
Configure capture from HANA encrypted logs and log backupslink
NOTE: Perform this step only when you need to capture data from HANA encrypted logs and log backups.
To capture from HANA encrypted log files and log backups, the following must be configured:
In addition to the grants provided in section Configure capture, the Fivetran database user must be granted the following privilege to read the encrypted log files and log backups:
grant execute on _HVR."/HVR/ROOT_KEYS_EXTRACT" to <username>;
If using a multi-tenant configuration, the encryption configuration management should be delegated to tenants. Execute the following command on the system database to change control over to the tenant databases:
alter system encryption configuration controlled by local databases;
Set a password for the encryption root keys backup in the source database:
alter system set encryption root keys backup password <password>;
Make a note of the password set for the encryption root keys backup in the previous step. You will need it to configure the Root Keys backup Password option in the connector setup form (Step 12).
Configure capturelink
The following access configuration is required for capturing changes from NetWeaver on HANA:
- Fivetran needs access to data from system dictionaries. This access is provided via views created as HANA user system.
Log in to SAP.
Start transaction DBCO.
Create a secondary connection using HANA user
SYSTEM
.NOTE: This connection should only be used for creation of the views, never for the standard replication process.
Run the program
/HVR/SAPAPPCONNECT_HANASYSVIEW
using transactionSA38
.Populate the field SYS DB Connection with the secondary connection created earlier and select the show log option.
Execute the program.
NOTE: Views are created in the default SAP schema with the name beginning with
/HVR/<view_name>
and will therefore not conflict with other existing SAP standard and custom objects.
Install HVAlink
This section provides detailed instructions on how to install HVA and outlines the necessary requirements.
HVA requirementslink
Before proceeding with the installation of HVA, ensure that you have the following prerequisites in place:
Compatibility: Verify that the HVA version is compatible with your operating system and DBMS. Refer to the COMPATIBILITY section in the relevant release notes on the Downloads page of your Fivetran dashboard.
Sufficient disk space: Ensure that the machine where you want to install HVA has ample disk space. We recommend a minimum of 10 GB of available disk space.
HVA installation file: Download the HVA installation file from the Downloads page of your Fivetran dashboard. Select the installation file suitable for your database server's operating system.
IMPORTANT: You must install HVA on the same host where your HANA database is running.
Install and operate HVA under the database owner user account (for example,
hana
).
Follow the instructions below to install HVA.
Install HVA on Linuxlink
Expand for instructions
Create folder structure
i. Create the HVA installation directory (for example,
/opt/fivetran
) and set the appropriate owner (for example,hana
) and group (for example,oinstall
) for it.NOTE: This command must be executed by the
root
user.mkdir /opt/fivetran chown -R hana:oinstall /opt/fivetran
ii. In the
/opt/fivetran
directory, create three key subdirectorieshvr_home
,hvr_config
, andhvr_tmp
:mkdir -p /opt/fivetran/hvr_home mkdir -p -m 01775 /opt/fivetran/hvr_config /opt/fivetran/hvr_tmp
Configure environment
i. Configure the
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
environment variables to point to the relevant HVA installation subdirectories:export HVR_HOME=/home/myhvr/hvr_home export HVR_CONFIG=/home/myhvr/hvr_config export HVR_TMP=/home/myhvr/hvr_tmp
ii. Add the
$HVR_HOME/bin
executable directory path to the environment variablePATH
:export PATH=$PATH:$HVR_HOME/bin
iii. Add the environment variables and the executable directory path into the startup file (for example,
.profile
or.bash_profile
):export HVR_HOME=/opt/fivetran/hvr_home export HVR_CONFIG=/opt/fivetran/hvr_config export HVR_TMP=/opt/fivetran/hvr_tmp export PATH=$PATH:$HVR_HOME/bin
Install HVA
Navigate to the
$HVR_HOME
directory and extract the contents of the HVA installation file (for example,fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
).cd $HVR_HOME tar xzf /tmp/fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
Once completed, the HVA and all required components are installed into the
$HVR_HOME
directory (/opt/fivetran/hvr_home
).NOTE: The size of the installation is less than 150 MB. After installing (extracting), you can delete the installation file as it is no longer required.
Configure HVAlink
Follow the instructions below to configure HVA:
Create an HVA user to set up your Fivetran connector. In the following example, we use
hva_user
.NOTE: The minimum password length for the user must be 10 characters and not contain special characters.
hvragentuserconfig -c hva_user Password for 'hva_user': <enter a password> Retype password: <confirm the password>
Make a note of the username and password. You will need them to configure Fivetran.
Run the following command to disable the setup mode:
hvragentconfig Setup_Mode_Timed_Until=
Run the following command to extract the HVA public certificate:
hvragentconfig Agent_Server_Public_Certificate
Save the output value of the
Agent_Server_Public_Certificate
. You will need it to set up your connector.
Start HVAlink
To start the agent, run the hvragentlistener
command with the -d
flag followed by the port number that you want to use. In the example, the default port is 4343
.
cd $HVR_HOME/bin
./hvragentlistener -d 4343
After the initial setup, you must add the agent to the system auto startup/shutdown sequence to ensure the HVA starts and stops with the database host. For instructions, see section Autostart.
Autostartlink
The following steps should be performed as the root
user to configure systemd:
Create the systemd unit files
hvr.socket
andhvr@.service
in the/etc/systemd/system
directory. Thehvr.socket
file should contain the following:NOTE: Specify the
User
for which the agent is installed/running, for examplehana
.[Unit] Description=Fivetran agent service socket [Socket] ListenStream=4343 Accept=true TriggerLimitIntervalSec=1s TriggerLimitBurst=10000 MaxConnectionsPerSource=100 MaxConnections=500 KeepAlive=true [Install] WantedBy=sockets.target
NOTE:
TriggerLimitIntervalSec
is supported since systemd version 230.TriggerLimitBurst
is supported since systemd version 230.MaxConnectionsPerSource
is supported since systemd version 232. Thehvr@.service
should contain the following:[Unit] Description=Fivetran Agent service [Service] Environment="HVR_HOME=/opt/fivetran/hvr_home" Environment="HVR_CONFIG=/opt/fivetran/hvr_config" Environment="HVR_TMP=/opt/fivetran/hvr_tmp" User=hana ExecStart=/opt/fivetran/hvr_home/bin/hvragent StandardInput=socket KillMode=process [Install] WantedBy=multi-user.target
To enable and start the service, execute the following commands:
systemctl enable hvr.socket systemctl start hvr.socket
To verify whether the service is active, execute the following command:
systemctl status hvr.socket
A sample output:
hvr.socket - HVR service socket Loaded: loaded (/etc/systemd/system/hvr.socket; enabled; vendor preset: enabled) Active: active (listening) since Mon 2020-09-07 17:54:44 CEST; 5s ago Listen: [::]:4343 (Stream) Accepted: 0; Connected: 0
Finish Fivetran configurationlink
In the connector setup form, enter a Destination schema. This applies to the connector and cannot be changed once your connector is created.
Choose the NetWeaver Connection Type:
Application Server: a simple connection for a single server. If you choose this type, complete the following fields:
- Host: Hostname or IP-address of the server on which the SAP NetWeaver is running.
- Instance Number: Two-digit number (00-97) of the SAP instance within its host.
- Client ID: Three digit (000-999) identifier of the SAP client, which is sent to an AS ABAP upon logon.
Message Server: connect using load balancer, when more than one dialog instance is configured. If you choose this type, complete the following fields:
- Host: Hostname or IP-address of the server on which the SAP NetWeaver is running.
- Client Id: Three digit (000-999) identifier of the SAP client, which is sent to an AS ABAP upon logon.
- Choose Remote Service Identification used for establishing connection to a message server:
- System ID. If you choose this option, enter the value in the System ID field below. The value should be the unique identifier
<sapsid>
of the SAP system. - Service. If you choose this option, enter the value in the Service field below. The value should be the port number or service name (like
sapms<SID>
) available in local/etc/services
file. Specify this parameter only if the message server does not listen on the standard servicesapms<SysID>
or if this service is not defined in the services file, and you need to specify the network port directly.
- System ID. If you choose this option, enter the value in the System ID field below. The value should be the unique identifier
- (optional) Enter the name of the SAP Logon Group. The default value is
PUBLIC
.
Enter the Fivetran-specific User.
Enter the Password for the Fivetran-specific user.
Enter the RFC SDK library files directory.
(Optional) To use secure network connection, set the SNC toggle to ON and provide the following information:
- Name. This is a token or identifier representing the external RFC program. Another term for SNC name is Client SNC name (DataStage Server SNC Name). It is also referred to as client Personal Security Environment (PSE) Name. An example of the SNC name is
p:CN=MYUSER_AGENT, O=FIVETRAN
. - Partner Name. This is a token or identifier representing the backend system or the communication partner’s SNC name. For example,
p:CN=EC3, O=Kerberos
. - SNC Library Path. This is a path to the external security product’s library. For example,
/home/ec2-user/sec/libsapcrypto.so
.
- Name. This is a token or identifier representing the external RFC program. Another term for SNC name is Client SNC name (DataStage Server SNC Name). It is also referred to as client Personal Security Environment (PSE) Name. An example of the SNC name is
Choose a Capture Method:
NOTE: We support DirectDBMS Log Reading and Archive Only capture methods when the HVA and HANA database are located on the same node. However, we only support the Archive Only capture method if they are on separate nodes.
- DirectDBMS Log Reading: Captures changes directly from HANA's log segments and log backups.
- Archive Only: Captures changes from HANA's log backups only. Selecting this method reveals the following fields:
- (Optional) Backup Directory: Enter the directory path containing log backups on the machine where HVA is installed. If this field is left empty, High-Volume Agent will detect the location of the log backups automatically.
- (Optional) FileName Format: Specify the filename format (template) of log backups stored in the directory specified in the Backup Directory field. Fivetran will only read the log backups that match the specified format. This field accepts the following format variables:
%v: log volume ID
%p: log partition ID
%s: start sequence number
%e: end sequence number
%t: start timestamp (in milliseconds since UNIX epoch)
%%: matches %
*: wildcard, matches zero or more characters
NOTE: The %s, %e, and %t format variables are mandatory. If the format is not defined, the default format is log_backup_%v_%p_%s_%e.%t.
(Optional) Backint Executable Path: Directory path for the Backint application installed on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(Optional) Backint Configuration Path: Directory path for the Backint configuration on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(For encrypted logs and log backups only) Enter your Root Keys backup Password created in Step 5.
Choose your Connection Method:
Connect directly
Connect via an SSH tunnel: Provide the following information:
- SSH Host (do not use a load balancer's IP address/hostname)
- SSH Port (the default port number is
22
) - SSH User
Connect via PrivateLink
Connect via proxy agent: Choose the necessary proxy agent from the Proxy agents drop-down list (if available) or configure a new proxy agent.
In the Agent Host field, enter the hostname or IP address of the HVA. In most cases, it should match the database hostname or IP address. However, it may vary depending on the connection method you have chosen. For more details, refer to High-Volume Agent network configuration options.
NOTE: Ensure that your Firewall/Security Group allows Fivetran's IPs for your database's region.
In the Agent Port field, enter the HVA's port number. The default port number is
4343
.In the Agent User ID field, enter the name of the HVA created in Step 10.
In the Agent User Password, enter the password of the HVA user created in Step 10.
In the Agent Public Cert, enter the HVA's Server Public Certificate you found in Step 10.