High-Volume Agent SAP ECC on HANA Setup Guide Private Previewlink
Follow these instructions to replicate your SAP HANA database to your destination using Fivetran High-Volume Agent (HVA) connector.
IMPORTANT:
- In Private Preview, the connector only supports Snowflake, BigQuery, and Databricks as destinations.
- Fivetran supports SAP HANA 2.0 SPS 07.
Prerequisiteslink
To connect your HANA database using the High-Volume Agent (HVA) connector to Fivetran, you need:
- A Fivetran account with an Enterprise or Business Critical plan
- SAP HANA version 1.0 SPS 11 to version 2.0 SPS 06
- The IP address (for example,
1.2.3.4
) or domain (for example,your.server.com
) of your database host - Your SAP instance number (a two-digit number)
- Your database name
- High-Volume Agent (HVA) installed on your SAP HANA database host
- The agent's IP address and port number (usually
4343
) - HANA client with the HANA ODBC driver installed on the SAP HANA database host
- unixODBC installed on the SAP HANA database host, if there is no default ODBC installation
- The
odbcinst.ini
file must contain information about the HANA ODBC driver under heading[HDBODBC]
or[HDBODBC32]
- The
- ODBC Driver Manager library path:
- For default installation, the library is available in
/usr/lib64
- If UnixODBC is installed, for example, in
/opt/unixodbc
, the value for this field will be/opt/unixodbc/lib
- For default installation, the library is available in
- ODBC Sys Ini directory path where the
odbc.ini
andodbcinst.ini
files are located:- For default installation, these files are available in
/etc
- If UnixODBC is installed, for example, in
/opt/unixodbc
, the value for this field will be/opt/unixodbc/etc
- For default installation, these files are available in
- Table types must be Column Store
Setup instructionslink
Choose connection methodlink
First, decide whether to connect Fivetran to your HANA database:
IMPORTANT: No matter which option you select, you must install the Fivetran High-Volume Agent on your database host.
Connect directlylink
Fivetran connects directly to the High-Volume Agent installed on the database host. All communication between Fivetran and the agent is encrypted. The agent connects directly to your SAP HANA database.
Connect using SSHlink
Fivetran connects to a separate server in your network that provides an SSH tunnel to your database host. You must connect through SSH if your database is in an inaccessible subnet.
To connect using SSH, follow our SSH connection instructions.
NOTE: If Reverse SSH Tunnel is required, contact our support team as additional steps are required in the connector setup.
Connect using Proxy Agentlink
Fivetran connects to your database through the Proxy Agent, providing secure communication between Fivetran processes and your database host. The Proxy Agent is installed in your network and creates an outbound network connection to the Fivetran-managed SaaS.
To learn more about the Proxy Agent, how to install it, and how to configure it, see our Proxy Agent documentation.
Connect using private networkinglink
IMPORTANT: You must have a Business Critical plan to use private networking.
Private networking enables communication between private networks and services without exposing traffic to the public internet. Private networking is the most secure connection method. We support the following providers:
- AWS PrivateLink – used for VPCs and either AWS-hosted or on-premises services. See our AWS PrivateLink setup guide for details.
- Azure PrivateLink – used for Virtual Networks (VNets) and either Azure-hosted or on-premises services. See our Azure PrivateLink setup guide for details.
- Google Cloud Private Service Connect – used for VPCs and either Google-hosted or on-premises services. See our Google Cloud Private Service Connect setup guide for details.
Configure capturelink
To capture changes from HANA, you need the following grants:
The user must have
SELECT
privileges on the HANA tables you want to replicate.grant select on table to user
Fivetran needs access to system views. However, granting permissions directly on system objects to any user is not possible in HANA. Instead, we use wrapper views that the user can access. To create these views:
Connect to the HANA database as the
SYSTEM
user.Create a schema called
\_HVR
.create schema _HVR;
Grant the
SELECT
privilege on this schema to the user.grant select on schema _HVR to <username>;
Execute the hvrhanaviews.sql script from the $HVR_HOME/sql/hana directory.
This will create a set of views that HVR uses to read from system dictionaries on HANA.
Configure log modelink
For Fivetran to capture changes from HANA, the automatic backup of transaction logs must be enabled in HANA. Fivetran reads changes from the 'online' transaction log file, but if interrupted (say for 2 hours) then it must be able to read from transaction log backup files to capture the older changes. Fivetran is not interested in full backups; it only reads transaction log backup file.
To enable automatic log backup in HANA, the log mode must be set to normal. The log mode can be changed using HANA Studio. Once the log mode is changed from overwrite to normal, a full data backup must be created.
For detailed steps, search for change log modes in SAP HANA Documentation.
Alternatively, you can execute the following SQL statement:
alter system alter configuration ('global.ini', 'SYSTEM') set ('persistence', 'log_mode') = 'normal' with reconfigure;
Transaction log (archive) retentionlink
If a backup process has already moved the transaction log files to the tape and deleted them, you must initiate a full re-sync. The required 'retention' period, measured in hours or days, depends on organizational factors (such as the desired level of real-time data) and practical considerations (such as the duration of a data refresh, whether it takes 1 hour or 24 hours).
Configure capture from Backintlink
NOTE: Perform this step only when you need to capture data from log files stored in Backint for SAP HANA.
To retrieve data from backups created using Backint for SAP HANA, the following configuration steps are required:
Configure options Backint Executable Path and Backint Configuration Path. This can be done when setting up the connector in Step 9.
Add the Backint application as a trusted external application:
a. Copy the
hvrosaccess_example.conf
file fromHVR_HOME/etc
toHVR_CONFIG/etc
and rename it tohvrosaccess.conf
.b. Edit the
hvrosaccess.conf
file to add the full path of the Backint application underAllowed_Plugin_Paths
. Typically, the default path is/usr/sap/HDB/SYS/global/hdb/opt
.Example:
{ ## Following section 'safelists' directories for command execution. Allowed_Plugin_Paths: [ /usr/sap/HDB/SYS/global/hdb/opt ] }
Configure capture from HANA encrypted logs and log backupslink
NOTE: Perform this step only when you need to capture data from HANA encrypted logs and log backups.
To capture from HANA encrypted log files and log backups, the following must be configured:
In addition to the grants provided in section Configure capture, the Fivetran database user must be granted the following privilege to read the encrypted log files and log backups:
grant execute on _HVR."/HVR/ROOT_KEYS_EXTRACT" to <username>;
If using a multi-tenant configuration, the encryption configuration management should be delegated to tenants. Execute the following command on the system database to change control over to the tenant databases:
alter system encryption configuration controlled by local databases;
Set a password for the encryption root keys backup in the source database:
alter system set encryption root keys backup password <password>;
Make a note of the password set for the encryption root keys backup in the previous step. You will need it to configure option Root Keys backup Password in the connector setup form (Step 9).
Install HVAlink
This section provides detailed instructions on how to install HVA and outlines the necessary requirements.
HVA requirementslink
Before proceeding with the installation of HVA, ensure that you have the following prerequisites in place:
Compatibility: Verify that the HVA version is compatible with your operating system and DBMS. Refer to the COMPATIBILITY section in the relevant release notes on the Downloads page of your Fivetran dashboard.
Sufficient disk space: Ensure that the machine where you want to install HVA has ample disk space. We recommend a minimum of 10 GB of available disk space.
HVA installation file: Download the HVA installation file from the Downloads page of your Fivetran dashboard. Select the installation file suitable for your database server's operating system.
IMPORTANT: You must install HVA on the same host where your HANA database is running.
Install and operate HVA under the database owner user account (for example,
hana
).
Follow the instructions below to install HVA.
Install HVA on Linuxlink
Expand for instructions
Create folder structure
i. Create the HVA installation directory (for example,
/opt/fivetran
) and set the appropriate owner (for example,hana
) and group (for example,oinstall
) for it.NOTE: This command must be executed by the
root
user.mkdir /opt/fivetran chown -R hana:oinstall /opt/fivetran
ii. In the
/opt/fivetran
directory, create three key subdirectorieshvr_home
,hvr_config
, andhvr_tmp
:mkdir -p /opt/fivetran/hvr_home mkdir -p -m 01775 /opt/fivetran/hvr_config /opt/fivetran/hvr_tmp
Configure environment
i. Configure the
HVR_HOME
,HVR_CONFIG
, andHVR_TMP
environment variables to point to the relevant HVA installation subdirectories:export HVR_HOME=/home/myhvr/hvr_home export HVR_CONFIG=/home/myhvr/hvr_config export HVR_TMP=/home/myhvr/hvr_tmp
ii. Add the
$HVR_HOME/bin
executable directory path to the environment variablePATH
:export PATH=$PATH:$HVR_HOME/bin
iii. Add the environment variables and the executable directory path into the startup file (for example,
.profile
or.bash_profile
):export HVR_HOME=/opt/fivetran/hvr_home export HVR_CONFIG=/opt/fivetran/hvr_config export HVR_TMP=/opt/fivetran/hvr_tmp export PATH=$PATH:$HVR_HOME/bin
Install HVA
Navigate to the
$HVR_HOME
directory and extract the contents of the HVA installation file (for example,fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
).cd $HVR_HOME tar xzf /tmp/fivetran-6.1.0_26-hub_and_agent-linux_glibc2.12-x64-64bit_ga_patch.tar.gz
Once completed, the HVA and all required components are installed into the
$HVR_HOME
directory (/opt/fivetran/hvr_home
).NOTE: The size of the installation is less than 150 MB. After installing (extracting), you can delete the installation file as it is no longer required.
Configure HVAlink
Follow the instructions below to configure HVA:
Create an HVA user to set up your Fivetran connector. In the following example, we use
hva_user
.NOTE: The minimum password length for the user must be 10 characters and not contain special characters.
hvragentuserconfig -c hva_user Password for 'hva_user': <enter a password> Retype password: <confirm the password>
Make a note of the username and password. You will need them to configure Fivetran.
Run the following command to disable the setup mode:
hvragentconfig Setup_Mode_Timed_Until=
Run the following command to extract the HVA public certificate:
hvragentconfig Agent_Server_Public_Certificate
Save the output value of the
Agent_Server_Public_Certificate
. You will need it to set up your connector.
Start HVAlink
Perform the following steps as the user who runs HVA.
To start the agent, you need to run the hvragentlistener
command with the -d
flag followed by the port number that you want to run on. In the example, the default port is 4343
.
hvragentlistener -d 4343
After the initial setup, you should add the agent to your system's autostart/shutdown sequence to ensure the agent starts and stops with the database host. For instructions, see section Autostart.
Advanced configurationlink
Autostartlink
Perform the following steps as the root
user to configure systemd:
Create the systemd unit files
hvr.socket
andhvr@.service
in the/etc/systemd/system
directory. Thehvr.socket
file should contain the following:[Unit] Description=Fivetran agent service socket [Socket] ListenStream=4343 Accept=true TriggerLimitIntervalSec=1s TriggerLimitBurst=10000 MaxConnectionsPerSource=100 MaxConnections=500 KeepAlive=true [Install] WantedBy=sockets.target
NOTE:
TriggerLimitIntervalSec
is supported since systemd version 230.TriggerLimitBurst
is supported since systemd version 230.MaxConnectionsPerSource
is supported since systemd version 232.
The
hvr@.service
should contain the following:NOTE: You need to specify the
User
for which the agent is installed/running.[Unit] Description=Fivetran Agent service [Service] Environment="HVR_HOME=/opt/fivetran/hvr_home" Environment="HVR_CONFIG=/opt/fivetran/hvr_config" Environment="HVR_TMP=/opt/fivetran/hvr_tmp" User=<username> ExecStart=/opt/fivetran/hvr_home/bin/hvragent StandardInput=socket KillMode=process [Install] WantedBy=multi-user.target
To enable and start the service, execute the following commands:
systemctl enable hvr.socket systemctl start hvr.socket
To verify whether the service is active, execute the following command:
systemctl status hvr.socket
A sample output:
hvr.socket - HVR service socket Loaded: loaded (/etc/systemd/system/hvr.socket; enabled; vendor preset: enabled) Active: active (listening) since Mon 2020-09-07 17:54:44 CEST; 5s ago Listen: [::]:4343 (Stream) Accepted: 0; Connected: 0
Finish Fivetran configurationlink
In the connector setup form, enter a Destination schema prefix of your choice. This applies to the connector and cannot be changed once your connector is created.
In the Host field, enter the hostname or IP address of your database.
Choose the Connection Mode to connect to the HANA server:
- Single container: Use this connection mode when there is only one container in the SAP HANA system. You can connect to the database using the host name and port number of the container.
- Multiple containers - Tenant database: Use this connection mode when there are multiple containers in the SAP HANA system, and each container has its own tenant database. You can connect to the database using the host name, port number, and tenant ID of the container.
- Multiple containers - System database: Use this connection mode when there are multiple containers in the SAP HANA system, and there is one system database that is shared by all containers. You can connect to the system database using the host name and port number of the system container.
- Manual port selection: Use this connection mode when you want to connect to the database using a port number that is different from the default port number. If you selected this option, enter the Port number.
Enter your SAP Instance Number.
Enter the name of your Database.
NOTE: This field is only available if the Multiple containers - Tenant database or Manual port selection was chosen as connection mode.
Enter the Fivetran-specific User.
Enter the Password for the Fivetran-specific user.
(Optional) Enter the ODBC manager library path.
- For default installation, the library is available in
/usr/lib64
. - If UnixODBC is installed elsewhere, for example, in
/opt/unixodbc
, the value for this field will be/opt/unixodbc/lib
.
- For default installation, the library is available in
(Optional) Enter the ODBC Sys Ini directory path where the
odbc.ini
andodbcinst.ini
files are located:- For default installation, these files are available in
/etc
. - If UnixODBC is installed in a different location, for example, in
/opt/unixodbc
, the value for this field will be/opt/unixodbc/etc
.
- For default installation, these files are available in
Enter the name of your Schema with SAP data.
Choose a Capture Method:
NOTE: We support DirectDBMS Log Reading and Archive Only capture methods when the HVA and HANA database are located on the same node. However, we only support the Archive Only capture method if they are on separate nodes.
- DirectDBMS Log Reading: Captures changes directly from HANA's log segments and log backups.
- Archive Only: Captures changes from HANA's log backups only. Selecting this method reveals the following fields:
- (Optional) Backup Directory: Enter the directory path containing log backups on the machine where HVA is installed. If this field is left empty, High-Volume Agent will detect the location of the log backups automatically.
- (Optional) FileName Format: Specify the filename format (template) of log backups stored in the directory specified in the Backup Directory field. Fivetran will only read the log backups that match the specified format. This field accepts the following format variables:
%v: log volume ID
%p: log partition ID
%s: start sequence number
%e: end sequence number
%t: start timestamp (in milliseconds since UNIX epoch)
%%: matches %
*: wildcard, matches zero or more characters
NOTE: The %s, %e, and %t format variables are mandatory. If the format is not defined, the default format is log_backup_%v_%p_%s_%e.%t.
(Optional) Backint Executable Path: Directory path for the Backint application installed on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(Optional) Backint Configuration Path: Directory path for the Backint configuration on the same node as the HVA. Define this field if you are capturing data from backups created using Backint for SAP HANA.
(For encrypted logs and log backups only) Enter your Root Keys backup Password created in Step 5.
Choose your Connection Method:
Connect directly
Connect via an SSH tunnel: Provide the following information:
- SSH Host (do not use a load balancer's IP address/hostname)
- SSH Port (the default port number is
22
) - SSH User
Make a note of the Public Key - you will need it to complete your setup.
Connect via PrivateLink
Connect via proxy agent: Choose the necessary proxy agent from the Proxy agents drop-down list (if available) or configure a new proxy agent.
In the Agent Host field, enter the hostname or IP address of the HVA. In most cases, it should match the database hostname or IP address. However, it may vary depending on the connection method you have chosen. For more details, refer to High-Volume Agent network configuration options.
NOTE: Ensure that your Firewall/Security Group allows Fivetran's IPs for your database's region.
In the Agent Port field, enter the HVA's port number. The default port number is
4343
.In the Agent User ID field, enter the name of the HVA created in Step 4.
In the Agent User Password, enter the password of the HVA user created in Step 4.
In the Agent Public Cert, enter the HVA's Server Public Certificate you found in Step 4.
Click Save & Test. Fivetran tests and validates the connection to your SAP HANA database. Upon successful completion of the setup tests, you can sync your data using Fivetran.