High-Volume Agent SAP
The Fivetran High-Volume Agent (HVA) connectors facilitate high-volume data replication for multiple SAP ERP services. The HVA allows to efficiently capture data from SAP databases and then process and write the data to your destination.
Supported services
We have HVA connectors for the following SAP ERP services:
High-Volume Agent overview
High-Volume Agent (HVA) connectors are designed to support high-volume data replication and use an on-premises software (agent) that typically resides on the same machine as a source database or file system to perform the task of capturing data. The on-premises agent sends the captured data back to Fivetran, so it can be processed and loaded to the destination.
We offer multiple network configuration options to connect your databases and services using HVA. For more information, see High-Volume Agent network configuration options.
On Unix and Linux, HVA runs as a system process (daemon). On Windows, HVA runs as a Windows service.
Fivetran connects to HVA using a specified TCP/IP port number (usually 4343).
Benefits
HVA connectors are most beneficial for databases that generate large amounts of data or have an extensive historical data repository. Common use cases include software product databases that track every event, or databases used as ERP backends where there may be thousands of data units to track and monitor.
The distributed architecture with agents provides security benefits and multiple performance and scalability advantages.
- Speed: The agent on a source machine filters data before sending it down the data pipeline. Databases always write more information to the transaction log than should be replicated, and in many cases, only a subset of the application’s tables is replicated to another system. Filtering data close to the source improves efficiency.
- Bandwidth: The agent compresses data before sending it. Sending compressed data across the wire requires less bandwidth and/or fewer data packets. The agent reduces network cost, and allows capture of changes directly from a DBMS logging system.
- Security: Communication between Fivetran and High-Volume Agent SQL Server connector is encrypted during transit. Authentication is locked down with an SSL certificate.
Resource consumption
CPU: HVA uses up to 1 CPU core in the system for batch runs. During the initial sync, up to four tables at a time are loading, each running its own process. Typically, the HVA process uses relatively few resources, but the associated database job uses a lot of resources to retrieve data (if parallel select operations run against a database, then the initial sync/load or full historical refresh may consume more CPU on the source database). Log parsing is generally the most resource intensive operation that can use up to 100% of a single CPU core when capture is running behind. HVA uses one log parser per database thread, and each database node in a SQL Server cluster constitutes one thread.
Memory: Memory consumption is up to 64 MB per transaction. Typically, memory usage does not reach 64 MB per transaction. However, this depends on the size of transactions and what portion of them are against tables that are part of the configuration. Note that you can adjust (upwards and downwards) the 64 MB threshold.
Storage space: An HVA installation is about 135 MB in size. While running, it does not use any additional disk space until it exceeds the 64 MB memory threshold and starts accumulating transaction files on disk. These files are compressed. If a database runs large batch jobs modifying multiple tables and only committing at the end, HVA may write more data to disk. We recommend starting with at least 5 GB for the configuration folder ($HVR_CONFIG).