Warning: W_JJ106E Wait for Response From Database Worker Exceeded
Issue
The HVR Scheduler periodically stops responding for about 15 minutes and the following warnings appear in the hub log (hvr.out):
W_JJ106E: Wait for response from database worker exceeded the 600 seconds defined by 'timeo_db' attribute. Waiting...
W_JJ1084: Response received from database worker after wait of 950 seconds.
The HVR UI or CLI can also report F_JJ1306: Server is not alive after 10 seconds. while the Scheduler is waiting.
Environment
- HVR 6
- Repository database: Oracle
- A firewall, NAT gateway, load balancer, or other network component (for example, in AWS) between the HVR Hub and the repository database
Resolution
To resolve this issue, configure the client-side keepalive setting (EXPIRE_TIME = 1) for the repository database connection:
In tnsnames.ora on the HVR Hub, add
(EXPIRE_TIME = 1)to the repository database's connect descriptor. The value is in minutes, so1sends a keepalive probe every minute. Use a value shorter than the idle timeout of the network component. Keep the existing values of your descriptor and add only the(EXPIRE_TIME = 1)line:<tns_alias> = (DESCRIPTION = (EXPIRE_TIME = 1) (ADDRESS = (PROTOCOL = TCP)(HOST = <repository_db_host>)(PORT = <listener_port>)) (CONNECT_DATA = (SERVICE_NAME = <repository_service_name>)) )The tnsnames.ora file is in the directory specified by the
TNS_ADMINenvironment variable of the HVR Hub Server, or ifTNS_ADMINis not set, in$ORACLE_HOME/network/admin.If the hub server property Oracle_TNS uses the
<host>:<port>/<service_name>form, set it to the net service name<tns_alias>instead. Otherwise, the tnsnames.ora entry is not used and the setting has no effect:hvrhubserverconfig Oracle_TNS=<tns_alias>Restart the HVR Hub Server.
Cause
This issue occurs when a network component between the HVR Hub and the repository database silently drops idle TCP connections (for example, after 350 seconds in AWS). The Scheduler's next request on the dropped connection is not answered, and the connection is only reestablished after the operating system's TCP retransmission timeout expires (about 15 minutes on Linux hubs).