Hvrmaint
Name
hvrmaint - Housekeeping script for HVR on the hub machine.
Synopsis
hvrmaint [optfile] [-options]
Description
Command hvrmaint is a script for regular housekeeping of the HVR on the hub machine. The script can run on the hub machine and can be scheduled on Unix using crontab or as a Windows scheduled task.
The hvrmaint can be used for three main purposes:
- Maintenance: Schedule hvrmaint nightly (or weekly) with options -stop and -start. These options instruct hvrmaint to restart the HVR Scheduler. Often other options can be used, such as -scan_hvr_out (scan log files for HVR errors) or -archive_files (move old log files to archive directory $HVR_CONFIG/logarchive/hub_name/day). Email alerts can be used to send an email with the status summary to operator(s). When used in this way, hvrmaint could be scheduled on Unix using crontab, and on Windows as a Windows Scheduled Task.
- Monitoring: Run hvrmaint frequently (e.g. every 15 minutes) with options -scan_hvr_out, -test_scheduler, and -check_logfile_growth to check if the HVR Scheduler is running and to scan the HVR log files for errors. Running hvrmaint this way does not interrupt the HVR Scheduler. There is option -email_only_when_errors to send emails only if an error has occurred.
- Backup: The last way to use hvrmaint is as part of a larger nightly or weekly batch script, which halts all server processes (including the DBMS), does a system backup and then restarts everything again. In this case, hvrmaint would be called at the top of the batch script with option -stop (stop the HVR Scheduler) and would then be called again near the bottom with option -start (restart the HVR Scheduler).
Command hvrmaint cannot process log files containing more than 12 months of data.
Options
This section describes the options available for command hvrmaint.
| Parameter | Description |
|---|---|
| -task_name=task | Task name is used internally by hvrmaint to locate its option file and name its offset files. This allows different tasks defined in the GUI to have a different state. e.g. so that a when a task for one channel has processed today's files a different task for a different channel still remembers to process today's files. |
| Scheduler checks | |
| -scan_hvr_out | Scan Scheduler log file hvr.out. Command hvrmaint writes a summary of HVR errors detected in this file to its output and to any emails that it sends. |
| -scan_channel=chn | Only scan the specified channel(s) chn for errors and warnings. Requires option -scan_hvr_out. |
| -scan_location=loc | Only scan the specified locations(s) loc for errors and warnings. Requires option -scan_hvr_out. |
| -scan_ignore=patt | Ignore log records which match specified pattern patt (can be regular expression). Requires option -scan_hvr_out. |
| -test_scheduler | Check that HVR Scheduler is actually running using hvrtestscheduler. If option -stop is also defined then this test is performed before the HVR Scheduler is stopped. If option -start is supplied then hvrmaint always checks that the HVR Scheduler is running using a test, regardless of whether or not option -test_scheduler is defined. |
| -check_logfile_growth | Check that logfile hvr.out has grown in size since the last time hvrmaint was run. If this file has not grown then an error message will be written. This option should be used with -scan_hvr_out. |
| -task_group=group | Task group allows different hvrmaint tasks to share the same state. So a nightly task that processes log files and gives a warning if the latency is >1 hour can use the same 'offset state' as a task that runs during the day which gives a warning if latency is >1 minute. |
| Latency checks | |
| -latency_limit=dur | Check for replication latencies and consider jobs over the limit erroneous. Value for dur can be specified in one of the following formats:
|
| -latency_channel=chn | Only check latencies of jobs in specified channel(s) chn. |
| -latency_location=loc | Only check latencies of jobs in specified location(s) loc. |
| Scheduler stop and start | |
| -stop | Stop HVR Scheduler. |
| -start | Start HVR Scheduler. |
| -quiesce_grace=secs | If jobs are still running when the HVR Scheduler must stop, allow seconds secs grace before killing them. The default is 60 seconds. This parameter is passed with the HVR Scheduler using the -q option. |
-start_if_not_runningSince v5.6.5/13 | Start HVR Scheduler if it is not already running.
|
| Logfile archives | |
-archive_files=patt | Move any files in directory $HVR_CONFIG/log/hub_name matching pattern patt to the archive directory ($HVR_CONFIG/logarchive/hub_name/day). Files that do not match pattern patt are deleted. Pattern patt is a regular expression. For example:
|
| -archive_keep_days=N | Retain files in archive directory ($HVR_CONFIG/logarchive/hub_name/day) for N number of days. The retained files are deleted after N number of days. Requires option -archive_files. If this option is not specified, then archived files are kept indefinitely. |
| -archive_compress | Compress HVR Scheduler log files while moving them to the archive directory ($HVR_CONFIG/logarchive/hub_name/day). For a Windows hub, this option can only be used if command gzip has been installed. |
| Journal purging | |
| -journal_keep_days=n | Retain HVR journal files in directory $HVR_CONFIG/jnl for n number of days. The files are deleted after n number of days. These files are written by integrate jobs if parameter Integrate /JournalRouterFiles is defined. |
| Logging | |
| -output_verbose | Prints latency information and error summaries to hvrmaint output. |
| Email alerts | |
| -email_to=addr1[;addr2] | Send the output from hvrmaint as email to the specified email address addr1 [and addr2]. Requires either option -smtp_server or option -mailer. Multiple email addresses can be specified with values separated by a semicolon or using multiple -email_to options (only in CLI). |
| -email_from=addr | Specify a sender address addr in email header. |
| -email_only_when_errors | Send an email if hvrmaint encountered an error itself or detected an HVR error while scanning hvr.out or the latency limit is exceeded. |
| -email_only_when_errors_or_warnings | Send an email if hvrmaint encountered an error itself or detected an HVR error or warning while scanning hvr.out or the latency limit is exceeded. |
| -error_limit=N | HVR errors reported is limited to N number. Default is 1000. This option prevents the generated emails becoming too large. |
| -smtp_server=server | SMTP server to use when sending an email. Value server can be either a node name or IP address. Requires option -email. |
-smtp_portSince v5.6.5/2 | SMTP port to use when sending an email. |
-smtp_starttlsSince v5.6.5/2 | Use the STARTTLS method to communicate with the SMTP server. |
| -smtp_user=user | Username user for authentication SMTP server if needed. |
| -smtp_pass=pass | Password pass used for authentication on the SMTP server if needed. |
| -mailer=cmd | Mailer command to use for sending emails, instead of sending them via an SMTP server. Requires option -email. String %s contained in cmd is replaced by the email subject and string %a is replaced by the intended recipients of the email. The body of the email is piped to cmd as stdin. E.g. on Linux: -mailer=/bin/mail -s %s %a |
-email_repeat_suppression=durSince v5.6.5/11 | Suppress repetition of the same email alert for the specified duration dur. By default, each time when hvrmaint encounters an error itself or detects an HVR error or warning while scanning hvr.out or the latency limit is exceeded, the hvrmaint sends out an alert until the issue is fixed. The number of the alerts sent depends on the frequency in which hvrmaint runs. As long as the issue is not resolved or the error/warning has not changed, hvrmaint will repeatedly send alerts for the same issue. To avoid repeatedly sending alerts for the same issue, this option forces hvrmaint to remain silent for specified duration dur after the first alert is sent out. Value for dur can be specified in one of the following formats:
|
| Slack alerts | |
| -slack_webhook_url=url | A webhook for a Slack channel in company MyCorp looks like https://hooks.slack.com/services/xxxx/yyyy. To generate a Slack webhook, sign into Slack, then navigate to Apps ▶ Manage apps ▶ Custom Integrations ▶ Incoming WebHooks ▶ Add Configuration. |
| -slack_channel=chn | Hvrmaint will send the message to the specified Slack user (@username) or channel chn. This optional field can be used to override the Slack user or channel defined in the Slack webhook (-slack_webhook_url). |
| -send_slack_only_when_errors | Send a Slack message if hvrmaint encountered an error itself or detected an HVR error while scanning hvr.out or the latency limit is exceeded. |
| -send_slack_only_when_errors_or_warnings | Send a Slack message if hvrmaint encountered an error itself or detected an HVR error or warning while scanning hvr.out or the latency limit is exceeded. |
-slack_repeat_suppression=durSince v5.6.5/11 | Suppress repetition of the same Slack alert for the specified duration dur. By default, each time when hvrmaint encounters an error itself or detects an HVR error or warning while scanning hvr.out or the latency limit is exceeded, the hvrmaint sends out an alert until the issue is fixed. The number of the alerts sent depends on the frequency in which hvrmaint runs. As long as the issue is not resolved or the error/warning has not changed, hvrmaint will repeatedly send alerts for the same issue. To avoid repeatedly sending alerts for the same issue, this option forces hvrmaint to remain silent for specified duration dur after the first alert is sent out. Value for dur can be specified in one of the following formats:
|
SNS alertsSince v5.6.5/2 | |
| -sns_notify | Send notification to Amazon Simple Notification Service (SNS). |
| -sns_destination | Amazon Resource Name (ARN) of the SNS topic. |
| -sns_only_when_errors | Send a notification to Amazon SNS if hvrmaint encountered an error itself or detected an HVR error while scanning hvr.out or the latency limit is exceeded. |
| -sns_only_when_errors_or_warnings | Send a notification to Amazon SNS if hvrmaint encountered an error itself or detected an HVR error or warning while scanning hvr.out or the latency limit is exceeded. |
-sns_repeat_suppression=durSince v5.6.5/11 | Suppress repetition of the same SNS alert for the specified duration dur. By default, each time when hvrmaint encounters an error itself or detects an HVR error or warning while scanning hvr.out or the latency limit is exceeded, the hvrmaint sends out an alert until the issue is fixed. The number of the alerts sent depends on the frequency in which hvrmaint runs. As long as the issue is not resolved or the error/warning has not changed, hvrmaint will repeatedly send alerts for the same issue. To avoid repeatedly sending alerts for the same issue, this option forces hvrmaint to remain silent for specified duration dur after the first alert is sent out. Value for dur can be specified in one of the following formats:
|
| -sns_access_key | Access key ID of the AWS IAM user. For more information about access key, refer to Managing Access Keys for IAM Users in AWS documentation. |
| -sns_secret_key | Secret access key of the AWS IAM user. For more information about secret key, refer to Managing Access Keys for IAM Users in AWS documentation. |
| SNMP alerts | |
| -snmp_notify | Send SNMP v1 traps or v2c notifications. The -snmp_community option is required. See $HVR_HOME/lib/mibs/HVR-MIB.txt |
| -snmp_version=vers | Specify SNMP version. Value vers can be 1 or 2c (default). |
| -snmp_heartbeat | Send a hvrMaintNotifySummary notification, even if there was nothing to report. |
| -snmp_hostname=host | SNMP agent hostname host. Default is localhost. |
| -snmp_port=port | SNMP agent trap port. Default is port 162. |
| -snmp_community=str | Community string str for SNMPv1/v2c transactions. |
| Disable | |
| -disable | Disable hvrmaint alerts. This option allows to disable the alerts without stopping the hvrmaint. This can be useful during a maintenance window when channels are being modified or stopped. An alternative is to stop running hvrmaint during the maintenance window and restart it after, but this can generate a lot of alerts caused by the maintenance. |
| -env=NAME=VALUE | Set environment variable. This option can be repeated to set multiple variables such as $HVR_HOME, $HVR_CONFIG, $HVR_TMP, $II_SYSTEM, $ORACLE_HOME etc. In HVRGUI, to view this option, click on the Environment button. |
| -hub=hub | Hub database for HVR Scheduler. This value has form user/pwd (for an Oracle hub), inghub (for an Ingres hub database), or hub for a (SQL Server hub database). For Oracle, passwords can be encrypted using command hvrcrypt. In HVRGUI, this option is only available inside the Text tab. |
| -sched_option=schedopt | Extra startup parameters for the HVR Scheduler service. Possible examples are -uuser/pwd (for a username), -hsqlserver (for the hub class) or -cclus/clusgrp (for Windows cluster group). In HVRGUI, this option is only available inside the Text tab. |
| -output=fil | Append hvrmaint output to file fil. If this option is not supplied, then output is sent to stdout. Output can also be sent to an operator using option -email. In HVRGUI, this option is only available inside the Text tab. |
Configuring HVR Email Alerts Using Gmail SMTP
This section describes the steps to set up HVR to send hvrmaint email alerts via Gmail SMTP server.
Prerequisite: You must generate App Password (-smtp_pass) for the Gmail account (-smtp_user) that will be used to authenticate with the Gmail SMTP server (-smtp_server). Also, ensure that the two-factor authentication is activated for the Gmail address (-smtp_user).
Perform the following steps in HVR GUI to create a maintenance task:
In the navigation tree pane, right-click Scheduler ▶ Maintenance Tasks.
In the Maintenance Tasks dialog, click Add in the left bottom. Type the name of the task and click OK.
Under the Email alerts section:
- Select -email_to and specify the email address(es) to which hvrmaint alerts will be sent.
- Select -smtp_server and specify the address of the Gmail SMTP server - smtp.gmail.com.
- Select -smtp_port and specify the Gmail SMTP server port for using TLS/STARTTLS - 587.
- Select -smtp_starttls to enable STARTTLS for secure connection.
- Select -smtp_user and specify the Gmail address to authenticate with the Gmail SMTP server. This is the Gmail account, from which the hvrmaint email alerts will be sent.
- Select -smtp_pass and specify the App Password you have generated.
Click Save and the task will be added to the list of tasks on the left pane.
To run the task manually, select the task in the list and click Run. This task will also run automatically if one of the conditions were defined: -email_only_when_errors or -email_only_when_errors_or_warnings.
Configuring HVR SNS Alerts
This section describes the steps to set up HVR to send hvrmaint SNS alerts.
Prerequisite: In AWS, you must Create an Amazon SNS topic and then Subscribe an endpoint to this newly created SNS topic.
Perform the following steps in HVR GUI to create a maintenance task:
- In the navigation tree pane, right-click Scheduler ▶ Maintenance Tasks.
- In the Maintenance Tasks dialog, click Add in the left bottom. Type the name of the task and click OK.
- Under the SNS alerts section:
- Select -sns_notify to enable SNS alerts.
- Select -sns_destination and specify ARN of the SNS topic created in AWS.
- Select either -sns_only_when_errors or -sns_only_when_errors_or_warnings. (optional)
- Select -sns_repeat_suppression and specify the duration. (optional)
- Select -sns_access_key and specify access key ID of the AWS IAM user.
- Select -sns_secret_key and specify secret access key of the AWS IAM user.
- Click SNS Test. (optional)
- Click Save and the task will be added to the list of tasks on the left pane.
- To run the task manually, select the task in the list and click Run. This task will also run automatically if one of the conditions were defined: -email_only_when_errors or -email_only_when_errors_or_warnings.
Configuring HVR SNMP Alerts
This section describes the steps to set up HVR to send hvrmaint SNMP alerts.
Prerequisite: You must have SNMP agent set up.
Perform the following steps in HVR GUI to create a maintenance task:
- In the navigation tree pane, right-click Scheduler ▶ Maintenance Tasks.
- In the Maintenance Tasks dialog, click Add in the left bottom. Type the name of the task and click OK.
- Under the SNMP alerts section:
- Select -snmp_notify to enable SNMP alerts.
- Select -snmp_version and specify your SNMP version.
- Select -snmp_heartbeat to send an hvrMaintNotifySummary notification.
- Select -snmp_hostname to specify the SNMP agent host
- Select -snmp_port and specify the agent trap port
- Select -snmp_community and specify the community string for SNMPv1/v2c transactions.
- Click SNMP Test. (optional)
Click Save and the task will be added to the list of tasks on the left pane.
To run the task manually, select the task in the list and click Run. This task will also run automatically if one of the conditions were defined: -email_only_when_errors or -email_only_when_errors_or_warnings.
Creating Maintenance Task
The behavior of hvrmaint is controlled by an option file, which can be generated using the HVR Maintenance Tasks dialog available from the context menu of the Scheduler.
Right-click the Scheduler node and select Maintenance Tasks from the context menu. The Maintenance Tasks dialog will open containing the list of tasks on the left pane (if they were previously created) and configuration options on the right pane.
Click the Add button at the bottom of the left pane to create a new maintenance task (option file). Type the name of the task and click OK.
Select the required options, specify parameters for them, where needed and click Save.
Click Run to run the hvrmaint script you created against the hub. You can click View Log to watch the output of the script.
The time options on the bottom pane allow you to schedule the task to run at a specific time, namely at regular intervals, daily or weekly.
Select Highest Privileges option to run the task with administrative permissions.
Steps to Schedule hvrmaint
This section describes the 'option files' and the steps to schedule hvrmaint in Unix/Linux and Windows.
Unix and Linux
hvrmaint can be scheduled to monitor the status of HVR every hour/minute/day and also to restart the HVR Scheduler and rotate log files. The environment for such batch programs is very limited, so many -env options are needed to pass it sufficient environment variables.
Create and save 'option files' in $HVR_CONFIG\files directory. Following are two sample option files.
The first option file (hvrmaint_hourly.opt) for hourly monitoring will just check for errors.
-hub=hvr/!\{s8Dhx./gsuWHUt\}! # Encrypted Oracle password -sched_option=-h oracle -env=HVR_HOME=/usr/hvr/hvr_home -env=HVR_CONFIG=/usr/hvr/hvr_config -env=HVR_TMP=/tmp -env=ORACLE_HOME=/distr/oracle/OraHome817 -env=ORACLE_SID=ORA817 -email_from=hvr@prod.mycorp.com -email_to=bob@mycorp.com;jim@mycorp.com -email_only_when_errors -snmp_server=snmp.mycorp.com -output=/usr/hvr/hvr_config/files/hvrmaint.log -scan_hvr_outThe second option file (hvrmaint_hourly.opt) for weekly monitoring will restart the HVR Scheduler and rotate the log files each week.
-hub=hvr/!\{s8Dhx./gsuWHUt\}! # Encrypted Oracle password -sched_option=-h oracle -env=HVR_HOME=/usr/hvr/hvr_home -env=HVR_CONFIG=/usr/hvr/hvr_config -env=HVR_TMP=/tmp -env=ORACLE_HOME=/distr/oracle/OraHome817 -env=ORACLE_SID=ORA817 -email_from=hvr@prod.mycorp.com -email_to=bob@mycorp.com;jim@mycorp.com -email_only_when_errors -snmp_server=snmp.mycorp.com -output=/usr/hvr/hvr_config/files/hvrmaint.log -scan_hvr_out -stop -archive_files=hvr.out # Only archive log file hvr.out -archive_compress -archive_keep_days=14 # Delete files after 2 weeks -journal_keep_days=4 -start
The following lines should be added to crontab for user hvr (these should be single lines without wrapping):
0 * * * * /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_hourly.opt 0 21 * * * /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_weekly.optAlternatively the following line could be added to crontab for root:
0 21 * * 6 su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_weekly.opt
Instead of scheduling hvrmaint on its own, it could also be used as part of a larger nightly batch script run by root which halts the HVR Scheduler and DBMS before doing a system backup. This batch script would roughly look like this:
su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /opt/hvrmaint.opt -stop -scan_hvr_out -archive_files=hvr.out su ingres -c /opt/ingres/utility/ingstop # Stop DBMS backup -f/dev/rmt/0m # Perform system backup su ingres -c /opt/ingres/utility/ingstart # Restart DBMS su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /opt/hvrmaint.opt -start
Windows
hvrmaint can be scheduled to monitor the status of HVR by adding it as a Windows Scheduled Task. Following are the steps to create Windows Schedule Task:
Create and save 'option files' in %HVR_CONFIG%\files directory. Following are two sample option files.
The first option file (hvrmaint_hourly.opt) for hourly monitoring will just check for errors.
-hub=hvr/!{s8Dhx./gsuWHUt}! # Encrypted Oracle password -sched_option=-h oracle -env=HVR_HOME=c:\\opt\\hvr_home -env=HVR_CONFIG=c:\\opt\\hvr_config -env=HVR_TMP=c:\\temp -env=ORACLE_HOME=c:\\distr\\oracle\\OraHome817 -env=ORACLE_SID=ORA817 -email_from=hvr@prod.mycorp.com -email=bob@mycorp.com;jim@mycorp.com -email_only_when_errors -snmp_server=snmp.mycorp.com -output=c:\\opt\\hvr_config\\files\\hvrmaint.log -scan_hvr_out<br /><br />The second option file (hvrmaint_hourly.opt) for weekly monitoring will restart the HVR Scheduler and rotate the log files each week.
-hub=hvr/!{s8Dhx./gsuWHUt}! # Encrypted Oracle password -sched_option=-h oracle -env=HVR_HOME=c:\\opt\\hvr_home -env=HVR_CONFIG=c:\\opt\\hvr_config -env=HVR_TMP=c:\\temp -env=ORACLE_HOME=c:\\distr\\oracle\\OraHome817 -env=ORACLE_SID=ORA817 -email_from=hvr@prod.mycorp.com -email=bob@mycorp.com;jim@mycorp.com -email_only_when_errors -snmp_server=snmp.mycorp.com -output=c:\\opt\\hvr_config\\files\\hvrmaint.log -scan_hvr_out -stop -archive_files=hvr.out # Only archive log file hvr.out -archive_keep_days=14 # Delete files after 2 weeks -journal_keep_days=4 -start
Create Windows Scheduled Tasks :
Open Task Scheduler. This can be accessed from Control Panel ▶ Administrative Tools
Alternatively, Task Scheduler can be accessed by running (Winkey+r) taskschd.msc.
Click Action ▶ Create Task
In General tab, enter task Name, and optionally a task Description.
In Actions tab, click New... to open New Action dialog.
- Browse and select the directory path for hvrmaint.exe (available in %HVR_HOME%\hvr_home\bin\) and click OK.
- Similarly, create separate actions to specify the directory path for the option file(s).
- Browse and select the directory path for hvrmaint.exe (available in %HVR_HOME%\hvr_home\bin\) and click OK.
In Triggers tab, click New... to open New Trigger dialog.
- Select the required frequency at which this task should be executed. For example, to execute this task every 10 minutes :
- Select the required frequency at which this task should be executed. For example, to execute this task every 10 minutes :
Click OK in Create Task dialog.
Alert Message Example
From: root@bambi.mycorp.com To: bob@mycorp.com; jim@mycorp.com Subject: hvrmaint detected 7 errors (323 rows in fail tables) for hub hvr/ on bambi 2017-11-01T21:00:01-06:30 hvrmaint: Starting hvrmaint c:\tools\hvrmaint.opt -hub=hvr/ -stop -start 2017-11-01T21:10:21-06:30 hvrmaint: Stopping HVR Scheduler 4.4.4/5 (windows-x64-64bit). 2017-11-01T21:10:33-06:30 hvrmaint: Scanning d:\hvr_config\log\hvr\hvr.out (2017-11-01T21:00:03-06:30). 2017-11-01T21:11:13-06:30 hvrmaint: 7 errors (323 rows in fail tables) were detected during scan. 2017-11-01T21:12:33-06:30 hvrmaint: 3 capture jobs for 1 location did 606 cycles. 2017-11-01T21:12:59-06:30 hvrmaint: 6 integrate jobs for 2 locations did 400 cycles and integrated 50 changes for 3 tables. 2017-11-01T21:13:53-06:30 hvrmaint: Archiving 9 log files to d:\hvr\archivelog\hvr_20050209. 2017-11-01T21:16:23-06:30 hvrmaint: Purging 0 archive directories older than 14 days. 2017-11-01T21:18:29-06:30 hvrmaint: Starting HVR Scheduler 4.4.4/5 (windows-x64-64bit). -------------------- Summary of errors detected during scan----------------- F_JD1034_RAISE_ERROR_P3 occurred 6 times between 2017-11-01T19:43:52-06:30 and 2017-11-01T20:14:24-06:30 F_JJ106E_TIMEO_DB occurred 1 time at 2017-11-01T21:10:03-06:30 -------------------------- Errors detected during scan--------------------------- 2017-11-01T19:43:52-06:30: channel-cap-d01: F_JD1034_RAISE_ERROR_P3: Error as raised by user during pl/sql procedure statement on Oracle SID. ----------------------- End of errors detected during scan ----------------------- 2017-11-01T21:19:01 hvrmaint: Sending e-mail to bob@mycorp.com; jim@mycorp.com
Files
