Troubleshoot the deployment of SiteScope templates

If the assignment of management templates/aspects containing SiteScope policy templates to the CI that you want to monitor fails, and you receive an error message about a missing connected server, perform the following troubleshooting steps:

Check that the connected server (the SiteScope server that executes the SiteScope monitors defined in the policy template) is configured. To do this, list connected servers by running ConnectedServer[.bat|.sh] -username <log-in name> -password <password> -list on the OMi system.

The output should be similar to:

Label               Server Type          ID
example.example.com site_scope           <id>

If the SiteScope server is not in the RTSM (not listed in the command output as a connected server), add the SiteScope server using the Monitoring Automation Node Editor.

The assignment can also fail because the selection script did not select a SiteScope server. Check the selection script log output to troubleshoot the behavior.

You can see the list of assignments which are linked to a specific SiteScope server, go to Administration > Setup and Maintenance > Connected Servers. You can then expand the list tab for your SiteScope connected server and click Launch report.

For troubleshooting, set cslevel=DEBUG in <OMi_HOME>/conf/core/Tools/log4j/jboss/opr-webapp.properties. Log output is in <OMi_HOME>/log/jboss/opr-configserver.log.

If you have imported a new policy version from SiteScope, but it is not yet assigned to all impacted CIs that are monitored by one SiteScope server, then the policy with the highest version is used and updates the template in the SiS runtime. However, parameters are also taken from the old assignments, which can lead to a parameter mix with sisconfig monitor update errors, for example when the new version has more parameters.

To avoid this issue, make sure that all assignments have the same policy version.

An assignment automatically triggers deployment jobs to the agent running on the SiteScope server. If the deployment succeeds, nothing is reported in the Deployment Jobs screen. If the deployment fails, check the deployment jobs screen, or use the command <OMi_HOME>/opr/bin/opr-jobs.

Deployment jobs are executed on OMi gateway systems. By default, a special job runs every 12 hours in the data processing server to re-calculate all assignments and, if needed, re-trigger deployment jobs.

Check the gateway logfile for troubleshooting: <OMi_HOME>/log/jboss/opr-configserver.log

If auto-assignment rules are involved, also check the data processing log file: <OMi_HOME>/log/jboss/opr-configserver.log

Second, on the SiteScope server, check that the HPE Operations Agent agent and the sisconfig configuration component are running and the policies are deployed correctly.

Execute the following commands:

<OvBinDir>/ovc -status

The output must contain the following line:

sisconfig     SiS Configuration         AGENT   (19106)  Running

<OvBinDir>/ovpolicy -list

Also, the output should contain lines similar to:

sitescope     "Oracle Database Availability (:Oracle Database Availability)"  enabled    0002.0000   

The SiteScope policies deployed from OMi are located in the following directories:

• UNIX or Linux: /var/opt/OV/datafiles/policies/sitescope

• Windows: C:\Program Files\HP\HP BTO Software\datafiles\policies\sitescope

The files ending with _params.xml contain the parameters to initiate SiteScope monitors and the files ending with _header.xml contain general information (such as a policy name, version, and so on). The files with the same UUID as the filename prefix are parts of the same policy. Each monitored instance (like a host or database instance) is reflected in a params.xml file by a section enclosed in <instance>...</instance>.To count the number of monitored instances, use a call like grep -c "<instance>" 0a65786b-6749-428e-896b-765970a03d5e_params.xml.

When enabling or disabling a SiteScope policy using <OVBinDir>/ovpolicy -enable -polname <polName> or the -disable option, a trigger is sent to SiteScope, which enables or disables the according monitors under the monitor group defined by <polName>.

With sisconfig component 1.00.080 and later, monitors are created enabled by default. sisconfig no longer performs an explicit API call to enable the monitor. This means, if you manually disable the monitor, it will not be automatically enabled at the next upload unless you enforce the old behavior by passing:

<OVBinDir>/ovconfchg -ns opr.sisconfig -set always_enable_group true

The sisconfig component is the adapter between the HPE Operations Agent and the SiteScope runtime. To check if sisconfig is running, execute the ovc -status sisconfig command on the SiteScope server. Typing ovconfget opr.sisconfig displays the following output:

log_enabled=true
sis_login=admin
sis_password=SZCDBfFiRR6UuEO/wZIqoQ==
sis_port=8080

The sis_login, sis_password and sis_port parameters are used to contact the local SiteScope runtime. They must be set in <OvLbinDir>\sisconfig\sisSetCredentials.bat on Windows systems and in <OvLbinDir>/sisconfig/sisSetCredentials.sh on UNIX systems.

More sisconfig-related utilities are located in the following directories:

• UNIX or Linux: /var/opt/OV/lbin/sisconfig

• Windows: C:\Program Files\HP\HP BTO Software\lbin\sisconfig

sisconfig logs to <OvDataDir>/log/system.<number>.<locale>, like:

Linux: /var/opt/OV/log/system.0.en_US

Windows: \ProgramData\HP\HP BTO Software\log\system.0.en_US

The configuration setting opr.sisconfig:log_enabled switches to the low-level logging that goes to the file <OvLogDir>/system.0.<locale>, for example: /var/opt/OV/log/system.0.en_US. This file provides various log statements, such as the information on how many monitors were newly deployed, updated or removed, for example:

Jul 4, 2014 4:42:26 AM;26;10;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.op\
r.sisconfig;INFO;Changed instances:  new: 1 changed: 0 deleted: 2

You can grep the sisconfig logfile for the string “SEVERE” to find the error entries.

Make sure that low level logging is enabled via:

<OvBinDir>/ovconfchg –ns opr.sisconfig –set log_enabled true

Sitescope web service calls are enclosed in log statements “BEGIN_API <APIname>” and “END_API <APIname> <result>”

Jun 28, 2016 3:26:50 PM;32;9;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;BEGIN_API deploySingleTemplateWithResult()

Jun 28, 2016 3:26:51 PM;33;9;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;END_API deploySingleTemplateWithResult() succeeded

The upload-, change-, and removal API call log entries are provided with a counter: “#<current>/<total>”. This enables you to estimate the remaining time.

The upload and change calls contain the parameter list passed to SiteScope.

Jun 28, 2016 3:26:50 PM;31;9;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;#3/3: calling deploySingleTemplateWithResult(); variables: {host=sis.example.net, cpu_queue_length=3, memory_virtual_swap_used=90,frequency=300, system_uptime=60 }, fullPathToTemplate: [Deployed from HP Monitoring Automation, example.example.net, Windows_Base_L1 (:POC_2016), Windows_Base_L1], fullPathToDestinationGroup: Deployed from HP Monitoring Automation/example.example.net/Windows_Base_L1 (:POC_2016)

When new data from the OMi server is coming, a statistic about new, changed, obsolete monitors is printed:

Jul 4, 2014 4:42:26 AM;26;10;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;Changed instances: new: 1 changed: 0 deleted: 2

For monitored instances, the number of instances and the specific instances that couldn’t be created are printed:

Jun 28, 2016 3:47:43 PM;77;9;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;instance creation/update failed for 3/3 items: [sis.example.net, example.example.net, ia2.example.net]

For each policy operation sent from ovconfd to sisconfig, an entry is logged mentioning the policyId:

Jun 29, 2016 1:18:34 PM;56;34;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;remotecheckpolicy (install or update): [0a65786b-6749-428e-896b-765970a03d5eM

For each policy operation, the execution time is measured:

Jun 28, 2016 3:27:36 PM;42;9;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;INFO;Policy operation done. Operation took: 82613 ms.

The purpose of the sisconfig file system cache is to find out when a monitor is to be removed in SiteScope, or when to update a monitor.

Cache file location: <OvDataDir>/datafiles/sisconfig/polcache.temp

The backup file in the same directory.

To list the monitors which were deployed from sisconfig to SiteScope use the following command:

<OvInstallDir>/lbin/sisconfig/sisconfigShowCacheFile.bat|sh

If a monitor is listed, then it was uploaded, independent of whether the monitor itself succeeds or fails. If it's not listed (but is mentioned in the policy's params.xml file), then there was a serious upload problem. In that case the upload can be retried using:

<OvBinDir>/ovc -restart sisconfig

If the cache file (+ backup file) is manually removed, all monitor instances mentioned in the policy's params.xml files are uploaded once again after sisconfig restart.

If a monitor is cleared manually in the SiteScope interface, the sisconfig cache is not informed about it.

To view details, like which parameter values were sent to SiteScope, use the following command:

<OvInstallDir>/lbin/sisconfig/sisconfigShowCacheFile.bat|sh -level 1

For sisconfig version 1.00.080 and later, monitors are created whenever possible, by default. This mean that the system does not check whether a remote server could be created or contacted by SiteScope, and it does not check if a monitor works. To change the default behavior, use the following command:

<OvBinDir>/ ovconfchg –ns opr.sisconfig –set connect_to_server true

This will check at sisconfig monitor creation time if a remote server is reachable.

To check at sisconfig monitor creation time if a monitor would fail, set the following:

<OvBinDir>/ovconfchg –ns opr.sisconfig –set test_remotes true

To re-send a monitor you can also remove the appropriate assignment in OMi and create it again.

Alternative ways to trigger the upload of failed monitors:

• For all policies without OMi server involvement:

  <OvBinDir>/ovc –restart sisconfig

• For a given policy (without OMi server involvement):

  <OvBinDir>/ovc/-notify CHECK_POLICY:sitescope –value <policyId>

The SiteScope runtime runs in its own Tomcat instance. Its default location is <host>:8080/sitescope.

The log files are located in <SiteScopeDir>/logs, for example: /opt/HP/SiteScope/logs. The following log files are a good starting point for troubleshooting: audit.log, error.log, SiteScope<CurrentDate>.log.

In the SiteScope UI, check that the monitors are initiated and are properly running (note that you can manually initiate them for test purposes). Refer to the following example:

The policy Oracle Database Availability (:Oracle Database Availability) was assigned to the CI for the database instance openview on <MyOmServer> and deployed to <MySiteScopeServer>. However, the SiteScope UI shows that an SQL SELECT statement failed and the monitor is not green. A possible cause of this problem may be that the standard user for the instance openview is used instead of the Oracle superuser, such as system and system password.

To solve this problem, in OMi, change the USER and PASSWORD parameters within the assignment of the policy Oracle Database Availability (:Oracle Database Availability) for the instance openview on <MyOmServer>. The following happens:

• An automatic deployment is triggered and the policy is updated on the SiteScope server’s HPE Operations Agent

• SiteScope is updated with the modified monitor

• The database availability monitor changes to green

Monitor creation typically fails due to the following issues:

• A problem with the template, for example, embedded paths, passwords, OS version, email recipients, and so on. Fix the template in SiteScope and import to OMi again.

• Preferences in SiteScope runtime don’t fit, for example, a host is not addable to SiteScope remote servers, a credential profile doesn’t exist or doesn’t contain required info, and so on. Fix in SiteScope, without modifying the template.

• Wrong template parameters. These can be overwritten in OMi by assignment (also possible as mass operation).

• Counter problems. Re-arrange the template and import to OMi again. See The counter problem: using patterns in monitors.

Some monitors use regular expressions to determine the counters (metrics), for example a list of directories or CPUs. To evaluate the expressions, a remote connection is needed when they are uploaded via sisconfig. These types of monitors will fail because sisconfig is set by default to use the flag connect_to_server=false. Such monitors will also fail when the flag was set to true using <OvBinDir>/ovconfget –ns opr.sisconfig –set connect_to_server true, but no connection to the remote host is possible.

The resulting sisconfig error looks like this:

MM;25;11;com.hp.ov.xpl.log.OvLogger;logMessage;com.hp.ov.opr.sisconfig;SEVERE;deploySingleTemplateWithResult() failed. SiteScope Error: <BR>Error: Parameter error occurred property

No counters selected, com.mercury.sitescope.api.configuration.exception.ExternalServiceAPIException

To fix the problem, the template must be modified so that it no longer contains patterns with wildcards.