Configure High Availability

This section describes the procedures for configuring a new High Availability (HA) configuration for NNMi. It contains the following topics:

When configuring HA, note the following general guidelines:

  • RHCS configuration requires a complete restart of the HA cluster daemons, including all applications, on each node in the HA cluster. Plan your configuration effort accordingly.
  • Do not use the RHCS luci Web interface to change the NNMi resource group. The luci Web interface removes the NNMi resource group global variables from /etc/cluster/cluster.conf if changes are made to the NNMi resource group. The NNMi resource group global variables are required for proper NNMi HA functionality.

  • By default, in an HA environment, the SNMP source address is set to a physical cluster node address. To set the SNMP source address to the NNM_INTERFACE (which is set to the virtual IP address), you must edit the ov.conf file and set the value for IGNORE_NNM_IF_FOR_SNMP to OFF. (By default, this setting is set to ON.)
  • When making file changes under High Availability (HA), you must make the changes on both nodes in the cluster. If the change requires you to stop and restart the NNMi management server, you must put the nodes in maintenance mode before running the ovstop and ovstart commands. See Maintenance Mode for more information.

Configure NNMi Certificates for High Availability

The NNMi installation process configures a self-signed certificate for secure communications between the NNMi console and the NNMi database. The process for configuring NNMi for High Availability (HA) correctly shares the self-signed certificate among the primary and secondary cluster nodes. You do not need to take any extra steps to use the default certificate with NNMi running under HA.

If you want to use a different self-signed certificate or a Certificate Authority (CA)-signed certificate for NNMi communications, you must do some additional work. After obtaining the new certificate, complete the steps shown in Working with Certificates in High-Availability Environments. You can complete this procedure before or after configuring NNMi for HA.

Configure NNMi for High Availability

The two distinct phases of configuring NNMi for High Availability (HA) are as follows:

  1. Copy the NNMi data files to the shared disk.

  2. Configure NNMi to run under HA.

Designate one HA cluster node as the primary NNMi management server. This is the node you expect to be active most of the time. Configure the primary node, and then configure all other nodes in the HA cluster as secondary nodes.

Caution You cannot configure NNMi for HA simultaneously on multiple cluster nodes. After the HA configuration process is completed on one cluster node, proceed with the HA configuration on the next node, and so forth until NNMi is configured for HA on all nodes in the cluster environment.

  During failover, the NNMi console is unresponsive. After failover completes, NNMi users must log on to continue their NNMi console sessions.

Note If you encounter errors during HA configuration, do the following:

  1. Unconfigure NNMi from the HA environment by running the nnmhaunconfigure.ovpl command.
  2. Correct the condition indicated by the error message(s).
  3. Reconfigure NNMi into the HA environment by running the nnmhaconfigure.ovpl command.

    (RHCS only) For the nnmhaconfigure.ovpl and nnmhaunconfigure.ovpl commands to work properly, the <failoverdomains/> tag must exist in the /etc/cluster/cluster.conf file.

    The <failoverdomains/> tag is embedded within the resource manager section, for example:

    ...
    ...
    <rm>
     <failoverdomains/>
    </rm>

    The nnmhaconfigure.ovpl command requires the<failoverdomains/> tag to create the NNMi resource group, using the following example structure:

    ...
    <rm>
      <failoverdomains>
        <failoverdomain name="<rg-name>-dom" nofailback="0"
    ordered="0" restricted="1">
    	<failoverdomainnode name="<node1>" priority="1"/>
    	<failoverdomainnode name="<node2>" priority="1"/>          </failoverdomain>
       </failoverdomains>
       <service autostart="1" domain="<rg-name>-dom" 
    exclusive="0" name="nnmha" recovery="relocate">
          <ip address="<addr>" monitor_link="1">
              <fs device="<nnmhalvol>" force_fsck="1" 
    force_unmount="1" fsid="" fstype="ext3" 
    mountpoint="<nnm-hamount>" name="nnmha-mount" 
    options="" self_fence="0">
              <NNMscript GLOBAL_VARIABLES="NNM_INTERFACE=
    <virtual hostname>;HA_LOCALE=en_US.UTF-8;
    HA_MOUNT_POINT=/<nnm-hamount>" 
    file="/var/opt/OV/hacluster/<rg-name>/nnmharhcs" 
    name="nnmha-APP"/>
              </fs>
         </ip>
      </service>
    </rm>

    The nnmhaunconfigure.ovpl command also requires the above structure to remove the node's failoverdomain entry.

    For more information, see the nnmhaunconfigure.ovpl and nnmhaconfigure.ovpl reference pages, or the Linux manpages.

NNMi High Availability Configuration Information

The High Availability (HA) configuration script collects information about the NNMi HA resource group. Please prepare the information listed in the following table before you configure NNMi HA. This information is needed to execute the HA script (nnmhaconfigure.ovpl) interactively, depending on your operating system or HA software.

NNMi HA Primary Node Configuration Information

HA Configuration Item

Description

HA resource group

The name of the resource group for the HA cluster that contains NNMi. This name must be unique, specific to NNMi, and not currently in use. See your HA system provider’s reference material for information on valid names.

Upon input of an HA resource group name, NNMi generates the following resources for Linux and Windows systems:

<resource group name>-IP

<resource group name>-Mount

<resource group name>-App

In addition, for Windows systems, the following resource is generated upon input of a vitual hostname:

<virtual hostname>

Virtual host short name

The short name for the virtual host. This hostname must map to the virtual IP address for the HA resource group. The nslookup command must be able to resolve the virtual host short name and the virtual IP address.

Note If NNMi is unable to resolve the virtual host short name or the virtual host IP address, the HA configuration script could leave the system in an unstable state. Therefore, recommends that you implement a secondary naming strategy (such as entering the information in the %SystemRoot%\system32\drivers\etc\hosts file on the Windows operating system or /etc/hosts file on UNIX operating systems) in case DNS is not available during NNMi HA configuration.

Virtual host netmask

The subnet mask that is used with the virtual host IP address, which must be an IPv4 address.

Virtual host network interface

The network interface on which the virtual host IP address is running. For example:

  • Windows: Local Area Connection
  • Linux: eth0

Shared file system type

The type of shared disk configuration being used for the HA resource group. Possible values are:

  • disk—The shared disk is a physically attached disk that uses a standard file system type. The HA configuration script can configure the shared disk. For more information, see the File system type entry in this table.
  • none—The shared disk uses a configuration other than that described for the disk option, such as NFS. After running the HA configuration script, configure the shared disk as described in Prepare the Shared Disk Manually in High Availability Environments.

File system type

(Linux only) The file system type of the shared disk (if the shared file system type is disk). The HA configuration scripts pass this value to the HA product so that it can determine how to validate the disk.

has tested the following shared disk formats:

Note HA products support other file system types. If you use a shared disk format that has not tested, prepare the disk before configuring NNMi to run under HA, and then specify none for the shared file system type while running the NNMi HA configuration script.

Disk information (disk group, volume group, and/or logical volume name, depending on the operating system used)

The name associated with the disk information for the NNMi shared file system.

Note When you create/attach a disk on UNIX platforms, for example, with vxfs or lvm, you create different items, such as: disk group, volume group, logical volume. The names for these items are assigned by the system administrator at the time of creation. NNMi does not enforce any naming conventions. Contact your system administrator for your company’s naming information.

Mount point

The directory location for mounting the NNMi shared disk. This mount point must be consistent between systems. (That is, each node must use the same name for the mount point.) For example:

  • Windows: S:\

    Note Specify the drive completely. S and S: are unacceptable formats and do not provide access to the shared disk.

  • Linux: /nnmmount

Configuring NNMi on the Primary Cluster Node

Complete the following procedure on the primary cluster node.

Note If you are using Oracle for the main NNMi database, see Configure NNMi for High Availability in an Oracle Environment first.

  1. If you have not already done so, complete the procedure for Verifying the Prerequisites to Configuring NNMi for High Availability.
  2. If you have not already done so, install NNMi (including the latest consolidated patch, if any), and then verify that NNMi is working correctly.
  3. If you expect to run any NNM iSPIs on this NNMi management server, see Configure NNM iSPIs for High Availability before continuing with this procedure.
  4. Use the nnmbackup.ovpl command, or another database command, to back up all NNMi data. For example:

    nnmbackup.ovpl -type offline -scope all -target nnmi_backups
  5. Define the disk device group (and logical volume), consisting of at least one shared disk for the NNMi HA resource group. For example:

    • WSFC: Use Disk Management to configure the disk mount point and format the disk.

    • VCS:

      Use VSF commands such as vxdiskadm, vxassist, and mkfs to add and initialize the disk, allocate disks by space, and create the logical volume.

    • RHCS:

      Use LVM commands such as pvcreate, vgcreate, and lvcreate to initialize the disk, create the volume group, and create the logical volume.

    Note NNMi requires RHCS clusters be configured such that the cluster node names specified in the /etc/cluster/cluster.conf file must be fully qualified for NNMi to correctly start and stop.

    For Linux operating systems, a reference web site is:
    http://www.unixguide.net/unixguide.shtml

  6. Create the directory mount point (for example, S:\ or /nnmmount), and then mount the shared disk:

    • Windows: Use the Windows Explorer and Disk Management tool to assign a drive letter.

      Caution Use the Disk Management tool make sure that the shared disk displays online. If it displays reserved, this indicates WSFC has control of the shared disk. Use the Delete action from the WSFC user interface to remove the shared disk from WSFC control. Also use the Disk Management tool to confirm that the reserve flag is changed to online.

    • Linux:

      • Use the mkdir and mount commands.
      • Verify that the shared disk directory mount point has been created with root as the user, sys as the group, and the permissions set to 755. For example:

        ls -l /nnmmount

        Caution After configuration, the HA product manages disk mounting. Do not update the files system table with this mount point.

  7. Stop NNMi:

    ovstop -c

    Note If NNMi is already installed on a node that you will include in this HA resource group, also run ovstop -c on that node at this time.

  8. Copy the NNMi database to the shared disk:

    • Windows:

      %NnmInstallDir%\misc\nnm\ha\nnmhadisk.ovpl NNM -to <HA_mount_point>
    • Linux:

      $NnmInstallDir/misc/nnm/ha/nnmhadisk.ovpl NNM -to <HA_mount_point>

    Note To prevent database corruption, run this command (with the -to option) only one time. For information about alternatives, see Re-Enable NNMi for High Availability after All Cluster Nodes are Unconfigured.

  9. (Linux only) Unmount the shared disk and deactivate the disk group:

    umount <HA_mount_point>
    vgchange -a n <disk_group>
  10. Verify that NNMi is not running:

    ovstop -c

  11. (RHCS only) Perform the following to add the necessary NNMscript resource to the /usr/share/cluster/cluster.rng file:

    1. Save a copy of the cluster.rng file.
    2. Edit the /usr/share/cluster/cluster.rng file as follows:
      1. Find <define name=”CHILDREN”>.
      2. Embed the contents of the file /opt/OV/misc/nnm/ha/NNMscript.rng ahead of the statement found in the previous step.

        For example go one line above <define name=”CHILDREN”>, and type:

        :r /opt/OV/misc/nnm/ha/NNMscript.rng
      3. In the CHILDREN XML block, add the text that is bold in the following:

        <define name=”CHILDREN”>
         <zeroOrMore>
          <choice>
               <ref name=”SCRIPT”/>                
               <ref name=”NNMSCRIPT”/>
               <ref name=”NETFS”/>
      4. Save the cluster.rng file.
    3. Copy the /opt/OV/misc/nnm/ha/NNMscript.sh file to /usr/share/cluster and ensure that it has 755 permissions with root:root ownership.
    4. Restart the ccsd service or reboot.
    5. If you rebooted the system in the previous step, before continuing with the cluster configuration, stopNNMi:

      ovstop -c
    6. Verify that NNMi is not running:

      ovstatus -c
  12. Configure the NNMi HA resource group:

    • Windows:

      %NnmInstallDir%\misc\nnm\ha\nnmhaconfigure.ovpl NNM

    • Linux:

      $NnmInstallDir/misc/nnm/ha/nnmhaconfigure.ovpl NNM

  13. (Linux only) By default, NNMi starts in the locale of the user who ran the nnmhaconfigure.ovpl command. To change the NNMi locale, run the following command:

    $NnmInstallDir/misc/nnm/ha/nnmhaclusterinfo.ovpl –config NNM –set HA_LOCALE <locale>
  14. In step 12, determine the value you specified for the shared file system type:

  15. Start the NNMi HA resource group:

    • Windows:

      %NnmInstallDir%\misc\nnm\ha\nnmhastartrg.ovpl NNM <resource_group>
    • Linux:

      $NnmInstallDir/misc/nnm/ha/nnmhastartrg.ovpl NNM <resource_group>

    If NNMi does not start correctly, see Troubleshooting the HA Configuration.

Caution Now that NNMi is running under HA, do not use the ovstart and ovstop commands for normal operation. Use these commands only when instructed to do so for HA maintenance purposes.

Configuring NNMi on the Secondary Cluster Nodes

Complete the following procedure on one secondary cluster node at a time.

  1. If you have not already done so, complete the procedure for Configuring NNMi on the Primary Cluster Node.
  2. If you have not already done so, complete the procedure for Verifying the Prerequisites to Configuring NNMi for High Availability.
  3. If you have not already done so, install NNMi (including the latest consolidated patch, if any), and then verify that NNMi is working correctly.
  4. Install the NNM iSPIs that you installed in step 3 of Configuring NNMi on the Primary Cluster Node.
  5. Stop NNMi:

    ovstop -c

  6. Create a mount point for the shared disk (for example, S:\ or /nnmmount).

    Note This mount point must use the same name as the mount point you created in step 6 of the procedure Configuring NNMi on the Primary Cluster Node.

  7. (RHCS only) Perform the following to add the necessary NNMscript resource to the /usr/share/cluster/cluster.rng file:

    1. Save a copy of the cluster.rng file.
    2. Edit the /usr/share/cluster/cluster.rng file as follows:

      1. Find <define name=”CHILDREN”>
      2. Embed the contents of the file /opt/OV/misc/nnm/ha/NNMscript.rng ahead of the statement found in the previous step.

        For example go one line above <define name=”CHILDREN”>, and type:

        :r /opt/OV/misc/nnm/ha/NNMscript.rng
      3. In the CHILDREN XML block, add the text that is bold in the following:

        <define name=”CHILDREN”>
         <zeroOrMore>
          <choice>
               <ref name=”SCRIPT”/>                
               <ref name=”NNMSCRIPT”/>
               <ref name=”NETFS”/>
      4. Save the cluster.rng file.
  8. (RHCS only) Copy the NNMi custom script into place, and then restart the HA cluster daemons.

    1. Copy the /opt/OV/misc/nnm/ha/NNMscript.sh file to the following location:

      /usr/share/cluster/NNMscript.sh
    2. Stop and then restart the /sbin/ccsd process.
  9. Configure the NNMi HA resource group:

    • Windows: %NnmInstallDir%\misc\nnm\ha\nnmhaconfigure.ovpl NNM
    • Linux: $NnmInstallDir/misc/nnm/ha/nnmhaconfigure.ovpl NNM

    Supply the HA resource group name when the command requests this information.

  10. Verify that the configuration was successful:

    • Windows:

      %NnmInstallDir%\misc\nnm\ha\nnmhaclusterinfo.ovpl 
      -group
      <resource_group> -nodes
    • Linux:

      $NnmInstallDir/misc/nnm/ha/nnmhaclusterinfo.ovpl 
      -group
      <resource_group> -nodes

    The command output lists all configured nodes for the specified HA resource group.

  11. Optionally, test the configuration by taking the NNMi HA resource group on the primary node offline and then bringing the NNMi HA resource group on the secondary node online.

Configure NNM iSPIs for High Availability

If you expect to run any NNM iSPIs on the NNMi management server, read this section before configuring NNMi to run under HA.

NNM iSPI Performance for Metrics, NNM iSPI Performance for QA, and NNM iSPI Performance for Traffic

The NNM iSPI Performance for Metrics can be installed on the NNMi management server or on a standalone server.

The NNM iSPI Performance for Traffic) has two different components (Traffic Master and Traffic Leaf), which can be installed on the NNMi management server or standalone servers, or a combination of both (one component on the NNMi management server and the other on a remote server).

Note  

  • If the NNM iSPI (or component) will be located on the NNMi management server, install the product before configuring NNMi to run under HA.
  • If the NNM iSPI (or component) will be located on a standalone server, configure NNMi to run under HA before installing the product. During the NNM iSPI installation process, supply the NNMi HA resource group virtual hostname as the NNMi management server name.

For more information on installing an NNM iSPI, see the appropriate NNM iSPI installation guide.

NNM iSPI Performance for QA, NNM iSPI for MPLS, NNM iSPI for IP Multicast, and NNM iSPI for IP Telephony

The NNM iSPI Performance for QA, NNM iSPI for MPLS, NNM iSPI for IP Multicast, and NNM iSPI for IP Telephony can be installed on the NNMi management server only.

For information about configuring the NNM iSPIs to run under HA, see the documentation for the appropriate NNM iSPI.

NNM iSPI Network Engineering Toolset Software and NNMi Running under HA

The NNM iSPI Network Engineering Toolset Software SNMP trap analytics and Microsoft Visio export functionality are automatically installed with the NNMi Premium or NNMi Ultimate products. No extra work is needed to run these tools under HA.

The NNM iSPI NET Diagnostics Server cannot be included in the NNMi HA resource group. Do not install this component on the NNMi management server. To run the NNM iSPI NET Diagnostics Server on a system that is outside the NNMi HA resource group, follow these steps:

The NNM iSPI NET Diagnostics Server requires an NNM iSPI NET or NNMi Ultimate license. See the NNM iSPI Network Engineering Toolset Software Interactive Installation and Upgrade Guide for information about how to install and configure this server.

  1. Completely configure the NNMi HA resource group.
  2. Install the NNM iSPI NET Diagnostics Server on a system that is outside the NNMi HA resource group. During the NNM iSPI NET Diagnostics Server installation process, supply the NNMi HA resource group virtual hostname as the NNM Server Hostname.

    For more information, see the NNM iSPI Network Engineering Toolset Software Planning and Installation Guide.

If the NNM iSPI NET Diagnostics Server is already installed on an NNMi management server that will run under HA, uninstall the NNM iSPI NET Diagnostics Server before configuring NNMi to run under HA.

Caution Uninstalling the NNM iSPI NET Diagnostics Server removes all existing reports.

Note   It might be possible to save existing reports, as described here, but the following procedure is untested:

  1. Use MySQL Workbench to perform a backup of the existing nnminet database.

    MySQL Workbench is available in the downloads area at dev.mysql.com.

  2. Uninstall the NNM iSPI NET Diagnostics Server.
  3. Configure NNMi to run under HA.
  4. Install the NNM iSPI NET Diagnostics Server on a separate system.
  5. Before running any flows, use MySQL Workbench to recover the nnminet database onto the new installation.

Configure NNMi for High Availability in an Oracle Environment

This sections presents a high-level overview of the process for configuring NNMi with an Oracle database to run under High Availability (HA).

Note The number of possible Oracle configurations is large, and the configuration process can vary according to the Oracle release. For the most accurate information about configuring Oracle to run under HA and creating an NNMi dependency on the Oracle HA resource group, see the HA product documentation. You can also go to the Oracle web site (www.oracle.com) for information about the appropriate Oracle configuration for your HA product.

NNMi Dependency on Oracle in High Availability Environments

When Oracle and NNMi both run under High Availability (HA), the NNMi HA resource group must include a shared disk for the NNMi data that is not stored in the Oracle database.

Additionally, consider the following information:

  • If the HA product supports dependencies, the recommended approach is to configure each product to run in a separate HA resource group. The Oracle HA resource group must be fully started before the NNMi HA resource group starts. If both HA resource groups are in the same HA cluster, you can modify the cluster configuration to set resource group ordering. If the HA resource groups are in different HA clusters, make sure that the NNMi HA resource group dependency on the Oracle HA resource group is met.
  • If the HA product does not support dependencies, include the Oracle systems and the NNMi systems in the NNMi HA resource group.

Configuring NNMi for High Availability in an Oracle Environment

  1. If you plan to run Oracle under High Availability (HA), complete that configuration first.
  2. Create an empty Oracle database instance for NNMi.
  3. On the primary NNMi node, install NNMi (including the latest consolidated patch, if any). During installation, do the following:

    1. Select the Oracle database type, and then select Primary Server Installation.
    2. Specify the virtual IP address or hostname for the Oracle HA resource group (if applicable).
  4. On the primary NNMi node, configure NNMi to run under HA as described in Configuring NNMi on the Primary Cluster Node.
  5. Set up the NNMi dependency on the Oracle HA resource group.

    For specific instructions, see the HA product documentation.

  6. On the secondary NNMi node, install NNMi (including the latest consolidated patch, if any). During installation, do the following:

    • Select the Oracle database type, and then select Secondary Server Installation.
    • Specify the virtual IP address or hostname for the Oracle HA resource group (if applicable).
  7. On the secondary NNMi node, configure NNMi to run under HA described in Configuring NNMi on the Secondary Cluster Nodes.
  8. For each additional secondary NNMi node, repeat step 6 and step 7.