Administer > Multimaster Mesh administration > MM Central Relocation

MM Central relocation

This section describes the process of relocating the MultiMaster (MM) Central, for an HPE SA mesh, from one core to another. Any HPE SA mesh is comprised of one Primary core (the core running the MM Central) and one or more Secondary cores. The procedures described here will allow SA administrators to:

  • Demote the Primary core to Secondary core role (in an SA mesh, mark the Primary core as Secondary core)
  • Promote a Secondary core to Primary core role (in an SA mesh, designate a Secondary core as Primary core)

Caution The steps should only be performed with assistance from HPE Support or HPE Professional Services.

For illustration, we are using an SA mesh with three cores: Core1, Core2, and Core3.

  • Core1 is the Primary core that will be moved to a Secondary core role.
  • Core2 is a Secondary core.
  • Core3 is a Secondary that will be promoted as Primary core.

The base steps are as follows:

  1. Check SA Mesh status.
  2. Update files required during the SA upgrade process.
    Note: The SA upgrade process is not in the scope of this document.
  3. Update files related to MM Central relocation process.
  4. Demote Core1 from Primary to Secondary core role.
  5. Promote Core3 to primary core role.
  6. Restart Core3.

Requirements

  • SA installed at version 10.0 or later
  • SA 10.51 installation media must be available, due to some needed scripts which have limitations or are not available in the 10.0x distribution.

Known limitations

  1. No content import is allowed during MM Central relocation. This includes windows patch import, Red Hat patch import, script import, etc.
  2. No Administrative operations should be performed during MM Central relocation, i.e. create user and user group, etc.

3-core mesh sample environment

  • The sample mesh represents a 10.2x GA environment.
  • SA 10.51 media has been exported from 192.168.160.100:/root/IMR2/T8900-15064

Core facility in the mesh

Core Facility Names

Facility ID

Core1 - Initial Primary core, will be demoted to Secondary core.

1 (First core was installed)

Core2 - Initial Secondary core.

2 (Second core was installed, 1st Secondary)

Core3 - Initial Secondary core, will be promoted to Primary.

3 (Third core was installed, 2nd Secondary)

Mesh configuration details

Core1, related server names

IP

Purpose

C1-DB-rose11.qa.opsware.com

192.168.76.11

Oracle DB

C1-Inf-rose12.qa.opsware.com

192.168.76.12

Infrastructure, Model Repository, Word, OS Prov

C1-S0-rose13.qa.opsware.com

192.168.76.13

Slice0

C1-S0-rose14.qa.opsware.com

192.168.76.14

Slice1

C1-Sat-rose15.qa.opsware.com

192.168.76.15

Satellite (Facility: Core1Sat)

C1-Ssvr-rose31.qa.opsware.com

192.168.76.31

Managed server, attached to Satellite Core1Sat.

C1-Ssvr-rose32.qa.opsware.com

192.168.76.32

Managed server, attached to Satellite Core1Sat.

C1-svr-rose33.qa.opsware.com

192.168.76.33

Managed server, attached to Core1.

Core2, related server names

IP

Purpose

C2-DB-rose16.qa.opsware.com

192.168.76.16

Oracle DB

C2-Inf-rose17.qa.opsware.com

192.168.76.17

Infrastructure, Model Repository, Word, OS Prov

C2-S0-rose18.qa.opsware.com

192.168.76.18

Slice0

C2-S0-rose19.qa.opsware.com

192.168.76.19

Slice1

C2-Sat-rose20.qa.opsware.com

192.168.76.20

Satellite (Facility: Core2Sat)

C2-Ssvr-rose34.qa.opsware.com

192.168.76.34

Managed server, attached to Satellite Core2Sat.

C2-Ssvr-rose35.qa.opsware.com

192.168.76.35

Managed server, attached to Satellite Core2Sat.

C2-svr-rose36.qa.opsware.com

192.168.76.36

Managed server, attached to Core2.

Core3, related server names

IP

Purpose

C3-DB-rose21.qa.opsware.com

192.168.76.21

Oracle DB

C3-Inf-rose22.qa.opsware.com

192.168.76.22

Infrastructure, Model Repository, Word, OS Prov

C3-S0-rose23.qa.opsware.com

192.168.76.23

Slice0

C3-S0-rose24.qa.opsware.com

192.168.76.24

Slice1

C3-Sat-rose25.qa.opsware.com

192.168.76.25

Satellite (Facility: Core3Sat)

C3-Ssvr-rose37.qa.opsware.com

192.168.76.37

Managed server, attached to Satellite Core3Sat.

C3-Ssvr-rose38.qa.opsware.com

192.168.76.38

Managed server, attached to Satellite Core3Sat.

C3-svr-rose39.qa.opsware.com

192.168.76.39

Managed server, attached to Core3.

Moving MM Central

Mesh status check

Note

Please resolve all failures encountered in the current section before proceeding to the next.

No.

Purpose

Target

Servers/Command/Output

1

Ensure SA processes are running

All SA core servers, except the DB server.

Log in to C1-Inf-rose12.qa.opsware.com as root.

$ /etc/init.d/opsware-sas health

Successfully performed "health" operation on Opsware SAS components.

Expected, Successfully performed "health"…

NOTE:  Any failures need to be resolved before continuing to the next step.

Repeat the same steps on:

C1-S0-rose13.qa.opsware.com

C1-S1-rose14.qa.opsware.com

C1-Sat-rose15.qa.opsware.com

C2-Inf-rose17.qa.opsware.com

C2-S0-rose18.qa.opsware.com

C2-S1-rose19.qa.opsware.com

C2-Sat-rose20.qa.opsware.com

C3-Inf-rose22.qa.opsware.com

C3-S0-rose23.qa.opsware.com

C3-S1-rose24.qa.opsware.com

C3-Sat-rose25.qa.opsware.com

2

Communication test passed

All SA Core servers and Managed servers, from SA Client on Primary slice server

Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has enough permissions.

SA Client -> Devices -> All managed Servers

Select all listed servers, right click -> Run -> Communication Test

Expected, Status: GREEN status are displayed on all checkup point for all servers.

NOTE:  Any failures need to be resolved before continuing to the next step.

3

System Diagnostics test passed

All Facilities, from SA Client on Primary slice server

Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential.

SA Client -> Administration -> Facilities

Right click on Core1 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core2 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core3 -> Run System Diagnosis -> Next -> Select All -> Start Job

Expected, Status: Succeeded on all items.

NOTE:  Any failures need to be resolved before continuing to the next step.

4

Mesh has No Conflict

The Mesh, from SA Clients on Primary slice server

Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential.

SA Client -> Administration -> MultiMaster Tools -> State View

Expected: No conflicts.

NOTE:  Any conflicts need to be resolved before continuing to the next step.

4a

Resolve Conflict

 

Note:  Please see section 8.1 for conflict analysis and resolution details.

The Mesh, from Primary Infrastructure server

  • From Step 4, find Source Facility of the conflict. 
  • Obtain the Source Facility’s Facility ID, i.e Facility_ID.

Log in to C1-Inf-rose12.qa.opsware.com as root

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib

$ /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc  --refresh --from $Facility_ID

Repeat the same process on all facilities which that have conflicts.

Updating files required during SA upgrade

  • After promoting a Secondary core to Primary core, certain files on the new Primary core have to match those on the original Primary to ensure a successful upgrade to a newer release of SA is possible. 
  • In our setup, ‘Core3’ is a Secondary core that will be promoted as new Primary core and ‘Core1’ will be demoted from Primary to Secondary core role.

No.

Purpose

Target

Servers/Command/Output

1

On the new Primary core, update the SA inventory file: /var/opt/opsware/install_opsware/inv/install.inv

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/inv

$ cp install.inv  install.inv _orig

$ vi install.inv

   Change

          %truth_mm_slave

   To

          %truth_mm_overlay

 

   Add

%word_uploads

build_id: opsware_50.0.37477.0

state:  complete

script: pre

script: post

 

Note: For build_id info check the old Primary inventory file.

2

On the old Primary core, update the SA inventory file: /var/opt/opsware/install_opsware/inv/install.inv

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/inv

$ cp install.inv  install.inv _orig

$ vi install.inv

   Change

          % truth_mm_overlay

   To

          % truth_mm_slave

 

   Remove

%word_uploads

build_id: opsware_50.0.37477.0

state:  complete

script: pre

script: post

 

Note: build_id info might have a different build number depending on the installed SA version.

 

3

From the old Primary copy to the new Primary the following directory and its contents: /var/opt/opsware/install_opsware/conf

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware

$ mv conf  conf_orig

$ mkdir  conf

$ cd conf

$ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/conf/* .

$ ls –l | wc

    111     992    6634

 

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/conf

$ ls –l | wc

    111     992    6634

Both output are matched.

4

From the old Primary copy to the new Primary the following directory contents: /var/opt/opsware/crypto/word_uploads

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/opsware/crypto/

$ ls word_uploads

ls: cannot access word_uploads: No such file or directory

$ mkdir word_uploads

$ chmod 0500  word_uploads

$ ls -lad /var/opt/opsware/crypto/word_uploads

dr-x------ 2 root root 4096 Sep 21 18:54 /var/opt/opsware/crypto/word_uploads

$ cd word_uploads

$ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/crypto/word_uploads/* .

$ ls –l

total 24

-r-------- 1 root root 1858 Sep 21 18:54 admin-ca.crt

-r-------- 1 root root 1862 Sep 21 18:54 agent-ca.crt

-r-------- 1 root root 1866 Sep 21 18:54 opsware-ca.crt

-r-------- 1 root root 3062 Sep 21 18:54 spin.srv

-r-------- 1 root root 3087 Sep 21 18:54 spog.pkcs8

-r-------- 1 root root 3062 Sep 21 18:54 wordbot.srv

 

 

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/crypto/word_uploads

$ ls -l

total 24

-r-------- 1 root root 1858 Sep 15 01:52 admin-ca.crt

-r-------- 1 root root 1862 Sep 15 01:52 agent-ca.crt

-r-------- 1 root root 1866 Sep 15 01:52 opsware-ca.crt

-r-------- 1 root root 3062 Sep 15 01:52 spin.srv

-r-------- 1 root root 3087 Sep 15 01:52 spog.pkcs8

-r-------- 1 root root 3062 Sep 15 01:52 wordbot.srv

 

word_uploads contents should be the same on the old and the new Primary.

5

On the old Primary remove: /var/opt/opsware/crypto/word_uploads

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/crypto/

$ mkdir backup

$ mv word_uploads ./backup/

6

On the new Primary, update the core definition file (cdf.xml)

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/cdf

$ cp cdf.xml cdf.xml_orig

$ vi cdf.xml

   Change

<PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_slave_typical</PublicProperty>

   To

<PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_master_typical</PublicProperty>

 

   Change

<OpswareComponent name="truth_mm_slave" ttl="3600" dc="2015-09-15T19:14:02Z">

   To

<OpswareComponent name="truth_mm_overlay" ttl="3600" dc="2015-09-15T19:14:02Z">

7

On the old Primary, update the core definition file (cdf.xml)

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/cdf

$ cp cdf.xml cdf.xml_orig

$ vi cdf.xml

   Change

<PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_master_typical </PublicProperty>

   To

<PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_slave_typical</PublicProperty>

 

   Change

<OpswareComponent name="truth_mm_overlay" ttl="3600" dc="2015-09-15T19:14:02Z">

   To

<OpswareComponent name="truth_mm_slave " ttl="3600" dc="2015-09-15T19:14:02Z">

8

On the new Primary, update the installation response file

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ ls –ltr

total 28

-rw------- 1 root root 2148 Sep 15 19:04 resp.2015-09-15.19:04:49

-rw------- 1 root root 2148 Sep 15 19:14 resp.2015-09-15.19:14:02

-rw------- 1 root root 2148 Sep 15 19:14 resp.2015-09-15.19:14:06

-rw------- 1 root root 2148 Sep 15 19:20 resp.2015-09-15.19:20:03

-rw------- 1 root root 2148 Sep 15 20:25 resp.2015-09-15.20:25:00

-rw------- 1 root root  828 Sep 15 20:25 resp_globals

-rw------- 1 root root 2148 Sep 15 20:30 resp.2015-09-15.20:30:17

 

Identify the latest good response (in the current setup: resp.2015-09-15.20:30:17).

$ cp "resp.2015-09-15.20:30:17" “resp.2015-09-15.20:30:17_orig”

$ vi "resp.2015-09-15.20:30:17"

   Change

%oi.components

osprov

infrastructure

slice

truth_mm_slave

   To

%oi.components

osprov

infrastructure

slice

truth_mm_overlay

word_uploads

   Delete

%masterCore.mgw_tunnel_listener_port

2001

 

$ ls -l resp.2015-09-15.20:30:17

-rw------- 1 root root 2120 Sep 21 20:17 resp.2015-09-15.20:30:17

9

On the new Primary, update the response file on each of the core slices

Core3’s slice servers

Log in to C3-S0-rose23.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ scp root@C3-Inf-rose22.qa.opsware.com:/var/opt/opsware/install_opsware/resp/resp.2015-09-15.20:30:17 .

$ ls –l resp.2015-09-15.20:30:17

-rw------- 1 root root 2120 Sep 21 20:17 resp.2015-09-15.20:30:17

 

 

Log in to C3-S1-rose24.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ scp root@C3-Inf-rose22.qa.opsware.com:/var/opt/opsware/install_opsware/resp/resp.2015-09-15.20:30:17 .

$ ls -l resp.2015-09-15.20:30:17

-rw------- 1 root root 2120 Sep 21 20:18 resp.2015-09-15.20:30:17

 

Expected: file size match on all servers.

10

On the old Primary, update the installation response file

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ ls –ltr

total 28

-rw------- 1 root root 2228 Sep 15 10:02 resp.2015-09-15.10:02:05

-rw------- 1 root root 2228 Sep 15 11:25 resp.2015-09-15.11:25:00

-rw------- 1 root root  933 Sep 15 11:25 resp_globals

-rw------- 1 root root 2228 Sep 15 11:30 resp.2015-09-15.11:30:22

 

Identify the latest good response (in the current setup: resp.2015-09-15.11:30:22).

$ cp " resp.2015-09-15.11:30:22" “ resp.2015-09-15.11:30:22_orig”

$ vi resp.2015-09-15.11:30:22

   Change

%oi.components

osprov

infrastructure

slice

truth_mm_overlay

word_uploads

   To

%oi.components

osprov

infrastructure

slice

truth_mm_slave

   Add

%masterCore.mgw_tunnel_listener_port

2001

 

$ ls -l resp.2015-09-15.11:30:22

-rw------- 1 root root 2232 Sep 21 20:20 resp.2015-09-15.11:30:22

11

On the old Primary, update the response file on each of the core slices

Core1’s slice servers

Log in to C1-S0-rose13.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/resp/ resp.2015-09-15.11:30:22 .

$ ls –l resp.2015-09-15.11:30:22

-rw------- 1 root root 2232 Sep 21 20:20 resp.2015-09-15.11:30:22

 

 

Log in to C1-S1-rose14.qa.opsware.com as root

$ cd /var/opt/opsware/install_opsware/resp

$ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/resp/ resp.2015-09-15.11:30:22 .

$ ls -l resp.2015-09-15.11:30:22

-rw------- 1 root root 2232 Sep 21 20:21 resp.2015-09-15.11:30:22

 

Expected: file size match on all servers.

12

On the new Primary core’s slices, create masterTwist file

Core3’s slice servers

Log in to C3-S0-rose23.qa.opsware.com as root

$ ls -l /var/opt/opsware/crypto/twist/masterTwist

ls: cannot access /var/opt/opsware/crypto/twist/masterTwist: No such file or directory

$ touch /var/opt/opsware/crypto/twist/masterTwist

$ chown twist:root /var/opt/opsware/crypto/twist/masterTwist

$ chmod 644 /var/opt/opsware/crypto/twist/masterTwist

$ ls -l /var/opt/opsware/crypto/twist/masterTwist

-rw-r--r-- 1 twist root 0 Sep 21 20:24 /var/opt/opsware/crypto/twist/masterTwist

 

 

Log in to C3-S1-rose24.qa.opsware.com as root

$ ls -l /var/opt/opsware/crypto/twist/masterTwist

ls: cannot access /var/opt/opsware/crypto/twist/masterTwist: No such file or directory

$ touch /var/opt/opsware/crypto/twist/masterTwist

$ chown twist:root /var/opt/opsware/crypto/twist/masterTwist

$ chmod 644 /var/opt/opsware/crypto/twist/masterTwist

$ ls -l /var/opt/opsware/crypto/twist/masterTwist

-rw-r--r-- 1 twist root 0 Sep 21 20:27 /var/opt/opsware/crypto/twist/masterTwist

13

On the old Primary core’s slices, remove masterTwist file

Core1’s slice servers

Log in to C1-S0-rose13.qa.opsware.com as root

$ rm -f /var/opt/opsware/crypto/twist/masterTwist

 

Log in to C3-S1-rose24.qa.opsware.com as root

$  rm -f /var/opt/opsware/crypto/twist/masterTwist

14

Optional: Check and,  if needed, update is_master.txt file

 

If /var/opt/opsware/OPSWpatch/is_master.txt exists on the old Primary core’s slice servers

then create /var/opt/opsware/OPSWpatch/is_master.txt on the new Primary core’s slice servers

Core3’s slice servers

 

Log in to C1-S0-rose13.qa.opsware.com as root

$ ls -l /var/opt/opsware/OPSWpatch/is_master.txt

If file exists then

 

$ scp /var/opt/opsware/OPSWpatch/is_master.txt root@C3-S0-rose23.qa.opsware.com: /var/opt/opsware/OPSWpatch

$ scp /var/opt/opsware/OPSWpatch/is_master.txt root@C3-S1-rose24.qa.opsware.com: /var/opt/opsware/OPSWpatch

 

Note:  The content of is_master.txt file should be: IS_MASTER_CORE=1

 

15

Optional: Check and,  if needed, update is_master.txt file

 

If /var/opt/opsware/OPSWpatch/is_master.txt exists on the old Primary core’s slice servers

then create /var/opt/opsware/OPSWpatch/is_master.txt on the new Primary core’s slice servers

Core1’s slice servers

Log in to C1-S0-rose13.qa.opsware.com as root

$ ls -l /var/opt/opsware/OPSWpatch/is_master.txt

If file exists then

$ vi /var/opt/opsware/OPSWpatch/is_master.txt

Change

IS_MASTER_CORE=1

To

IS_MASTER_CORE=0

 

Log in to C1-S1-rose14.qa.opsware.com as root

$ ls -l /var/opt/opsware/OPSWpatch/is_master.txt

If file exists then

$ vi /var/opt/opsware/OPSWpatch/is_master.txt

Change

IS_MASTER_CORE=1

To

IS_MASTER_CORE=0

 

 

16

Sync OGSH home directories from old Primary core

All other core’s infrastructure servers

Log in to C1-Inf-rose12.qa.opsware.com as root

$ tar -cpf /var/tmp/ogfs-home.tar  /var/opt/opsware/ogfs/export/store/home

tar: Removing leading `/' from member names

$ scp /var/tmp/ogfs-home.tar root@C2-Inf-rose17.qa.opsware.com:/var/tmp

$ scp /var/tmp/ogfs-home.tar root@C3-Inf-rose22.qa.opsware.com:/var/tmp

 

Log in to C2-Inf-rose17.qa.opsware.com as root

$ cd /

$ tar xpvf /var/tmp/ogfs-home.tar

 

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /

$ tar xpvf /var/tmp/ogfs-home.tar

Updating files related to MM Central relocation

  • Once primary changed, all related files for each core must be updated.

No.

Purpose

Target

Servers/Command/Output

1

Update MGW related files on Core1, the old Primary.

 

opswgw-mgw-CORE1/opswgw.custom

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE1

$ cp opswgw.custom opswgw.custom_orig

$ vi opswgw.custom

   Delete

opswgw.ForwardTCP=20002:CORE2-mm:192.168.76.16:1521

opswgw.ForwardTCP=20003:CORE3-mm:192.168.76.21:1521

   Add

opswgw.TunnelSrc=192.168.76.22:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE1/opswgw.pem

opswgw.TunnelSrc=192.168.76.17:2001:100:0:/var/opt/opsware/crypto/opswgw-mgw-CORE1/opswgw.pem

1a

Update MGW related files on Core1, the old Primary.

 

opswgw-mgw-CORE1/opswgw.properties

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE1

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.EgressFilter=tcp:192.168.76.12:7501:MM:CORE1-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE1-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm

   To

opswgw.EgressFilter=tcp:192.168.76.12:7501:MM:CORE3-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE3-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm

1b

Update MGW related files on Core1, the old Primary.

 

/var/opt/oracle/tnsnames.ora

Core1’s Infrastructure server

Log in to C1-Inf-rose12.qa.opsware.com as root

$ cd /var/opt/oracle

$ cp tnsnames.ora tnsnames.ora_orig

$ vi tnsnames.ora

   Delete

truth.Core2=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.12)(PORT=20002)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))

truth.Core3=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.12)(PORT=20003)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))

2

Update MGW related files on Core3, the new Primary.

 

opswgw-mgw-CORE3/opswgw.custom

Core3’s Infrastructure server

Logan to C3-Inf-rose22.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE3

$ cp opswgw.custom opswgw.custom_orig

$ vi opswgw.custom

   Delete

opswgw.TunnelSrc=192.168.76.12:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE3/opswgw.pem

opswgw.TunnelSrc=192.168.76.17:2001:100:0:/var/opt/opsware/crypto/opswgw-mgw-CORE3/opswgw.pem

   Add

opswgw.ForwardTCP=20002:CORE2-mm:192.168.76.16:1521

opswgw.ForwardTCP=20001:CORE1-mm:192.168.76.11:1521

2a

Update MGW related files on Core3, the new Primary.

 

opswgw-mgw-CORE3/opswgw.properties

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE3

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.EgressFilter=tcp:192.168.76.22:7501:MM:CORE1-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE1-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm

   To

opswgw.EgressFilter=tcp:192.168.76.22:7501:MM:CORE3-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE3-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm

2b

Update MGW related files on Core3, the new Primary.

 

/var/opt/oracle/tnsnames.ora

Core3’s Infrastructure server

Log in to C3-Inf-rose22.qa.opsware.com as root

$ cd /var/opt/oracle

$ cp tnsnames.ora tnsnames.ora_orig

$ vi tnsnames.ora

   Add

truth.Core1=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.22)(PORT=20001)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))

truth.Core2=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.22)(PORT=20002)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))

3

Update MGW related files on Core2 (unchanged Secondary).

 

opswgw-mgw-CORE2/opswgw.custom

Core2’s Infrastructure server

Log in to C2-Inf-rose17.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE2

$ cp opswgw.custom opswgw.custom_orig

$ vi opswgw.custom

   Change

opswgw.TunnelSrc=192.168.76.12:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE2/opswgw.pem

   To

opswgw.TunnelSrc=192.168.76.22:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE2/opswgw.pem

3a

Update MGW related files on Core2 (unchanged Secondary).

 

opswgw-mgw-CORE2/opswgw.properties

Core2’s Infrastructure server

Log in to C2-Inf-rose17.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-mgw-CORE2

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.EgressFilter=tcp:192.168.76.17:7501:MM:CORE1-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE1-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm

   To

opswgw.EgressFilter=tcp:192.168.76.17:7501:MM:CORE3-mm

opswgw.EgressFilter=tcp:*:1521:MMSPIN:CORE3-mm

opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm

opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm

4

Update CGW related files on Core1, the old Primary.

 

opswgw-cgws1-CORE1/opswgw.properties

 

opswgw-cgws2-CORE1/opswgw.properties

Core1’s Slice servers

Log in to C1-S0-rose13.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-cgws1-CORE1

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

 

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

     To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

 

Log in to C1-S1-rose14.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-cgws2-CORE1

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

   To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

5

Update CGW related files on Core2(unchanged Secondary).

 

opswgw-cgws1-CORE2/opswgw.properties

 

opswgw-cgws2-CORE2/opswgw.properties

Core2’s Slice servers

Log in to C2-S0-rose18.qa.opsware.com as root

$ /etc/opt/opsware/opswgw-cgws1-CORE2

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

   To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

 

Log in to C2-S1-rose19.qa.opsware.com as root

$ /etc/opt/opsware/opswgw-cgws2-CORE2

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

   To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

6

Update CGW related files on Core3, the new Primary.

 

opswgw-cgws1-CORE3/opswgw.properties

 

opswgw-cgws2-CORE3/opswgw.properties

Core3’s Slice servers

Log in to C1-S0-rose13.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-cgws1-CORE3

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

   To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

 

Log in to C1-S1-rose14.qa.opsware.com as root

$ cd /etc/opt/opsware/opswgw-cgws2-CORE3

$ cp opswgw.properties opswgw.properties_orig

$ vi opswgw.properties

   Change

opswgw.ForceRouteService=mmspin:1004:CORE1-mm

   To

opswgw.ForceRouteService=mmspin:1004:CORE3-mm

System check before demoting the old primary core

No.

Purpose

Target

Servers/Command/Output

1

System Diagnostics test passed

All Facilities, from SA Client

Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential.

SA Client -> Administration -> Facilities

Right click on Core1 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core2 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core3 -> Run System Diagnosis -> Next -> Select All -> Start Job

 

Expected, Status: Succeeded on all items.

NOTE:  If encounter any Failure, must resolve the problem before continue.

2

Mesh has No Conflict

The Mesh, from SA Clients

Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential.

SA Client -> Administration -> Multimaster Tools -> State View

 

Expected: No conflicts.

NOTE:  If encounter any conflict, must resolve the conflict before continue.

2a

Resolve Conflict

 

Note:  Please see section 9.1 for conflict analysis and resolution details.

The Mesh, from Primary Infrastructure server

  • From Step 4, find Source Facility of the conflict. 
  • Obtain the Source Facility’s Facility ID, that is, Facility_ID.

 

Log in to C1-Inf-rose12.qa.opsware.com as root

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib

$ /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc  --refresh --from $Facility_ID

Repeat the same process on all facilities which has conflict.

Demoting the old primary core

No.

Purpose

Target

Servers/Command/Output

1

Demote the old Primary core

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) on slice: C1-S0-rose13.qa.opsware.com as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central

Select the current Member: C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed)

From Actions menu, select Cut member(s).

Select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) -> Secondary.

From Actions menu, select Paste member.

For SA 10.22 and previous versions:

Log into the SA Client (on slice: C1-S0-rose13.qa.opsware.com) as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed)

Check the box next to C1-Inf-rose12.qa.opsware.com and select Tasks -> Re-Assign Node:

Select: Service Levels / Opsware / spin / Multimaster Central -> Select

select Opsware

select spin

select Secondary

Click Re-Assign.

2

Verify Core Demotion

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) with a user that has system administrator permissions.

Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

For SA 10.22 and previous versions:

In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Levell -> Opsware -> spin -> Multimaster Central -> Members.

 

Expected: No server is displayed.

Promoting a secondary core to primary core

No.

Purpose

Target

Servers/Command/Output

1

Promote a Secondary core to Primary core role

Core 1, from SA Client / SA Web Client

 

For SA 10.50 and later:

Log into the SA Client (not the web client) as a user that has system administrator permissions.

In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Model Repository, Multimaster Component (vault).

(Three servers are displayed)

C1-Inf-rose12.qa.opsware.com

C2-Inf-rose17.qa.opsware.com

C3-Inf-rose22.qa.opsware.com

Make a note of the server that will be promoted to Multimaster Central role. In this example we have selected: C3-Inf-rose22.qa.opsware.com

In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) ->Multimaster Central

From the Actions menu, select Add member(s)

From the new window, choose All Managed Servers, then Select the previously noted server: C3-Inf-rose22.qa.opsware.com

Select Data Access Engine (spin) node from the left side and right click on the server that is now Multimaster Central (C3-Inf-rose22.qa.opsware.com) -> Delete Member(s).

For SA 10.22 and previous versions:

Log into the SA Client as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Secondary

(Three servers are displayed):

C1-Inf-rose12.qa.opsware.com

C2-Inf-rose17.qa.opsware.com

C3-Inf-rose22.qa.opsware.com

 

Select C3-Inf-rose22.qa.opsware.com and

Tasks -> Re-Assign Node:

Choose Service Levels / Opsware / spin -> Select

select Opsware

select spin

select Multimaster Central

Re-Assign

2

Core Core Promotion

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) with a user that has system administrator permissions.

Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

For SA 10.22 and previous versions:

In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Level -> Opsware -> spin -> Multimaster Central -> Members.

Expected:  C3-Inf-rose22.qa.opsware.com is displayed.

Restart Mesh

No.

Purpose

Target

Servers/Command/Output

1

Restart mesh

 

All mesh SA servers, except Oracle.

  • Execute process in sequence.
  • The action in each section can be executed at the same time.

 

Log in to C1-Sat-ros15.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C2-Sat-rose20.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C3-Sat-rose25.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

 

Log in to C3-S1-rose24.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C3-S0-rose23.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C2-S1-rose19.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C2-S0-rose18.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

 

Log in to C3-Inf-rose22.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C2-Inf-rose17.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

 

Log in to C1-S1-rose14.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

Log in to C1-S0-rose13.qa.opsware.com as root

$ /etc/init.d/opsware-sas stop

 

Log in to C1-Inf-rose12.qa.opsware.com as root

$ /etc/init.d/opsware-sas restart

 

Log in to C2-Inf-rose17.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C3-Inf-rose22.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C1-S0-rose13.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C1-S1-rose14.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

 

Log in to C2-S0-rose18.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C2-S1-rose19.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C3-S0-rose23.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C3-S1-rose24.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C1-Sat-ros15.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C2-Sat-rose20.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Log in to C3-Sat-rose25.qa.opsware.com as root

$ /etc/init.d/opsware-sas start

Conflict analysis and resolution

  1. What are “normal” conflicts?
    1. Certain operations can generate conflicts when not all mesh cores are running.  Those conflicts can be resolved without issues.
    2. Automated job runs related conflicts may happen for role classes, compliance job runs, session, etc.
    3. Conflicts generated from human interaction (i.e. new user creation, modified device group, etc.) are not normal and need to be investigated.
    4. While upgrading a core of the mesh, admin activities should not be performed to avoid conflict generation.
  2. What is the impact on conflict?
    1. In an SA mesh, a Core upgrade will fail in case conflict(s) exist.  So, all conflict(s) must be resolved before upgrading a core.
  3. How to resolve conflicts
    1. Identify the Source Facility of the conflict.  Then, obtain the Facility ID, that is, Facility_ID, of the Source Facility.
      1. Conflicts maybe found on more than one Source Facility.  Obtain all Facility_ID.
    2. Log in to any core’s infrastructure server, prefer Primary core’s Infrastructure server, run
      1. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib
      2. /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc  --refresh --from Facility_ID
        1. May need to run multiple times to resolve all conflicts
        2. Repeat on all Facility_ID, facility has conflict.