Search for	Example	Results
A single word	`cat`	Topics that contain the word "cat". You will also find its grammatical variations, such as "cats".
A phrase. You can specify that the search results contain a specific phrase.	`"cat food"` (quotation marks)	Topics that contain the literal phrase "cat food" and all its grammatical variations. Without the quotation marks, the query is equivalent to specifying an OR operator, which finds topics with one of the individual words instead of the phrase.

Search for	Operator	Example
Two or more words in the same topic	`AND` `and` `+` (plus symbol) `&` (ampersand)	`cat AND dog` `"cat food"+milk` `"cat food"&"dog food"`
Either word in a topic	`OR` `or` `\|` (pipe)	`cat OR dog` `cat \| dog`
Topics that do not contain a specific word or phrase	`NOT` `not` `!` (exclamation point)	`NOT cat` `! dog`
Topics that contain one string and do not contain another	`^` (caret)	`cat ^ mouse`
A combination of search types	`( )` parentheses	`cat + (dog \| mouse)` `cat \| dog + (! mouse)`

Server Automation10.60

Administer > Multimaster Mesh administration > MM Central Relocation

MM Central relocation

This section describes the process of relocating the MultiMaster (MM) Central, for an HPE SA mesh, from one core to another. Any HPE SA mesh is comprised of one Primary core (the core running the MM Central) and one or more Secondary cores. The procedures described here will allow SA administrators to:

Demote the Primary core to Secondary core role (in an SA mesh, mark the Primary core as Secondary core)
Promote a Secondary core to Primary core role (in an SA mesh, designate a Secondary core as Primary core)

Caution The steps should only be performed with assistance from HPE Support or HPE Professional Services.

For illustration, we are using an SA mesh with three cores: Core1, Core2, and Core3.

Core1 is the Primary core that will be moved to a Secondary core role.
Core2 is a Secondary core.
Core3 is a Secondary that will be promoted as Primary core.

The base steps are as follows:

Check SA Mesh status.
Update files required during the SA upgrade process.
Note: The SA upgrade process is not in the scope of this document.
Update files related to MM Central relocation process.
Demote Core1 from Primary to Secondary core role.
Promote Core3 to primary core role.
Restart Core3.

Requirements

SA installed at version 10.0 or later
SA 10.60 installation media must be available, due to some needed scripts which have limitations or are not available in the 10.0x distribution.

Known limitations

No content import is allowed during MM Central relocation. This includes windows patch import, Red Hat patch import, script import, etc.
No Administrative operations should be performed during MM Central relocation, i.e. create user and user group, etc.

3-core mesh sample environment

The sample mesh represents a 10.2x GA environment.
SA 10.60 media has been exported from 192.168.160.100:/root/IMR2/T8900-15064

Core facility in the mesh

Core Facility Names	Facility ID
Core1 - Initial Primary core, will be demoted to Secondary core.	1 (First core was installed)
Core2 - Initial Secondary core.	2 (Second core was installed, 1^st Secondary)
Core3 - Initial Secondary core, will be promoted to Primary.	3 (Third core was installed, 2^nd Secondary)

Mesh configuration details

Core1, related server names	IP	Purpose
C1-DB-rose11.qa.opsware.com	192.168.76.11	Oracle DB
C1-Inf-rose12.qa.opsware.com	192.168.76.12	Infrastructure, Model Repository, Word, OS Prov
C1-S0-rose13.qa.opsware.com	192.168.76.13	Slice0
C1-S0-rose14.qa.opsware.com	192.168.76.14	Slice1
C1-Sat-rose15.qa.opsware.com	192.168.76.15	Satellite (Facility: Core1Sat)
C1-Ssvr-rose31.qa.opsware.com	192.168.76.31	Managed server, attached to Satellite Core1Sat.
C1-Ssvr-rose32.qa.opsware.com	192.168.76.32	Managed server, attached to Satellite Core1Sat.
C1-svr-rose33.qa.opsware.com	192.168.76.33	Managed server, attached to Core1.
Core2, related server names	IP	Purpose
C2-DB-rose16.qa.opsware.com	192.168.76.16	Oracle DB
C2-Inf-rose17.qa.opsware.com	192.168.76.17	Infrastructure, Model Repository, Word, OS Prov
C2-S0-rose18.qa.opsware.com	192.168.76.18	Slice0
C2-S0-rose19.qa.opsware.com	192.168.76.19	Slice1
C2-Sat-rose20.qa.opsware.com	192.168.76.20	Satellite (Facility: Core2Sat)
C2-Ssvr-rose34.qa.opsware.com	192.168.76.34	Managed server, attached to Satellite Core2Sat.
C2-Ssvr-rose35.qa.opsware.com	192.168.76.35	Managed server, attached to Satellite Core2Sat.
C2-svr-rose36.qa.opsware.com	192.168.76.36	Managed server, attached to Core2.
Core3, related server names	IP	Purpose
C3-DB-rose21.qa.opsware.com	192.168.76.21	Oracle DB
C3-Inf-rose22.qa.opsware.com	192.168.76.22	Infrastructure, Model Repository, Word, OS Prov
C3-S0-rose23.qa.opsware.com	192.168.76.23	Slice0
C3-S0-rose24.qa.opsware.com	192.168.76.24	Slice1
C3-Sat-rose25.qa.opsware.com	192.168.76.25	Satellite (Facility: Core3Sat)
C3-Ssvr-rose37.qa.opsware.com	192.168.76.37	Managed server, attached to Satellite Core3Sat.
C3-Ssvr-rose38.qa.opsware.com	192.168.76.38	Managed server, attached to Satellite Core3Sat.
C3-svr-rose39.qa.opsware.com	192.168.76.39	Managed server, attached to Core3.

Moving MM Central

Mesh status check

Note

Please resolve all failures encountered in the current section before proceeding to the next.

No.	Purpose	Target	Servers/Command/Output
1	Ensure SA processes are running	All SA core servers, except the DB server.	Log in to C1-Inf-rose12.qa.opsware.com as root. $ /etc/init.d/opsware-sas health Successfully performed "health" operation on Opsware SAS components. Expected, Successfully performed "health"… NOTE: Any failures need to be resolved before continuing to the next step. Repeat the same steps on: C1-S0-rose13.qa.opsware.com C1-S1-rose14.qa.opsware.com C1-Sat-rose15.qa.opsware.com C2-Inf-rose17.qa.opsware.com C2-S0-rose18.qa.opsware.com C2-S1-rose19.qa.opsware.com C2-Sat-rose20.qa.opsware.com C3-Inf-rose22.qa.opsware.com C3-S0-rose23.qa.opsware.com C3-S1-rose24.qa.opsware.com C3-Sat-rose25.qa.opsware.com
2	Communication test passed	All SA Core servers and Managed servers, from SA Client on Primary slice server	Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has enough permissions. SA Client -> Devices -> All managed Servers Select all listed servers, right click -> Run -> Communication Test Expected, Status: GREEN status are displayed on all checkup point for all servers. NOTE: Any failures need to be resolved before continuing to the next step.
3	System Diagnostics test passed	All Facilities, from SA Client on Primary slice server	Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential. SA Client -> Administration -> Facilities Right click on Core1 -> Run System Diagnosis -> Next -> Select All -> Start Job Right click on Core2 -> Run System Diagnosis -> Next -> Select All -> Start Job Right click on Core3 -> Run System Diagnosis -> Next -> Select All -> Start Job Expected, Status: Succeeded on all items. NOTE: Any failures need to be resolved before continuing to the next step.
4	Mesh has No Conflict	The Mesh, from SA Clients on Primary slice server	Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential. SA Client -> Administration -> MultiMaster Tools -> State View Expected: No conflicts. NOTE: Any conflicts need to be resolved before continuing to the next step.
4a	Resolve Conflict Note: Please see section 8.1 for conflict analysis and resolution details.	The Mesh, from Primary Infrastructure server	From Step 4, find Source Facility of the conflict. Obtain the Source Facility’s Facility ID, i.e Facility_ID. Log in to C1-Inf-rose12.qa.opsware.com as root $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib $ /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc --refresh --from $Facility_ID Repeat the same process on all facilities which that have conflicts.

Updating files required during SA upgrade

After promoting a Secondary core to Primary core, certain files on the new Primary core have to match those on the original Primary to ensure a successful upgrade to a newer release of SA is possible.
In our setup, ‘Core3’ is a Secondary core that will be promoted as new Primary core and ‘Core1’ will be demoted from Primary to Secondary core role.

No.	Purpose	Target	Servers/Command/Output
1	On the new Primary core, update the SA inventory file: /var/opt/opsware/install_opsware/inv/install.inv	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/inv $ cp install.inv install.inv _orig $ vi install.inv Change %truth_mm_slave To %truth_mm_overlay Add %word_uploads build_id: opsware_50.0.37477.0 state: complete script: pre script: post Note: For build_id info check the old Primary inventory file.
2	On the old Primary core, update the SA inventory file: /var/opt/opsware/install_opsware/inv/install.inv	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/inv $ cp install.inv install.inv _orig $ vi install.inv Change % truth_mm_overlay To % truth_mm_slave Remove %word_uploads build_id: opsware_50.0.37477.0 state: complete script: pre script: post Note: build_id info might have a different build number depending on the installed SA version.
3	From the old Primary copy to the new Primary the following directory and its contents: /var/opt/opsware/install_opsware/conf	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware $ mv conf conf_orig $ mkdir conf $ cd conf $ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/conf/* . … $ ls –l \| wc 111 992 6634 Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/conf $ ls –l \| wc 111 992 6634 Both output are matched.
4	From the old Primary copy to the new Primary the following directory contents: /var/opt/opsware/crypto/word_uploads	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/opsware/crypto/ $ ls word_uploads ls: cannot access word_uploads: No such file or directory $ mkdir word_uploads $ chmod 0500 word_uploads $ ls -lad /var/opt/opsware/crypto/word_uploads dr-x------ 2 root root 4096 Sep 21 18:54 /var/opt/opsware/crypto/word_uploads $ cd word_uploads $ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/crypto/word_uploads/* . … $ ls –l total 24 -r-------- 1 root root 1858 Sep 21 18:54 admin-ca.crt -r-------- 1 root root 1862 Sep 21 18:54 agent-ca.crt -r-------- 1 root root 1866 Sep 21 18:54 opsware-ca.crt -r-------- 1 root root 3062 Sep 21 18:54 spin.srv -r-------- 1 root root 3087 Sep 21 18:54 spog.pkcs8 -r-------- 1 root root 3062 Sep 21 18:54 wordbot.srv Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/crypto/word_uploads $ ls -l total 24 -r-------- 1 root root 1858 Sep 15 01:52 admin-ca.crt -r-------- 1 root root 1862 Sep 15 01:52 agent-ca.crt -r-------- 1 root root 1866 Sep 15 01:52 opsware-ca.crt -r-------- 1 root root 3062 Sep 15 01:52 spin.srv -r-------- 1 root root 3087 Sep 15 01:52 spog.pkcs8 -r-------- 1 root root 3062 Sep 15 01:52 wordbot.srv word_uploads contents should be the same on the old and the new Primary.
5	On the old Primary remove: /var/opt/opsware/crypto/word_uploads	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/crypto/ $ mkdir backup $ mv word_uploads ./backup/
6	On the new Primary, update the core definition file (cdf.xml)	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/cdf $ cp cdf.xml cdf.xml_orig $ vi cdf.xml Change <PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_slave_typical</PublicProperty> To <PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_master_typical</PublicProperty> Change <OpswareComponent name="truth_mm_slave" ttl="3600" dc="2015-09-15T19:14:02Z"> To <OpswareComponent name="truth_mm_overlay" ttl="3600" dc="2015-09-15T19:14:02Z">
7	On the old Primary, update the core definition file (cdf.xml)	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/cdf $ cp cdf.xml cdf.xml_orig $ vi cdf.xml Change <PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_master_typical </PublicProperty> To <PublicProperty name="install_type" ttl="3600" dc="2015-09-15T19:14:02Z" encoding="str">slices_slave_typical</PublicProperty> Change <OpswareComponent name="truth_mm_overlay" ttl="3600" dc="2015-09-15T19:14:02Z"> To <OpswareComponent name="truth_mm_slave " ttl="3600" dc="2015-09-15T19:14:02Z">
8	On the new Primary, update the installation response file	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ ls –ltr total 28 -rw------- 1 root root 2148 Sep 15 19:04 resp.2015-09-15.19:04:49 -rw------- 1 root root 2148 Sep 15 19:14 resp.2015-09-15.19:14:02 -rw------- 1 root root 2148 Sep 15 19:14 resp.2015-09-15.19:14:06 -rw------- 1 root root 2148 Sep 15 19:20 resp.2015-09-15.19:20:03 -rw------- 1 root root 2148 Sep 15 20:25 resp.2015-09-15.20:25:00 -rw------- 1 root root 828 Sep 15 20:25 resp_globals -rw------- 1 root root 2148 Sep 15 20:30 resp.2015-09-15.20:30:17 Identify the latest good response (in the current setup: resp.2015-09-15.20:30:17). $ cp "resp.2015-09-15.20:30:17" “resp.2015-09-15.20:30:17_orig” $ vi "resp.2015-09-15.20:30:17" Change %oi.components osprov infrastructure slice truth_mm_slave To %oi.components osprov infrastructure slice truth_mm_overlay word_uploads Delete %masterCore.mgw_tunnel_listener_port 2001 $ ls -l resp.2015-09-15.20:30:17 -rw------- 1 root root 2120 Sep 21 20:17 resp.2015-09-15.20:30:17
9	On the new Primary, update the response file on each of the core slices	Core3’s slice servers	Log in to C3-S0-rose23.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ scp root@C3-Inf-rose22.qa.opsware.com:/var/opt/opsware/install_opsware/resp/resp.2015-09-15.20:30:17 . $ ls –l resp.2015-09-15.20:30:17 -rw------- 1 root root 2120 Sep 21 20:17 resp.2015-09-15.20:30:17 Log in to C3-S1-rose24.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ scp root@C3-Inf-rose22.qa.opsware.com:/var/opt/opsware/install_opsware/resp/resp.2015-09-15.20:30:17 . $ ls -l resp.2015-09-15.20:30:17 -rw------- 1 root root 2120 Sep 21 20:18 resp.2015-09-15.20:30:17 Expected: file size match on all servers.
10	On the old Primary, update the installation response file	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ ls –ltr total 28 -rw------- 1 root root 2228 Sep 15 10:02 resp.2015-09-15.10:02:05 -rw------- 1 root root 2228 Sep 15 11:25 resp.2015-09-15.11:25:00 -rw------- 1 root root 933 Sep 15 11:25 resp_globals -rw------- 1 root root 2228 Sep 15 11:30 resp.2015-09-15.11:30:22 Identify the latest good response (in the current setup: resp.2015-09-15.11:30:22). $ cp " resp.2015-09-15.11:30:22" “ resp.2015-09-15.11:30:22_orig” $ vi resp.2015-09-15.11:30:22 Change %oi.components osprov infrastructure slice truth_mm_overlay word_uploads To %oi.components osprov infrastructure slice truth_mm_slave Add %masterCore.mgw_tunnel_listener_port 2001 $ ls -l resp.2015-09-15.11:30:22 -rw------- 1 root root 2232 Sep 21 20:20 resp.2015-09-15.11:30:22
11	On the old Primary, update the response file on each of the core slices	Core1’s slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/resp/ resp.2015-09-15.11:30:22 . $ ls –l resp.2015-09-15.11:30:22 -rw------- 1 root root 2232 Sep 21 20:20 resp.2015-09-15.11:30:22 Log in to C1-S1-rose14.qa.opsware.com as root $ cd /var/opt/opsware/install_opsware/resp $ scp root@C1-Inf-rose12.qa.opsware.com:/var/opt/opsware/install_opsware/resp/ resp.2015-09-15.11:30:22 . $ ls -l resp.2015-09-15.11:30:22 -rw------- 1 root root 2232 Sep 21 20:21 resp.2015-09-15.11:30:22 Expected: file size match on all servers.
12	On the new Primary core’s slices, create masterTwist file	Core3’s slice servers	Log in to C3-S0-rose23.qa.opsware.com as root $ ls -l /var/opt/opsware/crypto/twist/masterTwist ls: cannot access /var/opt/opsware/crypto/twist/masterTwist: No such file or directory $ touch /var/opt/opsware/crypto/twist/masterTwist $ chown twist:root /var/opt/opsware/crypto/twist/masterTwist $ chmod 644 /var/opt/opsware/crypto/twist/masterTwist $ ls -l /var/opt/opsware/crypto/twist/masterTwist -rw-r--r-- 1 twist root 0 Sep 21 20:24 /var/opt/opsware/crypto/twist/masterTwist Log in to C3-S1-rose24.qa.opsware.com as root $ ls -l /var/opt/opsware/crypto/twist/masterTwist ls: cannot access /var/opt/opsware/crypto/twist/masterTwist: No such file or directory $ touch /var/opt/opsware/crypto/twist/masterTwist $ chown twist:root /var/opt/opsware/crypto/twist/masterTwist $ chmod 644 /var/opt/opsware/crypto/twist/masterTwist $ ls -l /var/opt/opsware/crypto/twist/masterTwist -rw-r--r-- 1 twist root 0 Sep 21 20:27 /var/opt/opsware/crypto/twist/masterTwist
13	On the old Primary core’s slices, remove masterTwist file	Core1’s slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ rm -f /var/opt/opsware/crypto/twist/masterTwist Log in to C3-S1-rose24.qa.opsware.com as root $ rm -f /var/opt/opsware/crypto/twist/masterTwist
14	Optional: Check and, if needed, update is_master.txt file If /var/opt/opsware/OPSWpatch/is_master.txt exists on the old Primary core’s slice servers then create /var/opt/opsware/OPSWpatch/is_master.txt on the new Primary core’s slice servers	Core3’s slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ ls -l /var/opt/opsware/OPSWpatch/is_master.txt If file exists then $ scp /var/opt/opsware/OPSWpatch/is_master.txt root@C3-S0-rose23.qa.opsware.com: /var/opt/opsware/OPSWpatch $ scp /var/opt/opsware/OPSWpatch/is_master.txt root@C3-S1-rose24.qa.opsware.com: /var/opt/opsware/OPSWpatch Note: The content of is_master.txt file should be: IS_MASTER_CORE=1
15	Optional: Check and, if needed, update is_master.txt file If /var/opt/opsware/OPSWpatch/is_master.txt exists on the old Primary core’s slice servers then create /var/opt/opsware/OPSWpatch/is_master.txt on the new Primary core’s slice servers	Core1’s slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ ls -l /var/opt/opsware/OPSWpatch/is_master.txt If file exists then $ vi /var/opt/opsware/OPSWpatch/is_master.txt Change IS_MASTER_CORE=1 To IS_MASTER_CORE=0 Log in to C1-S1-rose14.qa.opsware.com as root $ ls -l /var/opt/opsware/OPSWpatch/is_master.txt If file exists then $ vi /var/opt/opsware/OPSWpatch/is_master.txt Change IS_MASTER_CORE=1 To IS_MASTER_CORE=0
16	Sync OGSH home directories from old Primary core	All other core’s infrastructure servers	Log in to C1-Inf-rose12.qa.opsware.com as root $ tar -cpf /var/tmp/ogfs-home.tar /var/opt/opsware/ogfs/export/store/home tar: Removing leading `/' from member names $ scp /var/tmp/ogfs-home.tar root@C2-Inf-rose17.qa.opsware.com:/var/tmp $ scp /var/tmp/ogfs-home.tar root@C3-Inf-rose22.qa.opsware.com:/var/tmp Log in to C2-Inf-rose17.qa.opsware.com as root $ cd / $ tar xpvf /var/tmp/ogfs-home.tar Log in to C3-Inf-rose22.qa.opsware.com as root $ cd / $ tar xpvf /var/tmp/ogfs-home.tar

Updating files related to MM Central relocation

Once primary changed, all related files for each core must be updated.

No.	Purpose	Target	Servers/Command/Output
1	Update MGW related files on Core1, the old Primary. opswgw-mgw-CORE1/opswgw.custom	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE1 $ cp opswgw.custom opswgw.custom_orig $ vi opswgw.custom Delete opswgw.ForwardTCP=20002:CORE2-mm:192.168.76.16:1521 opswgw.ForwardTCP=20003:CORE3-mm:192.168.76.21:1521 Add opswgw.TunnelSrc=192.168.76.22:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE1/opswgw.pem opswgw.TunnelSrc=192.168.76.17:2001:100:0:/var/opt/opsware/crypto/opswgw-mgw-CORE1/opswgw.pem
1a	Update MGW related files on Core1, the old Primary. opswgw-mgw-CORE1/opswgw.properties	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE1 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.EgressFilter=tcp:192.168.76.12:7501:MM:CORE1-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE1-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm To opswgw.EgressFilter=tcp:192.168.76.12:7501:MM:CORE3-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE3-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm
1b	Update MGW related files on Core1, the old Primary. /var/opt/oracle/tnsnames.ora	Core1’s Infrastructure server	Log in to C1-Inf-rose12.qa.opsware.com as root $ cd /var/opt/oracle $ cp tnsnames.ora tnsnames.ora_orig $ vi tnsnames.ora Delete truth.Core2=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.12)(PORT=20002)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth))) truth.Core3=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.12)(PORT=20003)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))
2	Update MGW related files on Core3, the new Primary. opswgw-mgw-CORE3/opswgw.custom	Core3’s Infrastructure server	Logan to C3-Inf-rose22.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE3 $ cp opswgw.custom opswgw.custom_orig $ vi opswgw.custom Delete opswgw.TunnelSrc=192.168.76.12:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE3/opswgw.pem opswgw.TunnelSrc=192.168.76.17:2001:100:0:/var/opt/opsware/crypto/opswgw-mgw-CORE3/opswgw.pem Add opswgw.ForwardTCP=20002:CORE2-mm:192.168.76.16:1521 opswgw.ForwardTCP=20001:CORE1-mm:192.168.76.11:1521
2a	Update MGW related files on Core3, the new Primary. opswgw-mgw-CORE3/opswgw.properties	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE3 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.EgressFilter=tcp:192.168.76.22:7501:MM:CORE1-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE1-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm To opswgw.EgressFilter=tcp:192.168.76.22:7501:MM:CORE3-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE3-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm
2b	Update MGW related files on Core3, the new Primary. /var/opt/oracle/tnsnames.ora	Core3’s Infrastructure server	Log in to C3-Inf-rose22.qa.opsware.com as root $ cd /var/opt/oracle $ cp tnsnames.ora tnsnames.ora_orig $ vi tnsnames.ora Add truth.Core1=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.22)(PORT=20001)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth))) truth.Core2=(DESCRIPTION=(ADDRESS=(HOST=192.168.76.22)(PORT=20002)(PROTOCOL=tcp))(CONNECT_DATA=(SERVICE_NAME=truth)))
3	Update MGW related files on Core2 (unchanged Secondary). opswgw-mgw-CORE2/opswgw.custom	Core2’s Infrastructure server	Log in to C2-Inf-rose17.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE2 $ cp opswgw.custom opswgw.custom_orig $ vi opswgw.custom Change opswgw.TunnelSrc=192.168.76.12:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE2/opswgw.pem To opswgw.TunnelSrc=192.168.76.22:2001:5:0:/var/opt/opsware/crypto/opswgw-mgw-CORE2/opswgw.pem
3a	Update MGW related files on Core2 (unchanged Secondary). opswgw-mgw-CORE2/opswgw.properties	Core2’s Infrastructure server	Log in to C2-Inf-rose17.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-mgw-CORE2 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.EgressFilter=tcp:192.168.76.17:7501:MM:CORE1-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE1-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE1-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE1-mm To opswgw.EgressFilter=tcp:192.168.76.17:7501:MM:CORE3-mm opswgw.EgressFilter=tcp::1521:MMSPIN:CORE3-mm opswgw.EgressFilter=tcp:mmspin:1004:SPIN:CORE3-mm opswgw.EgressFilter=tcp:spin:1004:SPIN:CORE3-mm
4	Update CGW related files on Core1, the old Primary. opswgw-cgws1-CORE1/opswgw.properties opswgw-cgws2-CORE1/opswgw.properties	Core1’s Slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-cgws1-CORE1 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm Log in to C1-S1-rose14.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-cgws2-CORE1 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm
5	Update CGW related files on Core2(unchanged Secondary). opswgw-cgws1-CORE2/opswgw.properties opswgw-cgws2-CORE2/opswgw.properties	Core2’s Slice servers	Log in to C2-S0-rose18.qa.opsware.com as root $ /etc/opt/opsware/opswgw-cgws1-CORE2 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm Log in to C2-S1-rose19.qa.opsware.com as root $ /etc/opt/opsware/opswgw-cgws2-CORE2 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm
6	Update CGW related files on Core3, the new Primary. opswgw-cgws1-CORE3/opswgw.properties opswgw-cgws2-CORE3/opswgw.properties	Core3’s Slice servers	Log in to C1-S0-rose13.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-cgws1-CORE3 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm Log in to C1-S1-rose14.qa.opsware.com as root $ cd /etc/opt/opsware/opswgw-cgws2-CORE3 $ cp opswgw.properties opswgw.properties_orig $ vi opswgw.properties Change opswgw.ForceRouteService=mmspin:1004:CORE1-mm To opswgw.ForceRouteService=mmspin:1004:CORE3-mm

System check before demoting the old primary core

No.	Purpose	Target	Servers/Command/Output
1	System Diagnostics test passed	All Facilities, from SA Client	Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential. SA Client -> Administration -> Facilities Right click on Core1 -> Run System Diagnosis -> Next -> Select All -> Start Job Right click on Core2 -> Run System Diagnosis -> Next -> Select All -> Start Job Right click on Core3 -> Run System Diagnosis -> Next -> Select All -> Start Job Expected, Status: Succeeded on all items. NOTE: If encounter any Failure, must resolve the problem before continue.
2	Mesh has No Conflict	The Mesh, from SA Clients	Log in from SA Client to C1-S0-rose13.qa.opsware.com with user who has credential. SA Client -> Administration -> Multimaster Tools -> State View Expected: No conflicts. NOTE: If encounter any conflict, must resolve the conflict before continue.
2a	Resolve Conflict Note: Please see section 9.1 for conflict analysis and resolution details.	The Mesh, from Primary Infrastructure server	From Step 4, find Source Facility of the conflict. Obtain the Source Facility’s Facility ID, that is, Facility_ID. Log in to C1-Inf-rose12.qa.opsware.com as root $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib $ /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc --refresh --from $Facility_ID Repeat the same process on all facilities which has conflict.

No.

Purpose

Target

Servers/Command/Output

System Diagnostics test passed

All Facilities, from SA Client

SA Client -> Administration -> Facilities

Right click on Core1 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core2 -> Run System Diagnosis -> Next -> Select All -> Start Job

Right click on Core3 -> Run System Diagnosis -> Next -> Select All -> Start Job

Expected, Status: Succeeded on all items.

NOTE: If encounter any Failure, must resolve the problem before continue.

Mesh has No Conflict

The Mesh, from SA Clients

SA Client -> Administration -> Multimaster Tools -> State View

Expected: No conflicts.

NOTE: If encounter any conflict, must resolve the conflict before continue.

Resolve Conflict

Note: Please see section 9.1 for conflict analysis and resolution details.

The Mesh, from Primary Infrastructure server

From Step 4, find Source Facility of the conflict.
Obtain the Source Facility’s Facility ID, that is, Facility_ID.

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib

$ /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc --refresh --from $Facility_ID

Repeat the same process on all facilities which has conflict.

Demoting the old primary core

No.	Purpose	Target	Servers/Command/Output
1	Demote the old Primary core	Core 1, from SA Client / SA Web Client	For SA 10.50 and later: Log into the SA Client (not the web client) on slice: C1-S0-rose13.qa.opsware.com as a user that has system administrator permissions. Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central Select the current Member: C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed) From Actions menu, select Cut member(s). Select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) -> Secondary. From Actions menu, select Paste member. For SA 10.22 and previous versions: Log into the SA Client (on slice: C1-S0-rose13.qa.opsware.com) as a user that has system administrator permissions. Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central. C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed) Check the box next to C1-Inf-rose12.qa.opsware.com and select Tasks -> Re-Assign Node: Select: Service Levels / Opsware / spin / Multimaster Central -> Select select Opsware select spin select Secondary Click Re-Assign.
2	Verify Core Demotion	Core 1, from SA Client / SA Web Client	For SA 10.50 and later: Log into the SA Client (not the web client) with a user that has system administrator permissions. Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central. For SA 10.22 and previous versions: In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Levell -> Opsware -> spin -> Multimaster Central -> Members. Expected: No server is displayed.

No.

Purpose

Target

Servers/Command/Output

Demote the old Primary core

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) on slice: C1-S0-rose13.qa.opsware.com as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central

Select the current Member: C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed)

From Actions menu, select Cut member(s).

Select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) -> Secondary.

From Actions menu, select Paste member.

For SA 10.22 and previous versions:

Log into the SA Client (on slice: C1-S0-rose13.qa.opsware.com) as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

C1-Inf-rose12.qa.opsware.com is displayed. (The only server is displayed)

Check the box next to C1-Inf-rose12.qa.opsware.com and select Tasks -> Re-Assign Node:

Select: Service Levels / Opsware / spin / Multimaster Central -> Select

select Opsware

select spin

select Secondary

Click Re-Assign.

Verify Core Demotion

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) with a user that has system administrator permissions.

Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

For SA 10.22 and previous versions:

In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Levell -> Opsware -> spin -> Multimaster Central -> Members.

Expected: No server is displayed.

Promoting a secondary core to primary core

No.	Purpose	Target	Servers/Command/Output
1	Promote a Secondary core to Primary core role	Core 1, from SA Client / SA Web Client	For SA 10.50 and later: Log into the SA Client (not the web client) as a user that has system administrator permissions. In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Model Repository, Multimaster Component (vault). (Three servers are displayed) C1-Inf-rose12.qa.opsware.com C2-Inf-rose17.qa.opsware.com C3-Inf-rose22.qa.opsware.com Make a note of the server that will be promoted to Multimaster Central role. In this example we have selected: C3-Inf-rose22.qa.opsware.com In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) ->Multimaster Central From the Actions menu, select Add member(s) From the new window, choose All Managed Servers, then Select the previously noted server: C3-Inf-rose22.qa.opsware.com Select Data Access Engine (spin) node from the left side and right click on the server that is now Multimaster Central (C3-Inf-rose22.qa.opsware.com) -> Delete Member(s). For SA 10.22 and previous versions: Log into the SA Client as a user that has system administrator permissions. Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Secondary. (Three servers are displayed): C1-Inf-rose12.qa.opsware.com C2-Inf-rose17.qa.opsware.com C3-Inf-rose22.qa.opsware.com Select C3-Inf-rose22.qa.opsware.com and Tasks -> Re-Assign Node: Choose Service Levels / Opsware / spin -> Select select Opsware select spin select Multimaster Central Re-Assign
2	Core Core Promotion	Core 1, from SA Client / SA Web Client	For SA 10.50 and later: Log into the SA Client (not the web client) with a user that has system administrator permissions. Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central. For SA 10.22 and previous versions: In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Level -> Opsware -> spin -> Multimaster Central -> Members. Expected: C3-Inf-rose22.qa.opsware.com is displayed.

No.

Purpose

Target

Servers/Command/Output

Promote a Secondary core to Primary core role

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) as a user that has system administrator permissions.

In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Model Repository, Multimaster Component (vault).

(Three servers are displayed)

C1-Inf-rose12.qa.opsware.com

C2-Inf-rose17.qa.opsware.com

C3-Inf-rose22.qa.opsware.com

Make a note of the server that will be promoted to Multimaster Central role. In this example we have selected: C3-Inf-rose22.qa.opsware.com

In the navigation panel, select SA Client -> Administration ->System Configuration ->Service Level Members -> Data Access Engine (spin) ->Multimaster Central

From the Actions menu, select Add member(s)

From the new window, choose All Managed Servers, then Select the previously noted server: C3-Inf-rose22.qa.opsware.com

Select Data Access Engine (spin) node from the left side and right click on the server that is now Multimaster Central (C3-Inf-rose22.qa.opsware.com) -> Delete Member(s).

For SA 10.22 and previous versions:

Log into the SA Client as a user that has system administrator permissions.

Select Administration tab in the navigation panel > System Configuration > Service Level Members > Data Access Engine (spin) > Secondary.

(Three servers are displayed):

C1-Inf-rose12.qa.opsware.com

C2-Inf-rose17.qa.opsware.com

C3-Inf-rose22.qa.opsware.com

Select C3-Inf-rose22.qa.opsware.com and

Tasks -> Re-Assign Node:

Choose Service Levels / Opsware / spin -> Select

select Opsware

select spin

select Multimaster Central

Re-Assign

Core Core Promotion

Core 1, from SA Client / SA Web Client

For SA 10.50 and later:

Log into the SA Client (not the web client) with a user that has system administrator permissions.

Select SA Client > Administration > System Configuration > Service Level Members > Data Access Engine (spin) > Multimaster Central.

For SA 10.22 and previous versions:

In the navigation panel, select SA Web Client -> Navigation -> Environment -> Service Level -> Opsware -> spin -> Multimaster Central -> Members.

Expected: C3-Inf-rose22.qa.opsware.com is displayed.

Restart Mesh

No.	Purpose	Target	Servers/Command/Output
1	Restart mesh	All mesh SA servers, except Oracle.	Execute process in sequence. The action in each section can be executed at the same time. Log in to C1-Sat-ros15.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C2-Sat-rose20.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C3-Sat-rose25.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C3-S1-rose24.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C3-S0-rose23.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C2-S1-rose19.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C2-S0-rose18.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C3-Inf-rose22.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C2-Inf-rose17.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C1-S1-rose14.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C1-S0-rose13.qa.opsware.com as root $ /etc/init.d/opsware-sas stop Log in to C1-Inf-rose12.qa.opsware.com as root $ /etc/init.d/opsware-sas restart Log in to C2-Inf-rose17.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C3-Inf-rose22.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C1-S0-rose13.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C1-S1-rose14.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C2-S0-rose18.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C2-S1-rose19.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C3-S0-rose23.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C3-S1-rose24.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C1-Sat-ros15.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C2-Sat-rose20.qa.opsware.com as root $ /etc/init.d/opsware-sas start Log in to C3-Sat-rose25.qa.opsware.com as root $ /etc/init.d/opsware-sas start

No.

Purpose

Target

Servers/Command/Output

Restart mesh

All mesh SA servers, except Oracle.

Execute process in sequence.
The action in each section can be executed at the same time.

$ /etc/init.d/opsware-sas stop

$ /etc/init.d/opsware-sas restart

$ /etc/init.d/opsware-sas start

Conflict analysis and resolution

What are “normal” conflicts?
1. Certain operations can generate conflicts when not all mesh cores are running. Those conflicts can be resolved without issues.
2. Automated job runs related conflicts may happen for role classes, compliance job runs, session, etc.
3. Conflicts generated from human interaction (i.e. new user creation, modified device group, etc.) are not normal and need to be investigated.
4. While upgrading a core of the mesh, admin activities should not be performed to avoid conflict generation.
What is the impact on conflict?
1. In an SA mesh, a Core upgrade will fail in case conflict(s) exist. So, all conflict(s) must be resolved before upgrading a core.
How to resolve conflicts
1. Identify the Source Facility of the conflict. Then, obtain the Facility ID, that is, Facility_ID, of the Source Facility.
  1. Conflicts maybe found on more than one Source Facility. Obtain all Facility_ID.
2. Log in to any core’s infrastructure server, prefer Primary core’s Infrastructure server, run
  1. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opsware/lib
  2. /opt/opsware/bin/python2 /opt/opsware/mmtools/force_resolve.pyc --refresh --from Facility_ID
    1. May need to run multiple times to resolve all conflicts
    2. Repeat on all Facility_ID, facility has conflict.

Send Help Center feedback