RPM remediation - Using the mrc_calc Tool

SA currently has a scalability limitation in the RPM remediation subsystem when attempting to remediate large software policies. The internal data structures that we use grow as <#RPMs> X <#servers>. When the <#RPMs> value is small, we can remediate against a large number of servers. When the <#servers> value is small, we can remediate with a large number of RPMs.

Unfortunately, modern RHN channels contain a large number of RPMs. The SERVER6-x86_64 channel contains about 3,700 unique RPMs. Before the latest performance fixes available in SA 9.15, the SA limit was approximately 10 to 20 servers for a remediation. With any more servers, the SA mesh was subject to immediate catastrophic crash (depending on mesh slice/CPU/RAM configuration). With the performance fixes, scalability was extended to about 100 servers. However, several limitations, such as serial execution, long runtimes, and large RAM consumption, prevent this solution from being fully practical in some environments.

Solution and limitations

There have been continued performance improvements for RPM remediation in SA Release 9.15 and above, and Release 10.01 and above. However, using the mrc_calc tool as documented here remains a good practice.

System sizing

It is helpful to give as much memory to the slices as possible. These RPM remediations can cause large, short-term spikes in memory usage. A “leak” is not occurring; these spikes are expected. We have seen customers successfully run much larger <#RPMs> X <#servers> when the SA core is configured with a larger amount of memory (e.g., 56GB) and physical servers are used.

Performance fixes with SA 9.15 and above

We have made significant performance improvements to the RPM remediation subsystem. Details are provided in the white paper titled "Server Automation Alert: Addressing Remediation Issue with a Software or Patch Policy Against a Large Number of Servers." If you are currently using an older version of SA 9.1x or SA 9.0x, we highly recommend upgrading to SA 9.15 or above.

rhn_import Enhancement: mrc_calc tool

The crux of this solution is the introduction of an rhn_import supplementary tool: mrc_calc.

The mrc_calc tool is used to reduce channel policies created by the rhn_import tool. rhn_import creates channel policies that can contain more than 4000 packages. When attached and remediated across a large set of servers (e.g., more than 100), SA processes consume large amounts of RAM. SA then is unusable until the job is completed or some processes are restarted.

The mrc_calc tool takes the rhn_import tool’s channel policy and creates a third kind of RHN policy, the Minimum Relevant Channel (MRC) policy. The MRC policy is calculated by cross-referencing a specified channel policy against a set of servers in the mesh, creating MRC policies for device groups or a group of managed servers. The MRC policy contains fewer RPMs than a full RHN channel policy. With a smaller MRC policy, you can remediate a larger number of servers in a much shorter period of time, because much less redundant data is generated inside of SA.

The MRC policy contains only those RPMs that are already installed on one or more target servers whose version, release, epoch, or architecture is different from the latest RPM in the policy. For example, if a channel policy with {a-2, b-2, c-2} and {b-1, c-2} is installed on the target server, then the MRC policy will contain {b-2}. Then, if b-2 requires a-2, a-2 is automatically pulled in during remediation.

The mrc_calc tool creates only one policy for each OS/channel at a time. For example, if the customer provides a RHEL6 channel policy name at the mrc_calc.conf prompt, the mrc-calc tool produces one MRC_RHEL6_policy. The MRC policy is created in the same folder as the input policy, and all necessary packages are attached to the MRC policy.

We recommend that you implement the rhn_import enhancement from HPE as part of a new process for updating your managed Red Hat servers' RPMs. This process involves configuring the enhanced rhn_import to produce the new MRC policies, attaching these policies to your managed servers, and remediating with these MRC policies instead of the old, large channel policies. Aside from the performance and system utilization advantages, there should be a tremendous gain in operational efficiency. You will be able to perform Linux remediation for more servers in less time.

Note: Because of the way channel and errata policies are constructed, they can be detached from a managed server without any worries about RPMs being uninstalled.

• Enhancement pros: It will reduce the scalability issues described in this document.
• Enhancement cons: General unknown issues related to running the smaller MRC policies in your environment.

To develop this strategy, HPE worked with experts who can make RPM remediation based on channel policies work for other customers. One individual described how he reduced a channel policy manually from 3,700 to 1,400 RPMs by reviewing each channel policy and removing those RPMs that the customer did not use.

This rhn_import enhancement automates the manual scale-down logic so it is easier for all customers to use. This tool applies all SA releases (however, as noted, we recommended that you upgrade to 9.15.xxx to benefit from all performance improvements). 

Installing the mrc_calc tool

The mrc_calc tool ships as a gzipped tar file called mrc_calc.tgz, which you untar into a directory. The mrc_calc directory contains the following files: verify_dvc_rpms.py, README, mrc_calc_with_rpmutils.py, mrc_calc.py, mrc_calc.log, mrc_calc.conf, mrc_calc.

Configuring and running the mrc_calc tool

Simple configuration

This configuration enables the mrc_calc tool to create one MRC policy for an entire environment of one version of Red Hat devices. This configuration considers all devices of one type of OS associated with a configured policy.

To create MRCs for RHEL5 and RHEL6:

1. Copy the mrc_calc.conf file as mrc_calc_rhel5.conf and mrc_calc_rhel6.conf.
2. In mrc_calc_rhel5.conf and mrc_calc_rhel6.conf, modify the [software policy] section to include both the channel policy that was created by rhn_import and the resultant policy.

 

      

To execute MRC for RHEL5, run mrc_calc as shown:

<install_dir>/mrc_calc  mrc_calc_rhel5.conf

This creates the “MRC_Red Hat Enterprise Linux (v.5 for 64-bit x86_64) Policy.” Attach this policy instead of the “Red Hat Enterprise Linux (v.5 for 64-bit x86_64) Policy” to patch your Red Hat 5 systems.

To execute MRC for RHEL6, run mrc_calc as shown:

<install_dir>/mrc_calc  mrc_calc_rhel6.conf

This creates the “MRC_Red Hat Enterprise Linux (v.6 for 64-bit x86_64) Policy.” Attach this policy instead of the “Red Hat Enterprise Linux (v.6 for 64-bit x86_64) Policy” to patch your Red Hat 5 systems.

Advanced configuration

This configuration creates different MRC policies for different server groups. Use this configuration if your environment has several sets of servers, each requiring its own set of packages.

Example

Your environment could have many global organizations with several sets of packages installed on groups of RHEL5 servers:

• ORACLE_USA_GROUP has its own set of packages
• WEBLOGIC_EURO_GROUP has its own set of packages
• SAP_USA_GROUP has its own set of packages

The best practice is to create an MRC policy for each of these groups. This example describes how to create server groups and associate RHEL5 servers to those groups.

Configuring mrc_conf for ORACLE_USA_GROUP

To create MRCs for RHEL5:

1. Copy mrc_calc.conf file as mrc_calc_rhel5_ora_usa_group.conf.
2. In mrc_calc_rhel5_ora_usa_group.conf, modify the [software policy] section to contain both the channel policy that was created by rhn_import and the resultant policy name.
3. Uncomment the [devicegroups] section and add ORACLE_USA_GROUP. You can add multiple groups using comma separation.
4. Create a device group named ORACLE_USA_GROUP, and add all related devices to that group.

 

Configuring mrc_conf for WEBLOGIC_EURO_GROUP

To create MRCs for RHEL5:

1. Copy the mrc_calc.conf file as mrc_calc_rhel5_weblogic_euro_group.conf.
2. In mrc_calc_rhel5_euro_group.conf, modify the [software policy] section to contain both the channel policy that was created by rhn_import and the resultant policy name.
3. Uncomment the [devicegroups] section and add WEBLOGIC_EURO_GROUP. You can add multiple groups using comma separation.
4. Create a device group named WEBLOGIC_EURO_GROUP and add all related devices to that group.

 

To execute the MRC for ORACLE_USA_GROUP, run mrc_calc as shown:

<install_dir>/mrc_calc  mrc_calc_rhel5_ora_usa_group.conf

This creates the “MRC_ORA_USA_GROUP_Red Hat Enterprise Linux (v. 5 for 64-bit x86_64) Policy.” Attach this policy instead of the “Red Hat Enterprise Linux (v.5 for 64-bit x86_64) Policy” to patch oracle_usa_group systems.

To execute MRC for WEBLOGIC_EURO_GROUP, run mrc_calc as shown:

<install_dir>/mrc_calc  mrc_calc_rhel5_weblogic_euro_group.conf

This creates the “MRC_WEBLOGIC_EURO_Red Hat Enterprise Linux (v.6 for 64-bit x86_64) Policy.” Attach this policy instead of the “Red Hat Enterprise Linux (v.6 for 64-bit x86_64) Policy” to patch weblogic_euro_group systems.

Limitations

The mrc_calc tool currently works only with a policy associated with just one OS. If mrc_calc is configured to work with a policy attached to multiple OSs, it will fail.

Risks and risk mitigation

What if the MRC logic fails and removes RPMs the servers need? One concern with this strategy is that MRC generation logic may fail and remove RPMs that some servers need. The original full-sized channel policies can be used to check this problem. The full channel policy can be attached to any set of managed servers, and a software compliance scan executed against it. If the MRC generation logic works correctly and remediation is successful, then the software compliance scan against the full channel policy will show 100% compliance.

Adding new RHEL servers to SA core

When you add new RHEL servers to the SA core and want to bring them up to date with current patches, create a temporary new device group, add the new servers to this device group, and run the channel policy. The number of new servers should be few, so they do not drain system resources during remediation. You can then move the new servers to the existing device groups that are managed using the MRC policy.