Monthly UCMDB Maintenance Tasks

Overview

This document describes a collection of tasks that are recommended for administrators of the UCMDB Server application and its database, to help maintain and restore high performance to their UCMDB and Universal Discovery ecosystem.

Summary of Tasks

The following list provides a high level view of the actions that administrators can take in support of maintaining a highly effective CMDB using UCMDB.

Task	Function	Minimum Frequency
Evaluate Discovery Processing Statistics	Identify statistics that indicate slow performance regarding Data-In functionality.	Monthly
Rebuild Indexes of your database	Clean up index fragmentation of the UCMDB database	Monthly
Delete Unbound History Tables	Remove tables from the history database that are no longer in use	Monthly
Re-index CI types	Index the core UCMDB database tables associated with CI data	Monthly
Re-index Composite Indexes	Rebuild the CMDB ID for all composite CI tables	Monthly
Clean the Reconciliation Priority table	Delete rows from the reconciliation priorities table (LOA) that don’t have a corresponding CI in the model	Monthly
Align History Tables	Ensure that history tables stay healthy and able to accept data	Monthly
Remove Broken Links	Decrease server startup time	Monthly
De-Activate Queries associated with Enrichment Rules	Improve performance of the UCMDB Server	Monthly
Check for “Flipping” Data	Evaluate if attributes of CIs are “flipping”	Weekly
Check for Memory Dumps	Evaluate if your UCMDB system is crashing (and then restarting)	Weekly
Evaluate database for slow performance	Evaluate the performance of your database	Weekly, possibly Daily

Evaluate Discovery Processing Statistics

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method exportDiscoveryProcessingStatisticsToExcel (in the UCMDB:service=Reconciliation Services category) to determine the health of your Data-In system.

This JMX method generates an excel file that contains a collection of tabs, showing information about Daily Rates Per Job and Daily Rates overall.

Look at the DailyRatesPerJob tab and then sort the column named Rate (CIs/sec) per thread from smallest to largest. This information should be viewed as a trend over time, so just because its bad one day, does not mean you must take action. Look at the trend for a given discovery job over time, and if you see CIs throughput slowly getting worse, then consider adding more probes.

Look at the DailyRatesPerProbe tab and then sort the column named Total Time in Hours from largest to smallest. This information should be viewed as a trend over time, so just because its bad one day, does not mean you must take action. Look at the trend for a given probe over time, and if you see many hours per day over time, then consider splitting the load assigned to that probe to additional probes.

Look at the dailyrates tab, and the column named Daily Usage. This shows how busy your UCMDB is on any given day. A single day with high percentage usage is not a problem. But when you see a trend of many days of high percentage usage, you should consider either creating a High Available (HA) UCMDB environment, or adding another UCMDB server instance to the HA cluster.

Evaluate the database index fragmentation percentage

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method rebuildIndexes (in the UCMDB:service=DAL Services category).

Running the Rebuild Indexes JMX method rebuilds any table where the database index fragmentation percentage is 10% or greater.

You can run a database schema statistics report (see Appendix – Database Schema Statistics Reporting) to identify the index fragmentation of each table in the database, but in reality, this is not normally required.

The value in running the MSSQL or Oracle schema statistics is to track database index fragmentation over time. If you notice that database index fragmentation occurs very quickly after running the rebuildIndexes JMX method, place a support ticket to request Micro Focus to investigate further what is going wrong.

Delete Unbound History Tables

Frequency: Monthly

Unbound tables are database history tables without an appropriate history partition resource in User Resource Manager (URM) or without related class of any customer. There are typically some number of unbound history tables in your UCMDB database. They can be deleted without risk, because they would simply be created by UCMDB at a future point in time if data needs to be stored in them.

Deleting unbound tables improves the overall performance of your database.

You can identify the list of tables that are unbound by running the findUnboundHistoryTables (in the UCMDB:service=History DB Services category) method in JMX Console.

If the count of tables is 100 or more, you should delete the unbound tables, using the JMX method deleteUnboundHistoryTables (in the UCMDB:service=History DB Services category). If you want to see the impact first before running, you can run this method in preview mode to get a list of tables, which is the same as running the findUnboundHistoryTables method.

The deletion of unbound tables places these tables into the recycle bin (for Oracle). If there are too many tables in the Oracle recycle bin, this could harm the database performance.

To resolve this problem, it is recommended that a DBA empty the recycle bin for the user associated with the UCMDB database using the SQL command PURGE RECYCLEBIN.

Re-index CI types

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method reindexCiType (in the UCMDB:service=Topology Search Services category).

Re-indexing the tables associated with all CI types improves search times for CIs and the overall database performance.

When you run this method, leave the ciType entry blank to direct UCMDB to re-index ALL CI types.

Re-index Composite Indexes

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method modifyCompositeIndexes (in the UCMDB:service=DAL services category). This task takes time to complete because of the scope of the action. This action is only necessary when using MSSQL or Oracle database back ends. It should not be used with the PostgreSQL database.

Be sure to identify the class name as all and to set the composite indexes to false, which forces the index to be dropped and recreated.

Re-indexing the tables associated with Composites improves search times for CIs and the overall database performance.

It is possible for the Rebuild Indexes work to be scheduled to occur on a regular basis. In the UCMDB UI, go to Administration > Scheduler > Job Scheduler, select the RebuildIndexes job, and click Edit. Modify the job definition and scheduler information as necessary to meet your needs and click OK.

Clean the Reconciliation Priority table

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method cleanLOATable (in the UCMDB:service=Reconciliation Service category). This task removes rows from the LOA table used for reconciliation priorities, that do not have a corresponding CI in the model.

Cleaning this table improves the reconciliation data-in time.

Align History Tables

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method alignHistoryForType (in the UCMDB:service=History DB Services category). This task ensures that the history tables are aligned to all the class types contained in the UCMDB database.

This action ensures that history tables stay healthy and can accept data.

Remove Broken Links

Frequency: Monthly

On a regular basis (monthly at a minimum), run the JMX method findBrokenLinks (in the UCMDB:service=Troubleshooting services category). This task makes visible any links (for example, relationships) in UCMDB that do not have two endpoints.

When a CI is deleted in your UCMDB, any relationships that reference this CI should be deleted. This is not always the case. This process cleans up your database of links that have one or zero endpoints.

Deleting broken links improves server startup time (because the whole CI map is loaded into memory and the system slows down when it encounters a broken link).

If the report shows any broken links, run the corresponding JMX method deleteBrokenLinks (in the UCMDB:service=Troubleshooting services category) to remove them.

De-Activate queries associated with Enrichment Rules

Frequency: Monthly

Ensure that queries used by the Enrichment Manager are not active, because they are run only when the enrichment is triggered. The query does not need to run in the background (that is, be active).

Navigate to the Enrichment Manager in the UCMDB Admin GUI and look for any enrichment rules that do NOT have the red X next to their titles. These enrichment rules are active.

Open the enrichment’s properties to capture the name of the query that supports the enrichment.

Repeat this process for all enrichment rules that are running.

Run the JMX method retrieveTqlNames (in the UCMDB:service=TQL Services category) to capture all the state of all TQLs in your system.

Search the list for each query that you identified as belonging to an active enrichment rule. When you find it, if the query is Active, click that query’s Deactivate button.

Repeat this process until you deactivate all queries associated with enrichment rules.

Check for “Flipping” Data

Frequency: Weekly

On a weekly basis, evaluate the state of attributes on data in CIs, to ensure that they are not flipping. Flipping occurs when two or more tasks change the same attribute in a CI, back and forth with each update. The tasks could be competing discovery jobs, integrations, or any combination thereof.

Run the CI Change Report as part of the UCMDB Admin GUI’s report capability. This report shows that the history of CI changes over time, and makes it more visible when an attribute is flipping.

If you find that there are two or more competing activities that each update the same attribute on a CI to be different values, identify each of the activities.

You can use the Reconciliation Priority function of UCMDB to prioritize which job has higher priority over others. Evaluate which of the competing tasks has the correct data, and then set this task as the highest priority in your UCMDB. From that point forward, all jobs have the ability to initially populate that attribute, but when the highest priority job populates the attribute, no other job can change it.

Starting with CMS 2018.11, you can now set reconciliation priorities by individual discovery job, allowing you to prioritize one discovery job as having priority over others. A common use case would be the Host Connections by Shell versus Host Connections by SNMP. You would not want an SNMP job, which typically has lower access privileges, to overwrite data that full shell discovery can obtain.

Check for Memory Dumps

Frequency: Weekly

Navigate to the \bin folder on the server and probe systems. If memory dump files exist, the issue needs to be investigated by Micro Focus support. Open a support ticket and provide the dump file to start determining the cause of the issue causing the memory dumps.

Evaluate database for slow performance

Frequency: Weekly/Daily

On a regular basis, check the last day’s database performance in the cmdb.dal.slow.log file, for database operation durations.

There are always some situations where queries have a duration greater than five seconds. However, you look for extensive instances of queries with durations longer than five seconds, over a short period of time, for which you cannot determine a reasonable cause.

At this point, generate a database schema statistics report for your database (see Appendix – Database Schema Statistics Reporting), open a ticket, and then provide the statistics report as part of the data that you use to populate your ticket.