Configure the Enhanced Search Engine

The Enhanced CI search engine translates free text queries to TQL queries, with much better performance and accuracy than the legacy search engine and flexible searching language. The syntax of the search queries is based on the class model.

The search engine can be configured in the following three areas:

• Indexing. Configure what is indexed, which CI types are "searchable", which attributes are "searchable", and what are types of the attributes. This configuration is performed with the Search_Indexer_Configuration_XML and Search_Ranking_Configuration_XML files.

  This configuration file includes a list of class types that are indexed and a list of attributes for each one of the class types. This configuration influences the manner in which the search by property condition is performed.

• Parsing. Configure class model synonyms. Synonyms translate the user input string query into a graph structure, TQL. This configuration is performed with the Search_Parser_Configuration_XML file.

  This configuration file includes a set of class synonyms, attribute synonyms, date synonyms, relation synonyms, and a set of redundant words.

  Note The list of redundant words must contain only single words, with no spaces (no phrases are permitted).

• Ranking. Configure which CI types are presented and the order of presentation. Prioritization is according to class model. This configuration is performed with the Search_Ranking_Configuration_XML file.

  This configuration file contains a list of attributes that are associated with five data fields, called data0, data1, data2, data3, and data4. These fields are prioritized, where data4 has the highest priority and data0 has the lowest priority.

  This configuration influences the order of CIs in the search results.

The CMS UI can be used out-of-the-box without initial configuration of the search engine. However, if an attribute is not in the Indexer or Ranking lists, it does not appear in the search results.

Modify the currently indexed list

Enable/Disable the Enhanced Search Engine

1. Go to JMX Console > UCMDB:service=Topology Search Services.

2. Choose one or more of the following operations:

   • editIndexerConfiguration – displays and enables editing of the Search_Indexer_Configuration_XML file.

   • editParserConfiguration – displays and enables editing of the Search_Parser_Configuration_XML file.

   • editRankingConfiguration – displays and enables editing of the Search_Ranking_Configuration_XML file.

3. For each operation, enter the relevant customer ID and click Invoke.

For the enhanced search engine to work, it must be enabled in UCMDB. By default, it is enabled (unless it was disabled during UCMDB installation).

To change the enable/disable setting:

1. Go to JMX Console > UCMDB:service=Settings Services > setGlobalSettingValue.
2. In the name field enter cmdb.search.enabled.

3. In the value field enter:

   true: If you want the search enabled.

   false: If you want the search disabled.

4. Click Invoke.
5. Restart the UCMDB server.

Enable/Disable Searching for Federated Data

The enhanced search engine can be configured to perform searches on federated data. By default, it is disabled. To enable searching for federated data, you must set the "federated search enable" flag to True in the JMX Console and in the UCMDB Infrastructure Settings Manager.

Enable or disable searching for federated data (Enhanced Search Engine)

You can enable or disable searching for federated data when the Enhanced Search Engine is enabled, in the JMX Console or in the UCMDB Infrastructure Settings Manager.

Enable or disable searching for federated data in the JMX console:

1. Go to JMX Console > UCMDB:service=Settings Services > setSettingValue.
2. In the name field enter cmdb.federation.search.enabled.

3. In the value field enter:

   true: If you want to enable searching federated data.

   false: If you want to disable searching federated data.

4. Click Invoke.
5. Restart the UCMDB server.

Enable or disable searching for federated data in the UCMDB Infrastructure Settings Manager:

1. In UCMDB, go to Administration > Infrastructure Settings Manager, and choose General Settings.
2. In the Name column, choose Federation in Search Engine Enabled.

3. In the Value column, choose True or False from the drop-down list.

4. Click Save .

Enable or disable searching for federated data in the UCMDB Infrastructure Settings Manager (Legacy Search Engine and widgets)

Follow the instructions in the section "Specify Data stores Used for Data Loading" in the Administer section of the UCMDB Help.

Enable searching of federated CIs by CI display name

To search for a federated CI by the CI display name (or sub-string of the display name), the CI type must be assigned the qualifier CMS_BROWSER_SEARCH in the CI Type Manager.

Specific Configuration Options

Configure Search by Data

In the Search_Ranking_Configuration_XML file, you can associate attributes of a CI to a data field according to the data field priority. This enables a search of the values of those CI attributes and affects the order of CIs returned in the search results.

Configure Search by Topology

In the Search_Parser_Configuration_XML file, you can configure relationship synonyms that associate phrases with class types. For example, in the example below, the phrase "owned by" is associated with the class type "person".

<relationship>
   <synonym>owned by</synonym> 
   <className>person</className>
</relationship> 

As a result of the above configuration, if the user enters the search query "all windows owned by John", the search engine looks for all windows CIs that are connected to a CI of type 'person' with the name 'John'.

Configure Search by Path

In the Search_Parser_Configuration_XML file, you can define compound classes by inserting a list of class types. In the example below, all the class types in the <string> tag make up the compound class, which triggers the search by path. If this list is empty, no search by path is initiated.

You can change the number of linked paths through which you want to search, starting from the original search term. To do this, go to the Infrastructure Settings Manager in UCMDB and change the Search Engine Compound Depth setting to the value you want. For details, see "Infrastructure Settings Manager" in the Administer section of the UCMDB Help.

<compoundClasses>
   <string>msdomain</string>
   <string>person</string>
   <string>osuser</string>
   <string>location</string>
   <string>business_element</string>
   <string>cluster</string> 
   <string>party</string>
</compoundClasses>

Configure Search by Cardinality Condition

In the Search_Parser_Configuration_XML file, you can define a set of cardinality synonyms that associate phrases with cardinality types. For example, if you define

<cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="with at least"/>

and then a user enters the search query "all NT with at least 2 CPUs", the search engine looks for all CIs with class type 'NT' that have two or more CIs of type CPU related to them.

The out-of-the-box cardinality synonyms are as follows:

<cardinalitySynonyms>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="min"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="with minimum"/>
   <cardinalitySynonym cardinalityType="EQUAL" cardinalityPhrase="with"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="max"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="with maximum"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="minimum"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="before"/>
   <cardinalitySynonym cardinalityType="MAX_EXCLUDED" cardinalityPhrase="with less than"/>
   <cardinalitySynonym cardinalityType="EQUAL" cardinalityPhrase="with exactly"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="with at least"/>
   <cardinalitySynonym cardinalityType="MIN_EXCLUDED" cardinalityPhrase="with more then"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="since"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="from"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="at least"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="till"/>
   <cardinalitySynonym cardinalityType="NOT" cardinalityPhrase="without"/>
   <cardinalitySynonym cardinalityType="MIN_EXCLUDED" cardinalityPhrase="with more than"/>
   <cardinalitySynonym cardinalityType="MIN" cardinalityPhrase="after"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="with at most"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="maximum"/>
   <cardinalitySynonym cardinalityType="MAX_EXCLUDED" cardinalityPhrase="with less then"/>
   <cardinalitySynonym cardinalityType="MAX" cardinalityPhrase="at most"/>
</cardinalitySynonyms>

Configure Search by Property Condition

This search is based on attribute synonyms. In the Search_Parser_Configuration_XML file, you can define synonyms for the CI attribute names that are defined in UCMDB. In the following example, the CI attribute display_label has four different synonyms defined for it:

<attributeSynonym>
   <synonym>name</synonym>
   <attributes>
      <attributeName>display_label</attributeName>
   </attributes>
</attributeSynonym>
<attributeSynonym>
   <synonym>display label</synonym>
   <attributes>
      <attributeName>display_label</attributeName>
   </attributes>
</attributeSynonym>
<attributeSynonym>
   <synonym>display</synonym>
   <attributes>
      <attributeName>display_label</attributeName>
   </attributes>
</attributeSynonym>
<attributeSynonym>
   <synonym>label</synonym>
   <attributes>
      <attributeName>display_label</attributeName>
   </attributes>
</attributeSynonym>

Another type of search by property condition occurs when a cardinality synonym is followed by a number, which can also include a unit. For example, if the user enters the search input text windows with at least 4GB memory, since memory is a synonym for the attribute types memory_size and nt_physical_memory as shown in the code example below, this triggers a search by property condition.

<attributeSynonym>
   <synonym>memory</synonym>
   <attributes>
      <attributeName>memory_size</attributeName>
      <attributeName>nt_physicalmemory</attributeName>
   </attributes>
</attributeSynonym>
.
.
.
<unit>
   <multiplyBy>1073741824</multiplyBy>
   <synonym>gigabytes</synonym>
   <synonym>g</synonym>
   <synonym>giga</synonym>
   <synonym>gb</synonym>
</unit>

You can add additional synonyms for cardinality phrases, synonyms for attribute names, and synonyms for unit types in order to refine this type of search. In the example above, the memory size attribute is stored in bytes, and in order to search by gigabytes, the search engine uses the conversion defined in the <multiplyBy> tag (this definition is provided out-of-the-box). In addition to the conversion for gigabytes, out-of-the-box conversions are also provided for megabytes and kilobytes.

Configure Search by Category

In the Search_Parser_Configuration_XML file, you can specify labels to be used when searching for CIs. In the following example, the labels "category" and "color" enable you to search for the strings “all windows color green” and “all servers category editable”.

<categoryLabelSynonyms>
    <categoryLabelSynonym>category</categoryLabelSynonym>
</categoryLabelSynonyms>
<categoryColorSynonyms>
    <categoryColorSynonym>color</categoryColorSynonym>
</categoryColorSynonyms> 

Configure Class Name Conditions

In the Search_Parser_Configuration_XML file, you can create synonyms for class names. In the following example, the synonym "windows" was created for the class name "nt".

<classSynonym>
   <synonym>windows</synonym>
   <className>nt</className>
</classSynonym>

Configure String Replacement

In the Search_Parser_Configuration_XML file, you can configure string replacements. The code example below enables the search engine to retrieve all nodes that have the attribute Host is virtual set to True. The CI property 'host_isvirtual' is a boolean property, and is true if the node is virtual, but the user does not need to know this and this string replacement converts his search query without the user knowing these details.

<replace>
   <from>virtual node</from>
   <to>node host_isvirtual true</to>
</replace>

String replacement can be implemented using the '%' wildcard character. With the following code example, if the user enters the search query "linux nodes" or "linux machine", the search engine converts this to "nodes version linux" and “machine version linux”, respectively.

<replace>
   <from>linux %1</from>
   <to>%1 version linux</to>
</replace>

Configure Enriching Capability

The enriching mechanism executes enriching queries over the search results. Enriching queries are defined in two ways, one is user-defined and the other is based on existing folding rules.

The enriching mechanism reviews the CIs in the search results and extends the results by applying each one of the enriching queries as a perspective over the CI. The enriching process is executed repeatedly, where after each execution search results are received and then enriched again by the enriching TQLs, which in turn produces more search results. By default, this process is repeated 3 times, but this can be changed by means of the cmdb.search.enriching.depth parameter. For instructions on configuring this parameter, see Configure Repetition of the Enriching Mechanism.

User-Defined Enriching Queries

To create user-defined enriching queries:

1. In UCMDB, go to Modeling > Modeling Studio and click New .
2. From the drop-down list, select Query. The Query Definition editor opens.

3. Click Query Definition Properties . The Query Definition Properties dialog box opens.

4. In the Type field, select Perspective from the drop-down list.
5. In the Bundles field, click Select Bundles .
6. From the Bundles list, select search_result_enriching_tqls and click OK.
7. Define the query according to your system needs.
8. Define one of the nodes of the query as Set as Contact Query Node. The query definition is similar to the perspective definition and therefore should include at least one contact node, where the contact node is the CI that should be enriched.

There are also two out-of-the-box enriching queries provided in the Modeling Studio. You can view and edit them as follows:

1. Go to Modeling > Modeling Studio, and select the Resources tab.
2. From the Resources Type drop-down list, select Queries.
3. In the list of queries, go to Search Engine > Enriching TQLs.
4. Under Enriching TQLs, choose one of the enriching queries. You can view it and edit it according to you system needs.

Enriching Queries Based on Existing Folding Rules

As mentioned above, in addition to the user-defined enriching queries, there is an existing set of predefined enriching queries based on folding rules.

To view, add, or edit enriching queries based on folding rules:

1. Go to Modeling > CI Type Manager.
2. From the CI Types drop-down list, choose Calculated Relationships.
3. In the list of Calculated Links, select Folding Rules (Configuration Manager).

4. Select the Triplets tab on the upper right. The list of Triplets appears.

   From each one of the listed triplets there is an automatic mechanism that creates at run-time an enriching query. It enriches the Target CI type by the Source CI type. In this case, the Target CI type is the contact node of the query.

5. To add a new triplet, click Add . If you select the forward relationship direction (from Source to Target), the Target CI type is enriched by the Source CI type. If you select the backward relationship direction (from Target to Source), the Source CI type is enriched by the Target CI type.

6. To edit a triplet, select the triplet in the list and click .

Configure Repetition of the Enriching Mechanism

To configure the number of times that enriching is performed on search results:

1. Go to JMX Console > UCMDB:service=Settings Services > setSettingValue.
2. In the name field enter cmdb.search.enriching.depth.

3. In the value field enter the number of times that you want enriching to be repeated on search results.

4. Click Invoke.
5. Restart the UCMDB server.

Configure Redundant Words

There is a set of words that the search engine ignores when parsing the search query. To configure this list, go to the Search_Parser_Configuration_XML file and go to the <blacklist> section. Add to or edit the list as needed.

The current list of redundant words is a follows:

<blacklist>
   <blacklistItem>to</blacklistItem>
   <blacklistItem>such as</blacklistItem>
   <blacklistItem>with</blacklistItem>
   <blacklistItem>for</blacklistItem>
   <blacklistItem>or</blacklistItem>
   <blacklistItem>a</blacklistItem>
   <blacklistItem>the</blacklistItem>
   <blacklistItem>like</blacklistItem>
   <blacklistItem>in</blacklistItem>
   <blacklistItem>and</blacklistItem>
   <blacklistItem>of</blacklistItem>
   <blacklistItem>if</blacklistItem>
   <blacklistItem>an</blacklistItem>
   <blacklistItem>on</blacklistItem>
   <blacklistItem>linked</blacklistItem>
   <blacklistItem>all</blacklistItem>
</blacklist>

Configure Search of CIs Changed in a Specific Time Period

In the Search_Parser_Configuration_XML file, there are a sets of words defined as month-name synonyms and day-name synonyms. For example:

<monthSynonym>
    <monthName>january</monthName>
    <monthNumber>0</monthNumber>
</monthSynonym>
<monthSynonym>
    <monthName>jan</monthName>
    <monthNumber>0</monthNumber>
</monthSynonym>
.
.
.
<daySynonym>
    <dayName>sunday</dayName>
    <dayNumber>1</dayNumber>
</daySynonym>
<daySynonym>
    <dayName>sun</dayName>
    <dayNumber>1</dayNumber>
</daySynonym>

Values in the tags <monthNumber> and <dayNumber> should not be modified. These configurations allow support of search queries like “all windows created last Sunday” and “all nodes changes between August 8th 1980 and 25/5/2011”.

Configure Date Format

The search engine supports two dates formats: day-month-year (DMY) and month-day-year (MDY), which can be configured as follows:

1. Go to JMX Console > UCMDB:service=Settings Services > setSettingValue.
2. In the name field enter: cmdb.search.date.format.
3. In the value field enter the desired date format: DMY, MDY, or both.
4. Click Invoke.
5. Restart the UCMDB server.

Configure Autocompletion

In the Search_Autocompletion_Configuration_XML file, you can expand the CMS UI's search capabilities by configuring the UCMDB server to suggest matches for substrings of words in a search, including CI attribute values.

1. Go to JMX Console > UCMDB:service=URM Services > listResourceTypes and click Invoke.

2. Click Topology_AUTOCOMPLETION_CONFIGURATION.

3. Click Search_Autocompletion_Configuration_XML.

4. Enter the fields of classes that you want to index for autocompletion. Since the class name supports inheritance, you can enter the following:

   <supportedAutocompletionFields>
       <autocompletionField>
           <className>managed_object</className>
           <attributeName>display_label</attributeName>
       </autocompletionField>
   </supportedAutocompletionFields>
   

   or, for example:

   <supportedAutocompletionFields>
       <autocompletionField>
           <className>nt</className>
           <attributeName>display_label</attributeName>
       </autocompletionField>
       <autocompletionField>
           <className>unix</className>
           <attributeName>display_label</attributeName>
       </autocompletionField>
   </supportedAutocompletionFields>
   

5. Click Save resource.

Configure Query Search Filter

By default, the search algorithm filters out the queries that are too general. If you wish to turn off the filter, follow these steps:

1. Go to JMX Console > UCMDB:service=Settings Services > setSettingValue.
2. In the name field, enter cmdb.search.filter.queries.

3. In the value field, enter false.

   Note Enter true if you want to turn on the filter.

4. Click Invoke.

Perform a Full Reindex of All CIs

Use either of the following approaches to perform a full reindex of all CIs available for search in the CMS UI:

• Go to JMX Console > UCMDB:service=Topology Search Services and invoke the reindex() method.

  This method removes all CIs information stored inside the SOLR index files without removing any internal SOLR files.

  Note You can also invoke the reindexCiType() method to re-index all the CIs of a given CI type from the CMDB model database. For more information, refer to the JMX Reference section of the UCMDB Help.

• Stop the UCMDB server, delete the <UCMDB_Server_Home>/search folder, and then start the UCMDB server.

  The deletion of the folder forces a full reindex for all UCMDB customers and reverts the SOLR configuration to the out of the box values. This process removes all internal SOLR files and hence takes longer time than the previous one.

Troubleshooting - Configure the Enhanced Search Engine

UCMDB doesn’t start because of the search subsystem manager. What do I do?

Stop UCMDB, delete the folder <UCMDB installation folder>/search, then restart UCMDB.

If the search still does not start, disable it, as explained in Enable/Disable the Enhanced Search Engine and revert to the legacy search engine.

The search doesn’t return any results.

In the Topology Search JMX, invoke the following methods:

• restoreFactoryDefaults: This restores factory configuration for the search.

• reindex: This recreates a search index for CIs in the UCMDB model. Note, this can take up to several hours for large databases (approx 1M CIs/hour).

  You can also invoke the reindexCiType method to re-index all the CIs of a given CI type from the CMDB model database.

The search doesn't find CI types that I want.

There are several different possible causes for this. Check the following:

• Check that the attribute and CI type are indexable according to the indexing configuration. If they are not, add the class attributes configuration item as explained Modify the currently indexed list.
• Check that you have correct synonyms defined for the class in Class synonyms.
• Check that rating and pageItemCount for this CI are non-zero. Check for rating in the Attribute ranking and for pageItemCount in Presentable CI types.

Cardinality conditions don't work or return incorrect results.

In addition to checking attribute synonyms, check that the attribute type is defined as numeric in Indexing Configuration and that units configuration matches attribute units in Search_Parser_Configuration_XML.

The search presents too many unwanted results.

• Check if you are you using queries with natural language. This can limit results of the "best guess" of what the user intended.
• If you need to be 100% certain that your query returns results only of one specific CI type, use type: ci-type filter in the query.
• If the two suggestions above don't help, contact the R&D team with your use case and status report from JMX.

Problem with configuration - restore factory defaults

To restore the default configuration XML files from the factory content, go to JMX Console > UCMDB:service=Topology Search Services and invoke the restoreFactoryDefaults() method.

Logs and debugging info

Logs

search.log logs everything related to searches. Default log level is INFO, only statistics are printed. The log level and number of logs are configured with the search.loglevel variable in conf/log/cmdb.properties.

Status Report

The topology search JMX status report displays all current configuration tables and statistics for the search engine component. It is useful to include it when reporting issues to R&D.

Content of Solr Database

By default, the Solr search engine is embedded inside UCMDB server. To query it directly, go to JMX Console > UCMDB:service=Topology Search Services and invoke the debugSolrQuery() method.

Example queries:

• empty query returns all CIs

• "id:a6693cd46cfd1b4fab0c3551bac9289e" returns a CI with cmdbId a6693cd46cfd1b4fab0c3551bac9289e. This uses Solr/Lucene syntax.