Configure Structured Log File Policies

You can create or edit a Structured Log File policy using the Structured Log File policy editor, which you can open in the following ways.

• To open the editor from the Edit Aspect dialog box:

  1. Open the Management Templates & Aspects manager:

     Administration > Monitoring > Management Templates & Aspects

     Alternatively, click Management Templates & Aspects.

  2. In the Configuration Folders pane, expand the configuration folders.

  3. In the Management Templates & Aspects pane, click an aspect, and then click the  Edit Item button.

     The Edit Aspect dialog box opens.

  4. Click the Policy Templates tab, and then do one of the following:

     • To add a new policy template:

       1. Click the Add Policy Templates From List button. The Add Policy Templates From List dialog box opens.

       2. Click the  New Item button, and then click the Add New Policy Template or the Add New Policy Template (Raw Mode) button. The Select Type for New Policy Template dialog box opens.

       3. Select the appropriate type <DataType> from Structured Log File, and then click OK.

     • To edit an existing policy template, click the policy template in the list, click the button, and then click the Edit Policy Template or the Edit Policy Template (Raw Mode) button.

     The Operations Connector High Availability policy editor opens.

     The Structured Log File policy editor opens.

• To open the editor from the Policy Templates manager:

  1. Open the Policy Templates manager:

     Administration > Monitoring > Policy Templates

     Alternatively, click Policy Templates.

  2. In the Policy Template Groups pane, expand Policy Management > Templates grouped by type.

  3. Depending on the type of data collected by the particular policy, expand Configuration, Events, Generic, Metrics, or Topology. Each policy type can be used only for a subset of the available data types.

  4. Click the <DataType> from Structured Log File item in the appropriate folder, and then do one of the following:

     • To add a new policy template, in the Policy Templates pane, click the button, and then click the Add New Policy Template or the Add New Policy Template (Raw Mode) button.

       The New <DataType> from Structured Log File policy editor opens.

     • To edit an existing policy template, click the policy template in the Policy Templates pane, click the button, and then click the Edit Policy Template or the Edit Policy Template (Raw Mode) button.

       The Edit <DataType> from Structured Log File policy editor opens.

• The integrated product is properly configured for integration with OMi. For more information, see the Integrate section.

• The certificate request from the integrated product is approved in OMi.

• The integrated product is configured as a connected server in OMi.

A log file structure is defined by using the OM pattern-matching language, so that the dynamic parts of the text-based events can be extracted from any log file row, assigned to variables, and then used as parameters to build the event description or to set other attributes.

You can use an asterisk (<*>) wildcard in the Log File Path / Name field to match multiple file names. For example, to match the source file names events.1.log and events.2.log, use the pattern <Path>/events<*>.xml in the Log File Path / Name field. Note that the <*> wildcard is the only supported OM pattern in log file paths. For more information on pattern matching, see Pattern-Matching Details.

In addition to defining a log file structure by using the OM pattern-matching language, you can identify a log file structure by using the static fields.

The "Recurring fields" configuration parameter is useful in case when more than one performance value is present within a single log file line. This is actually a word list that contains the recurring part from the log line. Each recurrence creates a record in the store.

Line start indicator enables you to differentiate the structured log file entries based on their logical relationship, regardless of their span in the log file. When the log entry that represents a single logical unit has a span of more than one line in the log file, you can easily differentiate it from other log entries by identifying a line start indicator from a log file, and then specifying the matched line start pattern by using the OM pattern matching language.

For example, the following tomcat.log file excerpt contains four logically separated log entries that span multiple log lines, however all of them start with a timestamp (May 19, 2017 2:39:01 PM):

May 19, 2017 2:39:01 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-30000]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-30000]]
       at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
       at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
       at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
       at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:821)
       at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
       at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
       ... 12 more

May 19, 2017 2:39:01 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 3622 ms
May 19, 2017 2:39:01 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
May 19, 2017 2:39:01 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Example Application

Therefore, in this case, the line start indicator must match the timestamp pattern, as follows:

<*> <#>, <4#> <#>:<#>:<#> <2*>

In this instance, the following applies:

A custom variable consists of a map name, a field extracted from the log file by using the OM pattern matching language, and one or more source and target value pairs. For example, you can assign the pattern matching field host to the map name maphost, and add a source value of critical. You can then assign the target value serious to the variable so that the agent inserts the value critical into the event in all places where the variable is used and the source value is serious in the log file.

Input data references use the following syntax: <$DATA:<InputReferenceField>>

Field values originate from the structured log file pattern settings defined for the data source.

For example, the custom variable maphost has the pattern matching field host assigned.

Assigning a pattern matching field to a map name is optional. If you do not assign a pattern matching field to a variable, you must add the source value directly to the variable when you insert the variable in an event attribute.

The Sample Data tab is empty if no sample data has been loaded into the policy or if the sample data does not match the structured log file pattern.

The Sample Data tab shows the following information, if sample data is available:

• Input Data Properties

  If sample data is available, the Input Data Properties section shows all fields that match the structured log file pattern.

  The items in the Input Data Properties section are by default sorted alphabetically in ascending order.

• Values for <InputDataProperty>

  This section displays the values of a field selected in the Input Data Properties section. If a value appears more than once, click to show or hide duplicate values. To find values that belong to more than one group, select the value and click . The Sample Data window opens and shows all fields that have the selected value.

When you drag a field from the Input Data Properties list and drop it on the Default Value Mapping List, the policy editor automatically adds the default prefix map to the map name and inserts the pattern matching field. You can then drag one or more structured log file source values from the values list and drop them on the Source Value list. You then finally only have to type the target values.

In the Structured Log File policy editor, follow the steps:

1. In the Properties page, define general policy information.
   

   Type the name of the policy in the Name text box.

   Optional. Provide a description of the policy (Description), select the instrumentation that will be deployed with the policy onto the host system where the agent is running (Instrumentation), and select the operating systems with which the policy is compatible (OS Types).

   For more information, see Properties Page.

2. In the Source page, provide full path to the structured log file on the host system of the agent.
   

   • Optional for events and generic output data. In the Line Start Indicator field, provide a line start pattern that matches the line start indicator from the log file.

   • Mandatory for events. In the Logfile Pattern field, compose an OM pattern by which the log file is structured. You can reuse the newly structured data on the Structured Data tab of the Mappings and Defaults pages of the policy editor.

   • Mandatory for metrics and generic data. Additionally, in the Logfile Structure section:

     1. Choose the data field by which the log file structure is identified. You can choose to identify it by using either an OM pattern or static fields.
     2. Define recurring fields in the log file structure.

     3. Enter the data field separator.

     Optional. Load sample log file to OMi. The log file must reside on the system from where your are logged into OMi.

     Note When you load the data, OMi replaces the already loaded data with new data. This action does not affect the mappings that are defined based on previously available sample data.

   Click to load the log file with sample data to the policy editor. Once loaded, a notifications appears in the Sample Data section of the policy editor.

3. In the Mappings page, configure mappings of log file input data properties to custom variables.

   1. Create custom variables.

      If sample data is loaded in the policy editor, drag a field from the Input Data Properties list to the Map Name column. The policy editor automatically adds the default prefix map to the map name and inserts the column name.

      Alternatively, click above the Map Name column and type a variable name in the map name field. Fields are optional. If you do not assign a field to a variable, you must add the source value directly to the variable when you insert the variable in an event attribute.

   4. Add source and target value pairs to each custom variable.

      • If sample data is loaded in the policy editor, drag a value from the Values for '...' list to the Source Value column, and then type the target value in the corresponding field.

        Alternatively, click above the Source Value column and type the source and target values in the corresponding fields.

      • Optional. In the Indicators tab, add indicators to the source or target value fields. After loading the indicators from the OMi server, the Indicators tab shows a hierarchy of configuration item types with the associated health indicators (HIs) and event type indicators (ETIs).

        To insert an indicator in a source or target value field, drag the indicator state (for example, HTTPServer:Normal) from the Indicators tab and drop it on the corresponding field.

      • Optional. In the Policy Variables tab, add policy variables to data attributes. The agent replaces the variables with the appropriate values in the generated data.

        HPE recommends to surround variables with quotation marks, for example "<$MSG_NODE>" or "<$MSG_GEN_NODE>", at least for those variables whose values can contain space characters.

4. In the Schema page, configure mappings for key fields for the generic output data policies.

   1. Create key field mappings.

      If sample data is loaded in the policy editor, drag an input data property from the Meta Data list to the Input Field Qualifier column. The policy editor automatically extracts the eligible field name.

      Alternatively, click above the Input Field Qualifier column and type the qualifier name in the corresponding field.

   2. Optional. Change the Keep All Input Fields setting. By default, the option is selected and all fields are kept, regardless whether they are mapped or not. To keep only the mapped fields, clear the option.

   3. Add any additional fields as required.
5. Optional for events. In the Defaults page, configure default settings for all events collected by the policy.

   Note The default values configured in this page:

   • Apply only to a subset of event attributes. You can configure defaults of other attributes within individual policy rules.

   • Affect new as well as already existing rules. You can override the defaults in individual policy rules. If a rule contains empty event attributes, the agent will use the defaults for incoming events.

   1. Click Event Attributes to define default event attributes, such as severity and category.

      In the strings that define event defaults, you can use indicators, policy variables, and policy parameters.

      Tip After loading the indicators from the connected OMi server, the Indicators tab shows a hierarchy of configuration item types.

      To insert an indicator, drag the indicator with its state from the Indicators tab to the policy.

      To insert an indicator, drag the indicator from the Indicators tab to the policy. The policy editor inserts the indicator name.

   2. Click Event Correlation to set the type of duplicate event suppression and define the method used to suppress duplicate events.

   3. Click Custom Attributes to add additional information to all events generated by this policy. For example, you might add a company name, contact information, or a geographical location to an event.

   4. Click Advanced to define default advanced attributes such as legacy OM attributes and agent MSI (Message Stream Interface) settings.

   5. Optional. Use the Sample Data tab to drag the input data references to the attribute boxes. Alternatively, you can type the reference directly into the attribute box.

      Input data references use the following syntax: <$DATA:<InputReferenceField>>.

      Field names originate from the structured log file pattern settings defined for the data source.

      The agent replaces the pattern matching field at runtime with the value of the specified field. If you insert a field value, the value will be used.

      Note The Sample Data tab is empty if no sample data has been loaded into the policy or if the sample data does not match the structured log file pattern specified in the source page.

   6. Optional. Use the Mappings tab to add mapping definitions to the attribute boxes.

      Mappings are custom variables that you define in the Mappings page.

      Alternatively, type the custom variable into the attribute box using the following syntax:

      <$MAP(<CustomVariable>)> where <CustomVariable> is the map name of the variable (for example, <$MAP(maphost)>).

      If the custom variable does not have a input data reference assigned, use the following syntax:

      <$MAP(<CustomVariable>,<SourceValue>) where <SourceValue> can be one of the following:

      • Name of the input data reference, for example <$MAP(maphost)>,<$DATA:host>

      • The source value itself, for example <$MAP(maphost,Critical)>

   7. Optional. Use the Operators tab to apply operators to the attribute values. Two functions are available:

      $MATCH(), to test a string or a variable against a pattern. The $MATCH function accepts three or four parameters:

      - the input string

      - the pattern definition

      - the output string if pattern matches on the input string

      - the output string if the pattern does not match (optional)

      Example: The data of the input field hostname start always with "TEST" (for example "TESTABC"). The $MATCH function to use the string after "TEST" is as follows:

      $MATCH(<$DATA:hostname>,TEST<*.prefix>,<prefix>)

      • <$DATETIME(FORMAT,VALUE)>, to convert the format of dates from the common format to the UNIX systems time (Epoch time) format.

      Note To apply operators to the attribute values, you can drag and drop them to a text field in the left pane of the same policy editor page. The appropriate tooltips are shown while performing this operation, which describe the role of the dragged operator.

   8. Optional. Use the Indicators tab to add indicators to the source or target value fields. After loading the indicators from the connected OMi server, the Indicators tab shows a hierarchy of configuration item types with the associated health indicators (HIs) and event type indicators (ETIs).

   9. Optional. In the Policy Variables tab, add policy variables to data attributes. The agent replaces the variables with the appropriate values in the generated data.

      HPE recommends to surround variables with quotation marks, for example "<$MSG_NODE>" or "<$MSG_GEN_NODE>", at least for those variables whose values can contain space characters.

6. Optional for metrics. In the Defaults page, configure default settings for all metrics collected by the policy.

   Note You can use the default values configured in this page when defining the policy rules on the Rules page, where you can also override them.

   1. Define the metric attributes common to all metrics collected by this policy, such as metric class and name. All metrics in the Basic tab marked with a * are required. Advanced attributes are optional.

   2. Optional. Use the Sample Data tab to drag pattern matching groups and values to the attribute boxes. Alternatively, you can type the pattern matching group directly into the attribute box.

      Pattern matching groups use the following syntax: $<PatternMatchingGroup>

      <PatternMatchingGroup> is the name of the group assigned to a particular content match.

      The agent replaces the pattern matching group at runtime with the value of the specified group. If you insert a group value, the value will be used.

      Note  

      • The default pattern matching group host_name contains the fully qualified domain name of the host system of the agent.

      • The Sample Data tab is empty if no sample data has been loaded into the policy or if the sample data does not match the log file pattern specified in the source page.

   3. Optional. Use the Mappings tab to add mapping definitions to the attribute boxes.

      Mappings are custom variables that you define in the Mappings page.

      Alternatively, type the custom variable into the attribute box using the following syntax:

      <$MAP(<CustomVariable>)> where <CustomVariable> is the map name of the variable (for example, <$MAP(maphost)>).

      If the custom variable does not have a input data reference assigned, use the following syntax:

      <$MAP(<CustomVariable>,<SourceValue>) where <SourceValue> can be one of the following:

      • Name of the input data reference, for example <$MAP(maphost)>,<$DATA:host>

      • The source value itself, for example <$MAP(maphost,Critical)>

   4. Optional. Use the Operators tab to apply operators to the attribute values. Two functions are available:

      $MATCH(), to test a string or a variable against a pattern. The $MATCH function accepts three or four parameters:

      - the input string

      - the pattern definition

      - the output string if pattern matches on the input string

      - the output string if the pattern does not match (optional)

      Example: The data of the input field hostname start always with "TEST" (for example "TESTABC"). The $MATCH function to use the string after "TEST" is as follows:

      $MATCH(<$DATA:hostname>,TEST<*.prefix>,<prefix>)

      • <$DATETIME(FORMAT,VALUE)>, to convert the format of dates from the common format to the UNIX systems time (Epoch time) format.

      Note To apply operators to the attribute values, you can drag and drop them to a text field in the left pane of the same policy editor page. The appropriate tooltips are shown while performing this operation, which describe the role of the dragged operator.

   5. Optional. Use the Indicators tab to add indicators to the source or target value fields. After loading the indicators from the connected OMi server, the Indicators tab shows a hierarchy of configuration item types with the associated health indicators (HIs) and event type indicators (ETIs).

   6. Optional. In the Policy Variables tab, add policy variables to data attributes. The agent replaces the variables with the appropriate values in the generated data.

      HPE recommends to surround variables with quotation marks, for example "<$MSG_NODE>" or "<$MSG_GEN_NODE>", at least for those variables whose values can contain space characters.

7. In the events and metrics policies, in the Rules page, define policy rules.

   Events policies:

   Rules define the action a policy should take in response to a specific type of incoming event. Each rule consists of the following:

   • A condition for the incoming data

     The condition is the part of a policy that describes the data source.

   • Settings for the outgoing event

     The settings define the actual event data that the agent sends to OMi.

   A policy must contain at least one rule. If the policy contains multiple rules, they are evaluated consecutively. After the condition is matched in one rule, rule evaluation stops.

   Metrics policies:

   Rules define the action a policy should take in response to a specific type of incoming metric. Each rule consists of the following:

   • A condition for the incoming data

     The condition is the part of a policy that describes the data source.

   • Settings for the outgoing event

     The settings define the actual metric data that the agent sends to OMi.

   A policy must contain at least one rule. If the policy contains multiple rules, they are evaluated consecutively. After the condition is matched in one rule, rule evaluation stops.

   To define policy rules, follow the steps:

   1. In the Policy Rules list, click the button, and then select an appropriate value. For more information, see Policy Rules List.

   2. Click Rule Description and type a brief description of the rule.

   3. In the Rule Content section, use the Condition tab to define values that you want to evaluate against elements and attributes in the XML file.

      1. Click to create a new condition. New conditions by default use the equals operator.

      2. Click to expand the new condition.

      3. In the Property field, specify the input data property that the policy searches for.

         If you are working with sample data, you can drag and drop the input data property from the Input Data Properties list to the Property field.

      4. Select the pattern operator.

         If you select the matches operator, you can type a pattern in the Operand field.

      5. In the Operand field, type the value or pattern that you want the policy to compare with the XML property. If sample data is loaded in the policy editor, you can drag the value from the Values list and drop it in the Operand field.

         Tip For matching patterns, you can use standard pattern-matching rules of the agent. Select the matches operator and click the icon in the Operand field to open the pattern matching toolbox window. The toolbox includes the following sections:

         • Pattern Matching Expressions. Click an expression to insert it into the Operand text box.

         • Variable Bindings Options. Variable binding options include the setting of case sensitivity check and the field separators used in the rule. If you do not specify the pattern matching options for the rule, either the defaults (enabled case sensitivity check; the space and the tab character as the separators) or the default options set for the policy are used.

      For further details, see Condition Tab.

   4. Optional. In the Rule Content section, click any of the tabs except the Condition tab, and then click the Pattern Matching Variables tab to add variables created through pattern matching.

      Pattern matching variables insert matched strings that have previously been assigned to variables. To insert a pattern matching variable, type the variable name enclosed in angle brackets (for example, <VariableName>) or drag it from the Pattern Matching Variables list to the event or metric attribute.

   5. Optional for events. If you are creating a rule of the "Event on matched rule" type, set attributes for events that you want the policy to send. You can override the default event attributes here. You can also enter instructions that help operators handle the associated event (or add the name of the instruction text interface in order to retrieve event instructions from an external source), and configure actions to solve problems automatically or manually.

      In text boxes, you can specify sample data, mappings, pattern-matching variables, indicators, policy variables, and policy parameters.

      For details, see Event Attributes Tab, Event Correlation Tab, Custom Attributes Tab, Instructions Tab, Advanced Tab (Default Event Attributes), and Actions Tab.

   6. Optional for metrics. If you are creating a rule of the "Store on matched rule" type, set attributes for metric that you want the policy to store. You can override the default metric attributes here.

      In text boxes, you can specify sample data, mappings, pattern-matching variables, indicators, policy variables, and policy parameters.

      For details, see Basic Tab and Advanced Tab (Default Metric Attributes).

8. Optional for events and metrics. In the Options page, set additional policy options.
   

   In this page, you can:

   • Configure which rule matching actions are logged by the policy

   • Define how the integrated agent behaves on unmatched events or metrics

   • Set the pattern matching options

   For details, see Options Page.

9. Click Save and Close to save the policy template and exit the dialog.

The attached .zip file is an example of how multi-line log files for events are processed. The readme.txt includes details about how to install the policy on an OMi server and on Operations Connector.

EventFromStructuredLogfile.zip

The ITOM Marketplace provides policy template examples for different policy types. You can use the Rapid Demo Generator to generate event, metric, and topology demo data: Rapid Demo Generator

The Defaults page enables you to set default attributes for all events that a policy sends. The event defaults only affect new rules. You can override the defaults for individual rules.

For details, see Event Attributes Tab, Event Correlation Tab, Custom Attributes Tab, Instructions Tab, and Advanced Tab (Default Event Attributes), and Actions Tab.

The Defaults page enables you to set default attributes for all metrics that a policy sends. The metric defaults only affect new rules. You can override the defaults for individual rules.

For details, see Basic Tab and Advanced Tab (Default Metric Attributes).

Specify if you want the instructions to accompany the event.

Events generated by a policy can include instructions that explain what to do when the event is generated. This instruction text can often help an operator to solve a problem when a particular type of event is received. The operator can view the instructions included with an event by checking the Event Details pane in the Event Browser. You can define default instructions for all rules in a policy. You can also override the default with different instructions for any rule.

The Rules page enables you to define one or more policy rules.

For details, see Policy Rules List, Condition Tab, Event Attributes Tab, Event Correlation Tab, Custom Attributes Tab, Instructions Tab, Advanced Tab (Default Event Attributes), and Actions Tab.

The Rules page enables you to define one or more policy rules.

For details, see Policy Rules List, Condition Tab, Basic Tab, and Advanced Tab (Default Metric Attributes).