Administer > Monitoring > Log Streaming

Log Streaming

Log Streaming enables you to collect structured logs from multiple applications and forward them to the required targets. In particular, log streaming enables you to collect structured logs and configure them for consumption by Operations Bridge Analytics (OBA), all while using Operations Bridge technologies. This way, you can benefit from the central configuration of log collection and normalization (via Operations Agent policies), and use OBA to perform log search, log analytics, and anomaly detection on the logs streamed by the Operations Agent.

For specific details on the OBA integration, see the OBA Help Center.

The following is the list of products or components required for Log streaming and Log analytics:

Products or Components Versions
OMi 10.61, 10.62
Operations Agent 12.03, 12.04
OBA 3.01, 3.02
OMi Management Pack for Infrastructure (Content) 2.02

Example of Log Analytics

In the IT environment, system administrators have the task to find the root cause and proactively manage and solve simple and complex IT operations problems. The best solution is to enable log search for quicker and easier root-cause identification and to find a problem, manage logs, and add analytics to derive the best results.

Log streaming capability provided by OMi and Operations Agent provides a robust solution to manage the logs in the IT environment. You can use this capability by integrating OBA with OMi and enable analytics for the required data.

An OMi user can easily configure Log Streaming on an application log on a remote Windows or Linux server by using Operations Agent. The first step is to configure a structured log file policy to collect the log data. This policy defines rules for collecting logs. These logs are extracted from the log file when a log file entry matches the fields defined in this policy by using the defined macros and mappings configuration.

Then, configure the data to push the logs to OBA for Analytics. Operations Agent ( collector) is installed on OBA server as well as on the node from which the logs will be collected and pushed to OBA.

OBA helps you to find the results and interesting data trends to monitor your environment better.

The following image depicts the flow to complete the tasks for Log streaming and Analytics:

To configure log streaming in your environment, you must configure the following policies

  • Generic Output from Structured Log File policy – This policy defines rules for collecting logs. Logs are extracted from the log file when a log file entry matches the fields defined in this policy by using the OM pattern-matching language. The extracted data is then normalized by applying the default values and rules defined in the policy. The structured log file policy supports similar and multiple pattern matching.
  • Data Forwarding policy – This policy configures targets that receive logs and defines rules for forwarding and discarding the logs.

Note Before setting up the policies, you should be:

  • Familiar with the format and syntax of the log file that the policy reads.
  • Clear about the purpose and usage of the forwarded logs.

After configuring the policies, you must deploy them to the monitored node to complete the log streaming configuration.

This section covers the following topics:

Note We recommend that you use the Log Streaming feature within the supported scale limits. The maximum supported throughput for each instance of Operations Agent is 700 KiB/minute. For example, a log file with approximately 3098 lines per minute, each line containing maximum 230 characters.

Tasks