Monitor log information for availability related messages

Service Manager writes configurable alert messages to the log file for availability monitoring. For more information, see System alerts.

There are two kinds of memory related error messages:

• Out of memory error messages: A session requests memory inside one of the memory pools; however, as resources are exhausted, the request cannot be served. This causes the RAD thread, the session, or the entire servlet to fail immediately.

  Shared memory

  RAD stack

  Java Heap

  Process Native Heap

• Proactive memory related messages. These messages are either pure informational or related to proactive mechanisms to prevent out of memory situations.

  Shared memory

  RAD stack

  Java Heap

  Process Native Heap

For more information, see Monitoring native heap and Java heap memory.

Resource exhaustion problems include the following:

• Shared memory resources exhausted

  When a block of shared memory is requested, there is no more shared memory available, the session cannot continue and terminates. The sm.log file contains messages about the failed allocation and a shared memory report.

  For more information, see Special parameters: reportshm.

• Rad stack exhausted

  The Rad Stack is required by System Language and resides in process private memory. Each RAD thread (that relates to open tabs in the Service Manager Windows or Web client) maintains a RAD stack. The agstackl parameter controls the number of stack frames.

  When the Rad Stack is exhausted, the RAD thread is terminated. The cause is most probably a recursive implementation. Increasing the number of stack frames typically only causes the issue to appear less frequently, while removing the recursive implementation is required for a permanent fix.

  The log file is showing a message of the failed request and an RAD application stack trace.

  For more information, see Startup parameter: agstackl.

• Process private memory exhausted

  When a RAD application requests a block of memory, which cannot be served, the session terminates with RAD application stack trace.

  When a JavaScript function running out of memory, the function is not reaching the return statement, so the returned value is null. A message is printed to the log file, including JavaScript and RAD stack. The session, however, is not terminated.

  Example:

  JS W Print ScriptTrace for OutOfMemory
  JS W MemWaster@'MemWaster':2,
  RAD W display:process.option:1,se.view.engine:display.go,se.list.engine:call.view.engine,se.search.engine:call.list.engine,database.engine:call.search.engine.1,database:call.database.engine,
  

  For more information, see the following help topics:

  Programming Guide

  Parameter: memorypollinterval

  Parameter: log4jDebug

  Monitoring memory in Service Manager processes

  Monitoring native heap and Java heap memory

• Java Heap exhausted

  When a Java heap memory request cannot be served, the session terminates with java exception stack.

  The java heap is used by the servlet to communicate to clients, integrations, or other servlets. The size of the request or response document is the biggest driver for java heap memory requirements.

  Resolving these issues requires analysis of the actual document and the implementation. A typical cause is that huge requests or response documents are required by integrations. A mechanism implemented to replace huge requests is web services pagination.

  For more information, see Retrieve data from Service Manager using Pagination.

Proactive memory pool messages include the following:

• Shared memory warnings

  When the first process of Service Manager is started on a host, it creates the shared memory. At this time, it compares the configured size in sm.ini with the minimum acceptable value of 8000000 (and automatically adjusts) and the recommended size.

  Example messages:

  RTE W Shared memory size of 4000000 too small, adjusted to 8000000
  RTE W The shared memory size 8000000 is less than the recommended size 134217728
  

• RAD stack warnings

  Service Manager logs warning messages when the RAD stack allocation exceeds a threshold.

  Example:

  RTE I RAD stack is 72% used, please exit out of current application.

• Process private memory warnings

  Service Manager monitors the native heap memory allocation and writes proactive messages to the log file before all resources are exhausted. In addition, the servlet will switch to a low memory mode to avoid total exhaustion and the servlet from completely terminating.

  When a servlet moves into low memory mode and is no more accepting new sessions and application threads for native memory until the garbage collection has released memory again.

  The memorypollinterval parameter may be used to influence the threshold when a servlet switches to low memory mode.

  When the allocation of native memory increases for a session, information about the allocated memory size is printed to the log file.

  Example:

  RTE I -Memory : S(392328) O(165668) MAX(1001004) - MALLOC's Total(4136)

  Here, the MAX value is relevant: Average here is for typical user sessions below 20M; if 40M is exceeded by a single session, investigations should be started. Possible causes are that users open a very large number of tabs in the client, or memory-intensive JavaScript implementations, or other problems.

  More frequent trace information is written to sm.log when trace parameter rtm:2 is turned on.

  Note Process native memory monitoring and the proactive messages are not implemented in Service Manager running on unix operating system.

  For more information, see Avoiding variable declaration inside a loop.

• Java Heap warnings

  Service Manager monitors the native heap memory allocation and writes proactive messages to the log file before all resources are exhausted. In addition, the servlet will switch to a low memory mode to avoid total exhaustion and the servlet from completely terminating.

  When a servlet moved into low memory mode and is no more accepting new sessions and application threads for java heap memory until garbage collection released memory again.

  In order to monitor java heap for low memory, set the following parameter in sm.ini:

  log4jdebug:com.hp.ov.sm.common.oom.LowMemoryHandler

  Example of low memory handler log messages:

  JRTE I Starting Memory Monitoring thread to check for memory every 15 seconds.
  JRTE D JavaMemory Max(123928576) Used(1011872) %Used(0.0)
  JRTE D NativeMemory Max(2147352576) Used(358121472) %Used(16.0)
  JRTE W Process Low on Java Memory. Max(123928576) Used(119603800) PercentUsed(96.0)
  JRTE W Send error response: Server is running low on memory try again.
  JRTE W Send error response: Server is running low on memory try again.
   RTE I Process Java Heap Memory is back to normal range.
  JRTE I Process Java Memory. Max(123928576)  Used(4256640) PercentUsed(3.0)
  

Related concepts

Availability monitoring tasks

Introduction to the Programming guide

Monitoring memory in Service Manager processes

Monitoring native heap and Java heap memory

Related references

Parameter: memorypollinterval

Parameter: log4jDebug

Dbdicts containing log information