Monitor log information for availability related messages

Search for	Example	Results
A single word	`cat`	Topics that contain the word "cat". You will also find its grammatical variations, such as "cats".
A phrase. You can specify that the search results contain a specific phrase.	`"cat food"` (quotation marks)	Topics that contain the literal phrase "cat food" and all its grammatical variations. Without the quotation marks, the query is equivalent to specifying an OR operator, which finds topics with one of the individual words instead of the phrase.

Search for	Operator	Example
Two or more words in the same topic	`AND` `and` `+` (plus symbol) `&` (ampersand)	`cat AND dog` `"cat food"+milk` `"cat food"&"dog food"`
Either word in a topic	`OR` `or` `\|` (pipe)	`cat OR dog` `cat \| dog`
Topics that do not contain a specific word or phrase	`NOT` `not` `!` (exclamation point)	`NOT cat` `! dog`
Topics that contain one string and do not contain another	`^` (caret)	`cat ^ mouse`
A combination of search types	`( )` parentheses	`cat + (dog \| mouse)` `cat \| dog + (! mouse)`

Monitor the server log for availability related messages

Monitor the following types of messages:

Alert messages

Error messages about exhausted resources

Proactive Memory pool messages

Error messages about exhausted resources

There are two kinds of memory related error messages:

Out of memory error messages: A session requests memory inside one of the memory pools; however, as resources are exhausted, the request cannot be served. This causes the RAD thread, the session, or the entire servlet to fail immediately.

Shared memory

RAD stack

Java Heap

Process Native Heap

Proactive memory related messages. These messages are either pure informational or related to proactive mechanisms to prevent out of memory situations.

Shared memory

RAD stack

Java Heap

Process Native Heap

Resource exhaustion problems include the following:

Shared memory resources exhausted

When a block of shared memory is requested, there is no more shared memory available, the session cannot continue and terminates. The sm.log file contains messages about the failed allocation and a shared memory report.

For more information, see Special parameters: reportshm.

Rad stack exhausted

The Rad Stack is required by System Language and resides in process private memory. Each RAD thread (that relates to open tabs in the Service Manager Windows or Web client) maintains a RAD stack. The agstackl parameter controls the number of stack frames.

When the Rad Stack is exhausted, the RAD thread is terminated. The cause is most probably a recursive implementation. Increasing the number of stack frames typically only causes the issue to appear less frequently, while removing the recursive implementation is required for a permanent fix.

The log file is showing a message of the failed request and an RAD application stack trace.

For more information, see Startup parameter: agstackl.

Process private memory exhausted

When a RAD application requests a block of memory, which cannot be served, the session terminates with RAD application stack trace.

When a JavaScript function running out of memory, the function is not reaching the return statement, so the returned value is null. A message is printed to the log file, including JavaScript and RAD stack. The session, however, is not terminated.

Example:

JS W Print ScriptTrace for OutOfMemory
JS W MemWaster@'MemWaster':2,
RAD W display:process.option:1,se.view.engine:display.go,se.list.engine:call.view.engine,se.search.engine:call.list.engine,database.engine:call.search.engine.1,database:call.database.engine,

For more information, see the following help topics:

Service Manager Programming

Parameter: memorypollinterval

Parameter: log4jDebug

Monitoring memory in Service Manager processes

Monitoring native heap and Java heap memory

Java Heap exhausted

When a Java heap memory request cannot be served, the session terminates with java exception stack.

The java heap is used by the servlet to communicate to clients, integrations, or other servlets. The size of the request or response document is the biggest driver for java heap memory requirements.

Resolving these issues requires analysis of the actual document and the implementation. A typical cause is that huge requests or response documents are required by integrations. A mechanism implemented to replace huge requests is web services pagination.

For more information, see Retrieve data from Service Manager using Pagination.

Proactive Memory pool messages

Proactive memory pool messages include the following:

Shared memory warnings

When the first process of Service Manager is started on a host, it creates the shared memory. At this time, it compares the configured size in sm.ini with the minimum acceptable value of 8000000 (and automatically adjusts) and the recommended size.

Example messages:

RTE W Shared memory size of 4000000 too small, adjusted to 8000000
RTE W The shared memory size 8000000 is less than the recommended size 134217728

RAD stack warnings

Service Manager logs warning messages when the RAD stack allocation exceeds a threshold.

Example:

RTE I RAD stack is 72% used, please exit out of current application.

Process private memory warnings

Service Manager monitors the native heap memory allocation and writes proactive messages to the log file before all resources are exhausted. In addition, the servlet will switch to a low memory mode to avoid total exhaustion and the servlet from completely terminating.

When a servlet moves into low memory mode and is no more accepting new sessions and application threads for native memory until the garbage collection has released memory again.

The memorypollinterval parameter may be used to influence the threshold when a servlet switches to low memory mode.

When the allocation of native memory increases for a session, information about the allocated memory size is printed to the log file.

Example:

RTE I -Memory : S(392328) O(165668) MAX(1001004) - MALLOC's Total(4136)

Here, the MAX value is relevant: Average here is for typical user sessions below 20M; if 40M is exceeded by a single session, investigations should be started. Possible causes are that users open a very large number of tabs in the client, or memory-intensive JavaScript implementations, or other problems.

More frequent trace information is written to sm.log when trace parameter rtm:2 is turned on.

Note Process native memory monitoring and the proactive messages are not implemented in Service Manager running on unix operating system.

For more information, see Avoiding variable declaration inside a loop.

Java Heap warnings

When a servlet moved into low memory mode and is no more accepting new sessions and application threads for java heap memory until garbage collection released memory again.

In order to monitor java heap for low memory, set the following parameter in sm.ini:

log4jdebug:com.hp.ov.sm.common.oom.LowMemoryHandler

Example of low memory handler log messages:

JRTE I Starting Memory Monitoring thread to check for memory every 15 seconds.
JRTE D JavaMemory Max(123928576) Used(1011872) %Used(0.0)
JRTE D NativeMemory Max(2147352576) Used(358121472) %Used(16.0)
JRTE W Process Low on Java Memory. Max(123928576) Used(119603800) PercentUsed(96.0)
JRTE W Send error response: Server is running low on memory try again.
JRTE W Send error response: Server is running low on memory try again.
 RTE I Process Java Heap Memory is back to normal range.
JRTE I Process Java Memory. Max(123928576)  Used(4256640) PercentUsed(3.0)

Note For detailed explanation of Garbage collection and Memory Pools, refer to the Oracle documentation: http://www.oracle.com/technetwork/systems/index-jsp-140228.html

Monitor dbdicts with log information for availability related messages

Review on a daily basis the following dbdicts, which contain log information, for error messages:

msglog

syslog

SMISlog

Related topics