WebSphere Application Server V8.5 Administration and Configuration Guide for the Full Profile

Chapter 31. Troubleshooting 1103

– WebSphere Application Server extensions for Dump Analyzer

Used for analyzing WebSphere Application Server JVM system dumps to diagnose

WebSphere specific problems.

Figure 31-8 shows the IBM Support Assistant with its add-ons for problem analysis.

Figure 31-8 IBM Support Assistant

For more information about the IBM Support Assistant add-ons, refer to the following website:

http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg27013116

31.4 Troubleshooting scenarios

Now that you have seen what are the most common issues and how to collect information to

further diagnose them, this section shows you simulated scenarios of common issues. This

section helps you understand what steps need to taken to identify the root cause of the

problem.

31.4.1 Hung threads

This section describes how to troubleshoot hung threads. These can occur due to many

reasons, and WebSphere Application Server monitors long running threads to warn about

possible issues. After the issue is detected, either by the warning messages in the log files or

by other monitoring tools, the first step is to take Java cores. A Java core is a snapshot of all

running threads in one specific application server along with the stack trace for each one of

them.

1104 WebSphere Application Server V8.5 Administration and Configuration Guide for the Full Profile

Creating Java cores

The following steps outline how to create a Java core using two different ways:

Using a command line:

1. Log into the server, which has the application server experiencing the issue.

2. Locate the process ID for the java process of that application server.

3. Issue the command kill -3 PID. Where PID is the process ID of the java process.

Using the administration console:

1. From the Web Administration Console, go to Troubleshooting  Java dumps and

cores.

2. Select the application server you want to create a Java core, and click Java core.

By default, the java cores are created in the profile_home directory with the naming

convention of javacore.<YEAR><MONTH><DAY>.<TIME>.<PID>.<SEQUENTIAL>.txt, for example:

javacore.20120628.171251.5424.0001.txt. The exact location of the log file is also shown in

the native_stderr.log log file. Example 31-8 shows this process and syntax.

Example 31-8 Location and naming of core file

JVMDUMP039I Processing dump event "user", detail "" at 2012/06/28 17:12:51 - wait.

JVMDUMP032I JVM requested Java dump using

'/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120628.171251.5424.0001

.txt' in response to an event

JVMDUMP010I Java dump written to

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120628.171251.5424.0001.

txt

JVMDUMP013I Processed dump event "user", detail "".

It is important that you create a few Java cores in regular intervals to be able to detect if a

thread is still hung or not. By comparing the java core files and matching with warning

messages in the logs, you can detect if a thread is still hung or not and what is the likely

cause.

Diagnosing hung thread issues

To diagnose hung thread issues:

1. After a thread is detected hung or if you suspect that something is hung, the first step is to

immediately take java cores as described in the section “Creating Java cores”. The

minimum number of java cores recommended to take is 3 (with an interval of 2 minutes

from each other).

2. Identify from the SystemOut.log file, what thread is reported as hung. Example 31-9

shows an example of a warning message saying that the thread WebContainer: 0 is hung:

Example 31-9 Thread WebContainer hung warning message

[6/28/12 18:28:19:227 EDT] 00000070 ThreadMonitor W WSVR0605W: Thread

"WebContainer : 0" (000000a1) has been active for 146,658 milliseconds and may

be hung. There is/are 1 thread(s) in total in the server that may be hung.

Chapter 31. Troubleshooting 1105

3. Download the java core files to a computer with IBM Support Assistance (ISA) installed.

Make sure you have the IBM Thread and Monitor Dump Analyzer for Java (TMDA)

installed in ISA.

4. From ISA, Click Launch Activity  Analyze problem

5. In the Tools Catalog, select IBM Thread and Monitor Dump Analyzer for Java (TMDA),

as shown in Figure 31-9.

Figure 31-9 Launching TMDA

6. In TMDA, open the Java core files by clicking File  Open Thread Dumps.

7. Select the Java core file, and click Open.

8. Select the Java core file from the list, and click the Thread Detail icon, as show in

Figure 31-10.

Figure 31-10 Viewing thread details

9. From the list of threads, select the thread identified as hung from step 2 on page 1104. On

the right side of the window, you will see the thread details, as shown in Figure 31-11 on

page 1106.