Skip to main content

Looking for ideas to analyze this hang, Java 6, Red Hat Linux Enterprise Setver release 6

No replies
MariD
Offline
Joined: 2011-09-22

Platform:

Red Hat Enterprise Linux Server release 6.0 (Santiago), 2.6.32-71.el6.x86_64

java -version: java version "1.6.0_27"
Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

I have a small program that reproduces a hang we're seeing intermittently during jvm initialization. A short script runs a java class 2000 times - the number as picked out of a hat. The jvm will hang before the 2000 iterations are over. kill -3 and kill -QUIT have no effect, we can do pstack <pid> (results below). I've also tried -Djava.compiler=none and running the jvm in 32 bit mode, without success. If I try starting the jvm with parameters to debug remotely (ie. -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=49000,server=y,suspend=n), the jvm then does not hang! Calling pstack for the process a second time after time has elapsed yields exactly the same output.

The shell script:

#!/bin/sh

ITERATION=1
while [ $ITERATION != 2000 ]
do
echo ITERATION == $ITERATION
/tools/linuxx86_64/java/jdk1.6.0_27/jre/bin/java -cp /users/sonic/davila/classes revolution.hostmanager.deploy.HelloWorld
ITERATION=`expr $ITERATION + 1`
done

The java class:

package revolution.hostmanager.deploy;

public class HelloWorld
{

/**
* @param args
*/
public static void main(String[] args)
{
System.err.println("Hello World");
}

}

The result of pstack:

Thread 11 (Thread 0x7f1a84dbc710 (LWP 9417)):
#0 0x00000034f7e06ea0 in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#1 0x00007f1a854c75e9 in os::create_thread(Thread*, os::ThreadType, unsigned long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a855d4f9b in WatcherThread::start() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a855da2da in Threads::create_vm(JavaVMInitArgs*, bool*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a852d75c0 in JNI_CreateJavaVM () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00000000400035f8 in InitializeJVM ()
#6 0x000000004000206e in JavaMain ()
#7 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f1a84108710 (LWP 9418)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e7c5 in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ef8e in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a852229b5 in GCTaskManager::get_task(unsigned int) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a85223cc3 in GCTaskThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f1a7cfff710 (LWP 9419)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e7c5 in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ef8e in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a852229b5 in GCTaskManager::get_task(unsigned int) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a85223cc3 in GCTaskThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f1a7ca07710 (LWP 9420)):
#0 0x00000034f7e0b7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccf26 in os::PlatformEvent::park(long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e8fb in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ef8e in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a85629b03 in VMThread::loop() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a856296fe in VMThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f1a7c906710 (LWP 9421)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a854bde6a in ObjectMonitor::wait(long, bool, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a855a2f13 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a853092a7 in JVM_MonitorWait () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a7d010d6e in ?? ()
#6 0x00007f1a800681e0 in ?? ()
#7 0x00007f1a800685b8 in ?? ()
#8 0x00007f1a80067800 in ?? ()
#9 0x00007f1a7c9057e8 in ?? ()
#10 0x00000000bee08998 in ?? ()
#11 0x00007f1a7c905858 in ?? ()
#12 0x00000000beea8610 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7f1a7c805710 (LWP 9422)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a854bde6a in ObjectMonitor::wait(long, bool, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a855a2f13 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a853092a7 in JVM_MonitorWait () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a7d010d6e in ?? ()
#6 0x00007f1a8006a110 in ?? ()
#7 0x00007f1a8006a4e8 in ?? ()
#8 0x00007f1a80069800 in ?? ()
#9 0x00007f1a000000b6 in ?? ()
#10 0x00007f1a7c804700 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7f1a7c704710 (LWP 9423)):
#0 0x00000034f7e0d720 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f1a854cdb75 in check_pending_signals(bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a854c613c in signal_thread_entry(JavaThread*, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a855d5dc1 in JavaThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#6 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f1a7c603710 (LWP 9424)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e7c5 in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ee2d in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a85162e5f in CompileQueue::get() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a85164c91 in CompileBroker::compiler_thread_loop() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a855dc8b9 in compiler_thread_entry(JavaThread*, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00007f1a855d5dc1 in JavaThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#8 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#9 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f1a7c502710 (LWP 9425)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e7c5 in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ee2d in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a85162e5f in CompileQueue::get() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a85164c91 in CompileBroker::compiler_thread_loop() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a855dc8b9 in compiler_thread_entry(JavaThread*, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00007f1a855d5dc1 in JavaThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#8 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#9 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f1a7c401710 (LWP 9426)):
#0 0x00000034f7e0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1a854ccdce in os::PlatformEvent::park() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#2 0x00007f1a8549e7c5 in Monitor::IWait(Thread*, long) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#3 0x00007f1a8549ef8e in Monitor::wait(bool, long, bool) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#4 0x00007f1a8543ead2 in LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#5 0x00007f1a855d5dc1 in JavaThread::run() () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#6 0x00007f1a854cd7af in java_start(Thread*) () from /tools/linuxx86_64/java/jdk1.6.0_27/jre/lib/amd64/server/libjvm.so
#7 0x00000034f7e077e1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000034f72e153d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f1a859c6700 (LWP 9416)):
#0 0x00000034f7e0803d in pthread_join () from /lib64/libpthread.so.0
#1 0x00000000400058fc in ContinueInNewThread ()
#2 0x0000000040001ee9 in main ()

Any insight on how to proceed, what to look at? Thanks.