Skip to main content

jdk1.6.0rc crash

18 replies [Last post]
sbogrett
Offline
Joined: 2004-12-01

We are seeing the crash report below while running a server application using jdk1.6.0rc (b104) over just a 3 hour period.

We found the bug #4916142 that referred to this same problem report but was marked as fixed. We have tried to use the work-around suggested in the bug by generally lowering the thread stack size to 256k (the -XX setting mentioned in the bug is no longer supported) but our application still crashes.

Is there a workaround for this issue?

#

# An unexpected error has been detected by Java Runtime Environment:

#

# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

#

# Internal Error (414C4C4F434154494F4E0E4350500065), pid=5064, tid=3368

#

# Java VM: Java HotSpot(TM) Client VM (1.6.0-rc-b104 mixed mode)

# If you would like to submit a bug report, please visit:

# http://java.sun.com/webapps/bugreport/crash.jsp

#

--------------- T H R E A D ---------------

Current thread (0x27f95400): JavaThread "CompilerThread0" daemon [_thread_in_native, id=3368]

Stack: [0x28450000,0x284a0000)

[error occurred during error reporting, step 110, id 0xc0000005]

Current CompileTask:

C1:678998 ! com.certive.cs.parsers.sql.SQLParserTokenManager.jjMoveNfa_0(II)I (3046 bytes)

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

0x2b635800 JavaThread "Ext-SQL-WT-43" [_thread_blocked, id=6096]

0x2b637c00 JavaThread "Ext-SQL-WT-42" [_thread_blocked, id=5900]

0x2b389800 JavaThread "Ext-SQL-WT-41" [_thread_blocked, id=2828]

0x294db000 JavaThread "Ext-SQL-WT-40" [_thread_blocked, id=4956]

0x291db800 JavaThread "SYS-WT-39" [_thread_blocked, id=5532]

0x29195000 JavaThread "SYS-WT-38" [_thread_blocked, id=4688]

0x29190c00 JavaThread "SYS-WT-37" [_thread_blocked, id=3672]

0x286e3800 JavaThread "SYS-WT-36" [_thread_blocked, id=4576]

0x286e5800 JavaThread "SYS-WT-35" [_thread_blocked, id=5420]

0x291a4c00 JavaThread "SYS-WT-34" [_thread_blocked, id=3264]

0x291a4400 JavaThread "SYS-WT-33" [_thread_blocked, id=4888]

0x29189000 JavaThread "SYS-WT-32" [_thread_blocked, id=5384]

0x29188000 JavaThread "SYS-WT-31" [_thread_blocked, id=3500]

0x2918bc00 JavaThread "SYS-WT-30" [_thread_blocked, id=5776]

0x29179400 JavaThread "SYS-WT-29" [_thread_blocked, id=3680]

0x2917bc00 JavaThread "SYS-WT-28" [_thread_blocked, id=5624]

0x28761000 JavaThread "SYS-WT-27" [_thread_blocked, id=2820]

0x00379000 JavaThread "DestroyJavaVM" [_thread_blocked, id=4456]

0x27fa3800 JavaThread "Scheduler-WT-26" [_thread_blocked, id=676]

0x285fb400 JavaThread "Scheduler-WT-25" [_thread_blocked, id=4892]

0x2908f400 JavaThread "SocketListener0-1" [_thread_blocked, id=2472]

0x2908e800 JavaThread "SocketListener0-0" [_thread_blocked, id=2188]

0x2909f800 JavaThread "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9000]" [_thread_in_native, id=4728]

0x2909f400 JavaThread "SessionScavenger" daemon [_thread_blocked, id=6076]

0x29081000 JavaThread "SessionScavenger" daemon [_thread_blocked, id=4156]

0x28f7ac00 JavaThread "STS-WT-24... Running: WSB-HeartBeat_Sender" [_thread_blocked, id=2984]

0x28f40000 JavaThread "STS-WT-23... Running: WSB-HeartBeat" [_thread_blocked, id=5184]

0x28fa2800 JavaThread "CacheMgr-WT-22... Running: Background Operations Daemon" [_thread_blocked, id=5248]

0x28f2f800 JavaThread "CacheMgr-WT-21... Running: Paging Daemon" [_thread_blocked, id=5240]

0x28fa8800 JavaThread "Master TCP/IP Port Handler-WT-20... Running: PortMaster-PacketHandler" [_thread_blocked, id=4800]

0x28e8d000 JavaThread "Master TCP/IP Port Handler-WT-19... Running: PortMaster-PacketHandler" [_thread_blocked, id=5832]

0x28f67400 JavaThread "Master TCP/IP Port Handler-WT-18... Running: PortMaster-PacketHandler" [_thread_blocked, id=3140]

0x28fa0c00 JavaThread "Master TCP/IP Port Handler-WT-17... Running: PortMaster-PacketHandler" [_thread_in_vm, id=4736]

0x28e8c000 JavaThread "TxnMgr-WT-16" [_thread_blocked, id=3728]

0x28f5e400 JavaThread "TxnMgr-WT-15... Running: DeadLockDetector" [_thread_blocked, id=4384]

0x28f5d000 JavaThread "SYS-WT-14" [_thread_blocked, id=4652]

0x28c67400 JavaThread "SYS-WT-13" [_thread_blocked, id=3900]

0x28eb8000 JavaThread "SYS-WT-12" [_thread_blocked, id=1056]

0x28c64000 JavaThread "SYS-WT-11... Running: Scheduler" [_thread_blocked, id=4232]

0x28f09800 JavaThread "SYS-WT-10" [_thread_blocked, id=4592]

0x28ebc800 JavaThread "SYS-WT-9" [_thread_blocked, id=5348]

0x28c6f800 JavaThread "SYS-WT-8... Running: PortMaster-SocketHandler" [_thread_in_native, id=4400]

0x28fac800 JavaThread "SYS-WT-7... Running: SessionCleaner" [_thread_blocked, id=2696]

0x28e81400 JavaThread "JavaScript-WT-6" [_thread_blocked, id=5352]

0x288db800 JavaThread "Ext-SQL-WT-5" [_thread_blocked, id=3036]

0x288d7000 JavaThread "Ext-SQL-WT-4" [_thread_blocked, id=1208]

0x288d2800 JavaThread "Ext-SQL-WT-3" [_thread_blocked, id=3632]

0x288c1c00 JavaThread "Ext-SQL-WT-2" [_thread_blocked, id=5116]

0x28620c00 JavaThread "Logging-WT-1... Running: AdvancedLogThreaded-0" [_thread_blocked, id=2424]

0x27f96400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=5792]

=>0x27f95400 JavaThread "CompilerThread0" daemon [_thread_in_native, id=3368]

0x27f88c00 JavaThread "JDWP Command Reader" daemon [_thread_in_native, id=3944]

0x27f87c00 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=5676]

0x27f85800 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_in_native, id=4248]

0x27f79c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=4632]

0x27f79000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3240]

0x27f69800 JavaThread "Finalizer" daemon [_thread_blocked, id=4796]

0x27f65400 JavaThread "Reference Handler" daemon [_thread_blocked, id=4816]

Other Threads:

0x27f62000 VMThread [id=5768]

0x27fb1000 WatcherThread [id=5540]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap

def new generation total 8192K, used 6152K [0x03ca0000, 0x04580000, 0x06400000)

eden space 7296K, 72% used [0x03ca0000, 0x041c2148, 0x043c0000)

from space 896K, 100% used [0x043c0000, 0x044a0000, 0x044a0000)

to space 896K, 0% used [0x044a0000, 0x044a0000, 0x04580000)

tenured generation total 106760K, used 59562K [0x06400000, 0x0cc42000, 0x23ca0000)

the space 106760K, 55% used [0x06400000, 0x09e2a910, 0x09e2aa00, 0x0cc42000)

compacting perm gen total 19456K, used 19450K [0x23ca0000, 0x24fa0000, 0x27ca0000)

the space 19456K, 99% used [0x23ca0000, 0x24f9e928, 0x24f9ea00, 0x24fa0000)

No shared spaces configured.

Dynamic libraries:

0x00400000 - 0x00423000 D:\sharedlibs\jdk1.6.0rc\bin\javaw.exe

0x7d600000 - 0x7d6f0000 C:\WINDOWS\system32\ntdll.dll

0x7d4c0000 - 0x7d5f0000 C:\WINDOWS\syswow64\kernel32.dll

0x77f50000 - 0x77fec000 C:\WINDOWS\syswow64\ADVAPI32.dll

0x7da20000 - 0x7db00000 C:\WINDOWS\syswow64\RPCRT4.dll

0x7d930000 - 0x7da00000 C:\WINDOWS\syswow64\USER32.dll

0x7d800000 - 0x7d890000 C:\WINDOWS\syswow64\GDI32.dll

0x7c340000 - 0x7c396000 D:\sharedlibs\jdk1.6.0rc\jre\bin\msvcr71.dll

0x6d7c0000 - 0x6da07000 D:\sharedlibs\jdk1.6.0rc\jre\bin\client\jvm.dll

0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll

0x71bc0000 - 0x71bc8000 C:\WINDOWS\system32\rdpsnd.dll

0x771f0000 - 0x77201000 C:\WINDOWS\system32\WINSTA.dll

0x77ba0000 - 0x77bfa000 C:\WINDOWS\syswow64\msvcrt.dll

0x71c40000 - 0x71c98000 C:\WINDOWS\syswow64\NETAPI32.dll

0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL

0x6d310000 - 0x6d318000 D:\sharedlibs\jdk1.6.0rc\jre\bin\hpi.dll

0x6d400000 - 0x6d429000 D:\sharedlibs\jdk1.6.0rc\jre\bin\jdwp.dll

0x6d6c0000 - 0x6d6c6000 D:\sharedlibs\jdk1.6.0rc\jre\bin\npt.dll

0x6d770000 - 0x6d77c000 D:\sharedlibs\jdk1.6.0rc\jre\bin\verify.dll

0x6d3b0000 - 0x6d3cf000 D:\sharedlibs\jdk1.6.0rc\jre\bin\java.dll

0x6d7b0000 - 0x6d7bf000 D:\sharedlibs\jdk1.6.0rc\jre\bin\zip.dll

0x6d280000 - 0x6d287000 D:\sharedlibs\jdk1.6.0rc\jre\bin\dt_socket.dll

0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
alanb
Offline
Joined: 2005-08-08

The fatal error log that you included in the original post suggests that the compiler thread in the VM was unable to allocate from the native heap. Failure to allocate from the native heap causes the VM to abort. There is no GC, and no out of memory error thrown. Even if there was sufficient memory available to invoke the GC then it would somehow need to know that if you wait a bit then the finalizers or threads servicing the reference queues will release native resources. I hope this helps.

sbogrett
Offline
Joined: 2004-12-01

I have been considering how to reply to your answers. While they are insightful and very helpful in understanding the failure they, quite frankly, scare the dickens out of me. Particularly since I found the opened bug #6543285. Anyone reading this thread or interested in the shortcomings of direct memory and/or the implementation of NIO should read this bug.

Because direct memory is held by a java object and is not release until the finalizer is execute there is always a chance, on any machine, that a direct memory crash can occur given a high volume use of NIO. The disconnect between the java objects holding the direct memory and the memory the objects hold is, in my opinion, a very fundamental design flaw.

What is even more freightening Alan is that the NIO classes allocate direct memory buffers and pool them behind the scenes even if I have chosen not to use direct memory in my application. Bug #6543285 describes a scenario that creates an out-of-memory error simply because of the implementation of NIO and the inability of the JVM to release direct memory within possibly collectable java objects in a timely manner.

If I use NIO and don't use direct buffers yet the package allocates temporary direct buffers which can then be the cause of a crash... well... this is scary. I have used all my bug votes on this bug. I suggest other people read this bug carefully and consider doing so as well.

Unless the JVM dramatically reduces the possibility of a direct memory crash arising from simply by using NIO, I am convinced that the only possible workaround for this is to quit using the NIO package.

As always I do appreciate your replies and time spent. I do not intend to be an alarmist nor to dissemble. However, I do find this situation troubling to say the least. (Scary to say a bit more).

Regards,
Steven

alanb
Offline
Joined: 2005-08-08

6543285 may not be good representative of what you are using. For starters, the allocation of a direct buffer will cause all unreachable buffers to be freed if there is insufficient virtual memory to allocate the new buffer. The second issue in that bug seems to related to the use of 256MB pages. We plan, in jdk7, to allow for direct buffers that aren't page aligned and this will significantly reduce the memory requirements of direct buffers on systems with large pages configured. As regards the buffer pool used in the NIO implementation - its a simple thread-local pool to avoid an expensive allocate/free per I/O operation.

sbogrett
Offline
Joined: 2004-12-01

I am bit confused and maybe you can educate me.

The only difference is whether the memory is on the heap or direct. Our memory setting for the heap is 512Mb max. We have 4Gb physical and during one of our test runs we added another 4Gb of swap. If GC happens correctly for the heap and we run sucessfully in 512Mb then any overflow buffers are being successfully collected.

If the JVM is willing to do a GC before throwing an out of memory error (as documented) during a java heap allocation why doesn't it do a GC during a direcct memory allocation? This would free the direct memory and the allocation would succeed... as it does during the heap allocation.

Regards,
Steve

mthornton
Offline
Joined: 2003-06-10

As I understand it, the GC marks a direct buffer as ready for release but doesn't actually do so. The release occurs a short time later on a 'cleanup' thread.

sbohne
Offline
Joined: 2005-04-26

This error message implies that you run out of virtual memory. It doesn't look like you have too many Java threads so stack size probably isn't the issue. However it could be the swap configuration. Can you check the Windows page file settings? See if the page file gets filled up during the application execution? If so, you may need to increase the size of the page file. If that doesn't help, your application may have a memory leak.

sbogrett
Offline
Joined: 2004-12-01

We are running on an AMD 4000+ / WinXP 64 with 4 Gig of RAM and nothing else significant running. Our JVM memory settings are -Xms256m -Xmx512m. Even if we have a memory leak we would expect an OutOfMemory exception, not a JVM crash. This seems like a serious JVM bug regardless of our memory management because it crashes instead of throwing an exception.

The bug that we found that referenced this exact same message indicated that there was a workaround using JVM settings.

Regards,
sbogrett

sbohne
Offline
Joined: 2005-04-26

Ok, if you have 4GB RAM, I'll assume you have enough swap, but I'd still encourage you to check and respond with the Windows pagefile settings.

Couple more things:

1. "The bug that we found that referenced this exact same message indicated that there was a workaround using JVM settings." You're referring to the message "java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?" It turns out that there can be multiple root causes associated with this message. So it's not guaranteed that the workaround will work for you.

2. The JVM will throw OutOfMemoryError whenever possible, however some situations we can't, and have to crash. In this case we failed to allocate memory on a VM-internal thread (a compiler thread) which is not a Java thread - so we can't throw an exception on it.

3. It looks like the logfile you posted is cut off. The last line in the logfile should be the "vm info" string. Are you sure the entire thing got posted?

4. There may still be a couple things you can try. The logfile indicates you're running with the client VM. You mentioned this was a server application - you'll get MUCH better performance in general using the server VM, and it might solve your memory problem, too. Note, for Windows you must download the entire JDK to get the server VM - it's not included with the JRE download. Try running with "-server" and see if that helps.

5. If THAT doesn't help, this could be a red herring, but the crash log indicates 99% perm gen usage. Normally if this was the problem I'd expect to see the OutOfMemory thrown normally on a Java thread, but you can give this a shot. Try running with -XX:PermSize=64m.

sbogrett
Offline
Joined: 2004-12-01

Thanks for you feedback/help.

Responses to your items:

1. Yes that's reasonable. We didn't really think the workaround given would fix the issue but we were willing to try it.

2. OK.

3. It might be truncated. I will reproduce and get you the full trace.

4. We have run this same test against the server and the client VM and both give us the same error. We just gave you the latest one. We certainly do not normally run with the client VM. We were just testing.

5. We will try this after we try a few other things. One thing that seems to be working but that we want to continue to test, is to actually exclude the method indicated in the trace from compilation using the .hotspot_compiler file and the exclusion command.

Again, thanks for all the help and attention.

Regards,
Steve

gregmccoy75
Offline
Joined: 2007-05-05

Steve,

I'm experiencing the same issue, were you able to resolve your issue? If so, how?

Greg

sbogrett
Offline
Joined: 2004-12-01

Sorry for the late reply but it has been awhile since I have visited this forum. I'm eagerly awaiting 1.6.0 u2.

Yes. In our application we made extensive use of NIO and DirectBuffers. When we stopped using DirectBuffers we saw this issue go away. We still use NIO but simply allocated java heap buffers instead of direct memory buffers. We tried every setting and configuration we could think of but none seemed to work so long as we used DirectBuffers. We used substantial sizes... approaching 32 megabytes of direct memory but as I indicated earlier we were running on a machine with plenty of memory... both virtual and physical.

We will try to use DirectBuffers again with 1.6.0 u2 but since we have a workaround we probably won't spend lots of time investigating if the failure reappers with direct memory.

Good luck and best wishes.

Regards,
Steve

alanb
Offline
Joined: 2005-08-08

There aren't any changes to the way that direct buffers are managed in 6.0u2. To help explain what you are seeing it would be useful if you could outline how you are using these buffers.

sbogrett
Offline
Joined: 2004-12-01

I understand that direct buffers haven't been changed significantly with any of the update releases. However, this problem has a flavor closer to compilation and/or gc rather than general direct buffer usage. Just hoping...

Here is a general outline...

We have a cache of Buffers which allocates and caches buffers of various sizes from 1k to 4Mb. Buffers are allocated at startup and can also be created at runtime and possibly added to the cache (or not...depending on cache capacity). We did this for performance reasons because of the slower allocation and gc of direct buffers. We have seen definite performance degradation when we allocate large numbers of direct buffers for short duration use.

Under intense loads we may request a buffer of a particular size, use it, and release it back to the cache 100s of times a second. The buffers are used for file i/o and socket i/o across many threads across many file channels and may socket channels. The reads/writes from the files and sockets are of varying sizes and are relatively atomic for a particular thread. Thus the pattern of... give me a buffer of a particular size, use it , release it.

We know of no race or deadlock issues with our cache and the use of the buffer is by a single thread only.

The allocation point for these buffers is a single method in our cache object. This is the only point of allocation for these buffers throughout the application. If we change the allocation point to allocate heap buffers everything is fine. If we allocate direct buffers then after approximately 3-4 hours we get the error mentioned above.

We would prefer to use direct buffers and we fully understand that caching of heap buffers isn't a good idea. However, we have left the cache in place hoping that we will be able to use direct buffers simply by changing the single allocation point to allocate direct buffers when we pass our stress tests.

If we are unable to pass our stress tests with direct buffers we will likely remove the caching of these buffers.

Regards,
Steve

alanb
Offline
Joined: 2005-08-08

If I read your reply correctly then at peak times there isn't buffers available in the cache so new buffers are allocated. What happens when you are finished with the buffers? From your think I think you are saying they are returned to the cache but it depends on the "capacity" of the cache. If the cache is full do you discard the buffers? If so, then that might give a hint to the issue. When direct buffers are discarded then the memory will not be reclaimed until the buffers become finalizable (ie: need a GC to discover that the buffers are unreachable). Maybe an approach to take is to limit the number of direct buffers in use and to use heap allocated buffers for any excess. Another possible saving is to allocate a very large buffer and to slice it into the smaller buffers that you need.

sbogrett
Offline
Joined: 2004-12-01

Absolutely. We allow the Cache to hold buffers up to a given capacity and then it simply does discards the buffer when it is released... thus allowing it to be gc'ed. All of our investigation concluded that it is not held by other references and does in fact become garbage.

All of your suggestions are reasonable and worthy of implementation. However, the generality here is that the JVM crashes when we allocate the direct buffers and doesn't when we allocate heap buffers.

As far as we can tell the only difference is what we allocate and not how we use it. Certainly there is an opportunity to tune this particular cache and we may choose to do so. But we have already spent many weeks determining that we could alleviate the crash by allocating heap buffers and we are only inclined to spend time now doing tests to see if the problem goes away with new JVM updates.

If we ever revisit the cache or decide to do away with it because we are unable to allocate direct memory then we will likely take a harder look.

As always thanks for the time and attention.

Regards,
Steve

alanb
Offline
Joined: 2005-08-08

When you "discard" a direct buffer then the memory is not released until the buffer become finalizable so you are dependent on timely GC/reference processing for the memory to be recovered. This can be a problem in some environments and in this case you run out of virtual memory before a GC occurs and this causes the VM to terminate.

trejkaz
Offline
Joined: 2004-01-05

There is a dirty hack to force it, though. Found this tip on a forum somewhere else. Not good to rely on Sun APIs, though... never know what might happen in JDK7.

if (buffer instanceof sun.nio.ch.DirectBuffer)
{
((sun.nio.ch.DirectBuffer) buffer).cleaner().clean();
}

alanb
Offline
Joined: 2005-08-08

"dirty hack" is putting it mildly. DO NOT DO THIS !! Aside from the fact that this is a Sun specific API you will also cause a crash or other reliability or security issues in the event that the buffer is still in use (by other threads or JNI code for example).