Skip to main content

OutOfMemoryError on jdk 5

12 replies [Last post]
Joined: 2006-07-20

hello All,
I have inherited a java enterprise app from another developer who left the company (probably a bad start ),
And i am noticing some OutOfMemory in the logs once in a while.

My environment is jdk5 update 11, jboss 4.05GA, Redhat 9.

I was thinking on how to go about troubleshooting this -
I added a flag -XX:+HeapDumpOnOutOfMemoryError, but we saw that this did not create a heap dump when the error happened.
Further the weird thing is that the jboss/jvm does not crash after this OutOfMemoryError and continues serving request normally
and even when we check the memory on the process everything looks normal.
(There is a delay of a couple of minutes from the time we get an EMail notification for it and when we check the logs)
We have just added jmxremote.port option for jvm startup so we can start a remote jconsole, but this happens probably once a week so until then we wait patiently
(or can we do something in the meanwhile).
I cant upgrade to jdk 6 (jdk 6 gives a stack trace with an outofmemory error which could be helpful)

So the questions i have in my mind are -
1. Can a jvm recover from an OutOfMemoryError ? via GC
2. Isnt it necessary for an OutOfMemoryError to crash the jvm ?
3. Is is possible that some lib is catching this error an ignoring it ?
4. Can I do something, to debug this other than wait for it to happen again?
5. How do i make sure i am not running into this -
"The parallel garbage collector (UseParallelGC) throws an out-of-memory exception if an excessive amount of time is being spent collecting a small amount of the heap."
documented here -

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Joined: 2004-01-07

Could it be that you're running out of permanent generation space?

lg Clemens

Joined: 2006-07-20

If that was the case wouldn't it say that specifically in OOME message?
Those errors usually say "OOME:Perm Gen Space"

Joined: 2007-10-22

Hi siddharthchhabra !
There are two different types of Memory allocation we are talking here.
1. Java Heap: This is where your objects live. This is limited by -Xmx
2. Native Memory: This is the Memory required by JVM itself. Remember, JVM itself is a process. This is limited only by the OS limitaion or by the available virtual memory (RAM + disk swap), whichever is the lower.

Your JVM will surely crash and die ONLY in the case of lack of number 2 above. (You will see a core file in JVM root directory). If it encouters Java Heap OOM, it will log a message first and then be handled either by the Application itself (if you are catching this exception) or by the Container (JBoss). I've seen several Weblogic and WebSphere Apps continuing to run even after Heap OOM. But, yeah, eventually you will need to restart the App to get it working properly.

got to run. Will look at your GC output and comment later.

- Karun

Joined: 2008-05-02

Hi Siddarth,

Did you find a reason/solution for this problem?Am facing the exact same situation in our Production weblogic server.

- Rajinder

Joined: 2007-05-01

Are you sure the HeapDumpOnOutOfMemory JVM option isn't working? You might have to look around a bit to find out where the heap dump file is located. Likely in the same dir java is running from.

When you finally do get a dump, the excellent free tool from SAP, Memory Analyzer, is the best I've found so far. Can even open huge 64-bit dumps on 32-bit Windows.

Since an OOME can result from a number problems (JVM heap size, perm size, threads, file descriptors), I recommend adding verbose GC logging and then after you encounter an OOME run the resulting log through a tool like HP's HPjmeter. That will give you some clues if the problem is indeed memory.

To answer your questions:
1. No, even though it might appear to recover and continue, it is suspect. You have no idea what might have gotten corrupted when the OOME occurred.

2. No.

3. Perhaps, but you said the error is getting logged 'once in a while'. GC log analysis will help you here if the problem is memory.

4. You need to find out what is triggering the OOME and then monitor that resource. Check things like file descriptor limits and current usage. See if FDs leak, or increase over time. Pull your GC log from time to time and run it though HPjmeter. Or since you have added the jmx port, connect up with jconsole and explore periodically.

5. GC log analysis.

Good luck, and please report back what you find.

Joined: 2006-07-20

Well, i searched the entire machine for java_* files, but couldnt find one.
Also i see OOME in logs [STDERR} and around that time i also see
"Exception in Thread-26" or some other thread number.

Also correct me if i am wrong here, but OOME should manifest itself in the GC logs as a AF(Allocation Failure), I dont see that?

i added the -verbose:gc flags and other -XX:-PrintGCDetails and -XX:-PrintGCTimeStamps

i saw the gc logs, but hpjmeter/hpjtune throws a numberformatexception on these logs.

I researched a bit and founds GC portal from sun, which is written for jdk 1.3 and tomcat 4( i think, see post )-
funnily enough i could not find the mentioned here -
and i did not have any patience left for going after Sun AppServer 7 ( i had tried 3 tomcat versions till then)

Finally i googled and found gcViewer here -
which did not choke on gc log analysis
[after spending 2 hours on GC Portal and HPTune/HPJmeter....ouch :(( ]
I just feel that this should have been given by Sun in the jdk (IMHO)

anyways now i am reading this -
and -
to make some sense from the out put.
If you are interested i can paste it here.

Let me know.

Joined: 2006-07-20

GC LOGS (just in case someones interested) -
6.622: [Full GC 6.622: [Tenured: 0K->5223K(349568K), 0.1237310 secs] 72721K->5223K(506816K), [Perm : 15061K->15061K(21248K)], 0.1238800 secs]
14.766: [GC 14.766: [DefNew: 139904K->7340K(157376K), 0.0581650 secs] 145127K->12563K(506944K), 0.0582850 secs]
17.290: [Full GC 17.290: [Tenured: 5223K->17383K(349568K), 0.2355890 secs] 86513K->17383K(506944K), [Perm : 26687K->26687K(26688K)], 0.2357420 secs]
18.947: [Full GC 18.947: [Tenured: 17383K->19975K(349568K), 0.2165020 secs] 48237K->19975K(507008K), [Perm : 32127K->32127K(32128K)], 0.2166360 secs]
26.412: [Full GC 26.412: [Tenured: 19975K->33516K(349568K), 0.4356840 secs] 152351K->33516K(507008K), [Perm : 37567K->37495K(37568K)], 0.4358550 secs]
29.232: [Full GC 29.232: [Tenured: 33516K->42099K(349568K), 0.3575290 secs] 90327K->42099K(507008K), [Perm : 42943K->42943K(42944K)], 0.3576940 secs]
33.955: [GC 33.955: [DefNew: 140032K->4939K(157504K), 0.0322210 secs] 182131K->47039K(507072K), 0.0323660 secs]
37.115: [GC 37.115: [DefNew: 144971K->5013K(157504K), 0.0270970 secs] 187071K->47113K(507072K), 0.0272300 secs]
41.054: [GC 41.055: [DefNew: 145045K->6081K(157504K), 0.0314170 secs] 187145K->48181K(507072K), 0.0315550 secs]
94.931: [Full GC 94.931: [Tenured: 42099K->52529K(349568K), 0.4246240 secs] 107730K->52529K(507072K), [Perm : 48383K->48383K(48384K)], 0.4247750 secs]
99.218: [Full GC 99.218: [Tenured: 52529K->54516K(349568K), 0.4248090 secs] 98063K->54516K(507072K), [Perm : 53823K->53823K(53824K)], 0.4249590 secs]
103.044: [Full GC 103.044: [Tenured: 54516K->55195K(349568K), 0.5637350 secs] 105811K->55195K(507072K), [Perm : 59263K->59175K(59264K)], 0.5638840 secs]
108.575: [Full GC 108.575: [Tenured: 55195K->60082K(349568K), 0.5196050 secs] 120998K->60082K(507072K), [Perm : 64703K->64703K(64704K)], 0.5197640 secs]
586.468: [GC 586.468: [DefNew: 140032K->5545K(157504K), 0.0403650 secs] 200114K->65627K(507072K), 0.0405230 secs]
806.546: [GC 806.546: [DefNew: 145577K->17472K(157504K), 0.0723870 secs] 205659K->84553K(507072K), 0.0725510 secs]
807.685: [GC 807.685: [DefNew: 103887K->1071K(157504K), 0.0675720 secs] 170968K->101714K(507072K), 0.0677220 secs]
808.620: [GC 808.620: [DefNew: 141103K->7204K(157504K), 0.1426170 secs] 241746K->172359K(507072K), 0.1427760 secs]
899.098: [Full GC 899.098: [Tenured: 165154K->93546K(349568K), 0.7205600 secs] 270368K->93546K(507072K), [Perm : 70143K->70143K(70144K)], 0.7207160 secs]
1220.718: [GC 1220.718: [DefNew: 140032K->6278K(157504K), 0.0456850 secs] 233578K->99824K(507072K), 0.0458490 secs]
1281.817: [GC 1281.817: [DefNew: 146310K->9027K(157504K), 0.0463190 secs] 239856K->102573K(507072K), 0.0464690 secs]
1294.832: [GC 1294.832: [DefNew: 149059K->8127K(157504K), 0.0569190 secs] 242605K->105515K(507072K), 0.0570710 secs]
1578.573: [GC 1578.573: [DefNew: 148159K->8451K(157504K), 0.0421320 secs] 245547K->105839K(507072K), 0.0422860 secs]
1598.302: [GC 1598.302: [DefNew: 148483K->7024K(157504K), 0.0323950 secs] 245871K->104412K(507072K), 0.0325470 secs]
1704.404: [GC 1704.404: [DefNew: 147056K->8703K(157504K), 0.0425000 secs] 244444K->106091K(507072K), 0.0426520 secs]
1770.288: [GC 1770.288: [DefNew: 148735K->16946K(157504K), 0.0964900 secs] 246123K->114334K(507072K), 0.0966420 secs]
1800.150: [GC 1800.150: [DefNew: 156978K->2498K(157504K), 0.0758100 secs] 254366K->116172K(507072K), 0.0759630 secs]
1811.243: [GC 1811.243: [DefNew: 142530K->3533K(157504K), 0.0240430 secs] 256204K->117206K(507072K), 0.0241980 secs]
1813.360: [GC 1813.360: [DefNew: 143565K->3183K(157504K), 0.0134500 secs] 257238K->116856K(507072K), 0.0135950 secs]
1814.333: [GC 1814.333: [DefNew: 143215K->3890K(157504K), 0.0177290 secs] 256888K->117564K(507072K), 0.0178770 secs]
1816.127: [GC 1816.127: [DefNew: 143922K->4080K(157504K), 0.0188270 secs] 257596K->117754K(507072K), 0.0189790 secs]
2187.194: [GC 2187.195: [DefNew: 144112K->3155K(157504K), 0.0139300 secs] 257786K->116828K(507072K), 0.0140800 secs]
2934.920: [GC 2934.920: [DefNew: 143187K->5008K(157504K), 0.0270460 secs] 256860K->118681K(507072K), 0.0272000 secs]
3025.177: [GC 3025.177: [DefNew: 145040K->4629K(157504K), 0.0295970 secs] 258713K->118302K(507072K), 0.0297910 secs]
3027.278: [GC 3027.278: [DefNew: 144661K->4860K(157504K), 0.0281080 secs] 258334K->118533K(507072K), 0.0282550 secs]
3028.563: [GC 3028.563: [DefNew: 144892K->3561K(157504K), 0.0156630 secs] 258565K->117234K(507072K), 0.0158100 secs]
3030.527: [GC 3030.527: [DefNew: 143593K->3747K(157504K), 0.0169270 secs] 257266K->117421K(507072K), 0.0170710 secs]
3035.918: [GC 3035.918: [DefNew: 143779K->4217K(157504K), 0.0200510 secs] 257453K->117890K(507072K), 0.0201940 secs]
3579.883: [GC 3579.883: [DefNew: 144249K->6981K(157504K), 0.0398810 secs] 257922K->120654K(507072K), 0.0400360 secs]
3589.057: [GC 3589.057: [DefNew: 147013K->14610K(157504K), 0.0810300 secs] 260686K->128284K(507072K), 0.0811820 secs]
3616.137: [GC 3616.137: [DefNew: 154642K->2697K(157504K), 0.0494870 secs] 268316K->124735K(507072K), 0.0496390 secs]
3631.523: [GC 3631.523: [DefNew: 142729K->2761K(157504K), 0.0166240 secs] 264767K->124799K(507072K), 0.0167790 secs]
3664.006: [GC 3664.007: [DefNew: 142793K->5953K(157504K), 0.0340570 secs] 264831K->127991K(507072K), 0.0342100 secs]
3670.323: [GC 3670.323: [DefNew: 145985K->3542K(157504K), 0.0166550 secs] 268023K->125579K(507072K), 0.0168010 secs]
3675.148: [GC 3675.148: [DefNew: 143574K->3511K(157504K), 0.0162390 secs] 265611K->125549K(507072K), 0.0163900 secs]
3680.503: [GC 3680.504: [DefNew: 143543K->3428K(157504K), 0.0160160 secs] 265581K->125466K(507072K), 0.0161620 secs]
4017.981: [GC 4017.981: [DefNew: 143460K->3013K(157504K), 0.0137330 secs] 265498K->125051K(507072K), 0.0138880 secs]
4255.020: [Full GC 4255.020: [Tenured: 122037K->87068K(349568K), 0.6278180 secs] 143381K->87068K(507072K), [Perm : 75583K->75583K(75584K)], 0.6279760 secs]
4327.103: [GC 4327.103: [DefNew: 140032K->3975K(157504K), 0.0332130 secs] 227100K->91043K(507072K), 0.0333650 secs]
4329.913: [GC 4329.913: [DefNew: 144007K->8023K(157504K), 0.0522510 secs] 231075K->95091K(507072K), 0.0524000 secs]
4350.119: [GC 4350.119: [DefNew: 148055K->10458K(157504K), 0.0557570 secs] 235123K->97526K(507072K), 0.0559080 secs]
4362.493: [GC 4362.493: [DefNew: 150490K->4660K(157504K), 0.0667160 secs] 237558K->101568K(507072K), 0.0668690 secs]
4450.053: [GC 4450.053: [DefNew: 144692K->2620K(157504K), 0.0163600 secs] 241600K->99528K(507072K), 0.0165020 secs]
4496.325: [GC 4496.325: [DefNew: 142652K->7208K(157504K), 0.0201140 secs] 239560K->104116K(507072K), 0.0202710 secs]
4497.174: [GC 4497.174: [DefNew: 120877K->17472K(157504K), 0.0309440 secs] 217785K->116124K(507072K), 0.0311050 secs]
4498.116: [GC 4498.117: [DefNew: 157504K->281K(157504K), 0.0715040 secs] 256156K->164699K(507072K), 0.0716540 secs]
4500.059: [GC 4500.060: [DefNew: 54130K->282K(157504K), 0.0048990 secs] 218548K->164701K(507072K), 0.0050500 secs]
4500.465: [GC 4500.465: [DefNew: 140314K->282K(157504K), 0.2080180 secs] 304733K->293725K(507072K), 0.2081560 secs]
4503.990: [GC 4503.990: [DefNew: 119739K->285K(157504K), 0.0056230 secs]4503.996: [Tenured: 293442K->207232K(349568K), 0.8589380 secs] 413182K->207232K(507072K), 0.8647650 secs]
4508.304: [GC 4508.304: [DefNew: 140032K->3K(157504K), 0.0052150 secs] 605313K->465284K(765124K), 0.0053530 secs]
4511.273: [GC 4511.273: [DefNew: 140035K->5K(157504K), 0.0050150 secs] 605316K->465286K(765124K), 0.0051430 secs]
4511.322: [GC 4511.322: [DefNew: 4241K->3K(157504K), 0.0050720 secs]4511.327: [Tenured: 465281K->336257K(607620K), 0.7955410 secs] 469522K->336257K(765124K), 0.8013200 secs]
4517.995: [GC 4517.995: [DefNew: 243200K->5K(273600K), 0.0072070 secs] 1095553K->852358K(1397320K), 0.0073650 secs]
4522.728: [GC 4522.728: [DefNew: 243205K->4K(273600K), 0.0071930 secs] 1095558K->852357K(1397320K), 0.0073520 secs]
4525.345: [GC 4525.345: [DefNew: 136765K->4K(273600K), 0.0072230 secs]4525.352: [Tenured: 852353K->594306K(1123720K), 1.0519200 secs] 989118K->594306K(1397320K), 1.0603710 secs]
4526.406: [Full GC 4526.407: [Tenured: 594306K->593892K(1403584K), 1.0970750 secs] 594306K->593892K(1909440K), [Perm : 69581K->69452K(76544K)], 1.0985590 secs]
4833.441: [GC 4833.441: [DefNew: 561472K->47980K(631616K), 0.1630440 secs] 1155364K->641872K(2035200K), 0.1632350 secs]
5729.568: [GC 5729.568: [DefNew: 609452K->27567K(631616K), 0.2383770 secs] 1203344K->636107K(2035200K), 0.2385250 secs]
6656.525: [GC 6656.525: [DefNew: 589039K->32970K(631616K), 0.1262260 secs] 1197579K->641510K(2035200K), 0.1263760 secs]
6931.350: [GC 6931.350: [DefNew: 594442K->44320K(631616K), 0.1989990 secs] 1202982K->652860K(2035200K), 0.1991500 secs]
7108.900: [GC 7108.900: [DefNew: 605792K->20103K(631616K), 0.1861710 secs] 1214332K->655942K(2035200K), 0.1863340 secs]
7195.760: [GC 7195.760: [DefNew: 581575K->20145K(631616K), 0.0845210 secs] 1217414K->655983K(2035200K), 0.0846810 secs]
8127.514: [Full GC 8127.514: [Tenured: 635838K->95763K(1403584K), 0.8589260 secs] 825939K->95763K(2035200K), [Perm : 75121K->69854K(75200K)], 0.8596950 secs]
8366.325: [GC 8366.325: [DefNew: 561472K->3446K(631616K), 0.0253580 secs] 657235K->99209K(2035200K), 0.0255170 secs]
8943.182: [GC 8943.182: [DefNew: 564918K->17085K(631616K), 0.0971910 secs] 660681K->112848K(2035200K), 0.0973460 secs]

Joined: 2007-05-01

Did an OOME occur during this GC log time span?

Sorry to hear about HPjmeter giving you grief, I've used it on GC logs from various Unix and Windows flavors without problem. Have you looked into jconsole that comes with the jdk?

Joined: 2006-07-20

In the logs, I did see an "ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space"
But the weird thing is i did not get my dump, despite the flag -XX:+HeapDumpOnOutOfMemoryError.
I was hoping that the GC logs would give me an AF(AllocationFailure), isnt that supposed to happen on OOME ???

If the GC logs look clean to you, then is this a spurious OOME ( a possible bug ?).
Will try jconsole (wasnt aware it could read GC logs) ..thanks for that tip.

Joined: 2007-05-01

Sorry to mislead you, JConsole is for monitoring performance and resource consumption of a live JVM. It will allow you to attach to the JVM and present some fairly nifty graphs and reports. Hopefully that will lead you to the source of your problem.

BTW, what JVM Option settings are you running with?

Joined: 2006-07-20

JAVA_OPTS="-XX:+HeapDumpOnOutOfMemoryError -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/home/schhabra/jboss/gc.log -
Xms512m -Xmx2056m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"


Joined: 2007-05-01

Thanks. Something quick to try is to set -Xms and -Xmx to 2056m (or 2g) and restart the application. If you can't allocate the 2g, then that rules out the OOME being triggered by growing the heap from start towards max.