Skip to main content

Frequent expensive full GCs (after the first full GC) with JDK_1.5.0_06

1 reply [Last post]
sacrengan
Offline
Joined: 2007-05-23

On load testing the application we see that

- everything works fine for the first 35 minutes till we hit the first full GC
after the first full GC, which happened after 35 minutes (after about 500 minor GCs), strangely the full GCs happen very frequently, once about every 2-3 minutes (once every 5-6 minor GCs).This degrades the performance significantly

- The frequency of minor GCs remains the same, about 1 every 15-30 seconds.

- Monitoring using visualGC and JConsole shows that till the first full GC, the minor GC was collecting a majority of the newly allocated objects with only a few spilling over to the Survivors and subsequently to the old gen. Hence the old gen was filling up very slowly, and it took 35 minutes for it to fill up and cause the first full GC.

- However, immediately after this first full GC (Note that the load exposed to the servers remained the same), it seems that the minor GCs started collecting very less objects and a huge number of objects were spilled over to the Survivors (which got completely filled up) and subsequently to the old gen which also got filled up quickly within 5-6 minor GCs, leading to frequenty major GCs, which were very expensive (took ~20 seconds)

Here is our JVM parameters:
-XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -Xms2560m -Xmx2560m -XX:NewSize=896m -XX:MaxNewSize=896m
-XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 -XX:InitialSurvivorRatio=8 -XX:MinSurvivorRatio=8 -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=512m

This is an urgent issue and help is very much appreciated

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
smoov
Offline
Joined: 2007-05-01

I've seen that type of behavior in load test scenarios when object creation outpaces the ability for the JVM to collect the garbage. Since this is in a test environment, you might be able to more easily try a few things.

If the minor GC times are acceptably short now, you could explore increasing the size of Young to 1024 or 1280.

If this is a dedicated server, I've had luck with the -XX:+AggressiveHeap option, but you will want to remove some of the other options you are using that might conflict. Something like this perhaps:
-XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+AggressiveHeap -Xms2560m -Xmx2560m -XX:PermSize=512m -XX:MaxPermSize=512m

You didn't mention whether the Full GC's were recovering much space for the Old gen, if not there is a possibility you may have unintentional object retention - a java style memory leak. You could add these two JVM options:
-XX:+HeapDumpOnCtrlBreak
-XX:+HeapDumpOnOutOfMemoryError

Then the next time you see these frequent Full GC's and a near full Old gen, issue a kill -3 on the PID of the Java process you are running. Then use a heap analysis tool to see what's what and who's holding onto who. Jhat is OK for that.

Please report back what you find. Cheers!