Skip to main content

1.6.0-b105 Server VM much slower than 1.5.0.11 -- Huge total compile time

4 replies [Last post]
pbeaman
Offline
Joined: 2003-06-22
Points: 0

We have a fairly complex application originally written for Java 5 that runs about 2.5x slower on the 1.6.0-b105 Server VM than under the 1.5.0.11 Server VM (measured a long time after startup, so we think the HotSpot warm-up time is factored out of this). We also see that the "Total compile time" shown in jconsole for 1.6 is huge. For example, from jconsole:

Uptime: 1 hour 1 minute
Process CPU time: 51 minutes
JIT Compiler: HotSpot Server Compiler
Total compile time: 53 minutes

Under 1.5.0.11, the total compile time is 6-10 minutes for this total duration. One possibility is that the measurement of the total compile time is just plain broken (as might be indicated by discrepancy between 51 and 53 minutes), but I suspect there really is an underlying problem since the application runs so much slower under 1.6.

The application uses about 220 threads to send and receive JMS messages (implemented through ActiveMQ) and perform lots of disk I/O to store and retrieve files. We spend a significant amount of CPU time compressing and encrypting the data stored in the files. We have loaded about 4,025 classes and have unloaded only 70 classes in the first hour of operation, so I don't think we are creating or loading an excessive number of new classes. The issue does not appear to be related to gc; we have spent only 1 minute in garbage collection in the first hour of run time. The application has created new threads at the rate of about 2 per second for the duration of the test, but this also happens when we run it under 1.5.0.11.

Thinking we might be exhausting the compiled code cache I set -XX:ReservedCodeCacheSize to 128m with no apparent effect. Are there any other VM settings we should try?

Many thanks in advance for any help or suggestions.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
alexlamsl
Offline
Joined: 2004-09-02
Points: 0

Just a suggestion before getting too carried away - have you tried profiling the application (e.g. attaching with NetBeans Profiler) when running on JavaSE 5 and 6 and compare the traces?

Some direct comparisons could help in narrowing down the regions of code which are causing the major slowdowns.

Alex.

linuxhippy
Offline
Joined: 2004-01-07
Points: 0

Just a guess (so most likely to be wrong because I am not an vm-engineer ;) ), maybe you hit a compile-deoptimize-compile loop?

Could you run the application with: -XX:+PrintCompilation

to see where all the compilation-time goes into?

Thank you in advance, lg Clemens

pbeaman
Offline
Joined: 2003-06-22
Points: 0

Ig -

Thanks for your reply and for all the help you offer in this forum.

It took some time due to external circumstances (including a large snow storm) to get results, but I now have some data from -XX:+PrintCompilation. It certainly appears that you are correct about some kind of compile/deoptimize loop, but that leaves the question of how to solve it. I also ran with -Xbatch, but as explained in http://blogs.sun.com/fatcatair/entry/calling_conventions, this may be less helpful in Java 6 than earlier versions. Steve's explanation is of general interest and I recommend his blog.

Here is a snippet of PrintCompilation results for a short interval long after the application started:

1766 made zombie sun.reflect.DelegatingMethodAccessorImpl::invoke (10 bytes)
1769 made zombie java.lang.ClassLoader::loadClass (58 bytes)
1775 made not entrant java.lang.ClassLoader::loadClass (7 bytes)
1777 !b java.net.URLClassLoader::findClass (29 bytes)
1777 made not entrant java.net.URLClassLoader::findClass (29 bytes)
1778 b sun.reflect.DelegatingMethodAccessorImpl::invoke (10 bytes)
1779 !b java.lang.reflect.Method::invoke (167 bytes)
1780 s!b java.lang.ClassLoader::loadClass (58 bytes)
1779 made not entrant java.lang.reflect.Method::invoke (167 bytes)
1772 made zombie sun.misc.Launcher$AppClassLoader::loadClass (40 bytes)
1778 made not entrant sun.reflect.DelegatingMethodAccessorImpl::invoke (10 bytes)
1780 made not entrant java.lang.ClassLoader::loadClass (58 bytes)
583 made not entrant java.util.ArrayList::remove (77 bytes)
1774 made zombie org.apache.activemq.util.ClassLoading::load (15 bytes)
1775 made zombie java.lang.ClassLoader::loadClass (7 bytes)
1771 made zombie sun.reflect.DelegatingMethodAccessorImpl::invoke (10 bytes)
1781 !b java.lang.reflect.Method::invoke (167 bytes)
1770 made zombie java.lang.reflect.Method::invoke (167 bytes)
1776 made not entrant org.apache.activemq.util.ClassLoading::loadClass (228 bytes)
1782 b org.apache.activemq.util.ClassLoading::load (15 bytes)

Overall the log lines appear at a rate of about 2 per second, so this excerpt represents about 10 seconds of clock time. One thing I didn't say in my first message is that I am testing this application on very low-power hardware (we are power-sensitive). The servers have 1Ghz VIA processors in them, so generally run slower than our fire-breathing 64-bit boxes. And I'm wondering if CPU speed has a bearing on the problem because we don't see such a disproportionately large amount of time spent in Compile Time on the faster processors. Perhaps the slow processor increases the likelihood that the "race" described in Steve's blog is lost much more frequently.

What shouts out from the log is that we are repeatedly invalidating and then recompiling methods related to serialization and reflection. This snippet is consistent with many thousands of additional lines, so clearly we are in some kind of cycle, although it does not repeatly exactly (to the eyeball).

Does anyone know the difference between "made zombie" and "made not entrant"? I had no luck with search engines.

And the major question, of course, is what can I do to help diagnose this problem?

Thanks again for your help.

Peter Beaman

linuxhippy
Offline
Joined: 2004-01-07
Points: 0

Well Hotspot-Server compiles very slowly, so it also could be a very long run-warm phase.
Hotspot server makes some assumptions and if they are broken at runtime code is recompiled with the broken assumptions de-optimized out. So maybe a lot of assumptions are broken and this means compilation takes a lot longer. I am no JVM engineer so I completly may be wrong. Maybe some optimizations are done in 6.0 which where missing in 5.0.

* What happens if you let it run for hours, not just one or two?
* Does the recompilation also happen on fast Systems?
* Does switching off biased locking help: -XX:-UseBiasedLocking ?
* Does the re-compilation also happen with 5.0?
* Do the last 7.0 builds work?

I would make sure that its really a JVM bug (let it run for days with PrintCompilation) and if it really recompiles all the time. If so, could you file a bug-report at Sun and submit a testcase - maybe a non-functional version of your software?
If you really need 6.0 you could also run with the client-compiler if the loss of performance is no problem.

Mfg Clemens