Skip to main content

Bad performance when moving from 1.5 to 1.6 with GC as ConcMarkSweepGC

5 replies [Last post]
sbaronia
Offline
Joined: 2009-04-22

After moving from JDK 1.5 to 1.6 using ConcMarkSweepGC, the program started losing msg coming from multicast cloud.

We have a hashMap which is growing upto 2 Millions.On average it receives 1 Millions msgs around half are added and other half removes transactions. In worse case, we have 2 to 4 million msgs. Since the message have to be serially handled, so we have three threads. one for retrieving msg from multicast socket.
other for processing msg.
last outputing on a multicast socket.

Processing thread is add /remove /modified hashMap. 45% msgs are add, 44% remove, and remaining is modifying.

500 M to 1.5 M messages are received in a day.

But we lose messages at the incoming multicast socket layer.

Linux Red Hat
JVM is 64 bit 1.6.3
66GB RAM
24 CPU

Options:
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:PrintCMSStatistics=1 -XX:+PrintGCTaskTimeStamps -XX:+PrintTenuringDistribution -XX:PrintFLSStatistics=1
-Xmx5G -Xms3G -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+UseParNewGC -XX:ParallelGCThreads=3 -XX:NewSize=300m -XX:MaxNewSize=300m

Garbage collector:

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
peter__lawrey
Offline
Joined: 2005-11-01

There is no JVM 1.6.3, perhaps you mean 1.6.0_03. In this case I suggest you try 1.6.0_13
The GC has only a very tenuous connection with UDP (In that you need to create objects to receive UDP traffic). There is no GC setting which relates with TCP or UDP traffic.

You say you have 2 million messages, but what is the rate of message per second?

sbaronia
Offline
Joined: 2009-04-22

Peter,

Usually the number message could vary, but I have seen program losing msgs even when the number of msgs are low i.e. 800K/min.
Peak is 100K msgs per sec
Avg is 22K msg per sec

I have tried 1.6.0_10 with similar or better result, but they never came close to 1.5.
I will try 1.6.0_13

Thanks
/Sharad

peter__lawrey
Offline
Joined: 2005-11-01

I have tended to avoid UDP when reliable transport is needed.
There are solutions which work with Java 1.6 which I have used, so I'm believe Java 1.6 can be made to work without having to change the GC settings.
I would need to be convinced the GC is cause, except you do have a high peak number of messages.
I may be that you have to use Java 1.5...

sbaronia
Offline
Joined: 2009-04-22

Thanks Peter.
We don't have option to go back to Java 1.5

Do you know a good tool for tuning GC? (other than visual GC, GC viewer, JConsole)

One thing I have notice that old generation increases memory by 8M for every minor GC collection.
After 2 or 3 full GC collection, it loses more messages. Is because of fragmentation in the memory.

I know Java is coming out with G1 Garbage collector which will replace CMS. G1 will handled memory fragmentation properly by design.
I have downloaded it Java 1.6 update 14 beta.

Do you thing it make sense to try G1?

/Sharad

linuxhippy
Offline
Joined: 2004-01-07

I have to agree with peter, nothing in your logs seems to indicate the GC is the root of the problem.
Especially I don't believe e.g. that because of fragmentation you loose messages, where do you see an evidence for that?

The only thing I could immagine are CMS failures causing a fallback to a stop-the-world collection (this can be caused by fragmentation), but your logs don't indicate that.

- Clemens