Skip to main content

Garbage collection - performance problem

2 replies [Last post]
ranjinie
Offline
Joined: 2008-11-23
Points: 0

Hi,

We are running an application in OC4j app server. When i restart the server,

Application runs normally. Each day pass, the garbage collection times increase and memory usage also increase. After 21days, here are the garbage collection time.
Garbage collector:
Name = 'PS MarkSweep', Collections = 15,294, Total time spent = 2 hours 39 minutes
Garbage collector:
Name = 'PS Scavenge', Collections = 685,935, Total time spent = 3 hours 12 minutes

When it reaches max memory, system throws out of memory error.

We have lots of Birt reports to generate by the system. It creates lots of objects.
Somehow the system is not releasing objects. is there anyway we can force the system to collect all objects end of the day. Can we set any parameter to run midnight to clean all objects.

Here are the results I was getting from the eclipse analyser.

Problem suspect 1:

The thread com.evermind.server.ApplicationServerThread @ 0x780f2928 AJPRequestHandler-RMICallHandler-213 keeps local variables with total size 66,662,744 (36.66%) bytes.

The memory is accumulated in one instance of "com.evermind.server.ApplicationServerThread" loaded by "oracle.classloader.PolicyClassLoader @ 0x74a35178".
The stacktrace of this Thread is available. See stacktrace.

Keywords
oracle.classloader.PolicyClassLoader @ 0x74a35178
com.evermind.server.ApplicationServerThread

Details »

Problem suspect 2:
One instance of "org.hibernate.impl.SessionFactoryImpl" loaded by "oracle.classloader.PolicyClassLoader @ 0x74a659a0" occupies 35,092,008 (19.30%) bytes. The memory is accumulated in one instance of "org.hibernate.impl.SessionFactoryImpl" loaded by "oracle.classloader.PolicyClassLoader @ 0x74a659a0".

Keywords
oracle.classloader.PolicyClassLoader @ 0x74a659a0
org.hibernate.impl.SessionFactoryImpl

Problem suspect 3:
One instance of "com.aafc.service.etl.impl.ETLServiceImpl" loaded by "oracle.classloader.PolicyClassLoader @ 0x74a659a0" occupies 31,032,768 (17.07%) bytes. The memory is accumulated in one instance of "java.util.HashMap$Entry[]" loaded by "".

Keywords
oracle.classloader.PolicyClassLoader @ 0x74a659a0
java.util.HashMap$Entry[]
com.aafc.service.etl.impl.ETLServiceImpl

Details »

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
grelf
Offline
Joined: 2007-11-14
Points: 0

It cannot be that the JVM is not releasing objects when it should. No JVM would be viable if it did that. No, the answer lies in your code I'm afraid. I think there's a clue in the last lines you showed: HashMap. A very common cause of memory leaks is that the programmer thinks every reference to an object should have gone but unfortunately there is still a collection (List, Map, whatever) holding a reference to the object. It is vital to clear such references. There are some classes that I believe are designed to help in exactly this situation, such as java.util.WeakHashMap, but make sure you fully understand them (test) before relying on them.

ranjinie
Offline
Joined: 2008-11-23
Points: 0

Hi

Now whatever processes we do in that site like create a report or upload a file. Server tries to do the work and hour later it times out . We have 3600 second setting in the server for time out. So it times out after one hour. We ran the same report in 2 minutes in dev environment.

I don't know what is hogging the memory and CPU.

I checked the jconsole. For garbage collection, it takes this much time. May be it takes too long for garbage collection (gc) that's why app times out because server wait for gc and then process app.
'PS MarkSweep', Collections = 306,866, Total time spent = 2 days 10 hours 22 minutes
'PS Scavenge', Collections = 3,010,212, Total time spent = 22 hours 29 minutes

i got the heap_dump. i analyse using Eclipse analyser. i have attached the file with the result.

HashMap and arraylist references are there. Those hashmaps are holding null values. i am not sure what i need to do to release the memory and CPU. If we restart the server, everything comes to normal for few weeks. How to fix without restarting the server. How I can check what is hogging in the CPU. Do i change the hashmap to weakHashmap where ever the hashmap hold values as null or change all hashmap to weakhashmap?

Can i send you the Heap dump file for your reference?

thank you!
Siva