Skip to main content

Glassfish Hang - Daily

15 replies [Last post]
badul
Offline
Joined: 2010-01-12

Guys,

We develop a Hospital Information System that manage patient registration, appoinment and billing. Currently we have deploy our system in 3 hospital and we got problem where we have to restart glassfish server on daily basis to avoid glassfish from getting hang.

Below is some information about our configuration and server :

OS : Red Hat Enterprise Linux 5.3 kernel 2.6.18.8.el5
Apps Server : Sun Glassfish Enterprise Server v2.1
RAM : 16GB

JVM Setting :

-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=50
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-verbose:gc
-XX:CMSInitiatingOccupancyFraction=1
-XX:LargePageSizeInBytes=256m
-XX:ParallelGCThreads=2
-Xmn2g
-Xms8G
-Xmx8G
-d64
-XX:-UseParallelOldGC
-XX:+UseConcMarkSweepGC
-XX:MaxPermSize=192m
-server
-Djava.endorsed.dirs=${com.sun.aas.installRoot}/lib/endorsed
-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dsun.rmi.dgc.client.gcInterval=3600000
-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext${path.separator}${com.sun.aas.derbyRoot}/lib
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
-Dcom.sun.enterprise.taglisteners=jsf-impl.jar
-XX:NewRatio=2

I also attach some snapshot from the server itself during hang happen. In the early stage of system going live we already come across this problem plus outofmemory error and already report it to local sun support and I also asked opinion from some java expert in my office. What they suggest just to increase the -Xms and -Xmx from 2G to 8G plus some other changes to jvm setting.

For me this is not a good suggestion and it will help us for temporary only. As I told you this thing already happen again. What I suspect is some serious memory leaks from the application itself (you can see in the attach files). I think the -Xms and -Xmx already high enough.

Additional info :
Our system run on office hour basis 9am to 6pm only. The screenshot is from 2 different sites.

My question :
1. Is the 8GB setting for jvm is normal for java application? (8GB for 1 site, another 2 site is using 12GB ; other setting would be the same)
2. What your opinion/suggestion for our problem.
3. If you think it is memory leaks, how can I proof it to the development team?

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
badul
Offline
Joined: 2010-01-12

Thanks guys for your help.

I manage to get the source code of the loginFilter from the development team. I attach the file with this post.

I already told them that something wrong with this code in the early stage but they dont believe me. Even when I show you guys this post they still playing defensive with me. Don't know how to tell them anymore.

Dominik Dorn

The doFilter() method is ~300 lines long... it's nothing special that
errors hide themselves there.. if there's any chance you can get them
working on
this again, they should split up the method in smaller chunks, doing
one thing at a time.
Its then easier to debug and easier to find the right place where this
lock is exactly happening.

also, the whole doFilter method is synchronized, leading at least to
poor performance
and maybe also to the lock mentioned... they should only synchronize
whats really necessary.

On Fri, Jan 15, 2010 at 5:35 AM, wrote:
> Thanks guys for your help.
>
> I manage to get the source code of the loginFilter from the development team. I attach the file with this post.
>
> I already told them that something wrong with this code in the early stage but they dont believe me. Even when I show you guys this post they still playing defensive with me. Don't know how to tell them anymore.
> [Message sent by forum member 'badul' (dzafarul@gmail.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=381081
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
Dominik Dorn
http://dominikdorn.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

culli
Offline
Joined: 2006-09-17

> 1. Change the compiler from "client" to "server"

I happened to read this yesterday:
http://wiki.glassfish.java.net/Wiki.jsp?page=Faq64bitConfig

According to that you need -d64 instead of -server to actually get the 64-bit JVM.

While you are not getting a PermGen space outofmemory error, you should probably monitor your PermGen memory usage too, which you can do with JVisualVM. We find that we can run out of PermGen easily, especially when deploying updates. Glassfish default configuration already has 192m PermGen, but you may need more.

Regards,
Jim

Felipe Gaúcho

3. If you think it is memory leaks, how can I proof it to the development team?

it sound like a memory leak.. it is hard to find in a general case..
what you can do is to keep the memory profiler turned on and repeat
some uses cases few times.. every time you finish a use case (client
session ends, for example) the memory used ruding the session should
be returned to the system ..... (perhaps take a little time due the
GC) .. and then, after repeating that use cases few times you will
have a proof about that......

if you are very lucky, you will find te problem doing that.. not easy...

otherwise, start the same workflow and measre each small step.. after
the user to login in, after the user to open an account, etc.

On Tue, Jan 12, 2010 at 9:28 AM, wrote:
> Guys,
>
> We develop a Hospital Information System that manage patient registration, appoinment and billing. Currently we have deploy our system in 3 hospital and we got problem where we have to restart glassfish server on daily basis to avoid glassfish from getting hang.
>
> Below is some information about our configuration and server :
>
> OS : Red Hat Enterprise Linux 5.3 kernel 2.6.18.8.el5
> Apps Server : Sun Glassfish Enterprise Server v2.1
> RAM : 16GB
>
> JVM Setting :
>
> -XX:+UseCMSInitiatingOccupancyOnly
>        -XX:+UseCMSCompactAtFullCollection
>        -XX:CMSFullGCsBeforeCompaction=50
>        -XX:+PrintGCTimeStamps
>        -XX:+PrintGCDetails
>        -verbose:gc
>        -XX:CMSInitiatingOccupancyFraction=1
>        -XX:LargePageSizeInBytes=256m
>        -XX:ParallelGCThreads=2
>        -Xmn2g
>        -Xms8G
>        -Xmx8G
>        -d64
>        -XX:-UseParallelOldGC
>        -XX:+UseConcMarkSweepGC
>        -XX:MaxPermSize=192m
>        -server
>        -Djava.endorsed.dirs=${com.sun.aas.installRoot}/lib/endorsed
>        -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
>        -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
>        -Dsun.rmi.dgc.server.gcInterval=3600000
>        -Dsun.rmi.dgc.client.gcInterval=3600000
>        -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
>        -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
>        -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext${path.separator}${com.sun.aas.derbyRoot}/lib
>        -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>        -Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
>        -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>        -Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
>        -Dcom.sun.enterprise.taglisteners=jsf-impl.jar
>        -XX:NewRatio=2
>
> I also attach some snapshot from the server itself during hang happen. In the early stage of system going live we already come across this problem plus outofmemory error and already report it to local sun support and I also asked opinion from some java expert in my office. What they suggest just to increase the -Xms and -Xmx from 2G to 8G plus some other changes to jvm setting.
>
> For me this is not a good suggestion and it will help us for temporary only. As I told you this thing already happen again. What I suspect is some serious memory leaks from the application itself (you can see in the attach files). I think the -Xms and -Xmx already high enough.
>
> Additional info :
> Our system run on office hour basis 9am to 6pm only. The screenshot is from 2 different sites.
>
> My question :
> 1. Is the 8GB setting for jvm is normal for java application? (8GB for 1 site, another 2 site is using 12GB ; other setting would be the same)
> 2. What your opinion/suggestion for our problem.
> 3. If you think it is memory leaks, how can I proof it to the development team?
> [Message sent by forum member 'badul' (dzafarul@gmail.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380228
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
------------------------------------------
Felipe Gaúcho
10+ Java Programmer
CEJUG Senior Advisor

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

badul
Offline
Joined: 2010-01-12

Guys,

I already capture the thread dump when our system hang today. Please help me to analyze it. I attach it with this post.

Thanks.

alexismp
Offline
Joined: 2005-01-06

Where can I get more information on com.htp.filter.LoginFilter.doFilter ?
Sounds fishy.

culli
Offline
Joined: 2006-09-17

I'll have to agree with Alex. What's going on in com.htp.filter.LoginFilter ? Sounds like there is some non-threadsafe code there. That's your code, right?

That's where several locks are waiting in your threads:
waiting to lock <0x00002aaaf923e988> (a com.htp.filter.LoginFilter)

badul
Offline
Joined: 2010-01-12

Thanks guys for your help.

I manage to get the source code of the loginFilter from the development team. I attach the file with this post.

I already told them that something wrong with this code in the early stage but they dont believe me. Even when I show you guys this post they still playing defensive with me. Don't know how to tell them anymore.

Alexis Moussine-Pouchkine

That's a lot of JVM tuning that you have here (some of which I've never seen before)...

What would be interesting is to get a heap-dump (kill -3, jstack) of the hung process if possible.
The other important data would be the reason for the outofmemory error (there could be multiple reasons for it).

Also when moving from a 2GB to an 8GB heap did you start using a 64-bit JVM or were you using it from the beginning?
If you don't require that you may want to consider going back to 2GB and a 32-bit JVM (the reason for the OOM error would help here).
64-bit JVMs are usually needed when you have large sets of data in memory.

Also I would recommend trying GlassFish 2.1.1 if possible.
Memory "leaks" can usually be observed when an explicit call to the GC doesn't reclaim all the expected memory.

-Alexis

On 12 janv. 2010, at 09:28, glassfish@javadesktop.org wrote:

> Guys,
>
> We develop a Hospital Information System that manage patient registration, appoinment and billing. Currently we have deploy our system in 3 hospital and we got problem where we have to restart glassfish server on daily basis to avoid glassfish from getting hang.
>
> Below is some information about our configuration and server :
>
> OS : Red Hat Enterprise Linux 5.3 kernel 2.6.18.8.el5
> Apps Server : Sun Glassfish Enterprise Server v2.1
> RAM : 16GB
>
> JVM Setting :
>
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseCMSCompactAtFullCollection
> -XX:CMSFullGCsBeforeCompaction=50
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDetails
> -verbose:gc
> -XX:CMSInitiatingOccupancyFraction=1
> -XX:LargePageSizeInBytes=256m
> -XX:ParallelGCThreads=2
> -Xmn2g
> -Xms8G
> -Xmx8G
> -d64
> -XX:-UseParallelOldGC
> -XX:+UseConcMarkSweepGC
> -XX:MaxPermSize=192m
> -server
> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/lib/endorsed
> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
> -Dsun.rmi.dgc.server.gcInterval=3600000
> -Dsun.rmi.dgc.client.gcInterval=3600000
> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext${path.separator}${com.sun.aas.derbyRoot}/lib
> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
> -Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
> -Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
> -Dcom.sun.enterprise.taglisteners=jsf-impl.jar
> -XX:NewRatio=2
>
> I also attach some snapshot from the server itself during hang happen. In the early stage of system going live we already come across this problem plus outofmemory error and already report it to local sun support and I also asked opinion from some java expert in my office. What they suggest just to increase the -Xms and -Xmx from 2G to 8G plus some other changes to jvm setting.
>
> For me this is not a good suggestion and it will help us for temporary only. As I told you this thing already happen again. What I suspect is some serious memory leaks from the application itself (you can see in the attach files). I think the -Xms and -Xmx already high enough.
>
> Additional info :
> Our system run on office hour basis 9am to 6pm only. The screenshot is from 2 different sites.
>
> My question :
> 1. Is the 8GB setting for jvm is normal for java application? (8GB for 1 site, another 2 site is using 12GB ; other setting would be the same)
> 2. What your opinion/suggestion for our problem.
> 3. If you think it is memory leaks, how can I proof it to the development team?
> [Message sent by forum member 'badul' (dzafarul@gmail.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380228
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

badul
Offline
Joined: 2010-01-12

The JVM tuning is based on suggestion by Local Sun Support and some from our system engineer itself. I think it is based on trial and error basis.

I will try the heap-dump if this thing happen again tomorrow. I attach the screenshot of garbage collection using YourKit Java Profiler. This screenshot I capture on testing server @my office. During the GC I'm the only person that use the system.

We are using the 64-bit JVM right now and start using it from the beginning as suggested by Sun. If we roll back to 2GB our system will become worst for sure.

I will provide information regarding GC tomorrow. For time being I provide the GC data for testing server at the office.

3.996: [GC [PSYoungGen: 503316K->12040K(1835008K)] 503316K->12040K(3932160K), 0.0597320 secs] [Times: user=0.09 sys=0.04, real=0.06 secs]
4.056: [Full GC (System) [PSYoungGen: 12040K->0K(1835008K)] [PSOldGen: 0K->11945K(2097152K)] 12040K->11945K(3932160K) [PSPermGen: 20964K->20964K(42240K)], 0.1537960 secs] [Times: user=0.13 sys=0.02, real=0.15 secs]

Thanks.

> That's a lot of JVM tuning that you have here (some
> of which I've never seen before)...
>
> What would be interesting is to get a heap-dump (kill
> -3, jstack) of the hung process if possible.
> The other important data would be the reason for the
> outofmemory error (there could be multiple reasons
> for it).
>
> Also when moving from a 2GB to an 8GB heap did you
> start using a 64-bit JVM or were you using it from
> the beginning?
> If you don't require that you may want to consider
> going back to 2GB and a 32-bit JVM (the reason for
> the OOM error would help here).
> 64-bit JVMs are usually needed when you have large
> sets of data in memory.
>
> Also I would recommend trying GlassFish 2.1.1 if
> possible.
> Memory "leaks" can usually be observed when an
> explicit call to the GC doesn't reclaim all the
> expected memory.
>
> -Alexis
>

Alexis Moussine-Pouchkine

On 12 janv. 2010, at 10:41, glassfish@javadesktop.org wrote:

> The JVM tuning is based on suggestion by Local Sun Support and some from our system engineer itself. I think it is based on trial and error basis.

Well, it's hard to get it right with all the knobs you have available...

> I will try the heap-dump if this thing happen again tomorrow.

Also don't forget to look for the exact OOM error message.

> I attach the screenshot of garbage collection using YourKit Java Profiler. This screenshot I capture on testing server @my office. During the GC I'm the only person that use the system.
>
> We are using the 64-bit JVM right now and start using it from the beginning as suggested by Sun. If we roll back to 2GB our system will become worst for sure.

It's possible that you need more than 4GB (the theoretical 32-bit limit) but it needs to be explained by the application behavior.
Is the app caching lots of data?

-Alexis

> I will provide information regarding GC tomorrow. For time being I provide the GC data for testing server at the office.
>
> 3.996: [GC [PSYoungGen: 503316K->12040K(1835008K)] 503316K->12040K(3932160K), 0.0597320 secs] [Times: user=0.09 sys=0.04, real=0.06 secs]
> 4.056: [Full GC (System) [PSYoungGen: 12040K->0K(1835008K)] [PSOldGen: 0K->11945K(2097152K)] 12040K->11945K(3932160K) [PSPermGen: 20964K->20964K(42240K)], 0.1537960 secs] [Times: user=0.13 sys=0.02, real=0.15 secs]
>
> Thanks.
>
>> That's a lot of JVM tuning that you have here (some
>> of which I've never seen before)...
>>
>> What would be interesting is to get a heap-dump (kill
>> -3, jstack) of the hung process if possible.
>> The other important data would be the reason for the
>> outofmemory error (there could be multiple reasons
>> for it).
>>
>> Also when moving from a 2GB to an 8GB heap did you
>> start using a 64-bit JVM or were you using it from
>> the beginning?
>> If you don't require that you may want to consider
>> going back to 2GB and a 32-bit JVM (the reason for
>> the OOM error would help here).
>> 64-bit JVMs are usually needed when you have large
>> sets of data in memory.
>>
>> Also I would recommend trying GlassFish 2.1.1 if
>> possible.
>> Memory "leaks" can usually be observed when an
>> explicit call to the GC doesn't reclaim all the
>> expected memory.
>>
>> -Alexis
>>
> [Message sent by forum member 'badul' (dzafarul@gmail.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380258
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

badul
Offline
Joined: 2010-01-12

Below is the setting that suggested by Sun Engineer during their visit to customer site in the early stage. FYI, the suggested configuration still didn't help and their already collect the thread dump during that time. But as I know they still didn't come back yet with the outcome.

1. Change the compiler from "client" to "server"
2. Added Initial heap size to "1024M" (-Xms settting)
3. Added parallel garbage collector setting -XX:+UseParallelOldGC
4. Added Concurrent Garbage Collector setting -XX:+UseConcMarkSweepGC

After that our engineer change the jvm setting to what I've given earlier and the OOM error don't show up but the system still hang till today.

This is the OOM message during that time.

[#|2009-11-10T14:35:43.738+0800|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=13;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=d4766bf0-6703-4a2e-8ddf-9ec11185c4ea;|ApplicationDispatcher[/sppekl] PWC1231: Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
at java.lang.Class.getDeclaredField(Class.java:1880)
at com.thoughtworks.xstream.mapper.AttributeMapper.getField(AttributeMapper.java:120)
at com.thoughtworks.xstream.mapper.AttributeMapper.getConverterFromAttribute(AttributeMapper.java:101)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:145)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
|#]

Some warning message that I capture this evening.

[#|2010-01-12T14:42:10.100+0800|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=httpSSLWorkerThread-8080-2;_RequestID=0636be83-81de-4690-be54-23a7749d99f2;|
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at com.htp.filter.LoginFilter.doFilter(LoginFilter.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
|#]

Is there any ways to know our application is caching lots of data?

Alexis Moussine-Pouchkine

The data I was looking for is "java.lang.OutOfMemoryError: Java heap space"
I trust the GC to do its work properly and I sounds like you need to chase a memory "leak".
A profiler like yourkit should help you track objects that are not GC'd on explicit calls to the GC but it's likely that you will need a developer familiar with the application.
-Alexis

On 12 janv. 2010, at 12:36, glassfish@javadesktop.org wrote:

> Below is the setting that suggested by Sun Engineer during their visit to customer site in the early stage. FYI, the suggested configuration still didn't help and their already collect the thread dump during that time. But as I know they still didn't come back yet with the outcome.
>
> 1. Change the compiler from "client" to "server"
> 2. Added Initial heap size to "1024M" (-Xms settting)
> 3. Added parallel garbage collector setting -XX:+UseParallelOldGC
> 4. Added Concurrent Garbage Collector setting -XX:+UseConcMarkSweepGC
>
> After that our engineer change the jvm setting to what I've given earlier and the OOM error don't show up but the system still hang till today.
>
> This is the OOM message during that time.
>
> [#|2009-11-10T14:35:43.738+0800|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=13;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=d4766bf0-6703-4a2e-8ddf-9ec11185c4ea;|ApplicationDispatcher[/sppekl] PWC1231: Servlet.service() for servlet jsp threw exception
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
> at java.lang.Class.getDeclaredField(Class.java:1880)
> at com.thoughtworks.xstream.mapper.AttributeMapper.getField(AttributeMapper.java:120)
> at com.thoughtworks.xstream.mapper.AttributeMapper.getConverterFromAttribute(AttributeMapper.java:101)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:145)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
> at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
> at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
> at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
> at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
> at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
> |#]
>
> Some warning message that I capture this evening.
>
> [#|2010-01-12T14:42:10.100+0800|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=httpSSLWorkerThread-8080-2;_RequestID=0636be83-81de-4690-be54-23a7749d99f2;|
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at com.htp.filter.LoginFilter.doFilter(LoginFilter.java:98)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
> at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
> at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
> at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
> |#]
>
> Is there any ways to know our application is caching lots of data?
> [Message sent by forum member 'badul' (dzafarul@gmail.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380280
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Wayne Fay

You may find some of the JavaOne 2009 session presentations useful for
dealing with these kinds of problems such as...

"Monitoring and Troubleshooting Java Platform Applications with JDK
Software" (BOF-4724)
http://blogs.sun.com/nbprofiler/entry/javaone_2009_slides_monitoring_and

"Monitoring and Troubleshooting GlassFish Application Server in the
Wild" (Community One)
don't have a link, sorry, assume it is posted online, check the aquarium?

Also, VisualVM 1.2.2 was just released a few days ago which might be helpful:
http://blogs.sun.com/nbprofiler/entry/visualvm_1_2_2_released

Wayne

On Tue, Jan 12, 2010 at 7:53 AM, Alexis Moussine-Pouchkine
wrote:
> The data I was looking for is "java.lang.OutOfMemoryError: Java heap space"
> I trust the GC to do its work properly and I sounds like you need to chase a memory "leak".
> A profiler like yourkit should help you track objects that are not GC'd on explicit calls to the GC but it's likely that you will need a developer familiar with the application.
> -Alexis
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Wayne Fay

> "Monitoring and Troubleshooting GlassFish Application Server in the
> Wild" (Community One)
> don't have a link, sorry, assume it is posted online, check the aquarium?

Found it:
http://www.slideshare.net/SteveMillidge/monitoring-and-tuning-glassfish-...

Wayne

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net