Skip to main content

100% CPU usage

12 replies [Last post]
egarcia_sms
Offline
Joined: 2009-08-13
Points: 0

Hello.

I'm having some problems with my Glassfish 3.0.1 Final installation. I've deployed a single WAR application with a few libraries. Application works fine, but after a few hours or days, Glassfish eats 100% CPU.

I've checked with VisualVM with Glassfish extensions to check the cause, and the most I can figure out is that GC is firing every second, and it's strangely enough because there is free memory.

I know it isn't too much info, but maybe someone can point out something to try to rule out the cause, because all is working fine until something makes the GC go crazy, even when is no usage of the application.

Thanks a lot in advance!

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
tmpsa
Offline
Joined: 2010-02-01
Points: 0

I'm having the same problem. We're running a single webapp (with EJB/JPA) with Glassfish 3.0.1 on a Linux Ubuntu box.
From time to time (about once a day or so) top suddenly shows 100% CPU for the java process, and Glassfish totally hangs for 10-20 minutes. After that, it resumes normal operation. Not good, this is a production system.
During this "hang time" it's totally impossible to reach Glassfish or the webapp (nor the admin console). Glassfish also stops talking to VisualVM and JConsole, so I can't get any useful information that way either. Also, Linux gets zonked too; It's not possible to create new processes (so diagnosing during 'hang time' is nearly impossible).
This happens like a flashbolt out of the blue sky without any previous warning signs in VisualVM or JConsole. Bam!
One side-effect is that our 10-15 client programs, who talks to the webapp via HTTPS, don't get any responses, and try to reconnect. So we see lots of TCP connections in CLOSE_WAIT state. First I thought this was the problem, but it's most likely a consequence of the hung Glassfish.
We also run Apache on the same Linux box, and it works just fine during 'hang time'. (I guess it does not fork any processes).
I have increased a number of parameters:

  • the <jvm-options> in domain.xml according to J-F Arcands recommendations
  • Thread count (Max Thread Pool Size) on http-thread-pool from its default 5 to 100
  • sysctl -w net.core.somaxconn=1024 (default 128)

But to no avail.
Any suggestions?

chrisika
Offline
Joined: 2010-12-01
Points: 0

Hi,
we also face the same problem that Glassfish (3.0, running on Linux Ubuntu) takes 100% CPU from time to time. We could manage to make a dump of the 100% java thread:
&quot;http-thread-pool-5180-(19)&quot; daemon prio=10 tid=0x0000000042c03800 nid=0x248d runnable [0x00007f16668e2000]    java.lang.Thread.State: RUNNABLE     at com.sun.jersey.json.impl.reader.JsonXmlStreamReader.next(JsonXmlStreamReader.java:442)     at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:192)     at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360)     at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)     at com.sun.jersey.json.impl.BaseJSONUnmarshaller.unmarshalJAXBElementFromJSON(BaseJSONUnmarshaller.java:103)     at com.sun.jersey.json.impl.BaseJSONUnmarshaller.unmarshalFromJSON(BaseJSONUnmarshaller.java:92)     at com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.readFrom(JSONRootElementProvider.java:98)     at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:105)     at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:410)     at com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:137)     at com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:79)     at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:126)     at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:173)     at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)     at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:208)     at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:115)     at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:75)     at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:115)     at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:67)     at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:775)     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:740)     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:731)     at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:372)     at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:452)     at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:633)     at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)     at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)     at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)     at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)     at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)     at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:332)     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:233)     at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)     at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)     at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)     at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)     at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)     at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)     at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)     at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)     at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)     at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)     at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)     at com.sun.grizzly.ContextTask.run(ContextTask.java:69)     at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)     at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)     at java.lang.Thread.run(Thread.java:662)
We use REST web services with JSON data format. It seems that the thread hangs when JSON data is deserialized. Any suggestions why?
Kind regards,
Christian

oleksiys
Offline
Joined: 2006-01-25
Points: 0

It could be a selector spinning problem, in case if Grizzly workaround is not recognizing your OS as linux.
Which JDK version are you using?
Can you pls. check what will be the result of java call System.getProperty("os.name"); ?

tmpsa
Offline
Joined: 2010-02-01
Points: 0

/usr/lib/jvm/java-6-sun points to /usr/lib/jvm/java-6-sun-1.6.0.22
I have a JSP that shows the result of System.getProperties(), where I see the following key-value pairs (among others):
[os.arch=amd64]
[os.name=Linux]
[os.version=2.6.32-22-server]

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hmm, it means if would be a selector spinning problem - Grizzly should workaround it.
During the selector spinning, usually, only one CPU would be 100% loaded, so entire environment shouldn't hang, which is not the case.
So, I guess it's different issue. Is it possible to take a threads dump, when it hangs (like kill -3 <pid>)?

tmpsa
Offline
Joined: 2010-02-01
Points: 0

There's only one CPU, I run Glassfish on a virtualized server. So, yes, the "only" CPU is 100% loaded.
It appears that it's also impossible to open files during a hang. Perhaps that's the reason why I can't start new processes
either. I have a hunch that the system also has run out of file descriptors, there are several entries in server.log about
too many open files. These occur just when Glassfish comes out of its 20-minute seizure. The first is:

[#|2010-11-24T21:24:57.326+0100|WARNING|glassfish3.0.1|sun.rmi.transport.tcp|_ThreadID=46;_ThreadName=Thread-1;|RMI TCP
Accept-8686: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8686] throws java.net.SocketException: Too
many open files

I had a hang yesterday (2010-11-24), and tried to make a thread dump with kill -3 <pid>. Unfortunately, jvm.log was not
written to until *after* the hang had ceded, so I don't know if the dump is useful. Or perhaps the dump was taken
immediately, but the output was buffered until the hang had ceded. (As I said, the system can't open files). The dump is
temporarily available at http://hacke.saeab.se/tmp/jvm.log .

Is there a way to check if I have ran into that "Grizzly spin" problem?

(I have been trying to respond to your post for many days, but get redirected to http://www.java.net/spam/denied all the
time!) :-(

Satyajit Tripathi

Hi Garcia,

The issue needs controlled investigation and detailed analysis. However,
here is just a suggestion if you are on Solaris 10 platform.

You may use DTrace toolkit (customizable) to debug and locate the cause
of the problem. If you need quick and simplified reference to DTrace,
please see
http://blogs.sun.com/stripathi/resource/Preso/SolarisDTrace_Simplified.pdf.
Specific DTrace probes are available for Hotspot JVM.

Please check the JVM options and other tuning schemes to influence the
current behavior of GC.

Thanks & regards
--Satya

On 09/07/10 13:44, glassfish@javadesktop.org wrote:
> Hello.
>
> I'm having some problems with my Glassfish 3.0.1 Final installation. I've deployed a single WAR application with a few libraries. Application works fine, but after a few hours or days, Glassfish eats 100% CPU.
>
> I've checked with VisualVM with Glassfish extensions to check the cause, and the most I can figure out is that GC is firing every second, and it's strangely enough because there is free memory.
>
> I know it isn't too much info, but maybe someone can point out something to try to rule out the cause, because all is working fine until something makes the GC go crazy, even when is no usage of the application.
>
> Thanks a lot in advance!
> [Message sent by forum member 'egarcia_sms']
>
> http://forums.java.net/jive/thread.jspa?messageID=482013
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SATYAJIT TRIPATHI
ISV Engineering APAC

Sun Microsystems India Private Limited
Bangalore 560025
DID : +91 80 66937865
Mobile: +91 9886019892
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

egarcia_sms
Offline
Joined: 2009-08-13
Points: 0

Thanks for your reply, too!

Unfortunately, I'm not on a Solaris platform, but a Linux one.
Maybe I'll try DTrace for Linux to see if I can get more info from the calls, don't know if this is possible, though. Thank you very much.

Deepu Syamaladevi Janardhananachary (UST, IND)

If you are creating any thread explicitly even though it is a web application?
If yes, check that portion.

Are you having any code part in which multiple creation of some connection or some thing like that, which should be better if created once?
If yes, use singleton method.

Regards,
Deepu Janardhananachary
[winmail.dat]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Martin Gainty

i would suggest synchronize your methods

Martin
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.

Date: Fri, 10 Sep 2010 22:16:52 +0530
From: Deepu.Janardhananachary@ust-global.com
To: users@glassfish.dev.java.net
Subject: RE: Re: 100% CPU usage

If you are creating any thread explicitly even though it is a web application?
If yes, check that portion.

Are you having any code part in which multiple creation of some connection or some thing like that, which should be better if created once?
If yes, use singleton method.

Regards,
Deepu Janardhananachary

--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net For additional commands, e-mail: users-help@glassfish.dev.java.net
[att1.html]

Craig Ringer

On 7/09/2010 4:14 PM, glassfish@javadesktop.org wrote:
> I've checked with VisualVM with Glassfish extensions to check the cause, and the most I can figure out is that GC is firing every second, and it's strangely enough because there is free memory.

Do you have any explicit calls to System.gc() in apps that're deployed
on the server ?

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

egarcia_sms
Offline
Joined: 2009-08-13
Points: 0

Thanks for your reply!

No, I'm not using any explicit call to GC