Skip to main content

glass fish hanging or unresponsive

12 replies [Last post]
madmac
Offline
Joined: 2006-04-26

We recently created a 2 node cluster with Glassfish 2.1. We are running SJSWS as the web server. We have noticed that a couple times a day the node agents become unresponsive (not usually at the same time) The node agents appear healthy so the web server continues to send requests to it, but it just hangs while trying process the request. Only restarting the node agent brings it up. We weren't sure if this was a cluster issue, so on a separate box, but identical hardware, we set up a stand alone app server. We are experiencing the same issue where the server appears to be up and healthy, no errors in the logs, but it just stops processing requests. Anyone else run into this? Many thanks to any assistance.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Dominik Dorn

It gets reloaded by me while I'm developing by touching the .reload file.

Somewhere there has to be a memory leak. However, even if such a leak
exists, imo it should
not destroy the applications configuration.

Is there a defined way to track such issues down and report it appropriately?

On Thu, Jan 14, 2010 at 3:45 PM, wrote:
> So the application seemed to get reloaded many times? How did you find this out? Anything in the server.log indicating this? Supposely if the ".reload" file timestamp has not changed, it should not trigger another reloading after the initial one.
> [Message sent by forum member 'hzhang_jn' (hong.zhang@sun.com)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380945
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
Dominik Dorn
http://dominikdorn.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

culli
Offline
Joined: 2006-09-17

FYI, VisualVM comes with Java 6 jdk. And there is a Glassfish plugin for it. The plugin is a very limited version of what you get when you buy support (Glassfish performance monitor). https://visualvm.dev.java.net/plugins.html

madmac
Offline
Joined: 2006-04-26

I've setup VisulaVM now I just need to wait for the crash. In looking at Procmon, I saw a bunch of failures related to Glassfish, specifically of reloading my apps. I didn't realize that autodeploy and reload were both enabled, so I've disabled those now. Hopefully that was the problem.

Dominik Dorn

Reloading is a bit broken in my opinion.. when doing heavy development
with some stateless session beans and reloading it around 20 times, I
get a PermGen-Error, having to kill glassfish by kill -9, then remove
the stuff it created in glassfish/domains/domain1/... for this
application and now deploy the application again.

On Wed, Jan 13, 2010 at 4:52 PM, wrote:
> I've setup VisulaVM now I just need to wait for the crash.  In looking at Procmon, I saw a bunch of failures related to Glassfish, specifically of reloading my apps.  I didn't realize that autodeploy and reload were both enabled, so I've disabled those now.  Hopefully that was the problem.
> [Message sent by forum member 'madmac' (jeff.mcguire@stanct.org)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380665
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
Dominik Dorn
http://dominikdorn.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hzhang_jn
Offline
Joined: 2005-07-22

So the application seemed to get reloaded many times? How did you find this out? Anything in the server.log indicating this? Supposely if the ".reload" file timestamp has not changed, it should not trigger another reloading after the initial one.

madmac
Offline
Joined: 2006-04-26

yes, it was getting reloaded every few minutes. We are on Windows and we used Procmon to monitor every process. Unfortunately we are still having the same problem with the server hanging, but it doesn't look like it's trying to reload the apps anymore.

Felipe Gaucho

May be resources ... Check db connections and jms... Check the log

On 14.01.2010, at 16:58, glassfish@javadesktop.org wrote:

> yes, it was getting reloaded every few minutes. We are on Windows
> and we used Procmon to monitor every process. Unfortunately we are
> still having the same problem with the server hanging, but it
> doesn't look like it's trying to reload the apps anymore.
> [Message sent by forum member 'madmac' (jeff.mcguire@stanct.org)]
>
> http://forums.java.net/jive/thread.jspa?messageID=380969
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

madmac
Offline
Joined: 2006-04-26

I was able to get a thread dump when it crashed yesterday. It doesn't look like anything was in monitor status, but I'm not really sure what to make of it. I've attached the thread dump in case anyone wants to take a look. Thanks for your help.

culli
Offline
Joined: 2006-09-17

I have run into this with RMI when the IIOP listener address was not configured (the default is 0.0.0.0). That's just a stab in the dark though. We'll need more info.

What type of application are you running: web client, or remote ejb?

Maybe pull a thread dump on the standalone when it's locked. https://visualvm.dev.java.net/threads.html
I bet you'll see a bunch of threads in monitor state (all waiting for some thread). The thread they're all waiting on can give a clue what's going on.

Regards,
Jim

madmac
Offline
Joined: 2006-04-26

Thanks for the response. I haven't done anything with the IIOP listener and am not sure how I would need to configure it.

We are running web applications. No ejb's at this point.

Thanks for the tip on the threads. I'll download the visulavm and take a look. I'll let you know what I find.

Jeff

km
Offline
Joined: 2005-10-28

When this happens, can you generate the jstack output (or kill -3 output) for the server that appears hung?
- find the process that's hung.
- find its pid (jps -lm | grep PE if this v2, | grep ASMain if it is v3).
- jstack
or kill -3
and send the jvm.log file or jstack output.

-Kedar

madmac
Offline
Joined: 2006-04-26

I should have mentioned this is running on a Windows 2008 server.