Skip to main content

Server stops responding due to Glassfish

68 replies [Last post]
kevinmacdonald
Offline
Joined: 2007-12-18

After a few hours HTTP requests consistently stop reaching the web application. The java application continues to run, but HTTP requests are not being passed to servlets. If I go into the admin console and fiddle with an HTTP Service setting, like enabling access logging for example, POOF!! the site comes back to life. I can only assume that doing that causes something to reset in Glassfish. Clearly this is a Glassfish issue and nothing to do with the application code. Maddening and unacceptable. My only option is to get off Glassfish and onto something more stable. I am stopping/starting the domain on my production servers constantly. If anyone has a suggestion to make this problem go away I will be very grateful.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
bkatnich
Offline
Joined: 2006-02-03

Roisin,

My company is interested in looking at this, and sooner rather than later. Can I get some response on these questions?

It's best coming from an engineer such as yourself. The front line support people appear to be clueless about these technical questions I ask through the subscription website.

Ryan de Laplante

Excellent! Your thread dump looks quite different than mine: I don't
see any threads that are "BLOCKED" and any mention of ResourceManager.

Now we have to wait for Jeanfrancois Arcand to look through this. He is
the Sun employee who created Grizzly and is the one who's been helping
me. Jeanfrancois lives in Canada and today is a holiday, so hopefully
he will have time to look at it tomorrow.

Thanks,
Ryan

glassfish@javadesktop.org wrote:
> Okay, I ran it with Grizzly just as before and waited for it crap out again. Attached is the output of the command:
>
> asadmin generate-jvm-report --type=thread > thread_dump.txt
> [Message sent by forum member 'bkatnich' (bkatnich)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275116
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Ryan de Laplante

I also attached your thread dump to this bug ticket which has other Sun
people on my support case watching it:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=5030

Ryan de Laplante wrote:
> Excellent! Your thread dump looks quite different than mine: I
> don't see any threads that are "BLOCKED" and any mention of
> ResourceManager.
> Now we have to wait for Jeanfrancois Arcand to look through this. He
> is the Sun employee who created Grizzly and is the one who's been
> helping me. Jeanfrancois lives in Canada and today is a holiday, so
> hopefully he will have time to look at it tomorrow.
>
> Thanks,
> Ryan
>
>
> glassfish@javadesktop.org wrote:
>> Okay, I ran it with Grizzly just as before and waited for it crap out
>> again. Attached is the output of the command:
>>
>> asadmin generate-jvm-report --type=thread > thread_dump.txt
>> [Message sent by forum member 'bkatnich' (bkatnich)]

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bkatnich
Offline
Joined: 2006-02-03

Just want to reiterate that the version I am using is:

Sun Java System Application Server Platform Edition 9.0_01 (build b02-p01) which I believe maps to Glassfish v1 update 1.

It's an older version I know, but upgrading to the newer 9.1 is not an option at the moment. If it turns out this has been fixed in a subsequent patch release of 9.0 can someone let me know how to get a hold of the patch?

Jeanfrancois Arcand

Hi,

glassfish@javadesktop.org wrote:
> Just want to reiterate that the version I am using is:
>
> Sun Java System Application Server Platform Edition 9.0_01 (build b02-p01) which I believe maps to Glassfish v1 update 1.

Right, this version isn't using NIO for SSL:

> Thread "SelectorThread-43921" thread-id 70 thread-stateRUNNABLERunning in native
> at: java.net.SocketInputStream.socketRead0(Native Method)
> at: java.net.SocketInputStream.read(SocketInputStream.java:129)
> at: com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at: com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
> at: com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
> at: com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
> at: com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
> at: com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)
> at: org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:124)
> at: com.sun.enterprise.web.connector.grizzly.SelectorThread.startBlockingMode(SelectorThread.java:1143)
> at: com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1095)
> at: com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1068)

Can you add in domain.xml:

-Dcom.sun.enterprise.server.ss.ASQuickStartup=false

That will fix the issue :-)

>
> It's an older version I know, but upgrading to the newer 9.1 is not an option at the moment. If it turns out this has been fixed in a subsequent patch release of 9.0 can someone let me know how to get a hold of the patch?
> [Message sent by forum member 'bkatnich' (bkatnich)]
If you still see the issue, then the problem has been reported already
but since we never released _02, you need to build the patch by yourself:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=1679

Thanks

--Jeanfrancois

>
> http://forums.java.net/jive/thread.jspa?messageID=275296
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bkatnich
Offline
Joined: 2006-02-03

Hi Jeanfrancois,

Thanks for the reply. Actually I have been running with that setting all along. That along with some others I have seen you comment on in past blogs. Here are my current jvm option settings:

-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false
-Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
-Dcom.sun.enterprise.taglisteners=jsf-impl.jar
-Djava.endorsed.dirs=${com.sun.aas.installRoot}/lib/endorsed
-Djava.ext.dirs=${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext${path.separator}${com.sun.aas.derbyRoot}/lib
-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
-Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-server
-Xmx1400m
-Xms1400m
-Xss128k
-XX:+AggressiveHeap
-XX:+DisableExplicitGC
-XX:ParallelGCThreads=2
-XX:+UseParallelOldGC
-XX:MaxPermSize=192m
-XX:NewRatio=2

And additionally my settings for the http service and listeners in question (the one I am having problems with is the listener with port 43921:











I will look at building and applying the additional patch.

Jeanfrancois Arcand

Salut,

yes I think you can get the patch from Sun if you have a support
contract. We have a page for that, but I can't remember where it is. Anyone?

A+

--Jeanfrancois

glassfish@javadesktop.org wrote:
> Hi Jeanfrancois,
>
> Thanks for the reply. Actually I have been running with that setting all along. That along with some others I have seen you comment on in past blogs. Here are my current jvm option settings:
>
> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
> -Dcom.sun.enterprise.server.ss.ASQuickStartup=false
> -Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
> -Dcom.sun.enterprise.taglisteners=jsf-impl.jar
> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/lib/endorsed
> -Djava.ext.dirs=${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext${path.separator}${com.sun.aas.derbyRoot}/lib
> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
> -Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
> -server
> -Xmx1400m
> -Xms1400m
> -Xss128k
> -XX:+AggressiveHeap
> -XX:+DisableExplicitGC
> -XX:ParallelGCThreads=2
> -XX:+UseParallelOldGC
> -XX:MaxPermSize=192m
> -XX:NewRatio=2
>
> And additionally my settings for the http service and listeners in question (the one I am having problems with is the listener with port 43921:
>
>
>
>

>
>
>
>

>
>
>
>

>
>
>
>

>
>
>
>
>
>
>
> I will look at building and applying the additional patch.
> [Message sent by forum member 'bkatnich' (bkatnich)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275516
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jerome Dochez

Alexis was describing this at

http://blogs.sun.com/alexismp/entry/support_for_glassfish_what_s

jerome

On May 20, 2008, at 10:33 AM, Jeanfrancois Arcand wrote:

> Salut,
>
> yes I think you can get the patch from Sun if you have a support
> contract. We have a page for that, but I can't remember where it is.
> Anyone?
>
> A+
>
> --Jeanfrancois
>
> glassfish@javadesktop.org wrote:
>> Hi Jeanfrancois,
>> Thanks for the reply. Actually I have been running with that
>> setting all along. That along with some others I have seen you
>> comment on in past blogs. Here are my current jvm option settings:
>> -
>> Dcom
>> .sun
>> .enterprise
>> .config
>> .config_environment_factory_class
>> =
>> com
>> .sun
>> .enterprise.config.serverbeans.AppserverConfigEnvironmentFactory >> jvm-options>
>> -
>> Dcom.sun.enterprise.server.ss.ASQuickStartup=false

>> -Dcom.sun.enterprise.taglibs=appserv-
>> jstl.jar,jsf-impl.jar

>> -Dcom.sun.enterprise.taglisteners=jsf-impl.jar >> jvm-options>
>> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/
>> lib/endorsed

>> -Djava.ext.dirs=${com.sun.aas.javaRoot}/jre/lib/
>> ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext$
>> {path.separator}${com.sun.aas.derbyRoot}/lib

>> -Djava.security.policy=$
>> {com.sun.aas.instanceRoot}/config/server.policy

>> -Djava.security.auth.login.config=$
>> {com.sun.aas.instanceRoot}/config/login.conf

>> -
>> Djavax
>> .management
>> .builder
>> .initial
>> =
>> com
>> .sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder >> jvm-options>
>> -Djavax.net.ssl.keyStore=$
>> {com.sun.aas.instanceRoot}/config/keystore.jks

>> -Djavax.net.ssl.trustStore=$
>> {com.sun.aas.instanceRoot}/config/cacerts.jks

>> -
>> Djdbc.drivers=org.apache.derby.jdbc.ClientDriver

>> -server
>> -Xmx1400m
>> -Xms1400m
>> -Xss128k
>> -XX:+AggressiveHeap
>> -XX:+DisableExplicitGC
>> -XX:ParallelGCThreads=2
>> -XX:+UseParallelOldGC
>> -XX:MaxPermSize=192m
>> -XX:NewRatio=2
>> And additionally my settings for the http service and listeners in
>> question (the one I am having problems with is the listener with
>> port 43921:
>>
>> >> rotation-interval-in-minutes="15" rotation-policy="time" rotation-
>> suffix="yyyy-MM-dd"/>
>>

>> >> blocking-enabled="false" default-virtual-server="server"
>> enabled="true" family="inet" id="http-listener-1" port="43922"
>> security-enabled="false" server-name="" xpowered-by="true"/>
>> >> blocking-enabled="false" default-virtual-server="server"
>> enabled="true" family="inet" id="http-listener-2" port="43921"
>> security-enabled="true" server-name="" xpowered-by="true"/>
>> >> blocking-enabled="false" default-virtual-server="__asadmin"
>> enabled="true" family="inet" id="admin-listener" port="43920"
>> security-enabled="false" server-name="" xpowered-by="true"/>
>>

>>
>>

>>

>>

>> >> id="__asadmin" log-file="${com.sun.aas.instanceRoot}/logs/
>> server.log" state="on">
>>

>>

>>

>> >> initial-thread-count="10" request-timeout-in-seconds="30" thread-
>> count="130" thread-increment="10"/>
>> >> in-seconds="5"/>
>> >> bytes="-1" receive-buffer-size-in-bytes="4096" send-buffer-size-in-
>> bytes="8192"/>
>> >> type="text/plain; charset=iso-8859-1" ssl-enabled="true"
>> version="HTTP/1.1"/>
>> >> transmission-enabled="false" globally-enabled="true" hash-init-
>> size="0" max-age-in-seconds="30" max-files-count="1024" medium-file-
>> size-limit-in-bytes="537600" medium-file-space-in-bytes="10485760"
>> small-file-size-limit-in-bytes="2048" small-file-space-in-
>> bytes="1048576"/>
>>
>> I will look at building and applying the additional patch.
>> [Message sent by forum member 'bkatnich' (bkatnich)]
>> http://forums.java.net/jive/thread.jspa?messageID=275516
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

rony_k
Offline
Joined: 2008-05-30

Hello,
I have the same problem. After this problem occurred I have generated jvmreport (see attached file). In server.log I can see only:

[#|2008-05-30T12:04:41.675+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=25;_ThreadName=SelectorThread-8080;_RequestID=50d69e6d-2493-4bae-8fee-a6c4fa5ddc7e;|WEB0756: Caught exception during HTTP processing.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAccept(SelectorThread.java:1460)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConnection(SelectorThread.java:1439)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1350)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247)
at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)
|#]

[#|2008-05-30T12:04:43.119+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=Timer-1;_RequestID=33af9590-57b2-49cd-b9c5-c50fb2ac9174;|
java.lang.NullPointerException
at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:964)
at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:197)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
at javax.management.timer.Timer.sendNotification(Timer.java:1234)
at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1203)
at javax.management.timer.TimerAlarmClock.run(Timer.java:1286)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
|#]

After restart of Glassfish all is working fine, but it would be better to find a solution. Any ideas?

Best regards
Ronald Kuczek

Jeanfrancois Arcand

Hi,

do you have a way to always reproduce the problem? The exception below
means too many simultaneous connections where active. Is GlassFish
servicing a large number of users?

The other problem you might suffer is all threads are blocked and
Grizzly pool requests (4096 is the maximum). If your OS doesn't have
enough file descriptor set (on unix/solaris you can see it by doing
ulimit -l), then you will get that exception. Could it be the case?

Thanks

-- Jeanfrancois

glassfish@javadesktop.org wrote:
> Hello,
> I have the same problem. After this problem occurred I have generated jvmreport (see attached file). In server.log I can see only:
>
> [#|2008-05-30T12:04:41.675+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=25;_ThreadName=SelectorThread-8080;_RequestID=50d69e6d-2493-4bae-8fee-a6c4fa5ddc7e;|WEB0756: Caught exception during HTTP processing.
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAccept(SelectorThread.java:1460)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConnection(SelectorThread.java:1439)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1350)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247)
> at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)
> |#]
>
> [#|2008-05-30T12:04:43.119+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=Timer-1;_RequestID=33af9590-57b2-49cd-b9c5-c50fb2ac9174;|
> java.lang.NullPointerException
> at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
> at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:964)
> at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:197)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
> at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
> at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
> at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
> at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
> at javax.management.timer.Timer.sendNotification(Timer.java:1234)
> at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1203)
> at javax.management.timer.TimerAlarmClock.run(Timer.java:1286)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> |#]
>
> After restart of Glassfish all is working fine, but it would be better to find a solution. Any ideas?
>
> Best regards
> Ronald Kuczek
> [Message sent by forum member 'rony_k' (rony_k)]
>
> http://forums.java.net/jive/thread.jspa?messageID=277507
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bjourne
Offline
Joined: 2009-01-26

Hi,

Did this problem ever get resolved? I have run into the exact same problem as Ronald with my GlassFish server. -Dcom.sun.enterprise.server.ss.ASQuickStartup=false is set and ulimit for file descriptors is 1024.

peppeme in this thread http://forums.java.net/jive/message.jspa?messageID=286954#286954 suggested to raise ulimit to 65536, but I doubt that should help. It should only delay the problem because something, either glassfish or the domain running on it must be leaking file descriptors it seem.

If patches exist, has they been integrated into GlassFish or do you need a support contract for that?

Ryan de Laplante

It was determined that there is not a problem with GlassFish. Either
the application code or one of its dependencies (such as a JDBC driver)
is locking the grizzly worker thread. Once all five worker threads are
locked up you experience a lockup. When you experience a lockup, run the
following command:

asadmin generate-jvm-report --type=thread > thread_dump.txt

Examine the file for blocked threads and see what they are waiting on.
The file is basically a stack dump for each thread. That is how I found
what was locking my application and fixed it.

A second lockup I recently encountered was also my fault. My app was
executing an SQL query with no WHERE clause on a many gigabyte sized
database table. It took two months to figure out what was happening.
JPA caches every entity it loads by default, and never expires them. I
disabled JPA caching and fixed the query to not allow it to return many
gigabytes of results.

Ryan

glassfish@javadesktop.org wrote:
> Hi,
>
> Did this problem ever get resolved? I have run into the exact same problem as Ronald with my GlassFish server. -Dcom.sun.enterprise.server.ss.ASQuickStartup=false is set and ulimit for file descriptors is 1024.
>
> peppeme in this thread http://forums.java.net/jive/message.jspa?messageID=286954#286954 suggested to raise ulimit to 65536, but I doubt that should help. It should only delay the problem because something, either glassfish or the domain running on it must be leaking file descriptors it seem.
>
> If patches exist, has they been integrated into GlassFish or do you need a support contract for that?
> [Message sent by forum member 'bjourne' (bjourne)]
>
> http://forums.java.net/jive/thread.jspa?messageID=328160
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

BJörn Lindqvist

Ah, great! But I'm still curious about the traceback. Both my and
Ronald Kuczek logs contain this trace:

[#|2008-05-30T12:04:43.119+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=26;_ThreadName=Timer-1;_RequestID=33af9590-57b2-49cd-b9c5-c50fb2ac9174;|
java.lang.NullPointerException
at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:964)
at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:197)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)

The traces are identical right down to the exact line numbers. How is
it possible that locking grizzly threads leads to
NullPointerExceptions from AutoAdminTask? I haven't been able to
reproduce the issue in over a month now so I can't create a jstack
report yet.

2009/1/26 Ryan de Laplante :
> It was determined that there is not a problem with GlassFish. Either the
> application code or one of its dependencies (such as a JDBC driver) is
> locking the grizzly worker thread. Once all five worker threads are locked
> up you experience a lockup. When you experience a lockup, run the following
> command:
>
> asadmin generate-jvm-report --type=thread > thread_dump.txt
>
> Examine the file for blocked threads and see what they are waiting on. The
> file is basically a stack dump for each thread. That is how I found what
> was locking my application and fixed it.
>
> A second lockup I recently encountered was also my fault. My app was
> executing an SQL query with no WHERE clause on a many gigabyte sized
> database table. It took two months to figure out what was happening. JPA
> caches every entity it loads by default, and never expires them. I disabled
> JPA caching and fixed the query to not allow it to return many gigabytes of
> results.
>
>
> Ryan
>
>
> glassfish@javadesktop.org wrote:
>>
>> Hi,
>>
>> Did this problem ever get resolved? I have run into the exact same problem
>> as Ronald with my GlassFish server.
>> -Dcom.sun.enterprise.server.ss.ASQuickStartup=false is set and ulimit for
>> file descriptors is 1024.
>> peppeme in this thread
>> http://forums.java.net/jive/message.jspa?messageID=286954#286954 suggested
>> to raise ulimit to 65536, but I doubt that should help. It should only delay
>> the problem because something, either glassfish or the domain running on it
>> must be leaking file descriptors it seem.
>> If patches exist, has they been integrated into GlassFish or do you need a
>> support contract for that?
>> [Message sent by forum member 'bjourne' (bjourne)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=328160
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
mvh Björn

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
> I corrected the JVM option as you noted.
>
> Assuming that Grizzly is the problem, is there a downside to using Coyote? Also, I don't see log entries specifically that say Coyote is being used instead of Grizzly. Should I expect such?
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]

If you execute a jstack , you will see there is some classes that
start with org.apache.tomcat

Thanks

-- Jeanfrancois

>
> http://forums.java.net/jive/thread.jspa?messageID=272320
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Ryan de Laplante

> After a few hours HTTP requests consistently stop reaching the web
> application. The java application continues to run, but HTTP requests
> are not being passed to servlets. If I go into the admin console and
> fiddle with an HTTP Service setting, like enabling access logging for
> example, POOF!! the site comes back to life. I can only assume that
> doing that causes something to reset in Glassfish. Clearly this is a
> Glassfish issue and nothing to do with the application code. Maddening
> and unacceptable. My only option is to get off Glassfish and onto
> something more stable. I am stopping/starting the domain on my
> production servers constantly. If anyone has a suggestion to make this
> problem go away I will be very grateful.
> How quickly do the waters become muddy. This thread now has multiple issues posted to it, which renders it pretty much useless to anyone trying to actually track down and fix a bug in Glassfish.
>
I would disagree. We are having the same exact issue, except I've been
having it for 6 months in production and it has several symptoms:

1) after some period of time, users get a blank screen forever waiting
for a response. We have to restart the Windows service to fix it.
2) With the same frequency, sometimes users get a "Maximum connections
reached (4096)" error immediately. We have to restart the Windows
service to fix it.
3) Same symptom as #1, but restarting app server doesn't help and
Windows has to be rebooted. This is the NP Pool bug in Windows 2003
server, and it has only happened a few times.

Through these emails, finally after 6 months someone has told me that
the first two issues are NOT caused by the NP Pool problem in Windows
2003 Server. Sun is giving me other ideas now, such as disabling
Grizzly and using Coyote with this JVM option:

-Dcom.sun.enterprise.web.useCoyoteConnector=true

These discussions used to happen off the mailing list because I was
communicating with Sun through our support contract. Now the world gets
to see these discussions. Nobody knows the answer to this problem yet,
so these discussions are the best thing you've got. Not even a Sun
support contract will help your resolve this issue.

I don't use a database in the web tier either, that happens in the EJB
tier. I'm not entirely convinced the problem is caused by database
connections or my application. An older version of the application used
to run on JBoss, same server, for over a year without these issues.

Today the system went down again (9 hours since last time) and I
remembered to try changing the HTTP Listener to see if it would come
back to life like you suggested (new idea to me). I first disabled it,
save, enable it, save. My app came back to life! I have a second HTTP
listener that uses SSL for web services, and disabling/re-enabling made
it come back with a response, but an HTTP 500 response. So I restarted
the app server.

This morning I've told the app server to use Coyote and we'll see how it
goes. I also captured what I can of a thread dump (999 lines max in
Windows console).

Thanks,
Ryan

glassfish@javadesktop.org wrote:
> The original issue that this thread was about has nothing whatever to do with database connections or deadlocked threads. My application doesn't even hit a database. The symptoms, and hints as to the source of the problem are to be found in my original post at the top of this thread. I wish the Glassfish team luck fixing that issue, because I very much believe it to be a bug in Glassfish, unlike deadlocked threads which is more likely to be application code.
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272165
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProto
col.java:281)
at: com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPS
ervice.java:574)
at: java.lang.Thread.run(Thread.java:619)

Thread "admin_ACCEPT" thread-id 64 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProto
col.java:281)
at: com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPS
ervice.java:574)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI RenewClean-[140.95.254.133:1093]" thread-id 62 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock@fdb2bd
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI TCP Accept-0" thread-id 61 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)

Thread "ClusterDiscoveryService" thread-id 60 thread-stateRUNNABLERunning in nat
ive
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.service.ClusterDiscoveryService.run
(ClusterDiscoveryService.java:318)
at: java.lang.Thread.run(Thread.java:619)

Thread "Broker Monitor" thread-id 59 thread-stateWAITINGWaiting on lock: java.ut
il.Collections$SynchronizedSet@1e1a630
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.messaging.jmq.jmsserver.core.cluster.BrokerConsumers.run(Mu
ltibrokerRouter.java:1342)
at: java.lang.Thread.run(Thread.java:619)

Thread "JMQPortMapper" thread-id 58 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.service.PortMapper.run(PortMapper.j
ava:485)
at: java.lang.Thread.run(Thread.java:619)

Thread "MQTimer-Thread" thread-id 57 thread-stateTIMED_WAITINGWaiting on lock: j
ava.util.TaskQueue@165aab0
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)

Thread "RMI RenewClean-[140.95.254.133:1087]" thread-id 52 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock@1a93cd1
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI TCP Accept-0" thread-id 50 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI TCP Accept-8686" thread-id 49 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)

Thread "Thread-12" thread-id 46 thread-stateWAITINGWaiting on lock: java.util.co
ncurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d7f9b0
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje
ct.await(AbstractQueuedSynchronizer.java:1925)
at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.jav
a:317)
at: com.sun.enterprise.management.support.LoaderRegThread.process(Loade
rRegThread.java:243)
at: com.sun.enterprise.management.support.LoaderRegThread.run(LoaderReg
Thread.java:154)

Thread "Thread-11" thread-id 45 thread-stateTIMED_WAITING
at: java.lang.Thread.sleep(Native Method)
at: com.sun.enterprise.management.support.LoaderBase.mySleep(LoaderBase
.java:241)
at: com.sun.enterprise.management.support.Loader$DeferredRegistrationTh
read.run(Loader.java:389)

Thread "Thread-9" thread-id 43 thread-stateWAITINGWaiting on lock: com.sun.corba
.ee.impl.javax.rmi.CORBA.KeepAlive@49af7b
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:857)

Thread "p: thread-pool-1; w: 2" thread-id 42 thread-stateRUNNABLERunning in nati
ve
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSo
cketImpl.java:259)
at: com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(
SocketOrChannelAcceptorImpl.java:250)
at: com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerT
hreadImpl.java:107)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThrea
d.run(ThreadPoolImpl.java:555)

Thread "p: thread-pool-1; w: 1" thread-id 41 thread-stateRUNNABLERunning in nati
ve
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSo
cketImpl.java:259)
at: com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(
SocketOrChannelAcceptorImpl.java:250)
at: com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerT
hreadImpl.java:107)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThrea
d.run(ThreadPoolImpl.java:555)

Thread "SelectorThread" thread-id 40 thread-stateRUNNABLERunning in native
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl
.java:274)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelect
orImpl.java:256)
at: sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:13
7)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at: com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:
283)

Thread "Timer-3" thread-id 39 thread-stateTIMED_WAITINGWaiting on lock: java.uti
l.TaskQueue@19fbd9e
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)

Thread "Timer-2" thread-id 38 thread-stateWAITINGWaiting on lock: java.util.Task
Queue@59e257
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.util.TimerThread.mainLoop(Timer.java:483)
at: java.util.TimerThread.run(Timer.java:462)

Thread "Timer-1" thread-id 37 thread-stateTIMED_WAITINGWaiting on lock: java.uti
l.TaskQueue@11ae4ea
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)

Thread "Timer-0" thread-id 36 thread-stateWAITINGWaiting on lock: java.util.Task
Queue@aacfa0
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.util.TimerThread.mainLoop(Timer.java:483)
at: java.util.TimerThread.run(Timer.java:462)

Thread "Thread-3" thread-id 35 thread-stateTIMED_WAITING
at: java.lang.Thread.sleep(Native Method)
at: com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClien
t.java:151)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI RenewClean-[140.95.254.133:1079,com.sun.enterprise.admin.server.core
.channel.LocalRMIClientSocketFactory@14d044]" thread-id 33 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock@25491c
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI Scheduler(0)" thread-id 32 thread-stateTIMED_WAITINGWaiting on lock:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e2ef31
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:1
98)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje
ct.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at: java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.t
ake(ScheduledThreadPoolExecutor.java:582)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.t
ake(ScheduledThreadPoolExecutor.java:575)
at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:946)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
or.java:906)
at: java.lang.Thread.run(Thread.java:619)

Thread "GC Daemon" thread-id 30 thread-stateTIMED_WAITINGWaiting on lock: sun.mi
sc.GC$LatencyLock@1c5894b
at: java.lang.Object.wait(Native Method)
at: sun.misc.GC$Daemon.run(GC.java:100)

Thread "RMI Reaper" thread-id 29 thread-stateWAITINGWaiting on lock: java.lang.r
ef.ReferenceQueue$Lock@1b8aa32
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at: sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at: java.lang.Thread.run(Thread.java:619)

Thread "RMI TCP Accept-0" thread-id 28 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)

Thread "Attach Listener" thread-id 4 thread-stateRUNNABLE

Thread "Finalizer" thread-id 3 thread-stateWAITINGWaiting on lock: java.lang.ref
.ReferenceQueue$Lock@adc2bb
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Thread "Reference Handler" thread-id 2 thread-stateWAITINGWaiting on lock: java.
lang.ref.Reference$Lock@1cba429
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

No deadlock found

Command generate-jvm-report executed successfully.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jeanfrancois Arcand

Ryan de Laplante wrote:
>> After a few hours HTTP requests consistently stop reaching the web
>> application. The java application continues to run, but HTTP requests
>> are not being passed to servlets. If I go into the admin console and
>> fiddle with an HTTP Service setting, like enabling access logging for
>> example, POOF!! the site comes back to life. I can only assume that
>> doing that causes something to reset in Glassfish. Clearly this is a
>> Glassfish issue and nothing to do with the application code. Maddening
>> and unacceptable. My only option is to get off Glassfish and onto
>> something more stable. I am stopping/starting the domain on my
>> production servers constantly. If anyone has a suggestion to make this
>> problem go away I will be very grateful.
>> How quickly do the waters become muddy. This thread now has multiple
>> issues posted to it, which renders it pretty much useless to anyone
>> trying to actually track down and fix a bug in Glassfish.
> I would disagree. We are having the same exact issue, except I've been
> having it for 6 months in production and it has several symptoms:
> 1) after some period of time, users get a blank screen forever waiting
> for a response. We have to restart the Windows service to fix it.
> 2) With the same frequency, sometimes users get a "Maximum connections
> reached (4096)" error immediately. We have to restart the Windows
> service to fix it.

This is expected to see such exception, and this confirm that all
threads are blocked somewhere.

Unfortunalty, the thread dump you sent contains nothing about grizzly
threads. You need to increase the win console length to 99999 by
clicking properties on a cmd windows (ouf)...Can you try it?

Thanks

-- Jeanfrancois

> 3) Same symptom as #1, but restarting app server doesn't help and
> Windows has to be rebooted. This is the NP Pool bug in Windows 2003
> server, and it has only happened a few times.
>
> Through these emails, finally after 6 months someone has told me that
> the first two issues are NOT caused by the NP Pool problem in Windows
> 2003 Server. Sun is giving me other ideas now, such as disabling
> Grizzly and using Coyote with this JVM option:
>
> -Dcom.sun.enterprise.web.useCoyoteConnector=true
>
> These discussions used to happen off the mailing list because I was
> communicating with Sun through our support contract. Now the world gets
> to see these discussions. Nobody knows the answer to this problem yet,
> so these discussions are the best thing you've got. Not even a Sun
> support contract will help your resolve this issue.
>
> I don't use a database in the web tier either, that happens in the EJB
> tier. I'm not entirely convinced the problem is caused by database
> connections or my application. An older version of the application used
> to run on JBoss, same server, for over a year without these issues.
>
> Today the system went down again (9 hours since last time) and I
> remembered to try changing the HTTP Listener to see if it would come
> back to life like you suggested (new idea to me). I first disabled it,
> save, enable it, save. My app came back to life! I have a second HTTP
> listener that uses SSL for web services, and disabling/re-enabling made
> it come back with a response, but an HTTP 500 response. So I restarted
> the app server.
>
> This morning I've told the app server to use Coyote and we'll see how it
> goes. I also captured what I can of a thread dump (999 lines max in
> Windows console).
>
>
> Thanks,
> Ryan
>
>
>
>
> glassfish@javadesktop.org wrote:
>> The original issue that this thread was about has nothing whatever to
>> do with database connections or deadlocked threads. My application
>> doesn't even hit a database. The symptoms, and hints as to the source
>> of the problem are to be found in my original post at the top of this
>> thread. I wish the Glassfish team luck fixing that issue, because I
>> very much believe it to be a bug in Glassfish, unlike deadlocked
>> threads which is more likely to be application code.
>> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=272165
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Ryan de Laplante

>> When the client (browsers) would attempt to access a page, it would
>> sit in "Waiting for response from web server..." forever, and people
>> would just close the browser. That is probably why we see those
>> errors.
>
> Right. If that's the case, it seems all threads are
> consumed/deadlocked. Most of the time this has nothing to do with the
> WebContainer, but with a db connection that is locked/slow. All
> Servlet try to get a db connection and they all wait of a lock. It
> happens when a db is slow usually...
Are you saying all of the previous transactions that completed
successfully are locking their threads on each page request? I'm
trying to understand how all of my database access code (JPA) that is
working fine is causing servlet threads to deadlock.

We use JDBC connection pools in GlassFish. I have two pools for SQL
Server 2005 databases, and one for a postgres database. The SQL Server
database uses the XADataSource, but with transactions disabled. I
don't remember the exact reason, other than this was the only way I
could get it to work when GlassFish started to enable distributed
transactions with JAX-WS by default. I could try changing that to
ConnectionPoolDataSource again, but after we get thread dumps.

> For sure getting a thread dump will tell us where all the thread dead
> lock. I don't think this is related to the nbpool....I bet you a dead
> lock with a bd connection :-) I will pay @ JavaOne if I'm wrong :-) :-)
Unfortunately I won't be at JavaOne this year. I was there last year
and attended one of your sessions. Anyway, I'll see what I can do to
get a thread dump. I like Scott Oak's idea about using :

asadmin generate-jvm-report --type=thread

Since the thread dump is so huge, it will probably go past the Windows
console buffer length. I'm trying to figure out how to pipe it to a
file. asadmin always asks me to login. I use --user admin but it
wants a password. I used asadmin login --host localhost --port 4848
to set up .asadminpass in my home directory, then ran the
generate-jvm-report command again and it still wants my username and
password. If I give it --user admin, it still wants a password. The
largest buffer size for the command prompt Windows will allow is 999.

>> Do you have a tool we can install on our test server to put the kind
>> of load you described on the system? I do not like the idea of doing
>> that on our production system. When you said it breaks, does the app
>> server not return any data when trying to access pages?
>
> No it throw an IOException. When the nbpool problems happens, the
> entire TCP stack is down on windows. Anything you will try to do on it
> will not work (reboot is the only operation). But I'm convinced this
> is not the nbpool.
>
>
> Does restarting the
>> app server solve the problem, or do you have to reboot Windows? For
>> us, just restarting the service solves the problem. We only reboot
>> once per month when installing Windows updates.
>
> That's means you aren't suffering the nbpool.

Good, I've been thinking the same.

>> console to interact with it. I like your suggestion of disabling
>> Grizzly from an other email:
>>
>> -Dcom.sun.enterprise.web.useCoyoteConnector=true
>
> You gonna face the same problem I suspect, or some requests will be
> dropped by Coyote if all the thread gets dead locked like It looks
> like. The difference between Grizzly & Coyote is Coyote drops requests
> (404), Grizzly queue them until it reach the max (4096 connection in
> the queue), and then close connection as well. The problem is if all
> Grizzly threads deadlock, then issuing a request will exactly produce
> what you are seeing, which is a browser spin. Now based on you
> exception, the queue is executed and Grizzly try to write response on
> closed connection. That means a thread has been released and grizzly
> is using it. So your Servlet seems to starts executing really slowly.
> We need to find why :-)
Ok I'll focus on getting a dump.

Thanks,
Ryan

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Ryan de Laplante

It went down again today! 5.5 hours since it went down last. This is a
new record. It also has a comparatively low NP Pool count of 383K (I've
seen it up to 2200K before), and is using only 304,504K memory. I
forgot to try tweaking a setting in HTTP listener to see if it comes
back to life or not. I did try to do a stack dump:

> jstack 5180
5180: Not enough storage is available to process this command

Then I tried using this tool to get a stack dump:

http://www.adaptj.com/main/download

5180 java.exe session:0 threads:131 parent:5744
The current version does not support processes running in a different
session.
Try any of the following options:
1) Run the StackTrace service in the same session with the target process.
2) Start the terminal client with "mstsc.exe /console"
3) Use VNC from http://www.realvnc.com/ as a remote client.

Attached are some Grizzly and NIO channels related exceptions from
server.log

We've had to write a program that checks the server every 10 minutes and
email us when it goes down. We're also now going to restart GlassFish
three times a week. Based on the discussions on this mailing list today
about linux users having these same problems, we are no longer convinced
that it can be blamed on the Windows 2003 NP Pool leak. Yes there is a
leak, but I think GlassFish has a serious problem too. We did not have
this problem with JBoss on the same server and OS a year ago.

Hopefully Sun will put more resources into this issue immediately. It
is the only issue we've had to use our support contract for, and we seem
to be getting nowhere with it after 6 months. My employer is not
satisfied and I'm wondering if he will renew the contract, or switch app
server vendors. This is a production server and it goes down all the time.

Ryan

Ryan de Laplante wrote:
> glassfish@javadesktop.org wrote:
>>> HTTP requests consistently stop reaching the web application
>>>
>>
>> Detect same on my server (linux), but not consistently and very rarely.
>> In that cases non of my webapplications are reachable, also admin gui.
>> Nothing to see in log files.
>>
>> Think this must be an "unnormal" issue.
>> Not familiar with that stuff, just guessing: could it be an problem
>> with broken connections, mean if client/user aborts
>> [Message sent by forum member 'hammoud' (hammoud)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=272085
>>
> This is concerning. Up until now I thought this problem was specific
> to Windows 2003 NP Pool leak. That might explain why I experience two
> similar but different issues:
>
> 1) Every week or two the web container would stop serving requests.
> Sometimes it would say "Maximum connections reached: 4096" even when
> there were only a couple of hundred transactions a day. Other times
> it would show nothing in the browser or not respond at all. My other
> http listener for web services also stops working. Usually the web
> admin console is the only http listener that is working.
> Restarting the SJSAS 9.1 Windows service solves the problem.
>
> 2) Every few months I find that restarting SJSAS 9.1 Windows service
> makes no difference. PostgreSQL also dies and you can't connect to
> it anymore. The only solution is to reboot Windows.
>
> I think issue #2 is related to the Windows 2003 Server NP Pool leak
> which may have been fixed now with Microsoft patches, but I doubt it
> since we have to restart SJSAS 9.1 more often since installing the
> patches.
> I think issue #1 is a GlassFish problem, since you experience it on
> linux and so does an other poster in this forum.
>
>
> Ryan

[#|2008-04-29T14:21:18.067-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=21;_ThreadName=httpSSLWorkerThread-81-0;_RequestID=90220d96-a07d-4826-b5d6-47cffa22a568;|executePhase(RENDER_RESPONSE 6,com.sun.faces.context.FacesContextImpl@529b3c) threw exception
javax.faces.FacesException
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:135)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
at com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
... 34 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
at org.apache.coyote.Response.doWrite(Response.java:599)
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
... 42 more
|#]

[#|2008-04-29T14:21:18.067-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=38;_ThreadName=httpSSLWorkerThread-81-3;_RequestID=b648f009-e520-4b2a-b5ec-ed31095856cd;|executePhase(RENDER_RESPONSE 6,com.sun.faces.context.FacesContextImpl@c669e6) threw exception
javax.faces.FacesException
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:135)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
at com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
... 34 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
at org.apache.coyote.Response.doWrite(Response.java:599)
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
... 42 more
|#]

[#|2008-04-29T14:21:18.067-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=23;_ThreadName=httpSSLWorkerThread-81-1;_RequestID=ff722f61-28f5-4b26-9792-dd5eec9fe3e1;|executePhase(RENDER_RESPONSE 6,com.sun.faces.context.FacesContextImpl@c669e6) threw exception
javax.faces.FacesException
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:135)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
at com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
... 35 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
at org.apache.coyote.Response.doWrite(Response.java:599)
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
... 43 more
|#]

[#|2008-04-29T14:21:18.067-0500|WARNING|sun-appserver9.1|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=24;_ThreadName=httpSSLWorkerThread-81-2;_RequestID=1618df7e-4a47-44b1-9c58-064c649693ac;|executePhase(RENDER_RESPONSE 6,com.sun.faces.context.FacesContextImpl@a3a8e9) threw exception
javax.faces.FacesException
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:135)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:270)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
at com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
... 34 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
at org.apache.coyote.Response.doWrite(Response.java:599)
at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
... 42 more
|#]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jeanfrancois Arcand

Hi Ryan,

thanks for the info...so far, the exception are expected. They just
means the client closed the connection before the server has a chance to
write a response:

> Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
> at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
> at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
> at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
> at org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
> at org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
> at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
> at com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
> at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
> ... 34 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
> at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
> at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
> at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
> at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
> at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
> at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
> at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
> at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
> at org.apache.coyote.Response.doWrite(Response.java:599)
> at org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
> ... 42 more
> |#]

This exception isn't the cause of the hangs. Can you try something? Can
you get a jstack every 2 hours to see how it goes? Also the nbpool
problem will happens faster if more http requests are made. I'm able to
reproduce it quite fast with a load of 300 users doing requests every
seconds...its takes less than 2 hours to break win32 :-)

Thanks

-- Jeanfrancois

Ryan de Laplante wrote:
> It went down again today! 5.5 hours since it went down last. This is a
> new record. It also has a comparatively low NP Pool count of 383K (I've
> seen it up to 2200K before), and is using only 304,504K memory. I
> forgot to try tweaking a setting in HTTP listener to see if it comes
> back to life or not. I did try to do a stack dump:
>
> > jstack 5180
> 5180: Not enough storage is available to process this command
>
> Then I tried using this tool to get a stack dump:
>
> http://www.adaptj.com/main/download
>
> 5180 java.exe session:0 threads:131 parent:5744
> The current version does not support processes running in a different
> session.
> Try any of the following options:
> 1) Run the StackTrace service in the same session with the target process.
> 2) Start the terminal client with "mstsc.exe /console"
> 3) Use VNC from http://www.realvnc.com/ as a remote client.
>
> Attached are some Grizzly and NIO channels related exceptions from
> server.log
>
> We've had to write a program that checks the server every 10 minutes and
> email us when it goes down. We're also now going to restart GlassFish
> three times a week. Based on the discussions on this mailing list today
> about linux users having these same problems, we are no longer convinced
> that it can be blamed on the Windows 2003 NP Pool leak. Yes there is a
> leak, but I think GlassFish has a serious problem too. We did not have
> this problem with JBoss on the same server and OS a year ago.
>
> Hopefully Sun will put more resources into this issue immediately. It
> is the only issue we've had to use our support contract for, and we seem
> to be getting nowhere with it after 6 months. My employer is not
> satisfied and I'm wondering if he will renew the contract, or switch app
> server vendors. This is a production server and it goes down all the time.
>
>
> Ryan
>
>
> Ryan de Laplante wrote:
>> glassfish@javadesktop.org wrote:
>>>> HTTP requests consistently stop reaching the web application
>>>>
>>>
>>> Detect same on my server (linux), but not consistently and very rarely.
>>> In that cases non of my webapplications are reachable, also admin gui.
>>> Nothing to see in log files.
>>>
>>> Think this must be an "unnormal" issue.
>>> Not familiar with that stuff, just guessing: could it be an problem
>>> with broken connections, mean if client/user aborts
>>> [Message sent by forum member 'hammoud' (hammoud)]
>>>
>>> http://forums.java.net/jive/thread.jspa?messageID=272085
>>>
>> This is concerning. Up until now I thought this problem was specific
>> to Windows 2003 NP Pool leak. That might explain why I experience two
>> similar but different issues:
>>
>> 1) Every week or two the web container would stop serving requests.
>> Sometimes it would say "Maximum connections reached: 4096" even when
>> there were only a couple of hundred transactions a day. Other times
>> it would show nothing in the browser or not respond at all. My other
>> http listener for web services also stops working. Usually the web
>> admin console is the only http listener that is working.
>> Restarting the SJSAS 9.1 Windows service solves the problem.
>>
>> 2) Every few months I find that restarting SJSAS 9.1 Windows service
>> makes no difference. PostgreSQL also dies and you can't connect to
>> it anymore. The only solution is to reboot Windows.
>>
>> I think issue #2 is related to the Windows 2003 Server NP Pool leak
>> which may have been fixed now with Microsoft patches, but I doubt it
>> since we have to restart SJSAS 9.1 more often since installing the
>> patches. I think issue #1 is a GlassFish problem, since you experience
>> it on linux and so does an other poster in this forum.
>>
>>
>> Ryan
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Ryan de Laplante

When the client (browsers) would attempt to access a page, it would sit
in "Waiting for response from web server..." forever, and people would
just close the browser. That is probably why we see those errors.

Do you have a tool we can install on our test server to put the kind of
load you described on the system? I do not like the idea of doing that
on our production system. When you said it breaks, does the app server
not return any data when trying to access pages? Does restarting the
app server solve the problem, or do you have to reboot Windows? For us,
just restarting the service solves the problem. We only reboot once per
month when installing Windows updates.

I wasn't able to do a jstack PID once, so I doubt I can do it every two
hours for you. Also, it runs as a Windows service so I can't see the
console to interact with it.

I like your suggestion of disabling Grizzly from an other email:

-Dcom.sun.enterprise.web.useCoyoteConnector=true

I'm going to ask for permission to try this setting in production. I
did a full transaction on my development computer using this setting. I
don't know how to confirm that Coyote was running instead of Grizzly,
but I did see that JVM parameter in the startup log messages.

Were there any changes in UR1 or UR2 that you think would affect this?
We're using the FCS + a patch you gave me in November that would
eventually be released as part of UR1.

Thanks,
Ryan

Jeanfrancois Arcand wrote:
> Hi Ryan,
>
> thanks for the info...so far, the exception are expected. They just
> means the client closed the connection before the server has a chance
> to write a response:
>
>> Caused by: ClientAbortException:
>> java.nio.channels.ClosedChannelException
>> at
>> org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
>>
>> at
>> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
>> at
>> org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
>> at
>> org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
>> at
>> org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
>>
>> at
>> org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
>>
>> at
>> com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
>>
>> at
>> com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
>>
>> at
>> com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
>>
>> ... 34 more
>> Caused by: java.nio.channels.ClosedChannelException
>> at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> at
>> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
>>
>> at
>> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
>>
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
>>
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
>>
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
>>
>> at
>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
>>
>> at
>> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
>>
>> at
>> org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
>>
>> at
>> org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
>>
>> at org.apache.coyote.Response.doWrite(Response.java:599)
>> at
>> org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
>>
>> ... 42 more
>> |#]
>
> This exception isn't the cause of the hangs. Can you try something?
> Can you get a jstack every 2 hours to see how it goes? Also the nbpool
> problem will happens faster if more http requests are made. I'm able
> to reproduce it quite fast with a load of 300 users doing requests
> every seconds...its takes less than 2 hours to break win32 :-)
>
> Thanks
>
> -- Jeanfrancois
>
>
> Ryan de Laplante wrote:
>> It went down again today! 5.5 hours since it went down last. This is
>> a new record. It also has a comparatively low NP Pool count of 383K
>> (I've seen it up to 2200K before), and is using only 304,504K
>> memory. I forgot to try tweaking a setting in HTTP listener to see
>> if it comes back to life or not. I did try to do a stack dump:
>>
>> > jstack 5180
>> 5180: Not enough storage is available to process this command
>>
>> Then I tried using this tool to get a stack dump:
>>
>> http://www.adaptj.com/main/download
>>
>> 5180 java.exe session:0 threads:131 parent:5744
>> The current version does not support processes running in a different
>> session.
>> Try any of the following options:
>> 1) Run the StackTrace service in the same session with the target
>> process.
>> 2) Start the terminal client with "mstsc.exe /console"
>> 3) Use VNC from http://www.realvnc.com/ as a remote client.
>>
>> Attached are some Grizzly and NIO channels related exceptions from
>> server.log
>>
>> We've had to write a program that checks the server every 10 minutes
>> and email us when it goes down. We're also now going to restart
>> GlassFish three times a week. Based on the discussions on this
>> mailing list today about linux users having these same problems, we
>> are no longer convinced that it can be blamed on the Windows 2003 NP
>> Pool leak. Yes there is a leak, but I think GlassFish has a serious
>> problem too. We did not have this problem with JBoss on the same
>> server and OS a year ago.
>>
>> Hopefully Sun will put more resources into this issue immediately.
>> It is the only issue we've had to use our support contract for, and
>> we seem to be getting nowhere with it after 6 months. My employer is
>> not satisfied and I'm wondering if he will renew the contract, or
>> switch app server vendors. This is a production server and it goes
>> down all the time.
>>
>>
>> Ryan
>>
>>
>> Ryan de Laplante wrote:
>>> glassfish@javadesktop.org wrote:
>>>>> HTTP requests consistently stop reaching the web application
>>>>>
>>>>
>>>> Detect same on my server (linux), but not consistently and very
>>>> rarely.
>>>> In that cases non of my webapplications are reachable, also admin gui.
>>>> Nothing to see in log files.
>>>>
>>>> Think this must be an "unnormal" issue.
>>>> Not familiar with that stuff, just guessing: could it be an problem
>>>> with broken connections, mean if client/user aborts
>>>> [Message sent by forum member 'hammoud' (hammoud)]
>>>>
>>>> http://forums.java.net/jive/thread.jspa?messageID=272085
>>>>
>>> This is concerning. Up until now I thought this problem was
>>> specific to Windows 2003 NP Pool leak. That might explain why I
>>> experience two similar but different issues:
>>>
>>> 1) Every week or two the web container would stop serving requests.
>>> Sometimes it would say "Maximum connections reached: 4096" even when
>>> there were only a couple of hundred transactions a day. Other times
>>> it would show nothing in the browser or not respond at all. My
>>> other http listener for web services also stops working. Usually
>>> the web admin console is the only http listener that is
>>> working. Restarting the SJSAS 9.1 Windows service solves the
>>> problem.
>>>
>>> 2) Every few months I find that restarting SJSAS 9.1 Windows service
>>> makes no difference. PostgreSQL also dies and you can't connect to
>>> it anymore. The only solution is to reboot Windows.
>>>
>>> I think issue #2 is related to the Windows 2003 Server NP Pool leak
>>> which may have been fixed now with Microsoft patches, but I doubt it
>>> since we have to restart SJSAS 9.1 more often since installing the
>>> patches. I think issue #1 is a GlassFish problem, since you
>>> experience it on linux and so does an other poster in this forum.
>>>
>>>
>>> Ryan
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jeanfrancois Arcand

Salut,

Ryan de Laplante wrote:
> When the client (browsers) would attempt to access a page, it would sit
> in "Waiting for response from web server..." forever, and people would
> just close the browser. That is probably why we see those errors.

Right. If that's the case, it seems all threads are consumed/deadlocked.
Most of the time this has nothing to do with the WebContainer, but with
a db connection that is locked/slow. All Servlet try to get a db
connection and they all wait of a lock. It happens when a db is slow
usually...

For sure getting a thread dump will tell us where all the thread dead
lock. I don't think this is related to the nbpool....I bet you a dead
lock with a bd connection :-) I will pay @ JavaOne if I'm wrong :-) :-)

>
>
> Do you have a tool we can install on our test server to put the kind of
> load you described on the system? I do not like the idea of doing that
> on our production system. When you said it breaks, does the app server
> not return any data when trying to access pages?

No it throw an IOException. When the nbpool problems happens, the entire
TCP stack is down on windows. Anything you will try to do on it will not
work (reboot is the only operation). But I'm convinced this is not the
nbpool.

Does restarting the
> app server solve the problem, or do you have to reboot Windows? For us,
> just restarting the service solves the problem. We only reboot once per
> month when installing Windows updates.

That's means you aren't suffering the nbpool.

>
> I wasn't able to do a jstack PID once, so I doubt I can do it every two
> hours for you.

I hate windows :-) Can you do a run without running it as a service? I
suspect using the asadmin start-domain --verbose will allow you do issue
a control-break or \, producing a thread dump.

Also, it runs as a Windows service so I can't see the
> console to interact with it.
> I like your suggestion of disabling Grizzly from an other email:
>
> -Dcom.sun.enterprise.web.useCoyoteConnector=true

You gonna face the same problem I suspect, or some requests will be
dropped by Coyote if all the thread gets dead locked like It looks like.
The difference between Grizzly & Coyote is Coyote drops requests (404),
Grizzly queue them until it reach the max (4096 connection in the
queue), and then close connection as well. The problem is if all Grizzly
threads deadlock, then issuing a request will exactly produce what you
are seeing, which is a browser spin. Now based on you exception, the
queue is executed and Grizzly try to write response on closed
connection. That means a thread has been released and grizzly is using
it. So your Servlet seems to starts executing really slowly. We need to
find why :-)

>
> I'm going to ask for permission to try this setting in production. I
> did a full transaction on my development computer using this setting. I
> don't know how to confirm that Coyote was running instead of Grizzly,
> but I did see that JVM parameter in the startup log messages.

Then it is there.

>
> Were there any changes in UR1 or UR2 that you think would affect this?

Difficult to say. One thing you may wan to try is to increase the number
of thread (...with more
thread, the problem will still happens, but will take more time.

> We're using the FCS + a patch you gave me in November that would
> eventually be released as part of UR1.

I think the issue is not with GlassFish :-) Let focus on trying to get a
thresd dump first :-)

A+

--Jeanfrancois

>
> Thanks,
> Ryan
>
>
> Jeanfrancois Arcand wrote:
>> Hi Ryan,
>>
>> thanks for the info...so far, the exception are expected. They just
>> means the client closed the connection before the server has a chance
>> to write a response:
>>
>>> Caused by: ClientAbortException:
>>> java.nio.channels.ClosedChannelException
>>> at
>>> org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:409)
>>>
>>> at
>>> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
>>> at
>>> org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:357)
>>> at
>>> org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:335)
>>> at
>>> org.apache.coyote.tomcat5.CoyoteResponse.flushBuffer(CoyoteResponse.java:638)
>>>
>>> at
>>> org.apache.coyote.tomcat5.CoyoteResponseFacade.flushBuffer(CoyoteResponseFacade.java:291)
>>>
>>> at
>>> com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:203)
>>>
>>> at
>>> com.sun.rave.web.ui.appbase.faces.ViewHandlerImpl.renderView(ViewHandlerImpl.java:320)
>>>
>>> at
>>> com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
>>>
>>> ... 34 more
>>> Caused by: java.nio.channels.ClosedChannelException
>>> at
>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>>> at
>>> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:94)
>>>
>>> at
>>> com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:67)
>>>
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:167)
>>>
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:202)
>>>
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:178)
>>>
>>> at
>>> com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:145)
>>>
>>> at
>>> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:851)
>>>
>>> at
>>> org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:149)
>>>
>>> at
>>> org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:626)
>>>
>>> at org.apache.coyote.Response.doWrite(Response.java:599)
>>> at
>>> org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:404)
>>>
>>> ... 42 more
>>> |#]
>>
>> This exception isn't the cause of the hangs. Can you try something?
>> Can you get a jstack every 2 hours to see how it goes? Also the nbpool
>> problem will happens faster if more http requests are made. I'm able
>> to reproduce it quite fast with a load of 300 users doing requests
>> every seconds...its takes less than 2 hours to break win32 :-)
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>>
>> Ryan de Laplante wrote:
>>> It went down again today! 5.5 hours since it went down last. This is
>>> a new record. It also has a comparatively low NP Pool count of 383K
>>> (I've seen it up to 2200K before), and is using only 304,504K
>>> memory. I forgot to try tweaking a setting in HTTP listener to see
>>> if it comes back to life or not. I did try to do a stack dump:
>>>
>>> > jstack 5180
>>> 5180: Not enough storage is available to process this command
>>>
>>> Then I tried using this tool to get a stack dump:
>>>
>>> http://www.adaptj.com/main/download
>>>
>>> 5180 java.exe session:0 threads:131 parent:5744
>>> The current version does not support processes running in a different
>>> session.
>>> Try any of the following options:
>>> 1) Run the StackTrace service in the same session with the target
>>> process.
>>> 2) Start the terminal client with "mstsc.exe /console"
>>> 3) Use VNC from http://www.realvnc.com/ as a remote client.
>>>
>>> Attached are some Grizzly and NIO channels related exceptions from
>>> server.log
>>>
>>> We've had to write a program that checks the server every 10 minutes
>>> and email us when it goes down. We're also now going to restart
>>> GlassFish three times a week. Based on the discussions on this
>>> mailing list today about linux users having these same problems, we
>>> are no longer convinced that it can be blamed on the Windows 2003 NP
>>> Pool leak. Yes there is a leak, but I think GlassFish has a serious
>>> problem too. We did not have this problem with JBoss on the same
>>> server and OS a year ago.
>>>
>>> Hopefully Sun will put more resources into this issue immediately.
>>> It is the only issue we've had to use our support contract for, and
>>> we seem to be getting nowhere with it after 6 months. My employer is
>>> not satisfied and I'm wondering if he will renew the contract, or
>>> switch app server vendors. This is a production server and it goes
>>> down all the time.
>>>
>>>
>>> Ryan
>>>
>>>
>>> Ryan de Laplante wrote:
>>>> glassfish@javadesktop.org wrote:
>>>>>> HTTP requests consistently stop reaching the web application
>>>>>>
>>>>>
>>>>> Detect same on my server (linux), but not consistently and very
>>>>> rarely.
>>>>> In that cases non of my webapplications are reachable, also admin gui.
>>>>> Nothing to see in log files.
>>>>>
>>>>> Think this must be an "unnormal" issue.
>>>>> Not familiar with that stuff, just guessing: could it be an problem
>>>>> with broken connections, mean if client/user aborts
>>>>> [Message sent by forum member 'hammoud' (hammoud)]
>>>>>
>>>>> http://forums.java.net/jive/thread.jspa?messageID=272085
>>>>>
>>>> This is concerning. Up until now I thought this problem was
>>>> specific to Windows 2003 NP Pool leak. That might explain why I
>>>> experience two similar but different issues:
>>>>
>>>> 1) Every week or two the web container would stop serving requests.
>>>> Sometimes it would say "Maximum connections reached: 4096" even when
>>>> there were only a couple of hundred transactions a day. Other times
>>>> it would show nothing in the browser or not respond at all. My
>>>> other http listener for web services also stops working. Usually
>>>> the web admin console is the only http listener that is
>>>> working. Restarting the SJSAS 9.1 Windows service solves the
>>>> problem.
>>>>
>>>> 2) Every few months I find that restarting SJSAS 9.1 Windows service
>>>> makes no difference. PostgreSQL also dies and you can't connect to
>>>> it anymore. The only solution is to reboot Windows.
>>>>
>>>> I think issue #2 is related to the Windows 2003 Server NP Pool leak
>>>> which may have been fixed now with Microsoft patches, but I doubt it
>>>> since we have to restart SJSAS 9.1 more often since installing the
>>>> patches. I think issue #1 is a GlassFish problem, since you
>>>> experience it on linux and so does an other poster in this forum.
>>>>
>>>>
>>>> Ryan
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Scott Oaks

>> I wasn't able to do a jstack PID once, so I doubt I can do it every
>> two hours for you.
>
> I hate windows :-) Can you do a run without running it as a service? I
> suspect using the asadmin start-domain --verbose will allow you do
> issue a control-break or \, producing a thread dump.
I hate Windows more than Jeanfrancois does. But another option is to use
asadmin generate-jvm-report --type=thread, which will also generate a
thread dump (as the output of the asadmin command). There are different
thread pools for the general HTTP requests and the asadmin requests, so
this should work even if the normal HTTP threads are all hung.

-Scott

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

> HTTP requests consistently stop reaching the web application

Detect same on my server (linux), but not consistently and very rarely.
In that cases non of my webapplications are reachable, also admin gui.
Nothing to see in log files.

Think this must be an "unnormal" issue.
Not familiar with that stuff, just guessing: could it be an problem with broken connections, mean if client/user aborts

Ryan de Laplante

glassfish@javadesktop.org wrote:
>> HTTP requests consistently stop reaching the web application
>>
>
> Detect same on my server (linux), but not consistently and very rarely.
> In that cases non of my webapplications are reachable, also admin gui.
> Nothing to see in log files.
>
> Think this must be an "unnormal" issue.
> Not familiar with that stuff, just guessing: could it be an problem with broken connections, mean if client/user aborts
> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272085
>
This is concerning. Up until now I thought this problem was specific to
Windows 2003 NP Pool leak. That might explain why I experience two
similar but different issues:

1) Every week or two the web container would stop serving requests.
Sometimes it would say "Maximum connections reached: 4096" even when
there were only a couple of hundred transactions a day. Other times it
would show nothing in the browser or not respond at all. My other http
listener for web services also stops working. Usually the web admin
console is the only http listener that is working. Restarting the
SJSAS 9.1 Windows service solves the problem.

2) Every few months I find that restarting SJSAS 9.1 Windows service
makes no difference. PostgreSQL also dies and you can't connect to it
anymore. The only solution is to reboot Windows.

I think issue #2 is related to the Windows 2003 Server NP Pool leak
which may have been fixed now with Microsoft patches, but I doubt it
since we have to restart SJSAS 9.1 more often since installing the
patches.

I think issue #1 is a GlassFish problem, since you experience it on
linux and so does an other poster in this forum.

Ryan

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

kevinmacdonald
Offline
Joined: 2007-12-18

Thus arises my unfortunate need to dispense with Glassfish and opt for something simpler and more stable. It is perhaps too heavy for my needs anyway. All I am doing is hosting a single java web application.

kevinmacdonald
Offline
Joined: 2007-12-18

I am running on Ubuntu. If there is something I can do to provide you more information can you give precise instructions please? I don't know what it means to "do a jstack".

Kevin

gpocecai
Offline
Joined: 2008-11-17

I'm experiencing more or less the same problem.
I think I totally miss the relation between acceptor threads, request processing and connection pool. I tried with different values but I got stuck, it seems connections are not going through to the jvm. I applied the tips on performance by Jean Francois, but it did not get better.
The same happened while gathering bits of information form various posts.
Everything is useless when you have not a clear picture in mind, you just guess, and this is no good.
Can sameone please help me in getting a clear picture.
thank you
salut
gianfranco

Ryan de Laplante

I'll give it a try. By default GlassFish provides only 5 threads for
processing all web requests. Jeanfrancois says they've done a lot of
performance testing and because of the way GlassFish works, that is plenty.

When all 5 threads are blocked waiting for the application to do
something and provide a response, new requests coming in are queued up
for the next available thread (instead of displaying an error message
like in Tomcat). In my case, the application was trying to establish a
connection to the database and for whatever reason the JDBC driver would
get stuck. After 4 more requests come in that also try to use a
database, all 5 threads would become locked and the http listener became
useless. Changing the JDBC driver fixed the problem for me.

Other people with this problem have said that they don't use a
database. Something in the application is causing the worker thread to
block and that is what you have to figure out.

Some people said that Tomcat never gave them this problem. That's
probably because tomcat has 100 worker threads by default and you can
cause a lot more threads to block before Tomcat displays an error
message (instead of being queued for the next available worker like in
GlassFish). Eventually threads become unblocked somehow and they never
saw all 100 threads become blocked.

Ryan

glassfish@javadesktop.org wrote:
> I'm experiencing more or less the same problem.
> I think I totally miss the relation between acceptor threads, request processing and connection pool. I tried with different values but I got stuck, it seems connections are not going through to the jvm. I applied the tips on performance by Jean Francois, but it did not get better.
> The same happened while gathering bits of information form various posts.
> Everything is useless when you have not a clear picture in mind, you just guess, and this is no good.
> Can sameone please help me in getting a clear picture.
> thank you
> salut
> gianfranco
> [Message sent by forum member 'gpocecai' (gpocecai)]
>
> http://forums.java.net/jive/thread.jspa?messageID=317170
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

tmpsa
Offline
Joined: 2010-02-01

Great explanation, Ryan! that should go into the official Glassfish documentation; it has little or no information on this.

The default value of 5 is, imho, not good. Let's say that you have 10 clients doing file upload over slow connections (think 'mobile'). Then the time before a HTTP request is served can easily become [i]minutes[/i]. Many clients (web browsers, etc) would give up and time out. Foo!

Also, it appears that [i]each[/i] HTTP listener has its own little thread pool of 5 threads. If so, it would be [i]much[/i] better if all three Listeners (HTTP, HTTP, Admin Console) would share the same little thread pool.

Even better, it would be nice to able able to configure a specific thread pool for a certain web app (such as in my horror story).

zukerman
Offline
Joined: 2008-09-24

Dear Jean Francois,
I have reconfigured the my production server according the performance tips from your articles Configuring Grizzly for performance part I and II and here are the results:
Configuration recommended by you:
-server
-XX:+AggressiveHeap
-Xmx1400m
-Xms1400m
-Xss128k
-XX:MaxPermSize=400m
-XX:+DisableExplicitGC
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false
-XX:ParallelGCThreads=2
-XX:+UseParallelOldGC
-XX:NewRatio=2

The server stops running one hour after being started throwing the following exception: java.lang.OutOfMemoryError: unable to create new native thread
I have increased the values for MaxPermSize but the problem still persists.

Configuration we used to have:
-server
-Xmx1200m
-Xms1200m
-Xss128k
-XX:MaxPermSize=400m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=1
-XX:+DisableExplicitGC
-Dcom.sun.enterprise.server.ss.ASQuickStartup=false
-XX:NewRatio=2

The difference between both configurations is the aggressive heap and the GC algorithm.
The problem with the second configuration is that the https port stops responding after receiving roughly 5000 requests. The strange thing is that the http port still works fine and there is no error message in Glassfish’s log file.

The hardware and software I’m using is as follows: processor Intel Core 2 Quad 2.50 GHz, 4 GB RAM, Windows Server 2008 Standard Edition, Glassfish V2.1 b51

I would really appreciate any comments or suggestions since I have a production server that is crashing ones a day.

Thank you for your time and help,
David

Jacob Kessler

That sounds like you're not releasing your old threads as you finish
with them, in both configurations, rather than a PermSize issue. The old
version may have been queuing incoming messages waiting for a thread,
while the new version simply runs out of allocatable threads. I'd check
your code to see if you're holding on to old connections or something
like that and, if possible, point something like JConsole at it to see
what your memory and thread usage is like to narrow down the potential
culprits.

glassfish@javadesktop.org wrote:
> Dear Jean Francois,
> I have reconfigured the my production server according the performance tips from your articles Configuring Grizzly for performance part I and II and here are the results:
> Configuration recommended by you:
> -server
> -XX:+AggressiveHeap
> -Xmx1400m
> -Xms1400m
> -Xss128k
> -XX:MaxPermSize=400m
> -XX:+DisableExplicitGC
> -Dcom.sun.enterprise.server.ss.ASQuickStartup=false
> -XX:ParallelGCThreads=2
> -XX:+UseParallelOldGC
> -XX:NewRatio=2
>
> The server stops running one hour after being started throwing the following exception: java.lang.OutOfMemoryError: unable to create new native thread
> I have increased the values for MaxPermSize but the problem still persists.
>
> Configuration we used to have:
> -server
> -Xmx1200m
> -Xms1200m
> -Xss128k
> -XX:MaxPermSize=400m
> -XX:+UseConcMarkSweepGC
> -XX:SoftRefLRUPolicyMSPerMB=1
> -XX:+DisableExplicitGC
> -Dcom.sun.enterprise.server.ss.ASQuickStartup=false
> -XX:NewRatio=2
>
> The difference between both configurations is the aggressive heap and the GC algorithm.
> The problem with the second configuration is that the https port stops responding after receiving roughly 5000 requests. The strange thing is that the http port still works fine and there is no error message in Glassfish’s log file.
>
> The hardware and software I’m using is as follows: processor Intel Core 2 Quad 2.50 GHz, 4 GB RAM, Windows Server 2008 Standard Edition, Glassfish V2.1 b51
>
> I would really appreciate any comments or suggestions since I have a production server that is crashing ones a day.
>
> Thank you for your time and help,
> David
> [Message sent by forum member 'zukerman' (zukerman)]
>
> http://forums.java.net/jive/thread.jspa?messageID=301259
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

zukerman
Offline
Joined: 2008-09-24

Dear Jacob,
Thank you very much for taking the time to answer my question.
I followed your advice and run the JConsole last Sunday.

The CPU and memory values look good:
Process CPU time: 3 hours
CPU usage average: 0.4%
Heap Memory: oscillates between 0.6Gb and 1.2Gb (which means that the GC is performing OK)

The thread values are odd: starting in 100 goes to 130 in 2 hours. Stays in 130 for 3 days and goes to 175. Stays on 175 for one day and goes to 200.
The thread count never goes down and seems to keep rising.

Here it’s my thread configuration:




...

I executed the generate-jvm-report --type=thread command and got the following result (I will not post the complete report because it may be annoying for the post’s readers):
Number of threads: 201
Number of daemon threads: 179
…..
No deadlock found

However, I did find 41 threads as follows:
Thread "httpSSLWorkerThread-443-59" thread-id 4,888 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@845a92
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)

and 69 threads as follows:
Thread "httpWorkerThread-80-54" thread-id 4,199 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline@1d6d430
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
at: com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)

Definitely, the issue I’m having is related with the threads. I checked the database connections and it seems to be closing and releasing resources OK (I’m using Connector/J 5.1 to access to a MySQL database that runs on a different server), so I can’t tell exactly what is the cause that prevents the thread count to drop and keep creating more threads. Is there any way to kill threads regardless of their state upon time out?

I would really appreciate any thought on this matter.

Thank you for your time and help,
David.

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
> Dear Jacob,
> Thank you very much for taking the time to answer my question.
> I followed your advice and run the JConsole last Sunday.
>
> The CPU and memory values look good:
> Process CPU time: 3 hours
> CPU usage average: 0.4%
> Heap Memory: oscillates between 0.6Gb and 1.2Gb (which means that the GC is performing OK)
>
> The thread values are odd: starting in 100 goes to 130 in 2 hours. Stays in 130 for 3 days and goes to 175. Stays on 175 for one day and goes to 200.
> The thread count never goes down and seems to keep rising.

Right. The number will grow until you reach the maximum, which you have
set to 400 (quite high IMO). Looking at the rate of thread creation, I
would guess 100 threads should be enough.

>
> Here it’s my thread configuration:
>
>
>
>
> ...
>

>
>
>
>
>

Thread-Pool is for Corba/others component, not the http runtime.

> I executed the generate-jvm-report --type=thread command and got the following result (I will not post the complete report because it may be annoying for the post’s readers):
> Number of threads: 201
> Number of daemon threads: 179
> …..
> No deadlock found
>
> However, I did find 41 threads as follows:
> Thread "httpSSLWorkerThread-443-59" thread-id 4,888 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline@845a92
> at: java.lang.Object.wait(Native Method)
> at: java.lang.Object.wait(Object.java:485)
> at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
> at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
>
> and 69 threads as follows:
> Thread "httpWorkerThread-80-54" thread-id 4,199 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline@1d6d430
> at: java.lang.Object.wait(Native Method)
> at: java.lang.Object.wait(Object.java:485)
> at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:294)
> at: com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:114)
>
> Definitely, the issue I’m having is related with the threads.

The above trace is expected. It just means Threads are waiting for work
and are idle.

I checked the database connections and it seems to be closing and
releasing resources OK (I’m using Connector/J 5.1 to access to a MySQL
database that runs on a different server), so I can’t tell exactly what
is the cause that prevents the thread count to drop and keep creating
more threads. Is there any way to kill threads regardless of their state
upon time out?
>

No. But I doubt is is related. Can you try with 100 threads and see how
it goes? When you see the issue, do the following:

% netstat -an| grep > fd.txt
% jstack PID > stack.txt

Let see what we can find from the two files.

A+

- Jeanfrancois

> I would really appreciate any thought on this matter.
>
> Thank you for your time and help,
> David.
> [Message sent by forum member 'zukerman' (zukerman)]
>
> http://forums.java.net/jive/thread.jspa?messageID=302967
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

chilak
Offline
Joined: 2006-04-10
eingfoan
Offline
Joined: 2008-02-07

hi *,

i just wanted to know if you have solved the issue now. i think i see really the same on my systems sometimes. can provide netstats threaddumps and all you need.

please drop me a line if you need or if this issue has been fixed already.

regards chris

pvenkataraju
Offline
Joined: 2008-05-29

Thakyou Ryan for having a look at the thread dump. My problem still persists. I deployed my application which is a Web Service and it uses Hibernate to retrieve data from MySql database in the form of Objects. These Java Objects are manipulated using Java code and the response is given out in the form of an XML. I am using JAXB for it.

My application is working fine when requested with a few requests. But if I send 30 or 40 requests at a time, the GlassFish is not responding. It keeps on waiting. I cannot even find error messages in the server log. At that particular moment the server status page is also not opening. Every time this problem occurs I just restart GlassFish and everything works fine.

Has any body faced such problem? Is it the problem with GlassFish settings or something else?

I will be grateful if some of you can look into this problem and help me out :)

jesont
Offline
Joined: 2008-08-05

To me you thread looks locked on SSL operation.

Is it SSL enabled(https)?

Have a look at this
http://forums.java.net/jive/thread.jspa?messageID=223176&#223176

Jeson

pvenkataraju
Offline
Joined: 2008-05-29

I tried to get the thread dump when the server didn't responded. Can some one take some pain to go through the attachment "thread_dump.txt" to find out the cause?

Ryan de Laplante

I had a quick look and see that it is definitely not the same problem
I've been having. There is no mention of ResourceManager being blocked
waiting for JDBC driver to establish a connection. For my problem I'm
trying the jTDS driver for SQL Server instead of Microsoft's one. We'll
see if that helps or not.

Sun will have to help you determine what the cause of your particular
issue is.

Ryan

glassfish@javadesktop.org wrote:
> I tried to get the thread dump when the server didn't responded. Can some one take some pain to go through the attachment "thread_dump.txt" to find out the cause?
> [Message sent by forum member 'pvenkataraju' (pvenkataraju)]
>
> http://forums.java.net/jive/thread.jspa?messageID=277518
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

pvenkataraju
Offline
Joined: 2008-05-29

I am also having the same problem. I am using Glassfish with Linux OS. I don't know much about this system administrative stuff. I deployed my application and after a day or so its not responding. Its not even throwing any errors. When I restart the app server, its working fine. My application is a WebService which listens to a WebServiceClient. I deployed both the WebService and WebServiceClient application. Users post requests to the client application and my client contacts the WebService and gives the response. Can anybody help me out why the application is not responding. At first I thought its the problem with the memory because my application is taking 30% of my memory. But after going through the posts in this thread I came to a conclusion that its not my application's problem. I am eagerly waiting for help since I have to get my application live in few days and while testing I got this problem.

roisinflannery
Offline
Joined: 2007-05-08

Hi Bkatnich and kenvinmacdonald,

I work in Sustaining Engineering and we are responsible for releasing patches containing fully tested and supported versions of these fixes. There are also a number of other great benefits that you get when you purchase support.

Along with fully supported patches, you will gain the benefits of getting all of your questions answered, and advice on how to get the best out of GlassFish for your deployment.

You can find more information here:
http://www.sun.com/service/applicationserversubscriptions/index.jsp
or else please feel free to reply to this thread and I will provide more information.

Kind regards,
Roisin

bkatnich
Offline
Joined: 2006-02-03

Hi Roisin,

I would be interested in looking at a support contract but have some questions about it first.

1) My immediate concern is the below bug would be fixed in available patches for SJSAS 9.0 u 1 as the report appears to say it is.

https://glassfish.dev.java.net/issues/show_bug.cgi?id=1679

2) Do the support contracts apply to a specific SJSAS version (9.0 for example) or do they cross over if I were to upgrade to a newer version (9.1 for example)?

If you can give me further information to proceed that would be great.

kevinmacdonald
Offline
Joined: 2007-12-18

>Are you using the default configuration (domain.xml)? Without any
>change, you are running fine with Coyote, right? That's impressive as
>you are only running with 5 threads.

I have thread-count set to 32 under the Request Processing section in GF. Note that this is "thread-count", not "acceptor-threads", which are two different things that I hear being constantly confused with each other.

>Yes, it is (Coyote isn't scaling very well).
I>'m not aware of any issues with Grizzly. I would be very interested to
>see if you can run GF with the log level set to finest to see if I can
>find something? Any application you can share to reproduce the issue?

But it was on this very thread that is was suggested to me to use Coyote to try to eliminate Glassfish as being a cause of my original problem (see the very first post on this thread). The conclusion was that changing to Coyote did in fact make a difference. So, doesn't that point the finger at Grizzly as a possible cause?

The fact that Coyote doesn't scale well brings me back to my initial conclusion that perhaps I have to move off of Glassfish. I can try switching back to Grizzly, but that puts me back to my original configuration where I was having the problem to begin with.

Jeanfrancois Arcand

glassfish@javadesktop.org wrote:
>> Are you using the default configuration (domain.xml)? Without any
>> change, you are running fine with Coyote, right? That's impressive as
>> you are only running with 5 threads.
>
> I have thread-count set to 32 under the Request Processing section in GF. Note that this is "thread-count", not "acceptor-threads", which are two different things that I hear being constantly confused with each other.

Ya, this is really confusion. Just let's that value set to the default
("1").

>
>> Yes, it is (Coyote isn't scaling very well).
> I>'m not aware of any issues with Grizzly. I would be very interested to
>> see if you can run GF with the log level set to finest to see if I can
>> find something? Any application you can share to reproduce the issue?
>
> But it was on this very thread that is was suggested to me to use Coyote to try to eliminate Glassfish as being a cause of my original problem (see the very first post on this thread). The conclusion was that changing to Coyote did in fact make a difference. So, doesn't that point the finger at Grizzly as a possible cause?

Yes, except I need more information to really say Yes Grizzly is the
issue. Turning off Grizzly turn off some features in GlassFish,
functionalities Coyote isn't supporting.

>
> The fact that Coyote doesn't scale well brings me back to my initial conclusion that perhaps I have to move off of Glassfish. I can try switching back to Grizzly, but that puts me back to my original configuration where I was having the problem to begin with.

Right. That would be really helpfull if you can turn it on, and when it
hangs, just to a jstack PIU and send me the file. From that I will be
able to find what's happening. Can you also send me your domain.xml?

Thanks

-- Jeanfrancois

> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272815
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bkatnich
Offline
Joined: 2006-02-03

Hi,

I've been following this issue as I've come across this problem as well. My details are:

- Sun Java System Application Server 9.0 update 1(so older than the one Kevin is using)

- Redhat Linux 5

I am running web services on my HTTPS port and after a while (it changes depending upon the load of course), usually a day or so, the calls to the HTTPS port just stop working. No exception, nothing. Calls to the HTTP port and the Admin port work fine.

So I first tried enabling/disabling some property on the http listener for the HTTPS port and sure enough it started working again right away.

Next I am going to try switching from Grizzly to Coyote as talked about here. I will post result after running with that for a while.

Ryan de Laplante

Restarting the HTTP listener creates new threads which is why it starts
working again. You need to figure out why the threads are locking in
the first place. When it locks up, please run this command:

asadmin generate-jvm-report --type=thread > thread_dump.txt

Send it to this mailing list. Someone from Sun will be able to help you
determine why the threads are blocked. As in my case, you will probably
find that Grizzly is not the cause. Grizzly was waiting on resource
manager to give me a database connection, and resource manager was
waiting on JDBC driver to log in. JDBC driver was waiting forever
because it had no timeout set. In this scenario, coyote will just
give you an HTTP error instead of waiting for one of the blocked threads
to become available.

Ryan

glassfish@javadesktop.org wrote:
> Hi,
>
> I've been following this issue as I've come across this problem as well. My details are:
>
> - Sun Java System Application Server 9.0 update 1(so older than the one Kevin is using)
>
> - Redhat Linux 5
>
> I am running web services on my HTTPS port and after a while (it changes depending upon the load of course), usually a day or so, the calls to the HTTPS port just stop working. No exception, nothing. Calls to the HTTP port and the Admin port work fine.
>
> So I first tried enabling/disabling some property on the http listener for the HTTPS port and sure enough it started working again right away.
>
> Next I am going to try switching from Grizzly to Coyote as talked about here. I will post result after running with that for a while.
> [Message sent by forum member 'bkatnich' (bkatnich)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275069
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bkatnich
Offline
Joined: 2006-02-03

Okay, I ran it with Grizzly just as before and waited for it crap out again. Attached is the output of the command:

asadmin generate-jvm-report --type=thread > thread_dump.txt

bkatnich
Offline
Joined: 2006-02-03

From the output it appears all the http threads (15 of them) for my ssl port (43921) are waiting for a lock:

Thread "httpWorkerThread-43921-0" thread-id 71 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline@83f892
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:279)
at: com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:73)

Jeanfrancois Arcand

glassfish@javadesktop.org wrote:
> From the output it appears all the http threads (15 of them) for my ssl port (43921) are waiting for a lock:
>
> Thread "httpWorkerThread-43921-0" thread-id 71 thread-stateWAITINGWaiting on lock: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline@83f892
> at: java.lang.Object.wait(Native Method)
> at: java.lang.Object.wait(Object.java:485)
> at: com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:279)
> at: com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:73)
> [Message sent by forum member 'bkatnich' (bkatnich)]

This is expected. All threads are waiting for works :-) Can you add:

-Dcom.sun.enterprise.server.ss.ASQuickStartup=false

in domain.xml and restart?

Thanks

-- Jeanfrancois

>
> http://forums.java.net/jive/thread.jspa?messageID=275117
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

kevinmacdonald
Offline
Joined: 2007-12-18

After switching form Grizzly to Coyote it looks like my original problem of the server no longer responding to HTTP has not recurred after several days of operation. Now, there was some speculation as to whether or not my issue was also related to the fact that the maximum number of open files (set via ulimit -n) was set at only 1024. I tried to increase it to 8192, but discovered that the setting does not persist. If I close the Putty session into the server and re-open it I have discovered that the setting reverts back to 1024. These settings are viewable by running "ulimit -a". So, two things seem clear to me. 1) I don't understand how ulimit is supposed to work. Perhaps someone can tell me. 2) I don't see how this "max open files" setting can have anything to do with my problem because it reverts back to it's original setting.

It does appear however, that Grizzly vs. Coyote has made a difference. So, I ask, what is the downside, if any, of using Coyote? Is it slower? Why is the default behavior of Glassfish to use Grizzly in the first place?