Skip to main content

glassfish 3.1.2 - GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8181

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
4 replies [Last post]
djgerbavore
Offline
Joined: 2009-03-11

On my glassfish install, after about an day or two I start seeing this fill my server.log:

<br />
[#|2012-08-16T09:25:26.953-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8181(3).|#]</p>
<p>[#|2012-08-16T09:25:26.954-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8181(2).|#]</p>
<p>[#|2012-08-16T09:25:26.954-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8181(1).|#]</p>
<p>[#|2012-08-16T09:25:26.954-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=15;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8181(4).|#]<br />

It seem to be only happen on the SSL http-listener and not the regular http listener.

Doing a jstack reveals the thread in question is stuck in a runnable state:

<br />
"http-thread-pool-8181(2)" daemon prio=10 tid=0x000000005cdd1800 nid=0x152c runnable [0x0000000044876000]<br />
   java.lang.Thread.State: RUNNABLE<br />
        at java.lang.Throwable.fillInStackTrace(Native Method)<br />
        - locked <0x00000000f2b65800> (a java.lang.InterruptedException)<br />
        at java.lang.Throwable.<init>(Throwable.java:181)<br />
        at java.lang.Exception.<init>(Exception.java:29)<br />
        at java.lang.InterruptedException.<init>(InterruptedException.java:38)<br />
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)<br />
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:735)<br />
        at com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(POAImpl.java:390)<br />
        at com.sun.corba.ee.impl.oa.poa.POAImpl.readLock(POAImpl.java:422)<br />
        at com.sun.corba.ee.impl.oa.poa.POAImpl.enter(POAImpl.java:1743)<br />
        at com.sun.corba.ee.impl.protocol.FullServantCacheLocalCRDImpl.internalPreinvoke(FullServantCacheLocalCRDImpl.java:73)<br />
        at com.sun.corba.ee.impl.protocol.LocalClientRequestDispatcherBase.servant_preinvoke(LocalClientRequestDispatcherBase.java:240)<br />
        at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.servant_preinvoke(CorbaClientDelegateImpl.java:574)<br />
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:218)<br />
        at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)<br />
        at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)<br />
        at com.foobare.dao.__ClientDAO_Remote_DynamicStub.productForDatasetName(com/foobar/dao/__ClientDAO_Remote_DynamicStub.java)<br />
        at com.foobar.dao._ClientDAO_Wrapper.productForDatasetName(com/foobar/dao/_ClientDAO_Wrapper.java)<br />
        at com.foobar.ejb.UpdateManagerImpl.xmlForProducts(UpdateManagerImpl.java:2144)<br />
        at com.foobar.ejb.UpdateManagerImpl.getProductsForDevice(UpdateManagerImpl.java:2006)<br />
        at com.foobar.ejb.UpdateManagerImpl.getProductsForDevice(UpdateManagerImpl.java:1958)<br />
        at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)<br />
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br />
        at java.lang.reflect.Method.invoke(Method.java:597)<br />
        at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)<br />
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)<br />
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)<br />
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)<br />
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)<br />
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)<br />
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)<br />
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)<br />
        at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)<br />

It looks like threads are stuck in a runnable state, no deadlock, just spinning. If I run top the java process is pegged at 100%. I'm confused what is happening. It seem to be only happening to SSL request.

Any help would be appreciated.

Thanks,

Jerry

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
oleksiys
Offline
Joined: 2006-01-25

Please check this thread and comment [1].
You have to apply 2 patches: one for grizzly, another for corba.

Thanks.

WBR,
Alexey.

[1] http://forums.java.net/forum/topic/glassfish/glassfish/what-causes-grizz...

dlaudams
Offline
Joined: 2012-12-12

I've run into the same problem, but with HTTP (non-SSL) requests:

"http-thread-pool-8000(3)" daemon prio=6 tid=0x0000000013bfd000 nid=0x16f4 runnable [0x000000001544b000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00000000ecb333b8> (a java.lang.InterruptedException)
at java.lang.Throwable.<init>(Throwable.java:181)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.InterruptedException.<init>(InterruptedException.java:38)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:735)
at com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(POAImpl.java:390)
at com.sun.corba.ee.impl.oa.poa.POAImpl.readLock(POAImpl.java:422)
at com.sun.corba.ee.impl.oa.poa.POAImpl.exit(POAImpl.java:1788)
at com.sun.corba.ee.impl.protocol.FullServantCacheLocalCRDImpl.servant_postinvoke(FullServantCacheLocalCRDImpl.java:83)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.servant_postinvoke(CorbaClientDelegateImpl.java:582)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:261)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at xxxx.__XXXX_Remote_DynamicStub.entityAdded(xxxx__XXXX_Remote_DynamicStub.java)

GlassFish/Grizzly interrupts the request processing thread after a timeout expires.

The interrupt should not cause fillInStackTrace to get stuck. This looks like a JRE bug, but I have not yet been able to reproduce this outside of GlassFish.

The interrupts are not sent when running in debug mode which make this problem more difficult to reproduce/test.

You can avoid this bug by increasing the request timeout or disable it using -1.

&lt;network-config>
  &lt;protocols>
    &lt;protocol name="http-listener-1">
      &lt;http request-timeout-seconds="-1"
hvilekar
Offline
Joined: 2006-10-06

This looks related to the issue with
com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(). The issue reported
in http://java.net/jira/browse/GLASSFISH-16217 is fixed.

Harshad

On 12/12/2012 02:51 PM, forums@java.net wrote:
> I've run into the same problem, but with HTTP (non-SSL) requests:
> "http-thread-pool-8000(3)" daemon prio=6 tid=0x0000000013bfd000
> nid=0x16f4
> runnable [0x000000001544b000] java.lang.Thread.State: RUNNABLE at
> http://java.net/jira/browse/GLASSFISH-16217
> java.lang.Throwable.fillInStackTrace(Native Method) - locked
> <0x00000000ecb333b8> (a java.lang.InterruptedException) at
> java.lang.Throwable.(Throwable.java:181) at
> java.lang.Exception.(Exception.java:29) at
> java.lang.InterruptedException.(InterruptedException.java:38) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
>
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:735)
>
> at com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(POAImpl.java:390) at
> com.sun.corba.ee.impl.oa.poa.POAImpl.readLock(POAImpl.java:422) at
> com.sun.corba.ee.impl.oa.poa.POAImpl.exit(POAImpl.java:1788) at
> com.sun.corba.ee.impl.protocol.FullServantCacheLocalCRDImpl.servant_postinvoke(FullServantCacheLocalCRDImpl.java:83)
>
> at
> com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.servant_postinvoke(CorbaClientDelegateImpl.java:582)
>
> at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:261)
>
> at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
>
> at
> com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
>
> at
> xxxx.__XXXX_Remote_DynamicStub.entityAdded(xxxx__XXXX_Remote_DynamicStub.java)
>
> GlassFish/Grizzly interrupts the request processing thread after a
> timeout
> expires. The interrupt should not cause fillInStackTrace to get stuck.
> This
> looks like a JRE bug, but I have not yet been able to reproduce this
> outside
> of GlassFish. The interrupts are not sent when running in debug mode
> which
> make this problem more difficult to reproduce/test. You can avoid this
> bug by
> increasing the request timeout or disable it using -1.
> request-timeout-seconds="-1"
>
> --
>
> [Message sent by forum member 'dlaudams']
>
> View Post: http://forums.java.net/node/889225
>
>

dlaudams
Offline
Joined: 2012-12-12

I've run into the same problem, but with HTTP (non-SSL) requests:

"http-thread-pool-8000(3)" daemon prio=6 tid=0x0000000013bfd000 nid=0x16f4 runnable [0x000000001544b000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00000000ecb333b8> (a java.lang.InterruptedException)
at java.lang.Throwable.<init>(Throwable.java:181)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.InterruptedException.<init>(InterruptedException.java:38)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:735)
at com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(POAImpl.java:390)
at com.sun.corba.ee.impl.oa.poa.POAImpl.readLock(POAImpl.java:422)
at com.sun.corba.ee.impl.oa.poa.POAImpl.exit(POAImpl.java:1788)
at com.sun.corba.ee.impl.protocol.FullServantCacheLocalCRDImpl.servant_postinvoke(FullServantCacheLocalCRDImpl.java:83)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.servant_postinvoke(CorbaClientDelegateImpl.java:582)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:261)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
at xxxx.__XXXX_Remote_DynamicStub.entityAdded(xxxx__XXXX_Remote_DynamicStub.java)

GlassFish/Grizzly interrupts the request processing thread after a timeout expires.

The interrupt should not cause fillInStackTrace to get stuck. This looks like a JRE bug, but I have not yet been able to reproduce this outside of GlassFish.

The interrupts are not sent when running in debug mode which make this problem more difficult to reproduce/test.

You can avoid this bug by increasing the request timeout or disable it using -1.

&lt;network-config>
  &lt;protocols>
    &lt;protocol name="http-listener-1">
      &lt;http request-timeout-seconds="-1"