Skip to main content

"GRIZZLY0023: Interrupting idle Thread" filling log, web apps not responding

8 replies [Last post]
Anonymous

Hi all,

Using GF 3.1.2, we've started seeing that the server stops responding
to web requests, and the logs are filled with these:

[#|2012-03-31T10:50:38.589-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-443(2).|#]

[#|2012-03-31T10:50:38.589-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-443(5).|#]

[#|2012-03-31T10:50:38.590-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-443(3).|#]

[#|2012-03-31T10:50:38.590-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-443(1).|#]

[#|2012-03-31T10:50:38.590-0400|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-443(4).|#]

As you can see, we get about 5 of them/second, and we have to restart
the app server to get things working again. Hunting around, I gather
that this happens when a web request is running for more than 15
minutes -- is that correct? We don't have anything in the web app that
does this directly, but we are using an ICEPush feature to push
changes to the user, and maybe that's somehow causing this. We've only
started seeing the problem recently, though, which makes this more
curious (maybe we just have more users now).

Can someone give me a pointer about how I can debug the issue to see
what's really going on? I can try a thread dump on one of the servers
the next time it gets in this state, but is there any other suggestion
in the meantime? Is there any way to make this situation *not* make
the server unresponsive?

Thanks,
Bobby

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Bobby Bissett

On Mon, Apr 2, 2012 at 10:13 AM, Bobby Bissett wrote:
> [...] We don't have anything in the web app that
> does this directly, but we are using an ICEPush feature to push
> changes to the user, and [...]

Paying more attention to the logs now, I see these messages from
ICEPush at startup:

[#|2012-04-02T10:35:01.822-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=21;_ThreadName=Thread-2;|Adapting
to Servlet 3.0 AsyncContext environment|#]

[#|2012-04-02T10:35:01.824-0400|INFO|glassfish3.1.2|org.icepush.servlet.AsyncAdaptingServlet|_ThreadID=21;_ThreadName=Thread-2;|Using
Servlet 3.0 AsyncContext|#]

[#|2012-04-02T10:35:01.865-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=19;_ThreadName=Thread-2;|Adapting
to Servlet 3.0 AsyncContext environment|#]

[#|2012-04-02T10:35:01.866-0400|INFO|glassfish3.1.2|org.icepush.servlet.AsyncAdaptingServlet|_ThreadID=19;_ThreadName=Thread-2;|Using
Servlet 3.0 AsyncContext|#]

[#|2012-04-02T10:35:02.798-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=22;_ThreadName=Thread-2;|Falling
back to Thread Blocking environment|#]

Does that "Falling back to Thread Blocking environment" mean that push
isn't available by default in GF 3.1.2? This is starting to seem
vaguely familiar.... Certainly "thread blocking" sounds like something
that could cause a request to take more than 15 minutes. ;)

Thanks,
Bobby

oleksiys
Offline
Joined: 2006-01-25

Hi Bobby,

>> [...] We don't have anything in the web app that
>> does this directly, but we are using an ICEPush feature to push
>> changes to the user, and [...]
> Paying more attention to the logs now, I see these messages from
> ICEPush at startup:
>
> [#|2012-04-02T10:35:01.822-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=21;_ThreadName=Thread-2;|Adapting
> to Servlet 3.0 AsyncContext environment|#]
>
> [#|2012-04-02T10:35:01.824-0400|INFO|glassfish3.1.2|org.icepush.servlet.AsyncAdaptingServlet|_ThreadID=21;_ThreadName=Thread-2;|Using
> Servlet 3.0 AsyncContext|#]
>
> [#|2012-04-02T10:35:01.865-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=19;_ThreadName=Thread-2;|Adapting
> to Servlet 3.0 AsyncContext environment|#]
>
> [#|2012-04-02T10:35:01.866-0400|INFO|glassfish3.1.2|org.icepush.servlet.AsyncAdaptingServlet|_ThreadID=19;_ThreadName=Thread-2;|Using
> Servlet 3.0 AsyncContext|#]
>
> [#|2012-04-02T10:35:02.798-0400|INFO|glassfish3.1.2|org.icepush.servlet.EnvironmentAdaptingServlet|_ThreadID=22;_ThreadName=Thread-2;|Falling
> back to Thread Blocking environment|#]
>
> Does that "Falling back to Thread Blocking environment" mean that push
> isn't available by default in GF 3.1.2? This is starting to seem
> vaguely familiar.... Certainly "thread blocking" sounds like something
> that could cause a request to take more than 15 minutes. ;)
Looks like that. I wonder what's the reason to fallback to
thread-blocking...

Thanks.

WBR,
Alexey.

Bobby Bissett

>> Does that "Falling back to Thread Blocking environment" mean that push
>> isn't available by default in GF 3.1.2? This is starting to seem
>> vaguely familiar.... Certainly "thread blocking" sounds like something
>> that could cause a request to take more than 15 minutes. ;)
>
> Looks like that. I wonder what's the reason to fallback to
> thread-blocking...

Well, maybe it's because our web app was missing the
true element? Sigh.

Besides adding this in, do I need to do anything else? I've set these
values to true, but with very brief testing it doesn't appear to make
a difference:

asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.comet-support-enabled=true
asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.comet-support-enabled=true

I get a warning and stack trace the 1st time ICEPush tries to update,
but after that it looks like things are working (though I haven't
tested fully). The stack trace is below. Can you tell me if anything
looks familiar there? I know ICEPush != GF, but you know a lot more
about the Grizzly/web stack than I do. :)

Also, do I need to enable Comet support or not as written above?

Thanks,
Bobby

[#|2012-04-03T10:14:10.907-0400|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-2;|StandardWrapperValve[PPCDApplication]:
PWC1406: Servlet.service() for servlet PPCDApplication threw exception
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.IllegalStateException: Request already released from
asynchronous mode
at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.comet.CometEngine.executeServlet(CometEngine.java:444)
at com.sun.grizzly.comet.CometEngine.handle(CometEngine.java:308)
at com.sun.grizzly.comet.CometAsyncFilter.doFilter(CometAsyncFilter.java:87)
at com.sun.grizzly.arp.DefaultAsyncExecutor.invokeFilters(DefaultAsyncExecutor.java:171)
at com.sun.grizzly.arp.DefaultAsyncExecutor.interrupt(DefaultAsyncExecutor.java:143)
at com.sun.grizzly.arp.AsyncProcessorTask.doTask(AsyncProcessorTask.java:102)
at com.sun.grizzly.http.TaskBase.run(TaskBase.java:193)
at com.sun.grizzly.http.TaskBase.execute(TaskBase.java:175)
at com.sun.grizzly.arp.DefaultAsyncHandler.handle(DefaultAsyncHandler.java:145)
at com.sun.grizzly.arp.AsyncProtocolFilter.execute(AsyncProtocolFilter.java:210)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException:
java.lang.IllegalStateException: Request already released from
asynchronous mode
at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:133)
at org.icepush.BlockingConnectionServer.access$700(BlockingConnectionServer.java:41)
at org.icepush.BlockingConnectionServer$RunningServer.service(BlockingConnectionServer.java:198)
at org.icepush.BlockingConnectionServer.service(BlockingConnectionServer.java:76)
at org.icepush.ConfigurationServer.service(ConfigurationServer.java:87)
at org.icepush.servlet.AsyncAdaptingServlet.service(AsyncAdaptingServlet.java:47)
at org.icepush.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:51)
at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
at org.icepush.servlet.BrowserDispatcher$BrowserEntry.service(BrowserDispatcher.java:111)
at org.icepush.servlet.BrowserDispatcher.service(BrowserDispatcher.java:50)
at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
at org.icepush.servlet.MainServlet.service(MainServlet.java:72)
at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:62)
... 32 more
Caused by: java.lang.IllegalStateException: Request already released
from asynchronous mode
at org.apache.catalina.connector.Request.asyncComplete(Request.java:3986)
at org.apache.catalina.connector.AsyncContextImpl.complete(AsyncContextImpl.java:242)
at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:66)
at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:131)
... 44 more
|#]

oleksiys
Offline
Joined: 2006-01-25

On 04/03/2012 06:12 PM, Bobby Bissett wrote:
>>> Does that "Falling back to Thread Blocking environment" mean that push
>>> isn't available by default in GF 3.1.2? This is starting to seem
>>> vaguely familiar.... Certainly "thread blocking" sounds like something
>>> that could cause a request to take more than 15 minutes. ;)
>> Looks like that. I wonder what's the reason to fallback to
>> thread-blocking...
> Well, maybe it's because our web app was missing the
> true element? Sigh.
>
> Besides adding this in, do I need to do anything else? I've set these
> values to true, but with very brief testing it doesn't appear to make
> a difference:
>
> asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.comet-support-enabled=true
> asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.comet-support-enabled=true
>
> I get a warning and stack trace the 1st time ICEPush tries to update,
> but after that it looks like things are working (though I haven't
> tested fully). The stack trace is below. Can you tell me if anything
> looks familiar there? I know ICEPush != GF, but you know a lot more
> about the Grizzly/web stack than I do. :)
Well, it looks like ICEPush or your app tries to finish Request/Response
processing from the different threads. Normally only one thread may work
with a single Request/Response instance at the same time, and only one
thread has to complete/finish the Request/Response processing.
If you can create a simple testcase, which reproduces the issue - it
would help.

> Also, do I need to enable Comet support or not as written above?
If you don't use Comet and from the stacktraces it looks like you don't
- then there is no need to enable it.

Thanks.

WBR,
Alexey.

>
> Thanks,
> Bobby
>
> [#|2012-04-03T10:14:10.907-0400|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-2;|StandardWrapperValve[PPCDApplication]:
> PWC1406: Servlet.service() for servlet PPCDApplication threw exception
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Request already released from
> asynchronous mode
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
> at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
> at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
> at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
> at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
> at com.sun.grizzly.comet.CometEngine.executeServlet(CometEngine.java:444)
> at com.sun.grizzly.comet.CometEngine.handle(CometEngine.java:308)
> at com.sun.grizzly.comet.CometAsyncFilter.doFilter(CometAsyncFilter.java:87)
> at com.sun.grizzly.arp.DefaultAsyncExecutor.invokeFilters(DefaultAsyncExecutor.java:171)
> at com.sun.grizzly.arp.DefaultAsyncExecutor.interrupt(DefaultAsyncExecutor.java:143)
> at com.sun.grizzly.arp.AsyncProcessorTask.doTask(AsyncProcessorTask.java:102)
> at com.sun.grizzly.http.TaskBase.run(TaskBase.java:193)
> at com.sun.grizzly.http.TaskBase.execute(TaskBase.java:175)
> at com.sun.grizzly.arp.DefaultAsyncHandler.handle(DefaultAsyncHandler.java:145)
> at com.sun.grizzly.arp.AsyncProtocolFilter.execute(AsyncProtocolFilter.java:210)
> at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
> at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
> at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
> at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException:
> java.lang.IllegalStateException: Request already released from
> asynchronous mode
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:133)
> at org.icepush.BlockingConnectionServer.access$700(BlockingConnectionServer.java:41)
> at org.icepush.BlockingConnectionServer$RunningServer.service(BlockingConnectionServer.java:198)
> at org.icepush.BlockingConnectionServer.service(BlockingConnectionServer.java:76)
> at org.icepush.ConfigurationServer.service(ConfigurationServer.java:87)
> at org.icepush.servlet.AsyncAdaptingServlet.service(AsyncAdaptingServlet.java:47)
> at org.icepush.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:51)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.BrowserDispatcher$BrowserEntry.service(BrowserDispatcher.java:111)
> at org.icepush.servlet.BrowserDispatcher.service(BrowserDispatcher.java:50)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.MainServlet.service(MainServlet.java:72)
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:62)
> ... 32 more
> Caused by: java.lang.IllegalStateException: Request already released
> from asynchronous mode
> at org.apache.catalina.connector.Request.asyncComplete(Request.java:3986)
> at org.apache.catalina.connector.AsyncContextImpl.complete(AsyncContextImpl.java:242)
> at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:66)
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:131)
> ... 44 more
> |#]

Bobby Bissett

On Tue, Apr 3, 2012 at 12:50 PM, Oleksiy Stashok
wrote:
> Well, it looks like ICEPush or your app tries to finish Request/Response
> processing from the different threads. Normally only one thread may work
> with a single Request/Response instance at the same time, and only one
> thread has to complete/finish the Request/Response processing.
> If you can create a simple testcase, which reproduces the issue - it would
> help.

I finally sat down tonight to try a simple test case, and my test case
works just fine. (You'd think that'd be a good thing!) So I need to
figure out what the difference is between my test case and our real
app.

Thanks for the help so far.

Cheers,
Bobby

Bobby Bissett

> I finally sat down tonight to try a simple test case, and my test case
> works just fine. (You'd think that'd be a good thing!) So I need to
> figure out what the difference is between my test case and our real
> app.

I think what might have been happening is that I was doing the first
push before the initial requests to load the page were even done.
Whether or not that's the case, I can now start a sample app and our
production app with true and they
start without the problems I saw before.

But now I'm seeing two problems:

1. My sessions never expire, which we can't have. I know this isn't
the ICEPush forum, but in case you have an idea about it I wanted to
mention it.

2. I now see exceptions like the one below. Do you know what could be
causing this? I do have a small sample web app that reproduces it.
After the error below, then I start seeing other ones, such as
"java.lang.IllegalStateException: Response already closed" and I have
to restart the server to get the web app running again.

Again, this is with GF 3.1.2.

Thanks,
Bobby

[#|2012-05-01T12:43:35.623-0400|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=20;_ThreadName=Thread-3;|StandardWrapperValve[MainServlet]:
PWC1406: Servlet.service() for servlet MainServlet threw exception
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.IllegalStateException: Not Suspended
at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.RuntimeException:
java.lang.IllegalStateException: Not Suspended
at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:133)
at org.icepush.BlockingConnectionServer.access$700(BlockingConnectionServer.java:41)
at org.icepush.BlockingConnectionServer$RunningServer.service(BlockingConnectionServer.java:198)
at org.icepush.BlockingConnectionServer.service(BlockingConnectionServer.java:76)
at org.icepush.ConfigurationServer.service(ConfigurationServer.java:87)
at org.icepush.servlet.AsyncAdaptingServlet.service(AsyncAdaptingServlet.java:47)
at org.icepush.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:51)
at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
at org.icepush.servlet.BrowserDispatcher$BrowserEntry.service(BrowserDispatcher.java:111)
at org.icepush.servlet.BrowserDispatcher.service(BrowserDispatcher.java:50)
at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
at org.icepush.servlet.MainServlet.service(MainServlet.java:72)
at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:62)
... 25 more
Caused by: java.lang.IllegalStateException: Not Suspended
at com.sun.grizzly.tcp.Response.resume(Response.java:768)
at org.apache.catalina.connector.Request.asyncComplete(Request.java:3991)
at org.apache.catalina.connector.AsyncContextImpl.complete(AsyncContextImpl.java:242)
at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:66)
at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:131)
... 37 more
|#]

rlubke
Offline
Joined: 2003-08-21

On 5/1/12 9:56 AM, Bobby Bissett wrote:
>> I finally sat down tonight to try a simple test case, and my test case
>> works just fine. (You'd think that'd be a good thing!) So I need to
>> figure out what the difference is between my test case and our real
>> app.
> I think what might have been happening is that I was doing the first
> push before the initial requests to load the page were even done.
> Whether or not that's the case, I can now start a sample app and our
> production app withtrue and they
> start without the problems I saw before.
>
> But now I'm seeing two problems:
>
> 1. My sessions never expire, which we can't have. I know this isn't
> the ICEPush forum, but in case you have an idea about it I wanted to
> mention it.
Perhaps someone from the web container team will chime in here.
>
> 2. I now see exceptions like the one below. Do you know what could be
> causing this? I do have a small sample web app that reproduces it.
> After the error below, then I start seeing other ones, such as
> "java.lang.IllegalStateException: Response already closed" and I have
> to restart the server to get the web app running again.
Could you email me your app/reproduction steps? I'll take a look asap.
>
> Again, this is with GF 3.1.2.
>
> Thanks,
> Bobby
>
>
> [#|2012-05-01T12:43:35.623-0400|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=20;_ThreadName=Thread-3;|StandardWrapperValve[MainServlet]:
> PWC1406: Servlet.service() for servlet MainServlet threw exception
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Not Suspended
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
> at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
> at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
> at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
> at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
> at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
> at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
> at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
> at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
> at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
> at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
> at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.RuntimeException:
> java.lang.IllegalStateException: Not Suspended
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:133)
> at org.icepush.BlockingConnectionServer.access$700(BlockingConnectionServer.java:41)
> at org.icepush.BlockingConnectionServer$RunningServer.service(BlockingConnectionServer.java:198)
> at org.icepush.BlockingConnectionServer.service(BlockingConnectionServer.java:76)
> at org.icepush.ConfigurationServer.service(ConfigurationServer.java:87)
> at org.icepush.servlet.AsyncAdaptingServlet.service(AsyncAdaptingServlet.java:47)
> at org.icepush.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:51)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.BrowserDispatcher$BrowserEntry.service(BrowserDispatcher.java:111)
> at org.icepush.servlet.BrowserDispatcher.service(BrowserDispatcher.java:50)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.MainServlet.service(MainServlet.java:72)
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:62)
> ... 25 more
> Caused by: java.lang.IllegalStateException: Not Suspended
> at com.sun.grizzly.tcp.Response.resume(Response.java:768)
> at org.apache.catalina.connector.Request.asyncComplete(Request.java:3991)
> at org.apache.catalina.connector.AsyncContextImpl.complete(AsyncContextImpl.java:242)
> at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:66)
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:131)
> ... 37 more
> |#]

swchan2
Offline
Joined: 2005-03-29

On 4/3/12 9:12 AM, Bobby Bissett wrote:
>>> Does that "Falling back to Thread Blocking environment" mean that push
>>> isn't available by default in GF 3.1.2? This is starting to seem
>>> vaguely familiar.... Certainly "thread blocking" sounds like something
>>> that could cause a request to take more than 15 minutes. ;)
>> Looks like that. I wonder what's the reason to fallback to
>> thread-blocking...
> Well, maybe it's because our web app was missing the
> true element? Sigh.
From the stack trace below, servlet async is used. So, the above xml
element is required.
>
> Besides adding this in, do I need to do anything else? I've set these
> values to true, but with very brief testing it doesn't appear to make
> a difference:
>
> asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.comet-support-enabled=true
> asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.comet-support-enabled=true
Do you use/libraries comet, too? It would be good to figure out what is
used in the given library.
>
> I get a warning and stack trace the 1st time ICEPush tries to update,
> but after that it looks like things are working (though I haven't
> tested fully). The stack trace is below. Can you tell me if anything
> looks familiar there? I know ICEPush != GF, but you know a lot more
> about the Grizzly/web stack than I do. :)
>
> Also, do I need to enable Comet support or not as written above?
>
> Thanks,
> Bobby
>
> [#|2012-04-03T10:14:10.907-0400|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=24;_ThreadName=Thread-2;|StandardWrapperValve[PPCDApplication]:
> PWC1406: Servlet.service() for servlet PPCDApplication threw exception
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Request already released from
> asynchronous mode
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
> at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
> at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
> at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
> at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
> at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
> at com.sun.grizzly.comet.CometEngine.executeServlet(CometEngine.java:444)
> at com.sun.grizzly.comet.CometEngine.handle(CometEngine.java:308)
> at com.sun.grizzly.comet.CometAsyncFilter.doFilter(CometAsyncFilter.java:87)
> at com.sun.grizzly.arp.DefaultAsyncExecutor.invokeFilters(DefaultAsyncExecutor.java:171)
> at com.sun.grizzly.arp.DefaultAsyncExecutor.interrupt(DefaultAsyncExecutor.java:143)
> at com.sun.grizzly.arp.AsyncProcessorTask.doTask(AsyncProcessorTask.java:102)
> at com.sun.grizzly.http.TaskBase.run(TaskBase.java:193)
> at com.sun.grizzly.http.TaskBase.execute(TaskBase.java:175)
> at com.sun.grizzly.arp.DefaultAsyncHandler.handle(DefaultAsyncHandler.java:145)
> at com.sun.grizzly.arp.AsyncProtocolFilter.execute(AsyncProtocolFilter.java:210)
> at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
> at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
> at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
> at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
> at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
> at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException:
> java.lang.IllegalStateException: Request already released from
> asynchronous mode
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:133)
> at org.icepush.BlockingConnectionServer.access$700(BlockingConnectionServer.java:41)
> at org.icepush.BlockingConnectionServer$RunningServer.service(BlockingConnectionServer.java:198)
> at org.icepush.BlockingConnectionServer.service(BlockingConnectionServer.java:76)
> at org.icepush.ConfigurationServer.service(ConfigurationServer.java:87)
> at org.icepush.servlet.AsyncAdaptingServlet.service(AsyncAdaptingServlet.java:47)
> at org.icepush.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:51)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.BrowserDispatcher$BrowserEntry.service(BrowserDispatcher.java:111)
> at org.icepush.servlet.BrowserDispatcher.service(BrowserDispatcher.java:50)
> at org.icepush.servlet.PathDispatcher.service(PathDispatcher.java:45)
> at org.icepush.servlet.MainServlet.service(MainServlet.java:72)
> at org.vaadin.artur.icepush.ICEPushServlet.service(ICEPushServlet.java:62)
> ... 32 more
> Caused by: java.lang.IllegalStateException: Request already released
> from asynchronous mode
> at org.apache.catalina.connector.Request.asyncComplete(Request.java:3986)
> at org.apache.catalina.connector.AsyncContextImpl.complete(AsyncContextImpl.java:242)
> at org.icepush.servlet.AsyncAdaptingServlet$AsyncRequestResponse.respondWith(AsyncAdaptingServlet.java:66)
> at org.icepush.BlockingConnectionServer.respondIfPendingRequest(BlockingConnectionServer.java:131)
> ... 44 more
This indicates that the async context object is used "after" the request
is released.
Possibly, the async context complete is called too late in
BrowserDispatcher$BrowserEntry.service.
> |#]