Skip to main content

Glassfish unresponsive for prolonged period after "broken pipe"

2 replies [Last post]
maikhassel
Offline
Joined: 2005-10-07

I am running a number of Glassfish V3 servers on both Windows and Linux boxes. All servers run a single application, consisting of a JaxXWS based web service and some servlets, both of which utilize EJB3 for persisting and accessing a mysql database configured as a JNDI datasource.

On all server instances I observe the following issue:

Approx 2 - 3 times a day, we will observe a "broken pipe" exception (see below), which results in the whole application becoming unresponsive for up to 2 minutes: Calls to both servlets and the web service will simply time out. The logs show no incoming requests, packet sniffing on the web port shows no traffic.
All the while, ssh or even access to the admin console or other services on the box are not effected.

The issue seems independent of the sever load (average load is one ws call every 7 seconds, one servlet request every 5 sec), calls contain very little data.

The issue is not on the client side: I can e.g try and point a web browser at the servlet during the "outage" and it will time out, while being accessible again several minutes after

After a while (12 sec to several minutes) the app will resume normal operation. There is no observable change in application behavior (slowdown, etc) before or after the "outage", and no other error messages can be observe.

In short, it looks like the web container simply "pauses"....

Memory settings as well as different pool sizes for DB and number of accepting threads does not show any change in behavior.

The same issue could not be observed in older glassfish versions.

[#|2010-03-18T13:05:50.621-0700|WARNING|glassfishv3.0|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=30;_ThreadName=Thread-1;|StandardWrapperValve[PBDEnergyConsumptionCollectorService]:
PWC1406: Servlet.service() for servlet
PBDEnergyConsumptionCollectorService threw exception
javax.servlet.ServletException
at org.glassfish.webservices.JAXWSServlet.doGet(JAXWSServlet.java:220)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1523)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:188)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:641)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:97)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:85)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:185)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:332)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:233)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:165)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:791)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:330)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:309)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: java.io.IOException: Broken pipe
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:280)
at com.sun.xml.ws.transport.http.HttpAdapter.publishWSDL(HttpAdapter.java:583)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:252)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.publishWSDL(ServletAdapter.java:157)
at org.glassfish.webservices.JAXWSServlet.doGet(JAXWSServlet.java:213)
... 26 more
Caused by: com.ctc.wstx.exc.WstxIOException: java.io.IOException: Broken pipe
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1687)
at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:278)
... 30 more
Caused by: org.apache.catalina.connector.ClientAbortException:
java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:382)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:351)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1685)
... 32 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:106)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:321)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:392)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:370)
at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1174)
at com.sun.grizzly.tcp.Response.action(Response.java:266)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:378)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:351)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1685)
at com.ctc.wstx.sw.BaseStreamWriter.writeEndDocument(BaseStreamWriter.java:585)
at com.sun.xml.ws.server.SDDocumentImpl.writeTo(SDDocumentImpl.java:270)
... 30 more

Message was edited by: maikhassel

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
maikhassel
Offline
Joined: 2005-10-07

Update: The problem happens deterministically on a "broken pipe" situation, e.g. when I pull the network cable kill the client during a client request. It effectively "freezes" the web portion of the server for anywhere between 15sec and 10minutes, no servlet, webservice, or even simple http request is handled.

oaygun
Offline
Joined: 2009-02-19

Hi.
I have same problem with similar situation.

Did you find any solution