Skip to main content

Cannot stop domain

3 replies [Last post]
slominskir
Offline
Joined: 2010-11-15
Points: 0

I've had Glassfish 3.1.1 running for several months now with no issue and now when I try to shut it down with asadmin stop-domain the response is that it isn't running, even though it is (I can request pages just fine). If I navigate to the web console on port 4848 I get a blank page. The server.log shows a bunch of errors:

[#|2012-08-30T14:52:57.375-0400|WARNING|glassfish3.1.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=31;_ThreadName=Thread-2;|ApplicationDispatcher[] PWC1231: Servlet.service() for servlet FacesServlet threw exception
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException while attempting to process a 'beforeCreate' event for 'event1'.
        at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:422)
        at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:394)
        at com.sun.jsftemplating.layout.descriptors.LayoutComponent.beforeCreate(LayoutComponent.java:348)
        at com.sun.jsftemplating.layout.descriptors.LayoutComponent.getChild(LayoutComponent.java:288)
        at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:556)
        at com.sun.jsftemplating.layout.LayoutViewHandler.createView(LayoutViewHandler.java:255)
        at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:247)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:116)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:593)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
        at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:787)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:649)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:483)
        at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:454)
        at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:350)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:300)
        at org.glassfish.admingui.common.security.AdminConsoleAuthModule.validateRequest(AdminConsoleAuthModule.java:231)
        at com.sun.enterprise.security.jmac.config.GFServerConfigProvider$GFServerAuthContext.validateRequest(GFServerConfigProvider.java:1171)
        at com.sun.web.security.RealmAdapter.validate(RealmAdapter.java:1445)
        at com.sun.web.security.RealmAdapter.invokeAuthenticateDelegate(RealmAdapter.java:1323)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:551)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:623)
        at org.apache.catalina.core.StandardPipeline.doChainInvoke(StandardPipeline.java:600)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:96)
        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:327)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:232)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor659.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jsftemplating.layout.descriptors.handler.Handler.invoke(Handler.java:442)
        at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:420)
        ... 46 more
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketException: Unexpected end of file from server
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
        at org.glassfish.admingui.common.util.RestUtil.get(RestUtil.java:711)
        at org.glassfish.admingui.common.util.RestUtil.restRequest(RestUtil.java:190)
        at org.glassfish.admingui.common.handlers.RestApiHandlers.restRequest(RestApiHandlers.java:216)
        ... 51 more
Caused by: java.net.SocketException: Unexpected end of file from server
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:777)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:774)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        ... 58 more
|#]

[#|2012-08-30T14:53:34.542-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=29;_ThreadName=Thread-2;|GRIZZLY0039: Request URI is too large.
java.nio.BufferOverflowException
        at com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:765)
        at com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:402)
        at com.sun.grizzly.http.ProcessorTask.parseRequest(ProcessorTask.java:861)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:692)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-08-30T14:53:34.542-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=29;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception.
java.lang.NullPointerException
        at java.nio.CharBuffer.put(CharBuffer.java:896)
        at com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendRequestInfo(CommonAccessLogFormatterImpl.java:203)
        at com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendLogEntry(CommonAccessLogFormatterImpl.java:133)
        at com.sun.enterprise.web.accesslog.CombinedAccessLogFormatterImpl.appendLogEntry(CombinedAccessLogFormatterImpl.java:68)
        at com.sun.enterprise.web.PEAccessLogValve.postInvoke(PEAccessLogValve.java:592)
        at com.sun.enterprise.web.VirtualServer$2.onParsingError(VirtualServer.java:1698)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:709)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
|#]

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
rdarbh
Offline
Joined: 2011-08-18
Points: 0

Try to do a kill -9 on the PID (ps -ef | grep glassfish), if you are on Linux, or just end process on Windows, and then try start-domain.

slominskir
Offline
Joined: 2010-11-15
Points: 0

I was really looking for an explanation for why this is happening; This seems like a fairly serious bug. I don't want to abruptly kill my server! To abruptly kill the server I can use the --kill=true flag to asadmin.

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi,

pls. make a dump of a threads, when server hangs using jstack and
send the results here.

Thanks.

WBR,
Alexey.

On 09/02/2012 04:17 PM, forums@java.net wrote:
> I was really looking for an explanation for why this is happening;
> This seems
> like a fairly serious bug. I don't want to abruptly kill my server! To
> abruptly kill the server I can use the --kill=true flag to asadmin.
>
> --
>
> [Message sent by forum member 'slominskir']
>
> View Post: http://forums.java.net/node/889712
>
>

Anonymous

OK, executed stop domain and this time I get the following response:
./asadmin stop-domain java.net.SocketException: Unexpected end of file from
server and the log file shows:
[#|2012-09-06T15:50:47.381-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=21;_ThreadName=Thread-2;|GRIZZLY0039:
Request URI is too large. java.nio.BufferOverflowException at
com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:765)
at
com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:402)
at com.sun.grizzly.http.ProcessorTask.parseRequest(ProcessorTask.java:861) at
com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:692) at
com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019) at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662) |#]
[#|2012-09-06T15:50:47.381-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=21;_ThreadName=Thread-2;|GRIZZLY0051:
ProcessorTask exception. java.lang.NullPointerException at
java.nio.CharBuffer.put(CharBuffer.java:896) at
com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendRequestInfo(CommonAccessLogFormatterImpl.java:203)
at
com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendLogEntry(CommonAccessLogFormatterImpl.java:133)
at
com.sun.enterprise.web.accesslog.CombinedAccessLogFormatterImpl.appendLogEntry(CombinedAccessLogFormatterImpl.java:68)
at
com.sun.enterprise.web.PEAccessLogValve.postInvoke(PEAccessLogValve.java:592)
at
com.sun.enterprise.web.VirtualServer$2.onParsingError(VirtualServer.java:1698)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:709) at
com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019) at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662) |#] The server isn't really hung
though because it continues to service other requests just fine. I ran
jstack, but there are too many threads to include here. I don't see anything
all that interesting in the stack trace. Is there any particular thread that
I should be examining?

--

[Message sent by forum member 'slominskir']

View Post: http://forums.java.net/node/889712

slominskir
Offline
Joined: 2010-11-15
Points: 0

OK, executed stop domain and this time I get the following response:

./asadmin stop-domain
java.net.SocketException: Unexpected end of file from server

and the log file shows:

[#|2012-09-06T15:50:47.381-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=21;_ThreadName=Thread-2;|GRIZZLY0039: Request URI is too large.
java.nio.BufferOverflowException
        at com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:765)
        at com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:402)
        at com.sun.grizzly.http.ProcessorTask.parseRequest(ProcessorTask.java:861)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:692)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
|#]

[#|2012-09-06T15:50:47.381-0400|SEVERE|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=21;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception.
java.lang.NullPointerException
        at java.nio.CharBuffer.put(CharBuffer.java:896)
        at com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendRequestInfo(CommonAccessLogFormatterImpl.java:203)
        at com.sun.enterprise.web.accesslog.CommonAccessLogFormatterImpl.appendLogEntry(CommonAccessLogFormatterImpl.java:133)
        at com.sun.enterprise.web.accesslog.CombinedAccessLogFormatterImpl.appendLogEntry(CombinedAccessLogFormatterImpl.java:68)
        at com.sun.enterprise.web.PEAccessLogValve.postInvoke(PEAccessLogValve.java:592)
        at com.sun.enterprise.web.VirtualServer$2.onParsingError(VirtualServer.java:1698)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:709)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
|#]

The server isn't really hung though because it continues to service other requests just fine. I ran jstack, but there are too many threads to include here. I don't see anything all that interesting in the stack trace. Is there any particular thread that I should be examining?