Skip to main content

Migration to 3.12 from 2.11 fraught with issues.

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
No replies
preston001
Offline
Joined: 2008-11-19

Hello,

We're recently migrated our app from GF 2.11 to 3.12 and are running into some really strange stability issues.

For starters we're no longer able to connect Visual VM to the nodes running in the cluster. Once we connect it keeps asking for the admin credentials, which were already supplied. This is really hampering our ability to properly monitor resource usage. It appears there's another post in the forum regarding the same problem.

The biggest issue is that we have nodes just that quit responding to request and or logging anything. It's as if it's just frozen. If we shut the failing node down, then the problem seems to move to another node as it becomes unresponsive. This goes on until I've restarted the entire cluster. Also when this happens it seems as if the RMI fail over isn't working as everything starts to pile up on that node as it isn't being removed from the cluster automatically.

In reviewing the logs I'm getting thousands of these errors

[#|2012-05-07T05:51:11.386-0600|SEVERE|glassfish3.1.2|javax.enterprise.system.container.appclient.org.glassfish.appclient.server.core.jws|_ThreadID=53;_ThreadName=Thread-2;|Adapter[/___JWSappclient/___system] s1as/glassfish/modules/bean-validator.jar
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)

[#|2012-05-07T05:51:14.077-0600|SEVERE|glassfish3.1.2|javax.enterprise.system.container.appclient.org.glassfish.appclient.server.core.jws|_ThreadID=85;_ThreadName=Thread-2;|Adapter[/___JWSappclient/___system] s1as/glassfish/modules/connectors-runtime.jar
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:426)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:498)
at com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes0(SocketChannelOutputBuffer.java:381)
at com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:364)
at com.sun.grizzly.http.ajp.AjpOutputBuffer.realWriteBytes(AjpOutputBuffer.java:111)
at com.sun.grizzly.tcp.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:894)
at com.sun.grizzly.tcp.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:159)
at com.sun.grizzly.tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:661)
at com.sun.grizzly.tcp.Response.doWrite(Response.java:685)

I don't know if it's related or yet another issue.

If anyone has any insight into this please share your thoughts.

Thanks in advance!