Skip to main content

GF4, 502 Bad Gateway, OutOfMemoryError: Direct buffer memory

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
No replies
markbto
Offline
Joined: 2013-04-16

We recently upgraded from Glassfish 3.1 to Glassfish 4.0 (89) running on Java build 1.7.0_55-b13 on our live 64 bit centos 6.2 servers. We run around 30 small web applications ranging in age from 0-10 yrs old.

We are now experiencing an issue (on both of our servers upgraded) that repeats on a roughly once a fortnight basis.
Without any warning in the telemetry, the server will start responding to all requests with a 502 Bad Gateway, and start spewing exceptions into the server.log. This problems persists for 20-30 minutes and then resolves itself without any explanation.

Only other clues are a small bump in memory usage during the period of unavailability.

We tried adding the JVM config -XX:MaxDirectMemorySize=1024m but this didn't appear to alter the frequency of the error

If anyone has any idea of what might be causing this or how we might resolve it, let me know.
The exception and the JVM config are below

[2014-07-20T15:58:50.495+0100] [glassfish 4.0] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=621 _ThreadName=jk-connector(190)] [timeMillis: 1405868330495] [levelValue: 900] [[
Exception during FilterChain execution
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at org.glassfish.grizzly.memory.ByteBufferManager.allocateByteBuffer0(ByteBufferManager.java:356)
at org.glassfish.grizzly.memory.ByteBufferManager.allocateByteBufferAtLeast(ByteBufferManager.java:282)
at org.glassfish.grizzly.memory.ByteBufferManager.allocateAtLeast(ByteBufferManager.java:126)
at org.glassfish.grizzly.memory.ByteBufferManager.allocateAtLeast(ByteBufferManager.java:63)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:909)
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:75)
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
at java.lang.Thread.run(Thread.java:745)
]]


-Djava.awt.headless=true
-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
-Dfelix.fileinstall.disableConfigSave=false
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.poll=5000
-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
-XX:+UseCompressedOops
-Dosgi.shell.telnet.port=6666
-XX:+UnlockDiagnosticVMOptions
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
-XX:ParallelGCThreads=2
-Dgosh.args=--nointeractive
-XX:MaxPermSize=1g
-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-XX:+DisableExplicitGC
-Dfelix.fileinstall.bundles.startTransient=true
-Xmx10g
-XX:+UseParallelOldGC
-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
-Dfelix.fileinstall.bundles.new.start=true
-Xms10g
-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
-Dosgi.shell.telnet.ip=127.0.0.1
-Dfelix.fileinstall.log.level=2
-XX:NewRatio=2
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-XX:+AggressiveHeap
-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
-server
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Xss1m
-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-XX:MaxDirectMemorySize=1024m