Skip to main content

Mbean bottleneck on all instances in the cluster at the same time

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
No replies
yhzs8
Offline
Joined: 2009-03-09

Hi,

We are running:
Sun GlassFish Communications Server 2.0 ((v2.1 Patch18)(9.1_02 Patch24)) (build b02-p12)
Java version 1.6.0_22-b04

We have expreienced system freezing (response timeout) issue in a cluster with 10+ instances, the only way to fix it is to restart the cluster.

We are constantly taking thread dumps. Now when it happened again and according to the thread dump we see that the Glassfish Mbean-related code is the bottleneck (locking com.sun.jmx.interceptor.DefaultMBeanServerInterceptor which will cause many other threads waiting for the same object) but the thread holding the lock is not blocked and only does some simple operation:

Sample 1:
Sample thread dump on instance 1 for the thread responsible for the bottleneck:
"httpSSLWorkerThread-8080-120" daemon prio=10 tid=0x00002aabf999d800 nid=0x41c4 runnable [0x000000007e407000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.substring(String.java:1939)
at javax.management.ObjectName.getDomain(ObjectName.java:1589)
at com.sun.jmx.mbeanserver.Repository.retrieveNamedObject(Repository.java:326)
at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:492)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabbd9d290> (a com.sun.jmx.interceptor.DefaultMBeanServerInterceptor)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)'
... ...

Sample 2:
Smaple thread dump on instance 2 for the thread responsible for the bottleneck (taken at the same time as Sample 1):
"httpSSLWorkerThread-8080-23" daemon prio=10 tid=0x00002aabf8e69000 nid=0x7968 runnable [0x000000006b8dc000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.substring(String.java:1939)
at javax.management.ObjectName.getDomain(ObjectName.java:1589)
at com.sun.jmx.mbeanserver.Repository.retrieveNamedObject(Repository.java:326)
at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:492)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabaf83aa0> (a com.sun.jmx.interceptor.DefaultMBeanServerInterceptor)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
... ...

Sample 3:
Sample thread dump on instance 1 (taken 1 minute later) for the thread responsible for the bottleneck (now its another thread):
"httpSSLWorkerThread-8080-98" daemon prio=10 tid=0x00002aabf3d0b800 nid=0x41ae waiting for monitor entry [0x000000007c2e6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabbd9d290> (a com.sun.jmx.interceptor.DefaultMBeanServerInterceptor)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:833)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
... ...

Sample 4:
Sample thread dump on instance 2 (taken at the same time as Sample 3) for the thread responsible for the bottleneck (now its another thread):
"httpSSLWorkerThread-8080-50" daemon prio=10 tid=0x00002aabf9022000 nid=0x7983 runnable [0x000000006d3f8000]
java.lang.Thread.State: RUNNABLE
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1089)
- locked <0x00002aaabaf83aa0> (a com.sun.jmx.interceptor.DefaultMBeanServerInterceptor)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
... ...

This pattern continues until we restart the cluster.

I just wonder why all those instances got thread contention on Mbean at the same time, is there any network issue towards the DAS when synchronizing with the MbeanServer? Or could it be caused by some other issues?

In the DAS server log I do see no responding exception when trying to reaching the MbeanServer on the instances, but this error is printed out 30 minutes after the thread contention started:
[#|2013-02-07T05:46:23.404-0200|WARNING|sun-glassfish-comms-server2.0|javax.enterprise.system.stream.err|_ThreadID=37;_ThreadName=httpSSLWorkerThread-4848-2;_RequestID=b139b136-ee14-4ff8-94a2-d6dca64a1cec;|java.security.PrivilegedActionException: java.lang.reflect.InvocationTargetException
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.enterprise.jbi.AppServerContextImpl.getMBeanServerConnection(AppServerContextImpl.java:186)
at com.sun.jbi.framework.sun.SunASPlatformContext.getMBeanServerConnection(SunASPlatformContext.java:206)
at com.sun.jbi.management.facade.Facade.getMBeanServerConnection(Facade.java:218)
at com.sun.jbi.management.facade.Facade.mbeanRegistrationCheck(Facade.java:769)
at com.sun.jbi.management.facade.Facade.invokeRemoteOperation(Facade.java:277)
at com.sun.jbi.management.facade.Facade.getComponentLifeCycleMBeanName(Facade.java:1204)
at com.sun.jbi.management.facade.Facade.getComponentStateOnInstance(Facade.java:576)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnInstance(ComponentLifeCycle.java:324)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnCluster(ComponentLifeCycle.java:342)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnTarget(ComponentLifeCycle.java:308)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentState(ComponentLifeCycle.java:176)
at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentStateOnAllTargets(ComponentLifeCycle.java:762)
at com.sun.jbi.management.facade.ComponentLifeCycle.getCurrentState(ComponentLifeCycle.java:180)
at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at com.sun.enterprise.interceptor.DynamicInterceptor.getAttribute(DynamicInterceptor.java:192)
at com.sun.esb.management.base.services.AbstractServiceMBeansImpl.getMBeanAttribute(AbstractServiceMBeansImpl.java:663)
at com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.toUiComponentInfo(AbstractListStateServiceMBeansImpl.java:664)
at com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.toUiComponentInfoList(AbstractListStateServiceMBeansImpl.java:606)
at com.sun.esb.management.base.services.AbstractListStateServiceMBeansImpl.listComponents(AbstractListStateServiceMBeansImpl.java:272)
at com.sun.esb.management.impl.runtime.RuntimeManagementServiceMBeanImpl.listBindingComponents(RuntimeManagementServiceMBeanImpl.java:121)
at sun.reflect.GeneratedMethodAccessor1134.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at javax.management.StandardMBean.invoke(StandardMBean.java:391)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:174)
at com.sun.esb.management.base.services.AbstractServiceImpl.invokeMBeanOperation(AbstractServiceImpl.java:330)
at com.sun.esb.management.impl.runtime.RuntimeManagementServiceImpl.listBindingComponents(RuntimeManagementServiceImpl.java:144)
at com.sun.jbi.ui.client.JBIAdminCommandsClientImpl.listBindingComponents(JBIAdminCommandsClientImpl.java:581)
at com.sun.jbi.ui.client.JBIAdminCommandsClientImpl.listBindingComponents(JBIAdminCommandsClientImpl.java:549)
at com.sun.jbi.jsf.bean.ListBean.getListBindingComponents(ListBean.java:486)
at com.sun.jbi.jsf.util.JBITreeAdaptorImpl.getChildTreeNodeObjects(JBITreeAdaptorImpl.java:175)
at com.sun.jsftemplating.component.factory.tree.DynamicTreeNodeFactory.processNode(DynamicTreeNodeFactory.java:178)
at com.sun.jsftemplating.component.factory.tree.DynamicTreeNodeFactory.create(DynamicTreeNodeFactory.java:109)
at com.sun.jsftemplating.component.ComponentUtil.createChildComponent(ComponentUtil.java:397)
at com.sun.jsftemplating.layout.descriptors.LayoutComponent.getChild(LayoutComponent.java:278)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:538)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.buildUIComponentTree(LayoutViewHandler.java:567)
at com.sun.jsftemplating.layout.LayoutViewHandler.createView(LayoutViewHandler.java:237)
at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:208)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:102)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at com.sun.faces.extensions.avatar.lifecycle.PartialTraversalLifecycle.execute(PartialTraversalLifecycle.java:80)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
at com.sun.enterprise.tools.admingui.servlet.DelayedInitFacesServlet.service(DelayedInitFacesServlet.java:89)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:333)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.enterprise.jbi.AppServerContextImpl$2.run(AppServerContextImpl.java:189)
... 112 more
Caused by: com.sun.enterprise.admin.servermgmt.InstanceException: Not responding: service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at com.sun.enterprise.ee.admin.clientreg.InstanceRegistry.getInstanceConnection(InstanceRegistry.java:163)
... 116 more
Caused by: com.sun.enterprise.ee.admin.servermgmt.AgentException: Not responding: service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.getConnection(JMXConnectorRegistry.java:346)
at com.sun.enterprise.ee.admin.clientreg.InstanceRegistry.getInstanceConnection(InstanceRegistry.java:161)
... 116 more
Caused by: java.io.IOException: Not responding: service:jmx:rmi:///jndi/rmi://[someHostName]:38686/management/rmi-jmx-connector
at com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connectToServer(JMXConnectorRegistry.java:418)
at com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connect(JMXConnectorRegistry.java:371)
at com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.connect(JMXConnectorRegistry.java:224)
at com.sun.enterprise.ee.admin.clientreg.JMXConnectorRegistry.getConnection(JMXConnectorRegistry.java:324)
... 117 more
|#]