Deadlock in AppServerStartup and ConnectorsRecoveryResourceHandler during Glassfish 22.214.171.124 startup.
as already described in the forum entry http://www.java.net/forum/topic/glassfish/glassfish/deadlock-transaction..., we faced some strange behavior during Glassfish startup resulting in deadlock somewhere between transaction recovery and startup singletons. Unfortunately our application is quite big and we could not create a smaller application reproducing the bug.
But we did some investigations of the thread dumps and debugging of Glassfish core using Glassfish source code.
Here is what we found and what we found looks like a Glassfish bug ;):
RecoveryHelperThread started by Glassfish is blocked on com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:62) by trying to obtain ApplicationLoaderService from the habitat (ConnectorsRecoveryResourceHandler.java:217). Seems that it tries to get some connector stuff and for that it needs ApplicationLoaderService to wait for connector to initialize I think. We use ActiveMQ RAR, but it is deployed independently of our Web application as a separate connector application (so no big master EAR). So RecoveryHelperThread blocks on trying to obtain ApplicationLoaderService. And it can't get ApplicationLoaderService since it is already blocked by the main thread in AppServerStartup line 253. AppServerStartup tries at that moment to load our Web application and to execute our startup singletons using ApplicationLoaderService. Our startup singletons are doing some transactional stuff. So they try to open a transaction. And they block on com.sun.jts.CosTransactions.EventSemaphore.waitEvent since RecoveryHelperThread is not ready yet with its recovery. And it can't since it is blocked on ApplicationLoaderService which is blocked by main which is blocked on CosTransactions.EventSemaphore. Deadlock. Or am I wrong?
I attached the stack traces of RecoveryHelperThread and the main thread.