Skip to main content

Deadlock in AppServerStartup and ConnectorsRecoveryResourceHandler during Glassfish 3.1.2.2 startup.

2 replies [Last post]
sbarlabanov
Offline
Joined: 2011-09-24
Points: 0

Hi,
as already described in the forum entry http://www.java.net/forum/topic/glassfish/glassfish/deadlock-transaction..., we faced some strange behavior during Glassfish startup resulting in deadlock somewhere between transaction recovery and startup singletons. Unfortunately our application is quite big and we could not create a smaller application reproducing the bug.
But we did some investigations of the thread dumps and debugging of Glassfish core using Glassfish source code.
Here is what we found and what we found looks like a Glassfish bug ;):

RecoveryHelperThread started by Glassfish is blocked on com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:62) by trying to obtain ApplicationLoaderService from the habitat (ConnectorsRecoveryResourceHandler.java:217). Seems that it tries to get some connector stuff and for that it needs ApplicationLoaderService to wait for connector to initialize I think. We use ActiveMQ RAR, but it is deployed independently of our Web application as a separate connector application (so no big master EAR). So RecoveryHelperThread blocks on trying to obtain ApplicationLoaderService. And it can't get ApplicationLoaderService since it is already blocked by the main thread in AppServerStartup line 253. AppServerStartup tries at that moment to load our Web application and to execute our startup singletons using ApplicationLoaderService. Our startup singletons are doing some transactional stuff. So they try to open a transaction. And they block on com.sun.jts.CosTransactions.EventSemaphore.waitEvent since RecoveryHelperThread is not ready yet with its recovery. And it can't since it is blocked on ApplicationLoaderService which is blocked by main which is blocked on CosTransactions.EventSemaphore. Deadlock. Or am I wrong?
I attached the stack traces of RecoveryHelperThread and the main thread.

Best regards,
Sergiy

AttachmentSize
deadlock.txt15.48 KB

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
j.hedin
Offline
Joined: 2012-09-03
Points: 0

We have exactly the same problem with Glassfish 3.1.2.2. The code is a single JEE 6 EAR with session EJB and JPA calls in the @PostConstruct of an @Singleton, @Startup. The deadlock is in the same place in the RecoveryHelperThread.

Best regards

Johan

T.Bergkvist
Offline
Joined: 2013-05-08
Points: 0

The problem only occurs when transaction are started in the same thread as the @Startup bean. We solved it by starting a new thread, from within the startup bean. This new thread performs all transation related work.

Tommy