Skip to main content

JMS Calls Throw JTS5067: Unexpected error occurred in commit

6 replies [Last post]
rdarbh
Offline
Joined: 2011-08-18

I am on glassfish 3.1 build 43.

We use a couple of JMS Queues that worked all these days, and suddenly, they started throwing these errors

[#|2012-08-30T12:53:57.608-0600|WARNING|glassfish3.1|javax.enterprise.system.core.transaction.com.sun.jts.jtsxa|_ThreadID=153;_ThreadName=Thread-1;|JTS5067: Unexpected error occurred in commit
java.lang.IllegalStateException: File not synced.  You must call Iterator to play back log file.
        at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.writeRecord(FileTransactionLogWriter.java:841)
        at com.sun.messaging.jmq.util.txnlog.file.FileTransactionLogWriter.write(FileTransactionLogWriter.java:824)
        at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.writeTransactionEvent(TransactionLogManager.java:779)
        at com.sun.messaging.jmq.jmsserver.persist.file.TransactionLogManager.logTxn(TransactionLogManager.java:739)
        at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.logTxn(FileStore.java:2480)
        at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.logTxn(TransactionHandler.java:1480)
        at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doCommit(TransactionHandler.java:1056)
        at com.sun.messaging.jmq.jmsserver.data.handlers.TransactionHandler.doCommit(TransactionHandler.java:941)
        at com.sun.messaging.jmq.jmsserver.data.protocol.ProtocolImpl.commitTransaction(ProtocolImpl.java:700)
        at com.sun.messaging.jmq.jmsserver.service.imq.IMQDirectService.commitTransaction(IMQDirectService.java:1746)
        at com.sun.messaging.jms.ra.DirectXAResource.commit(DirectXAResource.java:221)
        at com.sun.jts.jtsxa.OTSResourceImpl.commit_one_phase(OTSResourceImpl.java:174)
        at com.sun.jts.CosTransactions.RegisteredResources.commitOnePhase(RegisteredResources.java:1588)
        at com.sun.jts.CosTransactions.TopCoordinator.commitOnePhase(TopCoordinator.java:2969)
        at com.sun.jts.CosTransactions.CoordinatorTerm.commit(CoordinatorTerm.java:321)
        at com.sun.jts.CosTransactions.TerminatorImpl.commit(TerminatorImpl.java:250)
        at com.sun.jts.CosTransactions.CurrentImpl.commit(CurrentImpl.java:623)
        at com.sun.jts.jta.TransactionManagerImpl.commit(TransactionManagerImpl.java:319)
        at com.sun.enterprise.transaction.jts.JavaEETransactionManagerJTSDelegate.commitDistributedTransaction(JavaEETransactionManagerJTSDelegate.java:173)
        at com.sun.enterprise.transaction.JavaEETransactionManagerSimplified.commit(JavaEETransactionManagerSimplified.java:873)
        at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:5115)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4880)
        at com.sun.ejb.containers.MessageBeanContainer.afterMessageDeliveryInternal(MessageBeanContainer.java:1207)
        at com.sun.ejb.containers.MessageBeanContainer.afterMessageDelivery(MessageBeanContainer.java:1180)
        at com.sun.ejb.containers.MessageBeanListenerImpl.afterMessageDelivery(MessageBeanListenerImpl.java:86)
        at com.sun.enterprise.connectors.inbound.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:143)
        at $Proxy275.afterDelivery(Unknown Source)
        at com.sun.messaging.jms.ra.OnMessageRunner.run(OnMessageRunner.java:328)
        at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:114)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
|#]

And this is the SEVERE error that shows up every time we call the JMS queues.

[#|2012-08-30T12:53:57.609-0600|SEVERE|glassfish3.1|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=153;_ThreadName=Thread-1;|JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [commit one phase] operation.|#]
[#|2012-08-30T12:53:57.610-0600|WARNING|glassfish3.1|javax.enterprise.system.container.ejb.mdb.com.sun.ejb.containers|_ThreadID=153;_ThreadName=Thread-1;|MDB00037: [ps:InvalidateMessageHandler]: Message-driven bean invocation exception: [javax.ejb.EJBException: Unable to complete container-managed transaction.]|#]

Our code has not changed in the last 7-8 months, and it works fine on our AT servers, but it fails on Production.

This is the code that calls the Queues.

javax.jms.Connection connection = connectionFactory.createConnection();
            Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
            MessageProducer messageProducer = session.createProducer(queue);
            TextMessage msg = session.createTextMessage();
            msg.setText("Manual Call");
            messageProducer.send(msg);
            connection.close();

Can anyone let me know if they encountered these errors, and if they know what to do?

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
amyk
Offline
Joined: 2007-09-10

Most likely the initialization of the file store failed at broker startup, then the init exception should be logged in the broker log. What does the broker log show ?

amyk
Offline
Joined: 2007-09-10
rdarbh
Offline
Joined: 2011-08-18

Went back to logs from Server start up and saw these errors:

[29/Aug/2012:23:37:39 MDT] [B1132]: Auto-creating destination PsQueue [Queue]
[29/Aug/2012:23:37:39 MDT] ERROR [B4147]: Failed to load message directory /hosts/ussds/glassfishv3/glassfish/domains/domain1/imq/instances/imqbroker/fs370/message/QPsQueue for destination Q:PsQueue:
java.io.StreamCorruptedException: /hosts/ussds/glassfishv3/glassfish/domains/domain1/imq/instances/imqbroker/fs370/message/QPsQueue/vrfile: Bad file magic number: 0; Expecting: 1431677610
[29/Aug/2012:23:37:39 MDT] [B3044]: Internal Error:
com.sun.messaging.jmq.jmsserver.util.BrokerException: Failed to load message directory /hosts/ussds/glassfishv3/glassfish/domains/domain1/imq/instances/imqbroker/fs370/message/QPsQueue for destination Q:PsQueue

Is there a way to remove this fs370 folder, and will it recreate it?

rdarbh
Offline
Joined: 2011-08-18

Fixed the issue. Thanks for pointing me in that direction.

It turns out that the vrfile (in /fs370/message) was corrupted. So every time the server got re-started, even after repeated deploys, it was failing with the JMS error because the file was persisted in the above folder.

Went ahead and deleted the folders in /message, and re-started the server. That fixed the issue. I still don't know what caused this vrfile to be corrupt though. Well, some other day.

amyk
Offline
Joined: 2007-09-10

It appears that the broker was abnormally terminated then restarted, during the restart, the broker's txnlog detects the abnormal shutdown and set replay required flag, but the file store init failed either before the replay or during the replay - probably for the same "StreamCorruptedException" reason, and despite that, due to bug MQ-199, the broker starts up as "normal", then on receiving client request that needs to write the txnlog, it gaves "java.lang.IllegalStateException: File not synced. You must call Iterator to play back log file"

So it's very likely the "StreamCorruptedException" in the vrfile is related to the nature of the abnormal termination of the broker process, likely rooted to disk IO.

rdarbh
Offline
Joined: 2011-08-18

Thanks for the reply. I undeployed the ear file, and then purged all the Queue entries using flush-jmsdest, but still seeing the error.

This is the log from /imqbroker/log when I make the call:

[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=9
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=8
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=7
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=6
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=5
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=4
[31/Aug/2012:10:20:00 MDT] [B1066]:   Closing: :0->jmsdirect:0 because "[B0059]: Client closed the connection". Count: service=0 broker=3
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=4 broker=4
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=5 broker=5
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=6 broker=6
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=7 broker=7
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=8 broker=8
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=9 broker=9
[31/Aug/2012:10:20:00 MDT] [B1065]: Accepting: :0->jmsdirect:0. Count: service=10 broker=10