Skip to main content

Lost JMS Messages

2 replies [Last post]
cmathrusse
Offline
Joined: 2008-04-15

We have a clustered instance of GlassFish installed in our production environment. This consists of 2 domains that reside on separate servers. We are currently utilizing JMS in our applications and the JMS servers are configured as embedded.

The issue we have now is that from within application 1, a JMS Message was submitted but appears to have been lost. This has happened a total of 4 times since yesterday. Making a connection, via JNDI, I was able to browse the queue but found no messages residing. I went to the file system for each server and located under each of the domains is an imq directory. Drilling down into the directory I was able to locate our queues and the vrfile.

/imq/instances/PACQSR11PASQSR12/fs370/message/QquotingDeliveryExchangeQ/vrfile

I was able to grep through the vrfile looking for our lost messages. I managed to locate all 4 messages on server1 but none of them were within the vrfile on server2.

This appears to be a synchronization issue as it is clear that the messages did get given to the JMS server but for some reason they did not replicate. Now they appear to be completely inaccessible.

Why did the messages get stuck on one server and how can I resolve this issue?

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
cmathrusse
Offline
Joined: 2008-04-15

We are also seeing a good amount of exceptions within the server.log file, pertaining to Corba. Could this be the issue causing our messages to be lost?

[#|2008-10-21T09:26:33.560-0700|WARNING|sun-appserver9.1|javax.enterprise.resource.corba.ee._CORBA_.rpc.transport|_ThreadID=2777;_ThreadName=p: thread-pool-1; w: 4333;Thread[p: thread-pool-1; w: 4333,5,main];org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No;_RequestID=4131bf98-ce6d-4ff8-bce4-f131889333ab;|"IOP00710311: (INTERNAL) Worker thread Thread[p: thread-pool-1; w: 4333,5,main] caught throwable org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No while executing work."
org.omg.CORBA.INTERNAL: vmcid: SUN minor code: 311 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7706)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7730)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:558)
|#]

[#|2008-10-21T09:26:33.595-0700|WARNING|sun-appserver9.1|javax.enterprise.resource.corba.ee._CORBA_.rpc.transport|_ThreadID=2778;_ThreadName=p: thread-pool-1; w: 4332;Thread[p: thread-pool-1; w: 4332,5,main];org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No;_RequestID=c172c3e8-ad83-4a9c-87b5-bafbc62dcb9a;|"IOP00710311: (INTERNAL) Worker thread Thread[p: thread-pool-1; w: 4332,5,main] caught throwable org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No while executing work."
org.omg.CORBA.INTERNAL: vmcid: SUN minor code: 311 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7706)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.workerThreadDoWorkThrowable(ORBUtilSystemException.java:7730)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:558)
|#]

suthagaran
Offline
Joined: 2008-11-12

We also getting the same exceptions, but we didn't experience any JMS message loss. But the exceptions keep on coming. If you found any solutions, Please post it.