Lost JMS Messages
We have a clustered instance of GlassFish installed in our production environment. This consists of 2 domains that reside on separate servers. We are currently utilizing JMS in our applications and the JMS servers are configured as embedded.
The issue we have now is that from within application 1, a JMS Message was submitted but appears to have been lost. This has happened a total of 4 times since yesterday. Making a connection, via JNDI, I was able to browse the queue but found no messages residing. I went to the file system for each server and located under each of the domains is an imq directory. Drilling down into the directory I was able to locate our queues and the vrfile.
I was able to grep through the vrfile looking for our lost messages. I managed to locate all 4 messages on server1 but none of them were within the vrfile on server2.
This appears to be a synchronization issue as it is clear that the messages did get given to the JMS server but for some reason they did not replicate. Now they appear to be completely inaccessible.
Why did the messages get stuck on one server and how can I resolve this issue?