Skip to main content

Writing a JMS client for a Message Queue Broker Cluster (embedded in GlassFish 3.1.2)?

18 replies [Last post]
kappert
Offline
Joined: 2003-06-12
Points: 0

We have a GlassFish cluster with two instances, hosted on Jelastic.com. The application produces messages on a queue "the_output_queue".

We want to consume the output messages with a simple and light-weight JMS client running somewhere else (this could maybe even be an Android app).

GlassFish sets up Message Queue as an embedded broker cluster, which in turn has also two broker instances corresponding directly to the two GlassFish instances.

If I understand this description of "Cluster Message Delivery" correctly, a JMS client should be able to connect to ANY of the two broker instances and be able to receive ALL messages written to the_output_queue:

The home broker is responsible for routing and delivering the messages to all consumers of the destination, whether these consumers are local (connected to the home broker) or remote (connected to other brokers in the cluster).

But this is not working as I hoped. Messages for the_output_queue are distributed in a round-robin manner to both brokers. With only one JMS client, connected to the first broker, I see only every other message. (With a second JMS client connected to the second broker I get the other half of the messages).

The JMS clients connect to the physical queue directly, not with JNDI. Basically:

  ConnectionFactory myFctry = new com.sun.messaging.ConnectionFactory();
  myFctry.setProperty(ConnectionConfiguration.imqAddressList, "thathost:27676");
  Connection myConnection = myFctry.createConnection();
  Session mySession = myConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);
  Destination myDest = new com.sun.messaging.Queue("the_output_queue");
  MessageConsumer messageConsumer = mySession.createConsumer(myDest);
  myConnection.start();
  Message incomingMsg = messageConsumer.receive();

Did I misunderstand how a broker cluster works?
How should a stand-alone JMS consumer connect to a broker cluster?

Or if it should be working the way I thought: What should I check in the GlassFish and Message Queue configurations? (Note: I am in Jelastic and have no command line utilities available, only the GlassFish Administration Console.).

[We never asked for a Message Queue broker cluster, but that is what you get with a GlassFish cluster. And a remote broker is currently not possible with Jelastic.]

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
nigeldeakin
Offline
Joined: 2007-10-12
Points: 0

> Did I misunderstand how a broker cluster works?

No, your understanding is completely correct. If the two message queue brokers are clustered (which they should be if the two GlassFish instances are clustered) then it shouldn't make any difference which broker a client is connected to.

Before looking for other explanations, the first thing to check is whether the two brokers are indeed connected together as a cluster. Have a look at the broker log for each GlassFish instance (which will be under the GlassFish instance in a directory glassfish3\glassfish\nodes\nodeName\instanceName\imq\instances\clusterNameinstanceName\log).

After the brokers start they will try to connect to each other and log a message to say that this has happened successfully. Look for messages similar to:

[timestamp] [B1150]: Waiting for cluster connection to mq://1.2.3.4:27677/?instName=???&brokerSessionUID=null to be established

. . .

[timestamp] [B1071]: Established cluster connection to broker mq://1.2.3.4:27677/?instName=cluster1instance2&brokerSessionUID=1739584606839549696[myhost/1.2.3.4:51198]

If the brokers can't connect for some reason you'll see an error message.

kappert
Offline
Joined: 2003-06-12
Points: 0

Thanks a lot for the response and sorry for the delay. For administrative reasons I had no access to a real Jelastic account for further tests.

With your confirmation in mind I tested a cluster on my PC and all works as expected. The client sees all messages, no matter to which instance of the cluster it is connected.

In the log on my working configuration I found this line:

[#|2013-06-17T14:53:24.888+0200|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=80;_ThreadName=Thread-2;|JMS010: ADDRESSLIST in setJmsServiceProvider: mq://localhost:27676/,mq://localhost:27677/|#]


On Jelastic I have to use the Log Viewer of the Admin Console. I searched for "ADDRESSLIST in setJmsServiceProvider" and got:

Instance 1: ADDRESSLIST in setJmsServiceProvider: mq://localhost:27676/,mq://172.16.2.21:27676/
Instance 2: ADDRESSLIST in setJmsServiceProvider: mq://172.16.2.21:27676/,mq://localhost:27676/

The "localhost" looked suspect and lead me to GLASSFISH-17278 Bug (closed).

I then tried to modify the first node's "Node Host" from localhost to its IP address. But that just gives an exception: Cannot update node localhost-domain1. It is the built-in localhost node.

Do you have any suggestions on how to proceed? Is there something Jelastic should do differently when setting up a GlassFish cluster or is it a GlassFish problem? Can I fix it myself?

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

Hello kappert,

How did you create the cluster with instances?

Yes, it is true that node localhost-domain1 is not udpatable. But you can try to create a new local node on the same machine of DAS, and then create local instance with reference to the node. For example,

asadmin create-node-config node1
asadmin create-local-instance --node node1 instance1

Then you can update the node host of node1.

kappert
Offline
Joined: 2003-06-12
Points: 0

Thanks for your input Liang!

The cluster was created by Jelastic (a cloud platform) because I selected the "High-availability" option. I do not have control over the way that Jelastic sets up the GlassFish cluster. I also do not have access to the command line.

All I have is the Admin Console. Your suggestion is good! I will try to create my own new local node and use that in the cluster. Hopefully it does not get in the way of something that Jelastic does (applications are deployed through their interface, for example).

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

Nodes and instances can be created via admin gui too.

kappert
Offline
Joined: 2003-06-12
Points: 0

I have created a new node "localhost-with-ip" of type "CONFIG" that specifies IP address 172.16.2.20 as "Node Host". Then I stopped the "localhost"-based instance of the cluster and added a new one, based on the "localhost-with-ip" node. I gave this instance the same properties (ports) like the old one had. Then I got rid of the old instance.

After some problems (I think they disappeared after I removed my applications and JMS queues) I finally had the two instances starting up (my new one 172.16.2.20, plus the remote 172.16.2.21).

Things looked promising in my opinion. From the logfiles:
INSTANCE 1:
JMS010: ADDRESSLIST in setJmsServiceProvider: mq://172.16.2.20:27676/,mq://172.16.2.21:27676/
GlassFish MQ JMS Resource Adapter Started:EMBEDDED

INSTANCE 2:
JMS010: ADDRESSLIST in setJmsServiceProvider: mq://172.16.2.21:27676/,mq://172.16.2.20:27676/
GlassFish MQ JMS Resource Adapter Started:EMBEDDED

BUT... when I click on "JMS Physical Destinations" for the cluster to recreate the queues I had to delete I get this error:
An error has occurred Unable to list JMS Destinations. Please ensure that the Message Queue Brokers are running.
(This happens in the GlassFish admin console: click on the cluster, then select tab "JMS Physical Destinations").

This does not make sense to me as the "JMS Service Type" is set to "EMBEDDED" in the cluster configuration, and the logs confirm this.
Also, I do not see any warning or exception in the log files. I am attaching all log files. They are not shortened and show a cluster start.

REMARK: The original instance (the one I replaced with a new instance) had an instance property "rendezvousOccurred=true". I was not sure what it exactly means and tried my new instance with and without that property. But it did not help. I cannot create queues because the admin console insists that JMS is not running.

I cannot spend more time on this at the moment and will fall back to a single instance. But if somebody has a solution, I would appreciate it!

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

kappert,

The error message that you saw "An error has occurred Unable to list JMS Destinations." looks like a defect to me. Could you file a JIRA defect here with reproducible steps? I will probably fix it in 4.0.1

A workaround is to create Queue at Resources -> JMS Resources -> Destination Resources -> New... and choose the cluster as the target through admin gui.

/david

kappert
Offline
Joined: 2003-06-12
Points: 0

Thanks David for the follow-up. The suggested workaround seems to work in the GUI, but the physical queue is not created. Instead the following exception appears in the domain log.

[#|2013-06-20T10:21:23.680+0000|INFO|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.deployer|_ThreadID=466;_ThreadName=Thread-2;|RAR8029: Resource [ jms/nq ] of type [ aor ] is not enabled|#]

If the target is "gfcluster" (the cluster, and nothing else), the exception appears in the domain log, but nothing in the instance logs.
If the target is "server" (and nothing else), the exception appears in both instance logs, but not in the domain log (strange enough).

For the records: Clicking on "JMS Physical Destinations" for the cluster produces this exception in the domain log:

[#|2013-06-20T10:06:40.556+0000|SEVERE|glassfish3.1.2|org.glassfish.admingui|_ThreadID=418;_ThreadName=Thread-2;|RestResponse.getResponse() gives FAILURE.  endpoint = 'https://localhost:4848/management/domain/clusters/cluster/gfcluster/list-jmsdest'; attrs = '{}'|#]

I did many steps that brought me to the current state and I have not documented them, except what you find in this thread. Essentially I replaced the "localhost" node in the cluster created through Jelastic's "high-availability" option with a clone of that node where I replaced "localhost" with the local IP address. Then things went really wild (inconsistent lock states of my queues) and I was about to delete the entire setup when I found out that these exceptions went away if I removed all physical queues (and undeployed the applications referring to them).

Unfortunately I cannot invest more time of our project into this problem at the moment. Let me know if you want me to file a JIRA defect with the information documented here. But Jelastic has taken this up as a bug and I assume they will follow up on it, probably with Oracle support channels.

Fortunately we do not depend on the cluster for the coming months. So the problem is not that urgent for us.

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

I cannot reproduce what you reported at my local env. Please file JIRA issue if you can steadily reproduce it (it is better to see it at your local glassfish installation instead of the Cloud host).

It looks like something is out of order after you did many configuration updates, can you try if restarting DAS helps?

/David

kappert
Offline
Joined: 2003-06-12
Points: 0

I also do not have the problem on my PC, it concerns only the Jelastic environment where the cluster instances are on different nodes (high-availability option).

I have restarted the DAS many times, but for the remaining problem it does not change anything.

For the time being I think it is best to leave this up to Jelastic to investigate. They have assigned bug JE-10808 to it.

Thanks a lot for your help!

/Beat (this is my actual first name :-)

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

Beat,

Is it available to share your domain.xml for my trying locally?

/David

kappert
Offline
Joined: 2003-06-12
Points: 0

David, I was able to copy & paste the domain.xml through the Jelastic interface and have attached it.

I have arrived at this configuration through manipulations in the GlassFish Admin Console. This is not the original Jelastic high-availability configuration anymore, although it was the starting point.

/Beat

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

Beat,

Your domain.xml looks OK and it doesn't have any jms resources defined. Can you please double check it with the following steps?

1) Make sure gfcluster is started up successfully. This can be verified by checking admin console tab "Cluster > gfcluster > Instances". Make sure both instance-2886730261 and instance-2886730260-with-ip are in Running status before step 2).

2) In admin console, go to "Cluster > gfcluster > JMS Physical Destinations" to list the jms resources.

3) If you see the error message "Unable to list JMS Destinations." again, please collect DAS server.log to see if any exceptions. Please share it to me if possible.

The thing is that the Message Queue Brokers are running is neccessary even though JMS is in EMBEDDED mode. And exceptions in DAS server.log would be very helpful for leading us to the root cause.

/David

kappert
Offline
Joined: 2003-06-12
Points: 0

Hello David

I have performed the steps you have mentioned:

1) "Cluster > gfcluster > Instances": both instances "Running", with a green check mark.

2) "Cluster > gfcluster > JMS Physical Destinations": ERROR -> "Unable to list JMS Destinations. Please ensure that the Message Queue Brokers are running."

3) These are the newly added log messages after step 2:

Domain log:
[#|2013-06-24T09:32:41.371+0000|SEVERE|glassfish3.1.2|org.glassfish.admingui|_ThreadID=247;_ThreadName=Thread-2;|RestResponse.getResponse() gives FAILURE. endpoint = 'https://localhost:4848/management/domain/clusters/cluster/gfcluster/list-jmsdest'; attrs = '{}'|#]

Instance 1 log:
[#|2013-06-24T09:32:39.032+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=46;_ThreadName=Thread-2;|JMS010: ADDRESSLIST in setJmsServiceProvider: mq://172.16.2.20:27676/,mq://172.16.2.21:27676/|#]

[#|2013-06-24T09:32:39.032+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=46;_ThreadName=Thread-2;|JMS08: JMS Service Connection URL is : mq://172.16.2.20:27676/,mq://172.16.2.21:27676/|#]

[#|2013-06-24T09:32:39.132+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version: 4.5.2 Patch 1 (Build 3-d) Compile: Thu Jun 7 10:46:15 PDT 2012|#]

[#|2013-06-24T09:32:39.138+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is EMBEDDED, connection mode is TCP|#]

[#|2013-06-24T09:32:40.437+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:EMBEDDED|#]

Instance 2 log:
[#|2013-06-24T09:32:36.786+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=46;_ThreadName=Thread-2;|JMS010: ADDRESSLIST in setJmsServiceProvider: mq://172.16.2.21:27676/,mq://172.16.2.20:27676/|#]

[#|2013-06-24T09:32:36.787+0000|INFO|glassfish3.1.2|javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system|_ThreadID=46;_ThreadName=Thread-2;|JMS08: JMS Service Connection URL is : mq://172.16.2.21:27676/,mq://172.16.2.20:27676/|#]

[#|2013-06-24T09:32:36.926+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version: 4.5.2 Patch 1 (Build 3-d) Compile: Thu Jun 7 10:46:15 PDT 2012|#]

[#|2013-06-24T09:32:36.926+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is EMBEDDED, connection mode is TCP|#]

[#|2013-06-24T09:32:38.849+0000|INFO|glassfish3.1.2|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=46;_ThreadName=Thread-2;|MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:EMBEDDED|#]

I hope you can see something in these messages that helps. Please remember: My entire setup might well be in contradiction with how Jelastic meant to configure both nodes (virtual machines). Maybe they cannot even exchange messages due to some security restriction.

BTW: This is a suspicious message I see before step 2, after starting the cluster:
[#|2013-06-24T09:24:40.226+0000|WARNING|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1063: missed GroupStartupComplete message. Timed out group startup after 240 secs|#]

liang.x.zhao
Offline
Joined: 2011-12-18
Points: 0

Beat,

The Domain log shows the problem is that the admin gui uses wrong url for the REST service when doing list jms destination:

[#|2013-06-24T09:32:41.371+0000|SEVERE|glassfish3.1.2|org.glassfish.admingui|_ThreadID=247;_ThreadName=Thread-2;|RestResponse.getResponse() gives FAILURE. endpoint = 'https://localhost:4848/management/domain/clusters/cluster/gfcluster/list-jmsdest'; attrs = '{}'|#]

The endpoint is HTTPS protocol, but as far as I know, it should be HTTP instead. If you open a browser window and access url of 'http://localhost:4848/management/domain/clusters/cluster/gfcluster/list-jmsdest'; attrs = '{}', then you can get the REST response.

I don't know the reason why the protocol is changed to be HTTPS for REST internally inside glassfish. So I would like to leave it to the AdminGui/Admin/REST team to comment on this.

/David

kappert
Offline
Joined: 2003-06-12
Points: 0

Thanks for analyzing the logs. As we have public IP addresses I tried to access that URL from my PC:

https://xx.xx.xxx.xxx:24848/management/domain/clusters/cluster/gfcluster...

xx.xx.xxx.xxx is the public IP of our first or second GF cluster instance (both give the same result).
24848 is the port defined as ASADMIN_LISTENER_PORT in the cluster configuration.

After entering the required credentials I got a WEB GUI (see attached screenshot). This looks interesting, but I do not succeed to get any queues listed, not even mq.sys.dmq, which I have not deleted (or did not mean to, at least). I also tried to create a queue indirectly as you have suggested earlier (by creating a JMS Destination Resource). But it does not show up either.

In the Web GUI I tried filling in destType as "Queue", "javax.jms.Queue", "Topic" and "javax.jms.Topic" and then pressed the list-jmsdest button. But I never got a result. Maybe you have an idea what parameters I should add to the URL or how to operate that little Web GUI?

Otherwise I am fine with you leaving this up the AdminGui/Admin/REST team for further examination.

siruslan
Offline
Joined: 2010-02-10
Points: 0

> And a remote broker is currently not possible with Jelastic.
why? what is the limitation?

kappert
Offline
Joined: 2003-06-12
Points: 0

Jelastic currently does not offer a message broker that could be configured in their environment topology editor, although RabbitMQ is on the feature wishlist.
The Oracle Message Queue binaries are there because of GlassFish, but cannot be used to start an external broker as there is no commmand line access.

Indeed, with a VDS (virtual dedicated server) you could install any product. But a VDS is not free and brings with it the configuration and administration efforts.

For our purposes we would be happy with the embedded message broker. I think it also has a good reputation regarding performance. If only we could get it to work properly in the GlassFish cluster on Jelastic...