Skip to main content

GlassFish JMS Cluster Load Balancing Unpredictable

12 replies [Last post]
subodh_subu
Offline
Joined: 2009-08-04

Hi ,
I have a 2 machine GF Cluster(say M1 and M2).The instances are set to have equal weight (50-50) . I have a MDB listening to a Topic . I sent a Bulk of Messages(Approx. 3000) from another standalone Java Program to this topic. My problem here is the messages are not evenly distributed. One machine (say M1) gets a big chunk of messages ( aprrox 2200) and the other one(say M2) gets around 500 messages. I can see that CPU utilzation of M1 is almost 100% and for the M2 is almost none. My JMS settings are pretty much the default ones..JMS running in LOCAL mode ..I tried changing the Adress list behaviour priority to random., But to no noticable effect. Any help??

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
nigeldeakin
Offline
Joined: 2007-10-12

Hmm. I can't explain that. Could you give more information on what you are actually seeing?

To answer your other question: when you define a MDB, the connection used to consume messages is defined using an "activation spec", not a connection factory. The activation spec is defined in ejb-jar.xml or in annotation in your MDB.

Connection factories are normally only used for outbound connections.

Unfortunately, setting imqConsumerFlowLimit (or any arbitrary property) on an activation spec or connection factory won't be possible until MQ 4.4 (which will come with Glassfish 2.1.1 when it is released later this year). I plan to write a blog posting about this soon.

Nigel

(This answer relates to the JMSRA resource adapter shipped with Glassfish and used by default to connect to MQ)

subodh_subu
Offline
Joined: 2009-08-04

As per your previous reply

>>>You've set the destination property consumerFlowLimit for your topic "preprocessedQ" to 100.

>>The connection factory property imqConsumerFlowLimit is still its default value, which is 1000.

[b]How do I set imqConsumerFlowLimit ??[/b]

nigeldeakin
Offline
Joined: 2007-10-12

>
> >>>You've set the destination property
> consumerFlowLimit for your topic "preprocessedQ" to
> 100.
>
> >>The connection factory property
> imqConsumerFlowLimit is still its default value,
> which is 1000.
>
> [b]How do I set imqConsumerFlowLimit ??[/b]

You can't. That's a limitation of the jmsra resource adapter which will be fixed in 4.4/2.1.1. So we're stuck with the default of 1000.

However I would expect you to be able to configure consumer flow control by defining the corresponding property on the destination. According to the documentation, the lower value of the two is used.

(Just to reiterate, when configured in the broker the name of the property is slightly different: the property used for auto-created queues is imq.autocreate.queue.consumerFlowLimit. The property used for autocreated topics is imq.autocreate.topic.consumerFlowLimit. The destination property used for pre-configured queues and topics is consumerFlowLimit.)

Nigel

subodh_subu
Offline
Joined: 2009-08-04

thats exactly what I did.
I tried with consumerFlowLimit = 1 and 50
1 seems to be the worst , I would expect "1" to be bad in thoroughput but should be good in LB.
50 seems to do an OK job .. At least the CPU utilization seems to be at 70% in one machine and 100% on other.

Just some info on my set up.

M1 which is the DAS server and which claims as the MasterBroker and has one Instance , is Win 2003 Std - Xeon Dual. 4GB

M2 is a Windows XP-pro Desk top/2GB/Dual Core pentium.

M2 seems to do all the work, 100% CPU utilization and processes almost double the number of messages as M1.

Another interesting thing, when I do EJB loook up on this cluster, I seems to hit M2 more often than M1.

nigeldeakin
Offline
Joined: 2007-10-12

I wrote a blog article that described how MQ "load-balances" messages between Glassfish instances. This mentions how reducing the imqConsumerFlowLimit property can help. Please take a look:

http://openmessaging.blogspot.com/2009_03_01_archive.html

subodh_subu
Offline
Joined: 2009-08-04

Hi,
Let me thank you for pointing out that blog. its very helpful.
Couple of questions..
1)How can I set flow limit through the Glassfish Admin GUI( I tried adding as a new property to the JMS Settings., But it didnt work).
2) What is the property name to set through the admin GUI.
[b]imqConsumerFlowLimit[/b] or [b]imq.autocreate.topic.consumerFlowLimit[/b]

I am using [b]MDBs[/b] to consume messages from a topic in a 2 machine cluster

Thanks

Thanks again

nigeldeakin
Offline
Joined: 2007-10-12

> Couple of questions..
> 1)How can I set flow limit through the Glassfish
> Admin GUI( I tried adding as a new property to the
> JMS Settings., But it didnt work).

Unfortunately it's slightly less obvious than that: You need to set the "Start Arguments" field on the JMS Service node to -Dimq.autocreate.topic.consumerFlowLimit=50 (or whatever).

I've just written a blog posting about this:
http://openmessaging.blogspot.com/2009/08/how-to-set-arbitrary-broker-pr...

> 2) What is the property name to set through the admin
> GUI.
> [b]imqConsumerFlowLimit[/b] or
> [b]imq.autocreate.topic.consumerFlowLimit[/b]

It depends on whether your destination is auto-created or admin-created.

The difference between an auto-created destination and an admin created destinaiton is described here

If you're using an auto-created destination you need to set the broker property imq.autocreate.topic.consumerFlowLimit, as described above.

If you're using a admin-created destination you need to set imqConsumerFlowLimit on the destination. This is a property of the destination, not the broker. You can set it using imqcmd: More details here.

Nigel

subodh_subu
Offline
Joined: 2009-08-04

I still could nt get this to work..
I have set the consumerFlowLimit to 100 using "imqcmd update dst " command.. I verified the results by imq query command.. Here is my results.

[i]----------------------
Host Primary Port
----------------------
mayavi 37676

Destination Name preprocessedQ
Destination Type Topic
Destination State RUNNING
Created Administratively true

Current Number of Messages
Actual 0
Remote 0
Held in Transaction 0
Current Message Bytes
Actual 0
Remote 0
Held in Transaction 0
Current Number of Producers 0
Current Number of Producer Wildcards 0
Current Number of Consumers 4
Current Number of Consumer Wildcards 0

Max Number of Messages unlimited (-1)
Max Total Message Bytes unlimited (-1)
Max Bytes per Message unlimited (-1)
Max Number of Producers 100

Limit Behavior REJECT_NEWEST
Consumer Flow Limit 100
Is Local Destination false
Use Dead Message Queue true
XML schema validation enabled false
XML schema URI List -
Reload XML schema on failure false

Successfully queried the destination.[/i]

But still my LB is unpredicatable..
How ever when I look at my JMS client Log.. I see the consumerLimit to be 1000..
Here is the log snapshot

[i]Aug 7, 2009 7:27:08 AM com.sun.messaging.jms.ra.ManagedConnection
INFO: MQJMSRA_MC1101: constructor:Created mcId=20:xacId=8064859466712717056:Using xacf config={imqOverrideJMSPriority=false, imqConsumerFlowLimit=1000, imqOverrideJMSExpiration=false, imqAddressListIterations=3, imqLoadMaxToServerSession=true, imqConnectionType=TCP, imqPingInterval=30, imqSetJMSXUserID=false, imqConfiguredClientID=, imqSSLProviderClassname=com.sun.net.ssl.internal.ssl.Provider, imqJMSDeliveryMode=PERSISTENT, imqConnectionFlowLimit=1000, imqConnectionURL=http://localhost/imq/tunnel, imqBrokerServiceName=, imqJMSPriority=4, imqBrokerHostName=localhost, imqJMSExpiration=0, imqAckOnProduce=, imqEnableSharedClientID=false, imqAckTimeout=0, imqAckOnAcknowledge=, imqConsumerFlowThreshold=50, imqDefaultPassword=guest, imqQueueBrowserMaxMessagesPerRetrieve=1000, imqDefaultUsername=guest, imqReconnectEnabled=true, imqConnectionFlowCount=100, imqAddressListBehavior=RANDOM, imqReconnectAttempts=3, imqSetJMSXAppID=false, imqConnectionHandler=com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPStreamHandler, imqSetJMSXRcvTimestamp=false, imqBrokerServicePort=0, imqDisableSetClientID=false, imqSetJMSXConsumerTXID=false, imqOverrideJMSDeliveryMode=false, imqBrokerHostPort=7676, imqQueueBrowserRetrieveTimeout=60000, imqSetJMSXProducerTXID=false, imqSSLIsHostTrusted=false, imqConnectionFlowLimitEnabled=false, imqReconnectInterval=5000, imqAddressList=mq://molu:37676/,mq://MAYAVI:37676/, imqOverrideJMSHeadersToTemporaryDestinations=false}[/i]

My JMS client is a simple java class which does look up on the topic and send message..

and my Consumers are all MDBs in GF cluster.

Just wondering if I have to set some thing for the[b] Resource Adapter[/b] [jmsra] too?

Thanks

nigeldeakin
Offline
Joined: 2007-10-12

You've set the destination property consumerFlowLimit for your topic "preprocessedQ" to 100.

The connection factory property imqConsumerFlowLimit is still its default value, which is 1000.

These are separate properties which can hold different values. The former only applies to a specific destination whereas the latter could apply to any destination.

However according to the documentation, where this connection is used to consume messages from this particular former destination, the lower of the two values will be used:

The documentation for the connection factory property imqConsumerFlowLimit states:
( http://docs.sun.com/app/docs/doc/820-6740/aeooh?l=en&a=view)
"This limit can be overridden by a lower value set for a queue’s own consumerFlowLimit attribute"

Conversely, the documentation for the destination property consumerFlowLimit states:
( http://docs.sun.com/app/docs/doc/820-6740/aeooc?l=en&a=view )
"The client runtime can override this limit by specifying a lower value on the connection factory object.. "

So it sounds as if the lower value "wins".

However this doesn't explain why you don't get the load balancing you expect (which as my blog entry describes is determined by several properties together with the throughput of the indibvidual MDBs).

Have you tried even lower values of consumerFlowLimit, if only to confirm that changing it having an effect?

Nigel

subodh_subu
Offline
Joined: 2009-08-04

Yes I have tried with value 1.It didnt seems to have any effect.
I am yet to try the flowlimit for the connection factory.

I am bit confused abt this , since I am using MDBs as the consumers and you specify only the destination while creating the MDB.

Does the MDBs use the same connection factory used by the Message Producer to consume Messages. dont know if this make any sense

thanks
Subodh

nigeldeakin
Offline
Joined: 2007-10-12

Before we explore any further, a couple of quick questions:




What version of Glassfish are you using?




You mentioned you were using a Glassfish cluster. What "broker type" are your instances using? LOCAL, REMOTE or EMBEDDED?




(It you aren't sure, this is logged in the server log of each instance during startup. Look for a message of the form

MQJMSRA_RA1101: SJSMQ JMSRA Started:LOCAL)




Nigel

subodh_subu
Offline
Joined: 2009-08-04

I am using Glassfish version 2.1

Broker Type is LOCAL