Skip to main content

Auto Reply: OpenMQ unexpected shutdown

4 replies [Last post]
al130959
Offline
Joined: 2005-09-28
Points: 0

Ich bin vom 18.04.2011 bis 29.04.2011 in Urlaub und werde meine E-Mails während dieser Zeit nur unregelmäßig abrufen. Vielen Dank für Ihr Verständnis!

//

I am on vacation from Apr 18th to Apr 29th, 2011 and will therefore only read your mail with some delay. Many thanks for your understanding!

--

ORACLE
Andreas Loew | Senior Java Architect
Phone: +49 6103 752-515 | Fax: +49 6103 752-299 | Mobile: +49 172 8126863
Oracle Advanced Customer Services

ORACLE Deutschland B.V. & Co. KG | Ampèrestraße 6 | D-63225 Langen

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstraße 25 | D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167 | NL-3543 AS Utrecht | Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven

Oracle is committed to developing practices and products that help protect the environment

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
al130959
Offline
Joined: 2005-09-28
Points: 0

Ich bin vom 18.04.2011 bis 29.04.2011 in Urlaub und werde meine E-Mails während dieser Zeit nur unregelmäßig abrufen. Vielen Dank für Ihr Verständnis!

//

I am on vacation from Apr 18th to Apr 29th, 2011 and will therefore only read your mail with some delay. Many thanks for your understanding!

--

ORACLE
Andreas Loew | Senior Java Architect
Phone: +49 6103 752-515 | Fax: +49 6103 752-299 | Mobile: +49 172 8126863
Oracle Advanced Customer Services

ORACLE Deutschland B.V. & Co. KG | Ampèrestraße 6 | D-63225 Langen

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstraße 25 | D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167 | NL-3543 AS Utrecht | Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven

Oracle is committed to developing practices and products that help protect the environment

ebratt
Offline
Joined: 2005-03-07
Points: 0

Hi Márcio
I'm sorry to hear that you're still having trouble with this.

From what I can see:
GlassFish 2.1.1
MQ 4.4
Running Windows 2003
JDK 1.6.0_18

* It appears you are now running the broker as LOCAL (please correct
me, if that's wrong)
* I think you also said you never saw this problem in your own tests
* One interesting observation is, according to your first report,
this was also happening when you ran the broker with the EMBEDDED
configuration.

Could you try another possible work-around and configure the broker as
REMOTE. That would disable all "life-cycle" services from GlassFish --
so, if the problem is somehow related to an errant signal from
GlassFish, this should prevent that from occurring. (Not that we are
aware of any issue in this part of GlassFish, but this is certainly a
mystery so, trying different configurations could help us isolate this.)
If you can do this, and you can run for a long period of time, without
seeing this problem again, that could be significant. If you go this
route, you'll need to start-up and shutdown MQ separately, from GlassFish.

The server log doesn't cover the time of the first error, on April 21.
For the event on April 26th, sadly, I don't see anything you didn't
already highlight in the logs. I can't see a benefit from continuing
with FINEST logging since that hasn't exposed anything useful and, I'm
sure it's consuming lots of disk-space.

Can you review for us, the differences between what you ran in your
testing versus what is running at this customer site? (OS, JDK, GF, MQ
versions, do you use shell processes to start/stop, or are you using
Windows Service manager to start/stop?) It might help us to understand
what's different, and perhaps, why this wasn't apparent when you ran
your tests prior to delivery to your customer.

I'll ask around and see if anyone else here has any further ideas for
diagnosing this issue.

If you do reconfigure with REMOTE broker, and the problem goes away,
please let us know so that we can try to investigate this further. We
have seen occasional posts, similar to this on the lists, but I don't
recall a definitive conclusion ever being posted. Our presumption, in
the past, was that the user found the problem and didn't feel the need
for us to take further action. That may or may not be correct and we'd
certainly like to get to the bottom of this.

Thanks,
-- Ed

On 4/27/2011 10:03 AM, Márcio Geovani Jasinski wrote:
> Hi Ed,
>
> Mentioned issue is still happening. We registered 2 unexpected
> shutdown on April 20th and yesterday April 26th.
> In order to get more details of what´s happening I changed log level
> on several Glassfish components to finest.
>
> I did the same log adjustment in .ear application and there´s no
> signal of error neither context destroy information as we have coded.
> Moreover Glassfish did realize that something went wrong - but it
> tells that an admin requested a shutdown which nobody did...
> Glassfish is installed as a service and there´s no entry on Event
> Viewer about a service restart/stop.
>
> Heres the log (attached as well):
>
> [#|2011-04-26T16:58:25.579-0300|FINEST|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=com.sun.enterprise.web.PESessionLockingStandardPipeline;MethodName=unlockSession;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|finished
> unlocking session: sess
> =StandardSession[f52117d126e161b94ad468361421]|#]
>
> [#|2011-04-26T16:58:25.579-0300|FINEST|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=com.sun.enterprise.web.PESessionLockingStandardPipeline;MethodName=unlockSession;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|LOCK
> = _lockType= null
> foregroundRefCount= 0|#]
>
> [#|2011-04-26T16:58:25.579-0300|FINE|sun-appserver2.1|org.apache.coyote.tomcat5.OutputBuffer|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=org.apache.coyote.tomcat5.OutputBuffer;MethodName=realWriteBytes;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|realWrite(b,
> 0, 135) org.apache.coyote.Response@1fc9529|#]
>
> [#|2011-04-26T16:58:25.579-0300|FINEST|sun-appserver2.1|org.apache.coyote.tomcat5.InputBuffer|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=org.apache.coyote.tomcat5.InputBuffer;MethodName=recycle;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|recycle()|#]
>
> [#|2011-04-26T16:58:25.579-0300|FINE|sun-appserver2.1|org.apache.coyote.tomcat5.OutputBuffer|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=org.apache.coyote.tomcat5.OutputBuffer;MethodName=recycle;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|recycle()|#]
>
> [#|2011-04-26T16:58:25.579-0300|FINEST|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=48;_ThreadName=httpSSLWorkerThread-8585-3;ClassName=com.sun.enterprise.web.connector.grizzly.DefaultReadTask;MethodName=registerKey;_RequestID=f4b3f4f9-196d-49b5-a767-4a4e08836f78;|registerKey|#]
>
> [#|2011-04-26T16:58:25.767-0300|FINER|sun-appserver2.1|javax.resourceadapter.mqjmsra.outbound.connection|_ThreadID=56;_ThreadName=iMQEventHandler-1214;ClassName=com.sun.messaging.jms.ra.ConnectionEventListener;MethodName=onEvent();E201:[E201]:
> Connection closed due to admin requested shutdown:
> localhost:6767(1995),
> com.sun.messaging.jms.notification.ConnectionClosedEvent[source=BrokerAddress=localhost:6767(1995),
> ConnectionID=8082324740562242048, ReconnectEnabled: true,
> IsConnectedToHABroker:
> false];_RequestID=6dea1acb-2d02-4e22-aa9e-cc68fccb3956;|ENTRY
> E201:[E201]: Connection closed due to admin requested shutdown:
> localhost:6767(1995),
> com.sun.messaging.jms.notification.ConnectionClosedEvent[source=BrokerAddress=localhost:6767(1995),
> ConnectionID=8082324740562242048, ReconnectEnabled: true,
> IsConnectedToHABroker: false]|#]
>
>
> From OpenMQ, the issue is caught by the shutdown hook:
>
> [26/Apr/2011:16:58:17 GMT-03:00] Creating new Producer
> 8082324791354713856 on Q:COMMAND_OUT for connection
> 8082324740562242048
> [26/Apr/2011:16:58:18 GMT-03:00] Creating new Producer
> 8082324791355121920 on Q:COMMAND_OUT for connection
> 8082324740562242048
> [26/Apr/2011:16:58:20 GMT-03:00] Creating new Producer
> 8082324791355521792 on Q:COMMAND_OUT for connection
> 8082324740562242048
> [26/Apr/2011:16:58:25 GMT-03:00] Shutdown requested by
> brokerShutdownHook
> [26/Apr/2011:16:58:25 GMT-03:00] [B1047]: Shutting down broker...
> [26/Apr/2011:16:58:25 GMT-03:00] [B1077]: Broadcast good-bye to
> all connections ...
> [26/Apr/2011:16:58:25 GMT-03:00] [B1078]: Flushing good-bye
> messages ...
> [26/Apr/2011:16:58:25 GMT-03:00] Close Session 8082324680546840832
> [26/Apr/2011:16:58:25 GMT-03:00] Detaching Consumer
> [consumer:8082324680546844929, type=CLIENT_ACKNOWLEDGE] on
> connection 8082324680546829056 from Session 8082324680546840832
> last id was null
> [26/Apr/2011:16:58:25 GMT-03:00] Session: Pausing Session
> [8082324680546840832][1]Consumer.java: Detatch consumer A Consumer
> - Q:EVENT_IN:[consumer:8082324680546844929, type=CLIENT_ACKNOWLEDGE]
> [26/Apr/2011:16:58:25 GMT-03:00] Session: Resuming Session
> [8082324680546840832][0]Consumer.java: resuming after detatch
> Consumer - Q:EVENT_IN:[consumer:8082324680546844929,
> type=CLIENT_ACKNOWLEDGE]
> [26/Apr/2011:16:58:25 GMT-03:00] Cleaning up transactions on
> connection IMQConn[CLOSED,guest@127.0.0.1:2023
> ,null]
>
>
> We got access to the server as soon they realized the system was down
> and I could verify that Glassfish was not working.
> Following listeners didn´t respond at all after OpenMQ
> shutdown. Example: HTTP (8585) HTTP(4949), JMX (8686. Attached is
> all open ports at this time).
>
> Ed, is it possible to check what´s going on with those logs using
> finest level?
> We are looking forward to find a suitable solution since those
> shutdowns are stopping client employees to get access to their areas.
>
> Any thoughts will be very appreciated. Thanks.
>
> Regards,
> Márcio Geovani Jasinski
> Blumenau, Santa Catarina
> Fone: +55 47 9653 4899

marciogj
Offline
Joined: 2010-06-02
Points: 0

Hi Ed,

Thanks again for prompt answer and all insight.

The info you got from logs is absolutely right: Glassfish 2.1.1, MQ 4.4 and
JDK 1.6.0_18 (32 bits)
Few more details of server OS is Windows 2003 64 SP 2.

Right now we are using JMS Type Local but normally we use Embeeded (all
development and quality tests were done only with Embeeded).
At this client server MQ shutdown is happening in both configuration Local
and Embeeded.

Next week I'll try set up Remote mode on clients pre-prod environment and
see how is going to behave.I think it is a very interesting scenario to be
tested. In case everything goes well, I'll try same on production server and
let you about the results.

During this week we switched back the default log level. You guessed right
about disk consumption...
I also did a watchdog service which restart Glassfish automatically every
time JMS port is not listed on netstat command.
It's a workaround to reduce client impact regarding this issue and also send
me a email reporting that server went down.
This also will be helpful during any other verification like use Remote JMS
type.

At development side - we normally difference on OS side (Developers have
Windows 7 and Vista).
Although Glassfish and MQ version is always the same as used on client
side I think we have following differences between development team and
client environment:
- JDK can be slightly different since it's managed by Eclipse and each
developer can change it. Our company Eclipse default version uses JDK 1.5;
- Domain profile was development.

Before 'go to market phase' we also did several tests with quality group. On
this case most of the tests had pretty similar config as our client has.
- Cluster domain with Embeeded JMS Type;
- Glassfish, OpenMQ and JDK were same (2.11, 4.4 and 1.6.0_18)
- Windows 2003 32 bits and 64 bits
- Tests
- Main difference on this case would be infrastructure since those tests
were done in a lab with kind of isolated network and environment.
And also application database since internal tests used Oracle XE and SQL
Server Express.

Another thing that might be very useful Ed:
We have exactly same application running fine almost 2 months and it's using
Glassfish 2.1 with JDK 1.5 (at our own company).
This was considered 1st stage of 'go to market' and didn't reveal any issue
even until now.
Maybe this issue is related only to version 2.1.1 but it's pretty hard to
say.

Next week I'll try MQ as Remote and I'll let you know about the results. It
can take few weeks to have it but as soon I have some new I'll let you know.

Cheers,
Márcio Geovani Jasinski
Blumenau, Santa Catarina
Fone: +55 47 9653 4899

visik7
Offline
Joined: 2011-05-19
Points: 0

Hi,
I'm getting the same error using OpenMQ 4.5 as a standalone broker for a jboss cluster.
How did you solve it ?