Skip to main content

Urgent - EJB SSB Timeouts!

13 replies [Last post]
Anonymous

Hi All,

I have an urgent request for assistance. After a year and a half of development we're finally moving into product, but we have a crazy production bug.

We have a couple of seperate instances under management, one for the web tier and one for the ejb tier. On the web tier we have a couple of timers (java.util.Timers) that are started by context listeners, that every 20 minutes call an EJB to see if reports need to be generated and made available to the web users. Because of some bugs in glassfish with injecting ejbs in this scenario, we look them up using a remote initial context each time the timer runs...which works fine for about an hour and a half, then everything goes crazy. Every call to an ejb gives a time out, and our entire application stops working. There is nothing of note in the ejb server logs that I can see (definitely no errors anyway), but in the web tier logs we get the following:

[#|2009-06-22T19:28:09.398+1000|FINE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;ClassName=com.megajobindex.web.scheduling.ReportGenerationMonitor;MethodName=run;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|attempting to load unsecured operation bean from jndi ejb/UnsecuredOperationsBean|#]

[#|2009-06-22T20:17:39.393+1000|WARNING|sun-appserver2.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=24;_ThreadName=Timer-3;1800000;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|"IOP00410219: (COMM_FAILURE) Communications timeout waiting for response. Exceeded 1,800,000 milliseconds"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.ja
:198)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
|#]

[#|2009-06-22T20:17:39.400+1000|SEVERE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGeneration
Timer-3;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|Could not determine if reports need to be generated
javax.naming.NamingException: ejb ref resolution error for remote business interfacecom.megajobindex.ejb.UnsecuredOperationsRemote [Root exception is java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe]
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:425)
at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.ejb.EJBUtils.lookupRem

... 7 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
... 15 more
|#]

Can anyone shed some light onto why this may be ocurring...I'm hoping it's just a configuration issue.

Cheers
Adam

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Adam Jenkins

Maybe someone out there that has access to a system with EJBs can try a similar test...lookup the EJB using a remote InitialContext (rather than injection), call a method, sleep the thread for 20 (or 25 to be safe) minutes, then call it again...see if you get similar results.

that was we can rule out glassfish and figure out if it's a bug, or if it's something with my configuration/installation?

--- On Thu, 25/6/09, Adam Jenkins wrote:

> From: Adam Jenkins
> Subject: Re: Urgent - EJB SSB Timeouts!
> To: users@glassfish.dev.java.net
> Received: Thursday, 25 June, 2009, 5:56 AM
>
> Thanks Chris, I'll have a look at that now...but my
> thread-pool-1 has a max of 500 connections, and the loop
> only runs once before it errors, so I'd be surprised if I
> was running out of connections...but I'll do jstack and have
> a look.
>
> Interestingly, last night I ran the following code which
> starts with a one minute sleep and continues to increment
> the sleep by one minute each time:
>
>         System.out.println("running
> test");
>         for(int i = 0; i < 45;
> i++){
>            
> System.out.println("run " + i);
>             InitialContext
> remoteContext = RemoteContextLookup.lookup();
>             String opsJndi =
> "ejb/UnsecuredOperationsBean";
>            
> System.out.println("looking up ops");
>            
> UnsecuredOperationsRemote ops =
> (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
>            
> System.out.println("executing method");
>            
> System.out.println("Response: " +
> ops.isReadyForReportGeneration(30000l));
>            
> System.out.println("finsihed run " + i);
>            
> remoteContext.close();
>             long sleepTime =
> i * 60 * 1000;
>            
> System.out.println("Sleeping for " + i + " minutes");
>            
> Thread.sleep(sleepTime);
>         }
>         System.out.println("Finished
> test");
>
> Again, exactly at a 20 minute sleep I got an error, but
> this ti

> host".
>
> I'll keep investigating today, however it seems very
> strongly tied to 20 minutes sleep time (which coincidentally
> is the repeat delay I'm using for the timer thread). 
> I'm going to drop my timer thread down to a 10 minute
> schedule and see if that solves the problem.  It seems
> as long as I maintain a continuous (at least once every 19
> minutes) link to the ejb server it's fine, which is odd,
> because non of my settings on the ejb server (pool or cache
> timeouts or anything like that) are set to 20 minutes that I
> can see anyway.
>
> --- On Wed, 24/6/09, glassfish@javadesktop.org
>
> wrote:
>
> > From: glassfish@javadesktop.org
>
> > Subject: Re: Urgent - EJB SSB Timeouts!
> > To: users@glassfish.dev.java.net
> > Received: Wednesday, 24 June, 2009, 10:22 PM
> > > by the way, I think whatever's
> > happening is happening
> > > on the ejb server not the client, as I just tried
> to
> > > run the command line program below again and it
> > > didn't even get through the first loop, so JNDI
> on
> > > that server seems to have locked up
> >
> > Hi,
> >
> > seems to reflect my assumption that maybe the ORB has
> no
> > more connections/threads. I looked again at the
> domain.xml
> > and saw that the default orb listener uses
> thread-pool-1
> > which has the maximum thread number of 200. Maybe you
> can do
> > a jstack on the PID of the Glassfish and check if
> there are
> > a large number of suspicious threads. As a start you
> could
> > look for lines containing "thread-pool-1", e.g. "p:
> > thread-pool-1; w: 2" daemon prio=10 tid=0x0844d400
> > nid=0x3516 runnable [0x65773000..0x65773fc0]".
> >
> > Cheers
> > Chris.
> > [Message sent by forum member 'chrjohn' (chrjohn)]
> >
> > http://forums.java.net/jive/thread.jspa?messageID=352714
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > For additional commands, e-mail: users-help@
ess Yahoo!7 Mail on your mobile.
> Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Marina Vatkina

I remember some discussions about problems with connections to EJBs after X
minutes delay. Try to search this forum for possible scenarios.

Regards,
-marina

Adam Jenkins wrote:
> Maybe someone out there that has access to a system with EJBs can try a similar test...lookup the EJB using a remote InitialContext (rather than injection), call a method, sleep the thread for 20 (or 25 to be safe) minutes, then call it again...see if you get similar results.
>
> that was we can rule out glassfish and figure out if it's a bug, or if it's something with my configuration/installation?
>
> --- On Thu, 25/6/09, Adam Jenkins wrote:
>
>
>>From: Adam Jenkins
>>Subject: Re: Urgent - EJB SSB Timeouts!
>>To: users@glassfish.dev.java.net
>>Received: Thursday, 25 June, 2009, 5:56 AM
>>
>>Thanks Chris, I'll have a look at that now...but my
>>thread-pool-1 has a max of 500 connections, and the loop
>>only runs once before it errors, so I'd be surprised if I
>>was running out of connections...but I'll do jstack and have
>>a look.
>>
>>Interestingly, last night I ran the following code which
>>starts with a one minute sleep and continues to increment
>>the sleep by one minute each time:
>>
>> System.out.println("running
>>test");
>> for(int i = 0; i < 45;
>>i++){
>>
>>System.out.println("run " + i);
>> InitialContext
>>remoteContext = RemoteContextLookup.lookup();
>> String opsJndi =
>>"ejb/UnsecuredOperationsBean";
>>
>>System.out.println("looking up ops");
>>
>>UnsecuredOperationsRemote ops =
>>(UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
>>
>>System.out.println("executing method");
>>
>>System.out.println("Response: " +
>>ops.isReadyForReportGeneration(30000l));
>>
>>System.out.println("finsihed run " + i);
>>
>>remoteContext.close();
>> long sleepTime =
>>i * 60 * 1000;
>>
>>System.out.println("Sleeping for " + i + " minutes");
>>
>>Thread.sleep(sleepTime);
>> }
>> System.out.println("Finished
>>test");
>>
>>Again, exactly at a 20 minute sleep I got an error, but
>>this time it said "Connection forcibly closed by remote
>>host".
>>
>>I'll keep investigating today, however it seems very
>>strongly tied to 20 minutes sleep time (which coincidentally
>>is the repeat delay I'm using for the timer thread).
>>I'm going to drop my timer thread down to a 10 minute
>>schedule and see if that solves the problem. It seems
>>as long as I maintain a continuous (at least once every 19
>>minutes) link to the ejb server it's fine, which is odd,
>>because non of my settings on the ejb server (pool or cache
>>timeouts or anything like that) are set to 20 minutes that I
>>can see anyway.
>>
>>--- On Wed, 24/6/09, glassfish@javadesktop.org
>>
>>wrote:
>>
>>
>>>From: glassfish@javadesktop.org
>>
>>
>>
>>>Subject: Re: Urgent - EJB SSB Timeouts!
>>>To: users@glassfish.dev.java.net
>>>Received: Wednesday, 24 June, 2009, 10:22 PM
>>>
>>>>by the way, I think whatever's
>>>
>>>happening is happening
>>>
>>>>on the ejb server not the client, as I just tried
>>
>>to
>>
>>>>run the command line program below again and it
>>>>didn't even get through the first loop, so JNDI
>>
>>on
>>
>>>>that server seems to have locked up
>>>
>>>Hi,
>>>
>>>seems to reflect my assumption that maybe the ORB has
>>
>>no
>>
>>>more connections/threads. I looked again at the
>>
>>domain.xml
>>
>>>and saw that the default orb listener uses
>>
>>thread-pool-1
>>
>>>which has the maximum thread number of 200. Maybe you
>>
>>can do
>>
>>>a jstack on the PID of the Glassfish and check if
>>
>>there are
>>
>>>a large number of suspicious threads. As a start you
>>
>>could
>>
>>>look for lines containing "thread-pool-1", e.g. "p:
>>>thread-pool-1; w: 2" daemon prio=10 tid=0x0844d400
>>>nid=0x3516 runnable [0x65773000..0x65773fc0]".
>>>
>>>Cheers
>>>Chris.
>>>[Message sent by forum member 'chrjohn' (chrjohn)]
>>>
>>>http://forums.java.net/jive/thread.jspa?messageID=352714
>>>
>>>
>>
>>---------------------------------------------------------------------
>>
>>>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>>For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>
>>>
>>
>>
>> Access Yahoo!7 Mail on your mobile.
>>Anytime. Anywhere.
>>Show me how: http://au.mobile.yahoo.com/mail
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
>
>
> Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

ewernli
Offline
Joined: 2007-10-21

Similar post:
http://forums.java.net/jive/message.jspa?messageID=314619

We had this problem for some time using GF v2ur2. We improved the overal stability with the following actions.
1) apply latest patch of OS
2) use -server mode for the JVM
3) check "Allow non component caller" in the datasource
4) tune the ORB. We are now using -Dcom.sun.corba.ee.transport.ORBTCPTimeouts=500:30000:30:999999

Also, we sometimes injected EJBs using @Resource. It works but I suspect some issue with it. Make sure you use @EJB to inject beans.

Hope it helps...

Adam Jenkins

Thanks Chris, I'll have a look at that now...but my thread-pool-1 has a max of 500 connections, and the loop only runs once before it errors, so I'd be surprised if I was running out of connections...but I'll do jstack and have a look.

Interestingly, last night I ran the following code which starts with a one minute sleep and continues to increment the sleep by one minute each time:

System.out.println("running test");
for(int i = 0; i < 45; i++){
System.out.println("run " + i);
InitialContext remoteContext = RemoteContextLookup.lookup();
String opsJndi = "ejb/UnsecuredOperationsBean";
System.out.println("looking up ops");
UnsecuredOperationsRemote ops = (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
System.out.println("executing method");
System.out.println("Response: " + ops.isReadyForReportGeneration(30000l));
System.out.println("finsihed run " + i);
remoteContext.close();
long sleepTime = i * 60 * 1000;
System.out.println("Sleeping for " + i + " minutes");
Thread.sleep(sleepTime);
}
System.out.println("Finished test");

Again, exactly at a 20 minute sleep I got an error, but this time it said "Connection forcibly closed by remote host".

I'll keep investigating today, however it seems very strongly tied to 20 minutes sleep time (which coincidentally is the repeat delay I'm using for the timer thread). I'm going to drop my timer thread down to a 10 minute schedule and see if that solves the problem. It seems as long as I maintain a continuous (at least once every 19 minutes) link to the ejb server it's fine, which is odd, because non of my settings on the ejb server (pool or cache timeouts or anything like that) are set to 20 minutes that I can see anyway.

--- On Wed, 24/6/09, glassfish@javadesktop.org wrote:

> From: glassfish@javadesktop.org
> Subject: Re: Urgent -
Received: Wednesday, 24 June, 2009, 10:22 PM
> > by the way, I think whatever's
> happening is happening
> > on the ejb server not the client, as I just tried to
> > run the command line program below again and it
> > didn't even get through the first loop, so JNDI on
> > that server seems to have locked up
>
> Hi,
>
> seems to reflect my assumption that maybe the ORB has no
> more connections/threads. I looked again at the domain.xml
> and saw that the default orb listener uses thread-pool-1
> which has the maximum thread number of 200. Maybe you can do
> a jstack on the PID of the Glassfish and check if there are
> a large number of suspicious threads. As a start you could
> look for lines containing "thread-pool-1", e.g. "p:
> thread-pool-1; w: 2" daemon prio=10 tid=0x0844d400
> nid=0x3516 runnable [0x65773000..0x65773fc0]".
>
> Cheers
> Chris.
> [Message sent by forum member 'chrjohn' (chrjohn)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352714
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Adam Jenkins

by the way, I think whatever's happening is happening on the ejb server not the client, as I just tried to run the command line program below again and it didn't even get through the first loop, so JNDI on that server seems to have locked up

--- On Wed, 24/6/09, adamjenkinstmpredirect@yahoo.com.au wrote:

> From: adamjenkinstmpredirect@yahoo.com.au
> Subject: Re: Urgent - EJB SSB Timeouts!
> To: users@glassfish.dev.java.net
> Received: Wednesday, 24 June, 2009, 3:35 PM
>
>
> I have some more information regarding this problem if
> anyone can shed some light on things.  I know this is a
> long post, but it has steps to reproduce the error
> independent of my web code, so I'd really appreciate anyone
> wadding through it and any advice anyone could offer.
>
> First thing I did was turn on monitoring, to ensure that
> this wasn't a resource issue -- heaps of memory, so I
> progressed on to isolating the code.
>
> I took the code from the time task that runs in the web
> tier and pulled it out into a good old fashioned java main
> program and ran it from the command line...looping 10 times
> I ran the following...
>
>         System.out.println("running test");
>         for(int i = 0; i < 10; i++){
>             System.out.println("run " + i);
>             InitialContext remoteContext =
> RemoteContextLookup.lookup();
>             String opsJndi =
> "ejb/UnsecuredOperationsBean";
>             System.out.println("looking up ops");
>             UnsecuredOperationsRemote ops =
> (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
>             System.out.println("executing method");
>             System.out.println("Response: " +
> ops.isReadyForReportGeneration(30000l));
>             System.out.println("finsihed run " + i);
>         }
>         System.out.println("Finished test");
>
>
>
> That ran fine...connected and executed the method 10
> times.
>
>
> So then, to simulate the timer task, I put a 20 minute
> sleep into it
>
>         System.out.println("running test");
>         for(int i = 0; i < 10; i++){
>             System.out.println("run " + i);
>             InitialContext remoteContext =
> RemoteContextLookup.lookup();
>             String opsJndi =
> "ejb/UnsecuredOperationsBean";
>             System.out.println("looking up ops");
>             UnsecuredOperationsRemote ops =
> (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
>             System.out.println("executing method");
>             System.out.println("Response: " +
> ops.isReadyForReportGeneration(30000l));
>             System.out.println("finsihed run " + i);
>             System.out.println("Sleeping for 20
> minutes");
>
>             //make it sleep
> to simulate the timer task
>             Thread.sleep(1200000l);
>         }
>         System.out.println("Finished test");
>
>
> And I got the following output:
>
> running test
> run 0
> looking up ops
> executing method
> Response: false
> finsihed run 0
> Sleeping for 20 minutes
> run 1
> looking up ops
> 24/06/2009 15:19:19
> com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl
> waitForResponse
> WARNING: "IOP00410219: (COMM_FAILURE) Communications
> timeout waiting for response.  Exceeded 1,800,000
> milliseconds"
> org.omg.CORBA.COMM_FAILURE:   vmcid:
> SUN  minor code: 219 completed: Maybe
>         at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
>         at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
>         at
> com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
>         at
> com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
>         at
> com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
>         at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
>         at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
>         at
> com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
>         at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
>         at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
>         at
> com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
>         at
> com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at
> java.lang.reflect.Method.invoke(Method.java:597)
>         at
> com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
>         at
> com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
>         at
> javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
>         at
> com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
>         at
> javax.naming.InitialContext.lookup(InitialContext.java:392)
>         at
> scratchpad.Main.main(Main.java:105)
>
>
> So there seems to be some problem using a remote initial
> context with a delay in between.  I ran the same test
> calling remoteContext.close() after each use and it made no
> difference.  Interestingly, this one didn't even get a
> second run through the loop before it began timing out.
>
> The contents of the RemoteContext.lookup() method (that
> created the InitialContext) is below (note, I've replaced
> the host and port for security reasons):
>
>         Properties props = new
> Properties();
>        
> props.setProperty(Context.INITIAL_CONTEXT_FACTORY,
> "com.sun.enterprise.naming.SerialInitContextFactory");
>        
> props.setProperty(Context.PROVIDER_URL,"iiop://<>:<
>");
>        
> props.setProperty("org.omg.CORBA.ORBInitialHost",
> "<>");
>        
> props.setProperty("org.omg.CORBA.ORBServerHost",
> "<>");
>        
> props.setProperty("org.omg.CORBA.ORBInitialPort",
> "<
>");
>         return new
> InitialContext(props);
>
> Any assistance would be really really appreciated. 
> I'm at the end of my knowledge at the moment :(
>
>
>
> --- On Wed, 24/6/09, Adam Jenkins
> wrote:
>
> > From: Adam Jenkins
> > Subject: Re: Urgent - EJB SSB Timeouts!
> > To: users@glassfish.dev.java.net
> > Received: Wednesday, 24 June, 2009, 9:21 AM
> >
> > >>But you are using an EJB for it anyway,
> right?
> >
> > No, we're doing the generation in the timer task on
> the web
> > tier (I'm using java.util.Timer, spawned by a context
> > listener on startup of the web container)...I didn't
> bother
> > using an enterprise scheduler as that was a bit of
> overkill
> > for a simple 20 minute recursive timer thread).  It
> has
> > to be done on the web tier as it generates html files
> on the
> > local system which get included into a JSP using
> c:import
> > (we have to generate them as they're waaaaaaay to big
> and
> > complex to generate dynamically).
> >
> > The EJB call is just to see if reports need to be
> > generated.  It checks some JMS queues and looks up
> some
> > db tables to see if there is a new bunch of batching
> data
> > that is ready to be processed and returns either a
> true or a
> > false...then the calling web tier timer task either
> does the
> > generation (using more ejb calls to get the data -- it
> never
> > gets this far) or sleeps for another 20 minutes.
> >
> > This morning I turned on monitoring on all the tiers
> just
> > to make sure it wasn't a memory issue -- heaps of
> free
> > memory, so that's not it.
> >
> > I did find the following warning in the ejb tier:
> >
> >
> [#|2009-06-24T07:35:47.578+1000|WARNING|sun-appserver2.1|javax.jms|_ThreadID=22;_ThreadName=iMQReadChannel-36;_RequestID=5f3d6e80-3240-442e-9e85-ac7c26ddf53f;|[I500]:
> > Caught JVM Exception: java.io.EOFException|#]
> >
> > The EJB is referencing a queue
> >
> >    
> > @Resource(mappedName="jms/QueueConnectionFactory")
> >     private QueueConnectionFactory
> > queueConnectionFactory;
> >
> > so I am investigating the theory that the SSB EJBs
> are
> > being created too fast for the message queue (LOCAL
> JMS)
> > when the ejb pool resizes...I notice that just before
> the
> > error occurs (10 minutes or so before) the pool
> creates a
> > bunch more EJBs and it's the next call that fails. 
> So
> > I've turned down all my values (creating less EJBs on
> pool
> > resize) to see if that makes any difference.  I'll
> have
> > an answer on that in about 20 minutes after I
> finishing
> > deploying a new version of the ear and war (with more
> > logging)
> >
> > Other than that I'm completely stumped :(
> >
> > --- On Wed, 24/6/09, Marina Vatkina
> > wrote:
> >
> > > From: Marina Vatkina
> > > Subject: Re: Urgent - EJB SSB Timeouts!
> > > To: users@glassfish.dev.java.net
> > > Received: Wednesday, 24 June, 2009, 9:00 AM
> > > Adam Jenkins wrote:
> > > > Yeah, you're correct....it's only one
> timer,
> > calling
> > > one SSB method every 20 minutes.
> > > >
> > > > I'm using the default number of
> connections,
> > > 1024.  My thread pool has 50 minimum, initial
> ejb
> > pool
> > > size of 5 (max 500, resize 20).  So there should
> be
> > > ample threads/ejbs
> > > >
> > > > No idea why it didn't show up in
> testing...we
> > recently
> > > went from v2ur2 up to v2.1...and to be honest,
> the
> > test
> > > system never really ran for over an hour at a
> time :)
> > > >
> > > > We use EJB Timer service for all our other
> > timing
> > > needs and that works fine, but this process
> > specifically
> > > generates files into the web directory that then
> are
> > > referenced from other jsps using c:import...so
> this is
> > the
> > > only timer task that has to run on the web tier.
> > >
> > > But you are using an EJB for it anyway, right? If
> yes,
> > why
> > > can't the EJB Timer
> > > service do the same job?
> > >
> > > If it can't, which JDK api are you using? EJB
> Timer
> > Service
> > > reschedules the JDK
> > > timer task for each call (vs. scheduling the
> task
> > with
> > > multiple expirations).
> > >
> > > Regards,
> > > -marina
> > > >
> > > > --- On Tue, 23/6/09, glassfish@javadesktop.org
> > >
> > > wrote:
> > > >
> > > >
> > > >>From: glassfish@javadesktop.org
> > >
> > > >>Subject: Re: Urgent - EJB SSB Timeouts!
> > > >>To: users@glassfish.dev.java.net
> > > >>Received: Tuesday, 23 June, 2009, 7:23
> AM
> > > >>Hi,
> > > >>
> > > >>OK, so you say you call the EJB every 20
> > minutes,
> > > that
> > > >>means 3 times in an hour. Do you call
> only one
> > EJB
> > > every 20
> > > >>minutes or several EJBs every 20
> minutes?
> > > >>I am actually just guessing but I think
> that
> > the
> > > problem is
> > > >>not that the name of your EJB is not
> found
> > but
> > > maybe rather
> > > >>that some limit on the naming context
> lookup
> > object
> > > was
> > > >>reached or that maybe there were not any
> > pooled
> > > EJBs free to
> > > >>service your request.
> > > >>Looking at a default domain.xml config of
> a
> > > Glassfish I can
> > > >>see that the max-connections to the ORB
> are
> > 1024.
> > > Sounds
> > > >>high. ;)
> > > >>Maybe you can give more information how
> many
> > EJBs
> > > are
> > > >>called how often. Can you imagine why
> this
> > > behaviour did not
> > > >>show on the test system?
> > > >>
> > > >>Cheers
> > > >>Chris.
> > > >>[Message sent by forum member 'chrjohn'
> > (chrjohn)]
> > > >>
> > > >>http://forums.java.net/jive/thread.jspa?messageID=352409
> > > >>
> > >
> >
> >>---------------------------------------------------------------------
> > > >>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > > >>For additional commands, e-mail: users-help@glassfish.dev.java.net
> > > >>
> > > >>
> > > >
> > > >
> > > >
> > > >       Access Yahoo!7 Mail on
> > > your mobile. Anytime. Anywhere.
> > > > Show me how: http://au.mobile.yahoo.com/mail
> > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > > > For additional commands, e-mail: users-help@glassfish.dev.java.net
> > > >
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > > For additional commands, e-mail: users-help@glassfish.dev.java.net
> > >
> > >
> >
> >
> >       Access Yahoo!7 Mail on your mobile.
> > Anytime. Anywhere.
> > Show me how: http://au.mobile.yahoo.com/mail
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > For additional commands, e-mail: users-help@glassfish.dev.java.net
> >
> >
>
>
>
>       Access Yahoo!7 Mail on your mobile.
> Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

chrjohn
Offline
Joined: 2008-03-11

> by the way, I think whatever's happening is happening
> on the ejb server not the client, as I just tried to
> run the command line program below again and it
> didn't even get through the first loop, so JNDI on
> that server seems to have locked up

Hi,

seems to reflect my assumption that maybe the ORB has no more connections/threads. I looked again at the domain.xml and saw that the default orb listener uses thread-pool-1 which has the maximum thread number of 200. Maybe you can do a jstack on the PID of the Glassfish and check if there are a large number of suspicious threads. As a start you could look for lines containing "thread-pool-1", e.g. "p: thread-pool-1; w: 2" daemon prio=10 tid=0x0844d400 nid=0x3516 runnable [0x65773000..0x65773fc0]".

Cheers
Chris.

adamjenkinstmpredirect@yahoo.com.au

I have some more information regarding this problem if anyone can shed some light on things. I know this is a long post, but it has steps to reproduce the error independent of my web code, so I'd really appreciate anyone wadding through it and any advice anyone could offer.

First thing I did was turn on monitoring, to ensure that this wasn't a resource issue -- heaps of memory, so I progressed on to isolating the code.

I took the code from the time task that runs in the web tier and pulled it out into a good old fashioned java main program and ran it from the command line...looping 10 times I ran the following...

        System.out.println("running test");
        for(int i = 0; i < 10; i++){
            System.out.println("run " + i);
            InitialContext remoteContext = RemoteContextLookup.lookup();
            String opsJndi = "ejb/UnsecuredOperationsBean";
            System.out.println("looking up ops");
            UnsecuredOperationsRemote ops = (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
            System.out.println("executing method");
            System.out.println("Response: " + ops.isReadyForReportGeneration(30000l));
            System.out.println("finsihed run " + i);
        }
        System.out.println("Finished test");

That ran fine...connected and executed the method 10 times.

So then, to simulate the timer task, I put a 20 minute sleep into it

        System.out.println("running test");
        for(int i = 0; i < 10; i++){
            System.out.println("run " + i);
            InitialContext remoteContext = RemoteContextLookup.lookup();
            String opsJndi = "ejb/UnsecuredOperationsBean";
            System.out.println("looking up ops");
            UnsecuredOperationsRemote ops = (UnsecuredOperationsRemote)remoteContext.lookup(opsJndi);
            System.out.println("executing method");
            System.out.println("Response: " + ops.isReadyForReportGeneration(30000l));
            System.out.println("finsihed run " + i);
            System.out.print
("Sleeping for 20 minutes");

//make it sleep to simulate the timer task
            Thread.sleep(1200000l);
        }
        System.out.println("Finished test");

And I got the following output:

running test
run 0
looking up ops
executing method
Response: false
finsihed run 0
Sleeping for 20 minutes
run 1
looking up ops
24/06/2009 15:19:19 com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl waitForResponse
WARNING: "IOP00410219: (COMM_FAILURE) Communications timeout waiting for response. Exceeded 1,800,000 milliseconds"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
at com.sun.ejb.codegen._GenericEJBHom
(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at scratchpad.Main.main(Main.java:105)

So there seems to be some problem using a remote initial context with a delay in between. I ran the same test calling remoteContext.close() after each use and it made no difference. Interestingly, this one didn't even get a second run through the loop before it began timing out.

The contents of the RemoteContext.lookup() method (that created the InitialContext) is below (note, I've replaced the host and port for security reasons):

Properties props = new Properties();
props.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");
props.setProperty(Context.PROVIDER_URL,"iiop://<>:<
>");
props.setProperty("org.omg.CORBA.ORBInitialHost", "<>");
props.setProperty("org.omg.CORBA.ORBServerHost", "<>");
props.setProperty("org.omg.CORBA.ORBInitialPort", "<
>");
return new InitialContext(props);

Any assistance would be really really appreciated. I'm at the end of my knowledge at the moment :(

--- On Wed, 24/6/09, Adam Jenkins wrote:

> From: Adam Jenkins
> Subject: R
v.java.net
> Received: Wednesday, 24 June, 2009, 9:21 AM
>
> >>But you are using an EJB for it anyway, right?
>
> No, we're doing the generation in the timer task on the web
> tier (I'm using java.util.Timer, spawned by a context
> listener on startup of the web container)...I didn't bother
> using an enterprise scheduler as that was a bit of overkill
> for a simple 20 minute recursive timer thread).  It has
> to be done on the web tier as it generates html files on the
> local system which get included into a JSP using c:import
> (we have to generate them as they're waaaaaaay to big and
> complex to generate dynamically).
>
> The EJB call is just to see if reports need to be
> generated.  It checks some JMS queues and looks up some
> db tables to see if there is a new bunch of batching data
> that is ready to be processed and returns either a true or a
> false...then the calling web tier timer task either does the
> generation (using more ejb calls to get the data -- it never
> gets this far) or sleeps for another 20 minutes.
>
> This morning I turned on monitoring on all the tiers just
> to make sure it wasn't a memory issue -- heaps of free
> memory, so that's not it.
>
> I did find the following warning in the ejb tier:
>
> [#|2009-06-24T07:35:47.578+1000|WARNING|sun-appserver2.1|javax.jms|_ThreadID=22;_ThreadName=iMQReadChannel-36;_RequestID=5f3d6e80-3240-442e-9e85-ac7c26ddf53f;|[I500]:
> Caught JVM Exception: java.io.EOFException|#]
>
> The EJB is referencing a queue
>
>    
> @Resource(mappedName="jms/QueueConnectionFactory")
>     private QueueConnectionFactory
> queueConnectionFactory;
>
> so I am investigating the theory that the SSB EJBs are
> being created too fast for the message queue (LOCAL JMS)
> when the ejb pool resizes...I notice that just before the
> error occurs (10 minutes or so before) the pool creates a
> bunch more EJBs and it's the next call that fails.  So
> I've turned down all my values (creating less EJBs on pool
> resize) to see if that makes any difference.  I'll have
> a
r on that in about 20 minutes after I finishing
> deploying a new version of the ear and war (with more
> logging)
>
> Other than that I'm completely stumped :(
>
> --- On Wed, 24/6/09, Marina Vatkina
> wrote:
>
> > From: Marina Vatkina
> > Subject: Re: Urgent - EJB SSB Timeouts!
> > To: users@glassfish.dev.java.net
> > Received: Wednesday, 24 June, 2009, 9:00 AM
> > Adam Jenkins wrote:
> > > Yeah, you're correct....it's only one timer,
> calling
> > one SSB method every 20 minutes.
> > >
> > > I'm using the default number of connections,
> > 1024.  My thread pool has 50 minimum, initial ejb
> pool
> > size of 5 (max 500, resize 20).  So there should be
> > ample threads/ejbs
> > >
> > > No idea why it didn't show up in testing...we
> recently
> > went from v2ur2 up to v2.1...and to be honest, the
> test
> > system never really ran for over an hour at a time :)
> > >
> > > We use EJB Timer service for all our other
> timing
> > needs and that works fine, but this process
> specifically
> > generates files into the web directory that then are
> > referenced from other jsps using c:import...so this is
> the
> > only timer task that has to run on the web tier.
> >
> > But you are using an EJB for it anyway, right? If yes,
> why
> > can't the EJB Timer
> > service do the same job?
> >
> > If it can't, which JDK api are you using? EJB Timer
> Service
> > reschedules the JDK
> > timer task for each call (vs. scheduling the task
> with
> > multiple expirations).
> >
> > Regards,
> > -marina
> > >
> > > --- On Tue, 23/6/09, glassfish@javadesktop.org
> >
> > wrote:
> > >
> > >
> > >>From: glassfish@javadesktop.org
> >
> > >>Subject: Re: Urgent - EJB SSB Timeouts!
> > >>To: users@glassfish.dev.java.net
> > >>Received: Tuesday, 23 June, 2009, 7:23 AM
> > >>Hi,
> > >>
> > >>OK, so you say you call the EJB every 20
> minutes,
> > that
> > >>means 3 times in an hour. Do you call only one
> EJB
> > every 20
>
>>I am actually just guessing but I think that
> the
> > problem is
> > >>not that the name of your EJB is not found
> but
> > maybe rather
> > >>that some limit on the naming context lookup
> object
> > was
> > >>reached or that maybe there were not any
> pooled
> > EJBs free to
> > >>service your request.
> > >>Looking at a default domain.xml config of a
> > Glassfish I can
> > >>see that the max-connections to the ORB are
> 1024.
> > Sounds
> > >>high. ;)
> > >>Maybe you can give more information how many
> EJBs
> > are
> > >>called how often. Can you imagine why this
> > behaviour did not
> > >>show on the test system?
> > >>
> > >>Cheers
> > >>Chris.
> > >>[Message sent by forum member 'chrjohn'
> (chrjohn)]
> > >>
> > >>http://forums.java.net/jive/thread.jspa?messageID=352409
> > >>
> >
> >>---------------------------------------------------------------------
> > >>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > >>For additional commands, e-mail: users-help@glassfish.dev.java.net
> > >>
> > >>
> > >
> > >
> > >
> > >       Access Yahoo!7 Mail on
> > your mobile. Anytime. Anywhere.
> > > Show me how: http://au.mobile.yahoo.com/mail
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > > For additional commands, e-mail: users-help@glassfish.dev.java.net
> > >
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > For additional commands, e-mail: users-help@glassfish.dev.java.net
> >
> >
>
>
>       Access Yahoo!7 Mail on your mobile.
> Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anyw
ile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Adam Jenkins

>>But you are using an EJB for it anyway, right?

No, we're doing the generation in the timer task on the web tier (I'm using java.util.Timer, spawned by a context listener on startup of the web container)...I didn't bother using an enterprise scheduler as that was a bit of overkill for a simple 20 minute recursive timer thread). It has to be done on the web tier as it generates html files on the local system which get included into a JSP using c:import (we have to generate them as they're waaaaaaay to big and complex to generate dynamically).

The EJB call is just to see if reports need to be generated. It checks some JMS queues and looks up some db tables to see if there is a new bunch of batching data that is ready to be processed and returns either a true or a false...then the calling web tier timer task either does the generation (using more ejb calls to get the data -- it never gets this far) or sleeps for another 20 minutes.

This morning I turned on monitoring on all the tiers just to make sure it wasn't a memory issue -- heaps of free memory, so that's not it.

I did find the following warning in the ejb tier:

[#|2009-06-24T07:35:47.578+1000|WARNING|sun-appserver2.1|javax.jms|_ThreadID=22;_ThreadName=iMQReadChannel-36;_RequestID=5f3d6e80-3240-442e-9e85-ac7c26ddf53f;|[I500]: Caught JVM Exception: java.io.EOFException|#]

The EJB is referencing a queue

@Resource(mappedName="jms/QueueConnectionFactory")
private QueueConnectionFactory queueConnectionFactory;

so I am investigating the theory that the SSB EJBs are being created too fast for the message queue (LOCAL JMS) when the ejb pool resizes...I notice that just before the error occurs (10 minutes or so before) the pool creates a bunch more EJBs and it's the next call that fails. So I've turned down all my values (creating less EJBs on pool resize) to see if that makes any difference. I'll have an answer on that in about 20 minutes after I finishing deploying a new version of the ear and war (with more logging)

Other than that I'm completel
ped :(

--- On Wed, 24/6/09, Marina Vatkina wrote:

> From: Marina Vatkina
> Subject: Re: Urgent - EJB SSB Timeouts!
> To: users@glassfish.dev.java.net
> Received: Wednesday, 24 June, 2009, 9:00 AM
> Adam Jenkins wrote:
> > Yeah, you're correct....it's only one timer, calling
> one SSB method every 20 minutes.
> >
> > I'm using the default number of connections,
> 1024.  My thread pool has 50 minimum, initial ejb pool
> size of 5 (max 500, resize 20).  So there should be
> ample threads/ejbs
> >
> > No idea why it didn't show up in testing...we recently
> went from v2ur2 up to v2.1...and to be honest, the test
> system never really ran for over an hour at a time :)
> >
> > We use EJB Timer service for all our other timing
> needs and that works fine, but this process specifically
> generates files into the web directory that then are
> referenced from other jsps using c:import...so this is the
> only timer task that has to run on the web tier.
>
> But you are using an EJB for it anyway, right? If yes, why
> can't the EJB Timer
> service do the same job?
>
> If it can't, which JDK api are you using? EJB Timer Service
> reschedules the JDK
> timer task for each call (vs. scheduling the task with
> multiple expirations).
>
> Regards,
> -marina
> >
> > --- On Tue, 23/6/09, glassfish@javadesktop.org
>
> wrote:
> >
> >
> >>From: glassfish@javadesktop.org
>
> >>Subject: Re: Urgent - EJB SSB Timeouts!
> >>To: users@glassfish.dev.java.net
> >>Received: Tuesday, 23 June, 2009, 7:23 AM
> >>Hi,
> >>
> >>OK, so you say you call the EJB every 20 minutes,
> that
> >>means 3 times in an hour. Do you call only one EJB
> every 20
> >>minutes or several EJBs every 20 minutes?
> >>I am actually just guessing but I think that the
> problem is
> >>not that the name of your EJB is not found but
> maybe rather
> >>that some limit on the naming context lookup object
> was
> >>reached or that maybe there were not any pooled
domain.xml config of a
> Glassfish I can
> >>see that the max-connections to the ORB are 1024.
> Sounds
> >>high. ;)
> >>Maybe you can give more information how many EJBs
> are
> >>called how often. Can you imagine why this
> behaviour did not
> >>show on the test system?
> >>
> >>Cheers
> >>Chris.
> >>[Message sent by forum member 'chrjohn' (chrjohn)]
> >>
> >>http://forums.java.net/jive/thread.jspa?messageID=352409
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> >>For additional commands, e-mail: users-help@glassfish.dev.java.net
> >>
> >>
> >
> >
> >
> >       Access Yahoo!7 Mail on
> your mobile. Anytime. Anywhere.
> > Show me how: http://au.mobile.yahoo.com/mail
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> > For additional commands, e-mail: users-help@glassfish.dev.java.net
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Adam Jenkins

Yeah, you're correct....it's only one timer, calling one SSB method every 20 minutes.

I'm using the default number of connections, 1024. My thread pool has 50 minimum, initial ejb pool size of 5 (max 500, resize 20). So there should be ample threads/ejbs

No idea why it didn't show up in testing...we recently went from v2ur2 up to v2.1...and to be honest, the test system never really ran for over an hour at a time :)

We use EJB Timer service for all our other timing needs and that works fine, but this process specifically generates files into the web directory that then are referenced from other jsps using c:import...so this is the only timer task that has to run on the web tier.

--- On Tue, 23/6/09, glassfish@javadesktop.org wrote:

> From: glassfish@javadesktop.org
> Subject: Re: Urgent - EJB SSB Timeouts!
> To: users@glassfish.dev.java.net
> Received: Tuesday, 23 June, 2009, 7:23 AM
> Hi,
>
> OK, so you say you call the EJB every 20 minutes, that
> means 3 times in an hour. Do you call only one EJB every 20
> minutes or several EJBs every 20 minutes?
> I am actually just guessing but I think that the problem is
> not that the name of your EJB is not found but maybe rather
> that some limit on the naming context lookup object was
> reached or that maybe there were not any pooled EJBs free to
> service your request.
> Looking at a default domain.xml config of a Glassfish I can
> see that the max-connections to the ORB are 1024. Sounds
> high. ;)
> Maybe you can give more information how many EJBs are
> called how often. Can you imagine why this behaviour did not
> show on the test system?
>
> Cheers
> Chris.
> [Message sent by forum member 'chrjohn' (chrjohn)]
>
> http://forums.java.net/jive/thread.jspa?messageID=352409
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Y
Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Marina Vatkina

Adam Jenkins wrote:
> Yeah, you're correct....it's only one timer, calling one SSB method every 20 minutes.
>
> I'm using the default number of connections, 1024. My thread pool has 50 minimum, initial ejb pool size of 5 (max 500, resize 20). So there should be ample threads/ejbs
>
> No idea why it didn't show up in testing...we recently went from v2ur2 up to v2.1...and to be honest, the test system never really ran for over an hour at a time :)
>
> We use EJB Timer service for all our other timing needs and that works fine, but this process specifically generates files into the web directory that then are referenced from other jsps using c:import...so this is the only timer task that has to run on the web tier.

But you are using an EJB for it anyway, right? If yes, why can't the EJB Timer
service do the same job?

If it can't, which JDK api are you using? EJB Timer Service reschedules the JDK
timer task for each call (vs. scheduling the task with multiple expirations).

Regards,
-marina
>
> --- On Tue, 23/6/09, glassfish@javadesktop.org wrote:
>
>
>>From: glassfish@javadesktop.org
>>Subject: Re: Urgent - EJB SSB Timeouts!
>>To: users@glassfish.dev.java.net
>>Received: Tuesday, 23 June, 2009, 7:23 AM
>>Hi,
>>
>>OK, so you say you call the EJB every 20 minutes, that
>>means 3 times in an hour. Do you call only one EJB every 20
>>minutes or several EJBs every 20 minutes?
>>I am actually just guessing but I think that the problem is
>>not that the name of your EJB is not found but maybe rather
>>that some limit on the naming context lookup object was
>>reached or that maybe there were not any pooled EJBs free to
>>service your request.
>>Looking at a default domain.xml config of a Glassfish I can
>>see that the max-connections to the ORB are 1024. Sounds
>>high. ;)
>>Maybe you can give more information how many EJBs are
>>called how often. Can you imagine why this behaviour did not
>>show on the test system?
>>
>>Cheers
>>Chris.
>>[Message sent by forum member 'chrjohn' (chrjohn)]
>>
>>http://forums.java.net/jive/thread.jspa?messageID=352409
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
>
>
> Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Marina Vatkina

Adam,

Did you try to use EJB Timers and let the EJB container do the job of calling
your beans every 20 minutes?

thanks,
-marina

Adam Jenkins wrote:
> Hi All,
>
> I have an urgent request for assistance. After a year and a half of development we're finally moving into product, but we have a crazy production bug.
>
> We have a couple of seperate instances under management, one for the web tier and one for the ejb tier. On the web tier we have a couple of timers (java.util.Timers) that are started by context listeners, that every 20 minutes call an EJB to see if reports need to be generated and made available to the web users. Because of some bugs in glassfish with injecting ejbs in this scenario, we look them up using a remote initial context each time the timer runs...which works fine for about an hour and a half, then everything goes crazy. Every call to an ejb gives a time out, and our entire application stops working. There is nothing of note in the ejb server logs that I can see (definitely no errors anyway), but in the web tier logs we get the following:
>
>
>
> [#|2009-06-22T19:28:09.398+1000|FINE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;ClassName=com.megajobindex.web.scheduling.ReportGenerationMonitor;MethodName=run;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|attempting to load unsecured operation bean from jndi ejb/UnsecuredOperationsBean|#]
>
> [#|2009-06-22T20:17:39.393+1000|WARNING|sun-appserver2.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=24;_ThreadName=Timer-3;1800000;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|"IOP00410219: (COMM_FAILURE) Communications timeout waiting for response. Exceeded 1,800,000 milliseconds"
> org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
> at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
> at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
> at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
> at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
> at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
> at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
> at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
> at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
> at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
> at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
> at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
> at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
> at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> |#]
>
> [#|2009-06-22T20:17:39.400+1000|SEVERE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|Could not determine if reports need to be generated
> javax.naming.NamingException: ejb ref resolution error for remote business interfacecom.megajobindex.ejb.UnsecuredOperationsRemote [Root exception is java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
> org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe]
> at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:425)
> at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
> at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
> at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079707 Maybe; nested exception is:
> org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
> at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
> at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
> at com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
> ... 7 more
> Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 219 completed: Maybe
> at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
> at com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
> at com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
> at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
> at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
> at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
> at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
> at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
> at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
> ... 15 more
> |#]
>
>
> Can anyone shed some light onto why this may be ocurring...I'm hoping it's just a configuration issue.
>
> Cheers
> Adam
>
>
> Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Adam Jenkins

oh, and I'm running the following:

jdk1.6.0_14 - linux 32bit
glassfish v2.1 latest release (from the site 4 days ago)
ubuntu 2.6.27-11-server

--- On Tue, 23/6/09, Adam Jenkins wrote:

> From: Adam Jenkins
> Subject: Urgent - EJB SSB Timeouts!
> To: users@glassfish.dev.java.net
> Received: Tuesday, 23 June, 2009, 5:15 AM
>
> Hi All,
>
> I have an urgent request for assistance.  After a year
> and a half of development we're finally moving into product,
> but we have a crazy production bug.
>
> We have a couple of seperate instances under management,
> one for the web tier and one for the ejb tier.  On the
> web tier we have a couple of timers (java.util.Timers) that
> are started by context listeners, that every 20 minutes call
> an EJB to see if reports need to be generated and made
> available to the web users.  Because of some bugs in
> glassfish with injecting ejbs in this scenario, we look them
> up using a remote initial context each time the timer
> runs...which works fine for about an hour and a half, then
> everything goes crazy.  Every call to an ejb gives a
> time out, and our entire application stops working. 
> There is nothing of note in the ejb server logs that I can
> see (definitely no errors anyway), but in the web tier logs
> we get the following:
>
>
>
> [#|2009-06-22T19:28:09.398+1000|FINE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;ClassName=com.megajobindex.web.scheduling.ReportGenerationMonitor;MethodName=run;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|attempting
> to load unsecured operation bean from jndi
> ejb/UnsecuredOperationsBean|#]
>
> [#|2009-06-22T20:17:39.393+1000|WARNING|sun-appserver2.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.transport|_ThreadID=24;_ThreadName=Timer-3;1800000;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|"IOP00410219:
> (COMM_FAILURE) Communications timeout waiting for
> response.  Exceeded 1,800,000
s"
> org.omg.CORBA.COMM_FAILURE:   vmcid:
> SUN  minor code: 219 completed: Maybe
>     at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
>     at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
>     at
> com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
>     at
> com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
>     at
> com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
>     at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:192)
>     at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
>     at
> com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
>     at
> com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at
> java.lang.reflect.Method.invoke(Method.java:597)
>     at
> com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
>     at
> com.sun.ejb.containers.RemoteBusinessObjec
tory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
>     at
> javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
>     at
> com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
>     at
> javax.naming.InitialContext.lookup(InitialContext.java:392)
>     at
> com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
>     at
> java.util.TimerThread.mainLoop(Timer.java:512)
>     at
> java.util.TimerThread.run(Timer.java:462)
> |#]
>
> [#|2009-06-22T20:17:39.400+1000|SEVERE|sun-appserver2.1|com.megajobindex.web.scheduling.ReportGenerationMonitor|_ThreadID=24;_ThreadName=Timer-3;_RequestID=baabbe3b-6d79-47d2-95ad-5c57136ea92c;|Could
> not determine if reports need to be generated
> javax.naming.NamingException: ejb ref resolution error for
> remote business
> interfacecom.megajobindex.ejb.UnsecuredOperationsRemote
> [Root exception is java.rmi.MarshalException: CORBA
> COMM_FAILURE 1398079707 Maybe; nested exception is:
>    
> org.omg.CORBA.COMM_FAILURE:   vmcid:
> SUN  minor code: 219 completed: Maybe]
>     at
> com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:425)
>     at
> com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:74)
>     at
> javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
>     at
> com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
>     at
> javax.naming.InitialContext.lookup(InitialContext.java:392)
>     at
> com.megajobindex.web.scheduling.ReportGenerationMonitor.run(ReportGenerationMonitor.java:132)
>     at
> java.util.TimerThread.mainLoop(Timer.java:512)
>     at
> java.util.TimerThread.run(Timer.java:462)
> Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE
> 1398079707 Maybe; nested exception is:
>    
> org.omg.CORBA.COMM_FAILURE:   vmcid:
> SUN  minor code: 219 completed: Maybe
>     at
> com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:271)
>     at
> com.sun.c
lerImpl.privateInvoke(StubInvocationHandlerImpl.java:205)
>     at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
>     at
> com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:225)
>     at
> com.sun.ejb.codegen._GenericEJBHome_Generated_DynamicStub.create(com/sun/ejb/codegen/_GenericEJBHome_Generated_DynamicStub.java)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at
> java.lang.reflect.Method.invoke(Method.java:597)
>     at
> com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:372)
>     ... 7 more
> Caused by:
> org.omg.CORBA.COMM_FAILURE:   vmcid:
> SUN  minor code: 219 completed: Maybe
>     at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3180)
>     at
> com.sun.corba.ee.impl.logging.ORBUtilSystemException.communicationsTimeoutWaitingForResponse(ORBUtilSystemException.java:3195)
>     at
> com.sun.corba.ee.impl.transport.CorbaResponseWaitingRoomImpl.waitForResponse(CorbaResponseWaitingRoomImpl.java:198)
>     at
> com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.waitForResponse(SocketOrChannelConnectionImpl.java:1196)
>     at
> com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.waitForResponse(CorbaMessageMediatorImpl.java:291)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:389)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:357)
>     at
> com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:219)
>     at
> com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(Stub
ocationHandlerImpl.java:192)
>     ... 15 more
> |#]
>
>
> Can anyone shed some light onto why this may be
> ocurring...I'm hoping it's just a configuration issue.
>
> Cheers
> Adam
>
>
>       Access Yahoo!7 Mail on your mobile.
> Anytime. Anywhere.
> Show me how: http://au.mobile.yahoo.com/mail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

Access Yahoo!7 Mail on your mobile. Anytime. Anywhere.
Show me how: http://au.mobile.yahoo.com/mail

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

chrjohn
Offline
Joined: 2008-03-11

Hi,

OK, so you say you call the EJB every 20 minutes, that means 3 times in an hour. Do you call only one EJB every 20 minutes or several EJBs every 20 minutes?
I am actually just guessing but I think that the problem is not that the name of your EJB is not found but maybe rather that some limit on the naming context lookup object was reached or that maybe there were not any pooled EJBs free to service your request.
Looking at a default domain.xml config of a Glassfish I can see that the max-connections to the ORB are 1024. Sounds high. ;)
Maybe you can give more information how many EJBs are called how often. Can you imagine why this behaviour did not show on the test system?

Cheers
Chris.