Skip to main content

NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools

15 replies [Last post]
Anonymous

Folks,
Looking for some insight on this one. I'm using the
sun-jms-adapter as our connector to OpenMQ.
We are SunAS 9.1 ur2 in a clustered environment. At some point,
we started getting NPEs when
accessing our JMS objects (ConnectionFactories in particular).
This gets reported as as the following:
[#|2009-01-14T08:11:40.942-0600|WARNING|sun-appserver9.1|javax.en
terprise.system.stream.err|_ThreadID=2066;_ThreadName=httpSSLWork
erThread-50011-1;_RequestID=49b9cf44-0580-494d-93b2-f175a7edd24a;
|.stc.jmsjca.core.JConnection.createSessionByApplication(JConnect
ion.java:153)
at
com.stc.jmsjca.core.JConnection.createSession(JConnection.java:31
4)
at
com.stc.jmsjca.core.WConnection.createSession(WConnection.java:94
)
at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
ccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.jms.connection.TransactionAwareConnectionFact
oryProxy$TransactionAwareConnectionInvocationHandler.invoke(Trans
actionAwareConnectionFactoryProxy.java:258)
at $Proxy554.createSession(Unknown Source)
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAcce
ssor.java:196)
at
org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java
:450)
... 165 more
Caused by: java.lang.NullPointerException
After some investigation into the actual cause of the NPE (yes,
printing that stack trace would have been helpful),
I find the following:
Caused by: java.lang.NullPointerException
at
com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(P
oolManagerImpl.java:248)
at
com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManag
erImpl.java:176)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetCo
nnection(ConnectionManagerImpl.java:337)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConne
ction(ConnectionManagerImpl.java:189)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConne
ction(ConnectionManagerImpl.java:165)
at
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConne
ction(ConnectionManagerImpl.java:158)
at
com.stc.jmsjca.core.JConnection.createSessionByApplication(JConne
ction.java:131)
... 36 more
So its not the adapter - its the server. This code looks like
the following:
public ResourceHandle getResourceFromPool(ResourceSpec spec,
ResourceAllocator alloc, ClientSecurityInfo info, Transaction
tran) throws PoolingException {
ResourcePool pool = getPool( spec.getConnectionPoolName()
);
// pool.getResource() has been modified to:
// - be able to create new resource if needed
// - block the caller until a resource is acquired
or
// the max-wait-time expires
return pool.getResource(spec, alloc, tran); <<<<< this
is line 248
}
So clearly, getPool returns null and this code never checks for
that.
I've tried rebooting the cluster (which for us is stop/start
instance1 and then stop/start instance2) and I've tried rebooting
the DAS. I also believe that a full stop on the cluster (both
instances) has been done as well. I now get the NPEs even
after all the reboots.
I've turned on logging (FINEST) on the RAR and for Connectors and
I have also turned on Monitoring for Connectors
and have the logs for those startups. Monitoring doesn't seem to
work for this pool so that was pointless. And the logs
show all kinds of pools getting added but then removed with a
line about them being empty. We do have the pools
set to a steady state of 0 since the RAR actually pools the
connections/sessions. Its like they get created to deal with
JNDI
and then removed for some reason after that.
Things seem to be bound to the JNDI but the internal pool is not
there. I'm not sure how to get around this.
I've thought about adding a JSP to a web app that gets the pool
table and printing it out just to see.
I've also noticed a CVS commit from 3 months ago to
PoolManagerImpl (appsrv-core) related to cleaning up resources
after transaction commit. I'm nervous that key functionality is
not in our version but I don't understand the tags to know
which release is which.
I put together a sample program and when I went to deploy it, I
kept getting JNDI errors. I didn't understand since
everything was correct (targets/enabled state). It turned out
that my resource-ref was listed as false in the domain.xml
but true on the screen. This was easily reproducable by
asadmin-creating them into the server and then retargetting
them to the cluster and resetting enabled to true. If I add them
via the gui directly (no asadmin), then they go in correctly.
The common thread is that when enabled=false, I also see the
pools getting added and then removed. But the JNDI
is not there in one case and seems to still be there in the other
case.
Looking for clues as to what to do - let me know whatever you
might need. I can get it for you. Thanks in advance.
At this point I can't get my server to reboot and applications to
deploy properly.
Brian
[att1.html]

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Jagadish Prasath Ramu

Copying admin@gf as this seem to be related to synchronization between
DAS and NA.

Brian :
* I do not see NPEs w.r.t PoolManager in the server.logs
* The connection-pool configurations in domain.xml of DAS and Cluster's
instance are in-sync.

Thanks,
-Jagadish

On Thu, 2009-01-15 at 15:31 -0600, Brian Repko wrote:
> ZIP of the domain.xmls (DAS, one NA and one SI) and the logs from SI1 and SI2 are attached.
>
> What we noticed is that our NA domain.xml is drastically out of sync with the SI and DAS
>
>
> We are concerned that the instances are syncing or using the NA as a proxy for the DAS
> (das is rebooting) and that causes them to go delete the pools.
>
> At this point restarting things isn't working so if there is a way to shutdown and startup
> that you think we should try, I'd LOVE to hear it.
>
> Thanks,
> Brian
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

Jagadish,

Sorry about that - I only included the log up to the point of application deployment
(which shows the pools getting created and then destroyed).

Attached is a collection of log snippets that show the failures. These are previous
ones. Unfortunately, the sun-jms-adapter eats the original NPE but we get it somewhere
else.

Brian

----- Original message -----
From: "Jagadish Prasath Ramu"
To: "Brian Repko"
Cc: admin@glassfish.dev.java.net, "users"
Date: Fri, 16 Jan 2009 19:53:56 +0530
Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools

Copying admin@gf as this seem to be related to synchronization between
DAS and NA.

Brian :
* I do not see NPEs w.r.t PoolManager in the server.logs
* The connection-pool configurations in domain.xml of DAS and Cluster's
instance are in-sync.

Thanks,
-Jagadish

On Thu, 2009-01-15 at 15:31 -0600, Brian Repko wrote:
> ZIP of the domain.xmls (DAS, one NA and one SI) and the logs from SI1 and SI2 are attached.
>
> What we noticed is that our NA domain.xml is drastically out of sync with the SI and DAS
>
>
> We are concerned that the instances are syncing or using the NA as a proxy for the DAS
> (das is rebooting) and that causes them to go delete the pools.
>
> At this point restarting things isn't working so if there is a way to shutdown and startup
> that you think we should try, I'd LOVE to hear it.
>
> Thanks,
> Brian
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

[jmsFailures.log]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jagadish Prasath Ramu

Hi Brian,
Can you post your domain.xml snapshot & server.log from DAS & cluster
instances when you face the issue [FINEST for "connectors" &
"resource-adapter" module log levels] ?
If you have exact steps to reproduce the issue (or a test-case), that
would be helpful.

* Somehow the pool is removed (via delete-connector-resource,
delete-connector-connection-pool or disable connector resource). So, the
NPE is an after effect.
* Does your application cache the connection factory ? (eg: as an
instance variable and is used during a web request after the
resource/pool is removed)

Thanks,
-Jagadish

On Wed, 2009-01-14 at 15:30 -0600, Brian Repko wrote:
>
> Folks,
>
> Looking for some insight on this one. I'm using the sun-jms-adapter
> as our connector to OpenMQ.
> We are SunAS 9.1 ur2 in a clustered environment. At some point, we
> started getting NPEs when
> accessing our JMS objects (ConnectionFactories in particular). This
> gets reported as as the following:
>
> [#|2009-01-14T08:11:40.942-0600|WARNING|sun-appserver9.1|
> javax.enterprise.system.stream.err|
> _ThreadID=2066;_ThreadName=httpSSLWorkerThread-50011-1;_RequestID=49b9cf44-0580-494d-93b2-f175a7edd24a;|.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:153)
> at
> com.stc.jmsjca.core.JConnection.createSession(JConnection.java:314)
> at
> com.stc.jmsjca.core.WConnection.createSession(WConnection.java:94)
> at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:258)
> at $Proxy554.createSession(Unknown Source)
> at
> org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> at
> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> ... 165 more
> Caused by: java.lang.NullPointerException
>
> After some investigation into the actual cause of the NPE (yes,
> printing that stack trace would have been helpful),
> I find the following:
>
> Caused by: java.lang.NullPointerException
> at
> com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> at
> com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> at
> com.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:131)
> ... 36 more
>
> So its not the adapter - its the server. This code looks like the
> following:
>
> public ResourceHandle getResourceFromPool(ResourceSpec spec,
> ResourceAllocator alloc, ClientSecurityInfo info, Transaction tran)
> throws PoolingException {
> ResourcePool pool = getPool( spec.getConnectionPoolName() );
> // pool.getResource() has been modified to:
> // - be able to create new resource if needed
> // - block the caller until a resource is acquired or
> // the max-wait-time expires
> return pool.getResource(spec, alloc, tran); <<<<< this is
> line 248
> }
>
> So clearly, getPool returns null and this code never checks for that.
>
> I've tried rebooting the cluster (which for us is stop/start instance1
> and then stop/start instance2) and I've tried rebooting
> the DAS. I also believe that a full stop on the cluster (both
> instances) has been done as well. I now get the NPEs even
> after all the reboots.
>
> I've turned on logging (FINEST) on the RAR and for Connectors and I
> have also turned on Monitoring for Connectors
> and have the logs for those startups. Monitoring doesn't seem to work
> for this pool so that was pointless. And the logs
> show all kinds of pools getting added but then removed with a line
> about them being empty. We do have the pools
> set to a steady state of 0 since the RAR actually pools the
> connections/sessions. Its like they get created to deal with JNDI
> and then removed for some reason after that.
>
> Things seem to be bound to the JNDI but the internal pool is not
> there. I'm not sure how to get around this.
> I've thought about adding a JSP to a web app that gets the pool table
> and printing it out just to see.
>
> I've also noticed a CVS commit from 3 months ago to PoolManagerImpl
> (appsrv-core) related to cleaning up resources
> after transaction commit. I'm nervous that key functionality is not
> in our version but I don't understand the tags to know
> which release is which.
>
> I put together a sample program and when I went to deploy it, I kept
> getting JNDI errors. I didn't understand since
> everything was correct (targets/enabled state). It turned out that my
> resource-ref was listed as false in the domain.xml
> but true on the screen. This was easily reproducable by
> asadmin-creating them into the server and then retargetting
> them to the cluster and resetting enabled to true. If I add them via
> the gui directly (no asadmin), then they go in correctly.
>
> The common thread is that when enabled=false, I also see the pools
> getting added and then removed. But the JNDI
> is not there in one case and seems to still be there in the other
> case.
>
> Looking for clues as to what to do - let me know whatever you might
> need. I can get it for you. Thanks in advance.
> At this point I can't get my server to reboot and applications to
> deploy properly.
>
> Brian
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

Jagadish,

I'll get the domain.xml(s) and logs together. I've noticed issues when the DAS domain.xml and the instance/nodeagent domain.xmls get out of sync. I've also noticed issues where the admin says something different from the xml (enabled is true vs false). I don't have a test case - I have a sample app but it runs on my local cluster. I've tried to setup local as close to the failing cluster as possible but it still works for me. That's why I think its part of the domain.xml or our domains are setup in such a way that there is cross-talk (bad port assignments).

We are using Spring's JmsTemplate as you can see from the stack trace so we are holding on to the ConnectionFactory and the Destination.

I'll send files when I get them.

Brian

----- Original message -----
From: "Jagadish Prasath Ramu"
To: users@glassfish.dev.java.net
Date: Thu, 15 Jan 2009 17:19:54 +0530
Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools

Hi Brian,
Can you post your domain.xml snapshot & server.log from DAS & cluster
instances when you face the issue [FINEST for "connectors" &
"resource-adapter" module log levels] ?
If you have exact steps to reproduce the issue (or a test-case), that
would be helpful.

* Somehow the pool is removed (via delete-connector-resource,
delete-connector-connection-pool or disable connector resource). So, the
NPE is an after effect.
* Does your application cache the connection factory ? (eg: as an
instance variable and is used during a web request after the
resource/pool is removed)

Thanks,
-Jagadish

On Wed, 2009-01-14 at 15:30 -0600, Brian Repko wrote:
>
> Folks,
>
> Looking for some insight on this one. I'm using the sun-jms-adapter
> as our connector to OpenMQ.
> We are SunAS 9.1 ur2 in a clustered environment. At some point, we
> started getting NPEs when
> accessing our JMS objects (ConnectionFactories in particular). This
> gets reported as as the following:
>
> [#|2009-01-14T08:11:40.942-0600|WARNING|sun-appserver9.1|
> javax.enterprise.system.stream.err|
> _ThreadID=2066;_ThreadName=httpSSLWorkerThread-50011-1;_RequestID=49b9cf44-0580-494d-93b2-f175a7edd24a;|.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:153)
> at
> com.stc.jmsjca.core.JConnection.createSession(JConnection.java:314)
> at
> com.stc.jmsjca.core.WConnection.createSession(WConnection.java:94)
> at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:258)
> at $Proxy554.createSession(Unknown Source)
> at
> org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> at
> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> ... 165 more
> Caused by: java.lang.NullPointerException
>
> After some investigation into the actual cause of the NPE (yes,
> printing that stack trace would have been helpful),
> I find the following:
>
> Caused by: java.lang.NullPointerException
> at
> com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> at
> com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> at
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> at
> com.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:131)
> ... 36 more
>
> So its not the adapter - its the server. This code looks like the
> following:
>
> public ResourceHandle getResourceFromPool(ResourceSpec spec,
> ResourceAllocator alloc, ClientSecurityInfo info, Transaction tran)
> throws PoolingException {
> ResourcePool pool = getPool( spec.getConnectionPoolName() );
> // pool.getResource() has been modified to:
> // - be able to create new resource if needed
> // - block the caller until a resource is acquired or
> // the max-wait-time expires
> return pool.getResource(spec, alloc, tran); <<<<< this is
> line 248
> }
>
> So clearly, getPool returns null and this code never checks for that.
>
> I've tried rebooting the cluster (which for us is stop/start instance1
> and then stop/start instance2) and I've tried rebooting
> the DAS. I also believe that a full stop on the cluster (both
> instances) has been done as well. I now get the NPEs even
> after all the reboots.
>
> I've turned on logging (FINEST) on the RAR and for Connectors and I
> have also turned on Monitoring for Connectors
> and have the logs for those startups. Monitoring doesn't seem to work
> for this pool so that was pointless. And the logs
> show all kinds of pools getting added but then removed with a line
> about them being empty. We do have the pools
> set to a steady state of 0 since the RAR actually pools the
> connections/sessions. Its like they get created to deal with JNDI
> and then removed for some reason after that.
>
> Things seem to be bound to the JNDI but the internal pool is not
> there. I'm not sure how to get around this.
> I've thought about adding a JSP to a web app that gets the pool table
> and printing it out just to see.
>
> I've also noticed a CVS commit from 3 months ago to PoolManagerImpl
> (appsrv-core) related to cleaning up resources
> after transaction commit. I'm nervous that key functionality is not
> in our version but I don't understand the tags to know
> which release is which.
>
> I put together a sample program and when I went to deploy it, I kept
> getting JNDI errors. I didn't understand since
> everything was correct (targets/enabled state). It turned out that my
> resource-ref was listed as false in the domain.xml
> but true on the screen. This was easily reproducable by
> asadmin-creating them into the server and then retargetting
> them to the cluster and resetting enabled to true. If I add them via
> the gui directly (no asadmin), then they go in correctly.
>
> The common thread is that when enabled=false, I also see the pools
> getting added and then removed. But the JNDI
> is not there in one case and seems to still be there in the other
> case.
>
> Looking for clues as to what to do - let me know whatever you might
> need. I can get it for you. Thanks in advance.
> At this point I can't get my server to reboot and applications to
> deploy properly.
>
> Brian
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jagadish Prasath Ramu

Copying admin@gf for help.

Thanks,
-Jagadish

On Thu, 2009-01-15 at 07:25 -0600, Brian Repko wrote:
> Jagadish,
>
> I'll get the domain.xml(s) and logs together. I've noticed issues when the DAS domain.xml and the instance/nodeagent domain.xmls get out of sync. I've also noticed issues where the admin says something different from the xml (enabled is true vs false). I don't have a test case - I have a sample app but it runs on my local cluster. I've tried to setup local as close to the failing cluster as possible but it still works for me. That's why I think its part of the domain.xml or our domains are setup in such a way that there is cross-talk (bad port assignments).
>
> We are using Spring's JmsTemplate as you can see from the stack trace so we are holding on to the ConnectionFactory and the Destination.
>
> I'll send files when I get them.
>
> Brian
>
>
> ----- Original message -----
> From: "Jagadish Prasath Ramu"
> To: users@glassfish.dev.java.net
> Date: Thu, 15 Jan 2009 17:19:54 +0530
> Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools
>
> Hi Brian,
> Can you post your domain.xml snapshot & server.log from DAS & cluster
> instances when you face the issue [FINEST for "connectors" &
> "resource-adapter" module log levels] ?
> If you have exact steps to reproduce the issue (or a test-case), that
> would be helpful.
>
> * Somehow the pool is removed (via delete-connector-resource,
> delete-connector-connection-pool or disable connector resource). So, the
> NPE is an after effect.
> * Does your application cache the connection factory ? (eg: as an
> instance variable and is used during a web request after the
> resource/pool is removed)
>
> Thanks,
> -Jagadish
>
>
>
> On Wed, 2009-01-14 at 15:30 -0600, Brian Repko wrote:
> >
> > Folks,
> >
> > Looking for some insight on this one. I'm using the sun-jms-adapter
> > as our connector to OpenMQ.
> > We are SunAS 9.1 ur2 in a clustered environment. At some point, we
> > started getting NPEs when
> > accessing our JMS objects (ConnectionFactories in particular). This
> > gets reported as as the following:
> >
> > [#|2009-01-14T08:11:40.942-0600|WARNING|sun-appserver9.1|
> > javax.enterprise.system.stream.err|
> > _ThreadID=2066;_ThreadName=httpSSLWorkerThread-50011-1;_RequestID=49b9cf44-0580-494d-93b2-f175a7edd24a;|.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:153)
> > at
> > com.stc.jmsjca.core.JConnection.createSession(JConnection.java:314)
> > at
> > com.stc.jmsjca.core.WConnection.createSession(WConnection.java:94)
> > at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:258)
> > at $Proxy554.createSession(Unknown Source)
> > at
> > org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> > at
> > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> > ... 165 more
> > Caused by: java.lang.NullPointerException
> >
> > After some investigation into the actual cause of the NPE (yes,
> > printing that stack trace would have been helpful),
> > I find the following:
> >
> > Caused by: java.lang.NullPointerException
> > at
> > com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> > at
> > com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> > at
> > com.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:131)
> > ... 36 more
> >
> > So its not the adapter - its the server. This code looks like the
> > following:
> >
> > public ResourceHandle getResourceFromPool(ResourceSpec spec,
> > ResourceAllocator alloc, ClientSecurityInfo info, Transaction tran)
> > throws PoolingException {
> > ResourcePool pool = getPool( spec.getConnectionPoolName() );
> > // pool.getResource() has been modified to:
> > // - be able to create new resource if needed
> > // - block the caller until a resource is acquired or
> > // the max-wait-time expires
> > return pool.getResource(spec, alloc, tran); <<<<< this is
> > line 248
> > }
> >
> > So clearly, getPool returns null and this code never checks for that.
> >
> > I've tried rebooting the cluster (which for us is stop/start instance1
> > and then stop/start instance2) and I've tried rebooting
> > the DAS. I also believe that a full stop on the cluster (both
> > instances) has been done as well. I now get the NPEs even
> > after all the reboots.
> >
> > I've turned on logging (FINEST) on the RAR and for Connectors and I
> > have also turned on Monitoring for Connectors
> > and have the logs for those startups. Monitoring doesn't seem to work
> > for this pool so that was pointless. And the logs
> > show all kinds of pools getting added but then removed with a line
> > about them being empty. We do have the pools
> > set to a steady state of 0 since the RAR actually pools the
> > connections/sessions. Its like they get created to deal with JNDI
> > and then removed for some reason after that.
> >
> > Things seem to be bound to the JNDI but the internal pool is not
> > there. I'm not sure how to get around this.
> > I've thought about adding a JSP to a web app that gets the pool table
> > and printing it out just to see.
> >
> > I've also noticed a CVS commit from 3 months ago to PoolManagerImpl
> > (appsrv-core) related to cleaning up resources
> > after transaction commit. I'm nervous that key functionality is not
> > in our version but I don't understand the tags to know
> > which release is which.
> >
> > I put together a sample program and when I went to deploy it, I kept
> > getting JNDI errors. I didn't understand since
> > everything was correct (targets/enabled state). It turned out that my
> > resource-ref was listed as false in the domain.xml
> > but true on the screen. This was easily reproducable by
> > asadmin-creating them into the server and then retargetting
> > them to the cluster and resetting enabled to true. If I add them via
> > the gui directly (no asadmin), then they go in correctly.
> >
> > The common thread is that when enabled=false, I also see the pools
> > getting added and then removed. But the JNDI
> > is not there in one case and seems to still be there in the other
> > case.
> >
> > Looking for clues as to what to do - let me know whatever you might
> > need. I can get it for you. Thanks in advance.
> > At this point I can't get my server to reboot and applications to
> > deploy properly.
> >
> > Brian
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

ZIP of the domain.xmls (DAS, one NA and one SI) and the logs from SI1 and SI2 are attached.

What we noticed is that our NA domain.xml is drastically out of sync with the SI and DAS

We are concerned that the instances are syncing or using the NA as a proxy for the DAS
(das is rebooting) and that causes them to go delete the pools.

At this point restarting things isn't working so if there is a way to shutdown and startup
that you think we should try, I'd LOVE to hear it.

Thanks,
Brian
[sun.zip]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

We've tried stop/starting the NA (in order to get it to sync) and we still get
the same issue.

Does it matter if the Node Agent domain.xml is in sync or not?

We are not sure what to do at this point - clearly rebooting doesn't work for
us. We cannot get a JMS connection factory out of this adapter anymore because
GF is not managing this pool properly.

Brian

----- Original message -----
From: "Brian Repko"
To: users@glassfish.dev.java.net
Date: Thu, 15 Jan 2009 15:31:19 -0600
Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools

ZIP of the domain.xmls (DAS, one NA and one SI) and the logs from SI1 and SI2 are attached.

What we noticed is that our NA domain.xml is drastically out of sync with the SI and DAS

We are concerned that the instances are syncing or using the NA as a proxy for the DAS
(das is rebooting) and that causes them to go delete the pools.

At this point restarting things isn't working so if there is a way to shutdown and startup
that you think we should try, I'd LOVE to hear it.

Thanks,
Brian

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Steve Essery

Just some general points on synchronization.

If the node agent is configured to automatically start the instances it
controls when it itself is started, then those instances don't sync with
the DAS and their local repository caches remain unchanged.

The instances should only sync with the DAS - more accurately the Node
Agent fires off a separate process prior to starting the instance that
is going to need sync'ing prior to starting the instance - and its that
process that talks to the DAS.

Differences in system clocks on the DAS and NA machines can cause odd
sync behaviour (there are hidden timestamp files in the areas that get
sync'd (.com_sun_appserv_timestamp) that record the time of the last
sync - these are sent to the DAS so it can figure out what's new in its
master repository which is why having clocks in sync is important).

Stopping/Starting the instances explicitly, e.g. asadmin
stop-instance/start-instance or asadmin stop-cluster/start-cluster (as
the last time I looked it just did stop/start to each instance) will
make them sync.

You can also change the behaviour of the node agent with the
--syncinstances flag, so asadmin start-node-agent --syncinstances=true.

Finding, and deleting all the .com_sun_appserv_timestamp files from an
instance and then stopping and restarting the instance will force it to
resync everything with the DAS.

Might be worth creating a standalone instance on the NA on the DAS box
(if there is still one) using the same config as the other instances and
see when its started whether its configuration matches expectations.
The domain.xml files should be identifical. Making local changes to the
instances always runs the risk of them being overwritten sometime in the
future when a sync does occur.

Regards,
Steve

Brian Repko wrote:
> We've tried stop/starting the NA (in order to get it to sync) and we still get
> the same issue.
>
> Does it matter if the Node Agent domain.xml is in sync or not?
>
> We are not sure what to do at this point - clearly rebooting doesn't work for
> us. We cannot get a JMS connection factory out of this adapter anymore because
> GF is not managing this pool properly.
>
> Brian
>
>
> ----- Original message -----
> From: "Brian Repko"
> To: users@glassfish.dev.java.net
> Date: Thu, 15 Jan 2009 15:31:19 -0600
> Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools
>
>
> ZIP of the domain.xmls (DAS, one NA and one SI) and the logs from SI1 and SI2 are attached.
>
> What we noticed is that our NA domain.xml is drastically out of sync with the SI and DAS
>
>
> We are concerned that the instances are syncing or using the NA as a proxy for the DAS
> (das is rebooting) and that causes them to go delete the pools.
>
> At this point restarting things isn't working so if there is a way to shutdown and startup
> that you think we should try, I'd LOVE to hear it.
>
> Thanks,
> Brian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

Steve,

Does the node agent ever act as a proxy for the DAS w.r.t the servers?
So if the server can't connect to the DAS, then it goes to the NA?

I'm finding .synchronize files with the long value of the timestamp in
it. Are we on an different version than you are thinking? Does that
change the synch architecture?

The server domain.xmls seem fine (and are close to matching the das
domain.xml) but the node agent domain.xml is horribly out of sync.
Missing all kinds of stuff - particularly related to our RA and its
resources.

Can't we shutdown/copy-the-domain.xml/startup without problems?
Wouldn't that solve the 'overwritten' issue?

What is the point of the domain.xml in the nodeagent/config area?

Is there a way to get a NA or server to persist their domain.xml?
Is there a way to force a sync on a running NA or SI?

Brian

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

Sorry about that - I DO have

.com_sun_appserv_timestamp in the SI/config directories.

.synchronize is in the DAS/config directory.

The NA/config has a .domain.xml.timestamp file which I'm assuming is when its domain.xml was sync'd.

Brian

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Steve Essery

Brian Repko wrote:
> Sorry about that - I DO have
>
> .com_sun_appserv_timestamp in the SI/config directories.
>
Basically this timestamp is sent to the DAS, it finds all the files in
that area newer than the timestamp and zips them up along with a listing
of the area. This is passed back and the files unzip, a cleaning
process then looks at the file list and removes any files that are no
longer present on the DAS.
> .synchronize is in the DAS/config directory.
>
This is a temporary file - its used by the DAS to figure out if there is
any delta between its system clock and the files it will be
synchronizing - for example if those files reside on network storage of
some kind. Setting the SYNC logging to FINE should result in additional
logging showing it being calculated.
> The NA/config has a .domain.xml.timestamp file which I'm assuming is when its domain.xml was sync'd.
>
Yes, other files in the config directory can also have their own
timestamp files, such as any jks or NSS keystore (depending on whether
you have a developer/cluster or enterprise installation of the
application server). You should be able to remove this, or just touch
the domain.xml on the DAS (once you are happy the clocks are in sync) to
have the domain.xml copied over.
> Brian
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Brian Repko

Steve,

> This is a temporary file - its used by the DAS to figure out if there is
> any delta between its system clock and the files it will be
> synchronizing - for example if those files reside on network storage of
> some kind. Setting the SYNC logging to FINE should result in additional
> logging showing it being calculated.

Which module is SYNC logging? I don't see that in my console.

Brian

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Steve Essery

Brian Repko wrote:
> Steve,
>
>
>> This is a temporary file - its used by the DAS to figure out if there is
>> any delta between its system clock and the files it will be
>> synchronizing - for example if those files reside on network storage of
>> some kind. Setting the SYNC logging to FINE should result in additional
>> logging showing it being calculated.
>>
>
> Which module is SYNC logging? I don't see that in my console.
>
In V2 its in the console as config->[config-name]->Logger Settings->Log
Levels->Synchronization ( javax.ee.enterprise.system.tools.synchronization)

Don't have a V2.1 immediately to hand to check that's the same - should be.
> Brian
>
>

[att1.html]

Brian Repko

> In V2 its in the console as config->[config-name]->Logger
Settings->Log Levels->Synchronization (
javax.ee.enterprise.system.tools.synchronization)
> Don't have a V2.1 immediately to hand to check that's the same
- should be.
Sorry about that, mate. I was looking at my DAS-only domain and
that doesn't
list Synchronization on the list of Log Levels. Only in a
clustered domain.
Thanks!
[att1.html]

Brian Repko

Synchronization log levels turned up to FINEST on cluster-config
and admin-config.
Touched the domain.xml
I don't see anything except for these 2 lines in the DAS
server.log
[#|2009-01-19T09:53:45.872-0600|INFO|sun-appserver9.1|javax.enter
prise.system.tools.admin|_ThreadID=84;_ThreadName=httpSSLWorkerTh
read-9000-4;com.sun.enterprise.admin.event.LogLevelChangeEvent --
server [1 Change(s), Id:153, ts:1232380425872];|ADM1041:Sent the
event to
instance:[com.sun.enterprise.admin.event.LogLevelChangeEvent --
server [1 Change(s), Id:153, ts:1232380425872]]|#]
[#|2009-01-19T10:14:06.565-0600|INFO|sun-appserver9.1|javax.enter
prise.system.tools.admin|_ThreadID=37;_ThreadName=httpSSLWorkerTh
read-9000-2;com.sun.enterprise.admin.event.LogLevelChangeEvent --
server [1 Change(s), Id:154, ts:1232381646565];|ADM1041:Sent the
event to
instance:[com.sun.enterprise.admin.event.LogLevelChangeEvent --
server [1 Change(s), Id:154, ts:1232381646565]]|#]
Nothing related to Synchronization. I thought log levels changed
without reboot. Am I wrong on that?
What should I have seen?
Brian
----- Original message -----
From: "Brian Repko"
To: "Steve Essery"
Cc: users@glassfish.dev.java.net, admin@glassfish.dev.java.net
Date: Mon, 19 Jan 2009 08:10:16 -0600
Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connect
or Connection Pools

> In V2 its in the console as config->[config-name]->Logger
Settings->Log Levels->Synchronization (
javax.ee.enterprise.system.tools.synchronization)
> Don't have a V2.1 immediately to hand to check that's the same
- should be.
Sorry about that, mate. I was looking at my DAS-only domain and
that doesn't
list Synchronization on the list of Log Levels. Only in a
clustered domain.
Thanks!
[att1.html]