Synchronization errors in Glassfish SSH Cluster Setup
Synchronization errors in Glassfish SSH Cluster Setup
February 17, 2012 - 03:35
Hi out there.
My name is basti, I'm the new one ;-)
I think I am a advanced glassfish user, but this error drives me crazy.
Here is the setup, I try to setup:
All servers are windows 2008 64 bit Edition (don't ask...)
- glassfish version is 3.1.1
- 1 Server as Domain Admin Server
- 4 additional server as ssh-nodes, each server 1 instance
- these 4 servers are assigned to 2 clusters
- for the ssh communication I installed cygwin with a default host-config setup (ssh login is successful, the server fingerprints are sucessfully inserted in cygwin)
Now: When I setup these things with my asadmin batch script, the script prints out no errors, but a warning: " Warning: Synchronization with DAS failed, continuing startup..."
The script runs fully through and the described setup looks just fine...but it isn't!
Looking into a server.log of an instance, there are serveral errors:
- java.lang.IllegalArgumentException: WEB0314: virtual server server has an invalid docroot C:\glassfish3\glassfish\nodes\gibson-5\gibson-5-instance/docroot
at com.sun.enterprise.web.WebContainer.validateDocroot(WebContainer.java:1235)
at com.sun.enterprise.web.WebContainer.createHost(WebContainer.java:1171)
at com.sun.enterprise.web.WebContainer.createHosts(WebContainer.java:1142) >> there is no folder "docroot" for this instance... - when I try to restart these instances from DAS there is this message: DTX5016:Error in XA recovery. See logs for more details
java.lang.RuntimeException: Connector Runtime not initialized
at com.sun.enterprise.connectors.ConnectorRuntime.getRuntime(ConnectorRuntime.java:213)
at com.sun.enterprise.connectors.util.ResourcesUtil.getRuntime(ResourcesUtil.java:122)
at com.sun.enterprise.connectors.util.ResourcesUtil.getDomain(ResourcesUtil.java:115)Cannot start container web - After waiting a long time for the restart the instances are marked for "restart required"
- when I try to deploy a simple application I get this error in the DAS log:[#|2012-02-17T11:13:36.955+0100|SEVERE|glassfish3.1.1|grizzly|_ThreadID=101;_ThreadName=Thread-2;|doSelect exception
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 256
at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
at com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191)
at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)Googling for this errors brings me to several Bug issues from glassfish (http://java.net/jira/browse/GLASSFISH-13837, http://java.net/jira/browse/GLASSFISH-6942, http://java.net/jira/browse/GLASSFISH-16289). But all issues have one problem: there are oficially fixed in my glassfish version.
Do you have any Ideas or just links to fix these issues?
for the ssh communication I installed cygwin with a default host-config setup (ssh login is successful, the server fingerprints are sucessfully inserted in cygwin)
Re: Synchronization errors in Glassfish SSH Cluster Setup
February 17, 2012 - 03:56
Hi,
pls. check GF domain.xml file(s),
and search for the line:
max-queue-size="256"
replace 256 with some bigger value, say 16384.
Let us know if it helped.
Thanks.
WBR,
Alexey.
On 02/17/2012 11:36 AM, forums@java.net wrote:
> Hi out there.
>
> My name is basti, I'm the new one ;-)
>
> I think I am a advanced glassfish user, but this error drives me crazy.
>
> Here is the setup, I try to setup:
>
> All servers are windows 2008 64 bit Edition (don't ask...)
>
> * glassfish version is 3.1.1
> * 1 Server as Domain Admin Server
> * 4 additional server as ssh-nodes, each server 1 instance
> * these 4 servers are assigned to 2 clusters
> * for the ssh communication I installed cygwin with a default
> host-config
> setup (ssh login is successful, the server fingerprints are
> sucessfully
> inserted in cygwin)
>
> Now: When I setup these things with my asadmin batch script, the script
> prints out no errors, but a warning: " Warning: Synchronization with DAS
> failed, continuing startup..."
>
> The script runs fully through and the described setup looks just
> fine...but
> it isn't!
>
> Looking into a server.log of an instance, there are serveral errors:
>
> * java.lang.IllegalArgumentException: WEB0314: virtual server
> server has
> an invalid docroot
>
> C:\glassfish3\glassfish\nodes\gibson-5.tecis.hh\gibson-5.tecis.hh-instance/docroot
>
> at
>
> com.sun.enterprise.web.WebContainer.validateDocroot(WebContainer.java:1235)
>
> at
> com.sun.enterprise.web.WebContainer.createHost(WebContainer.java:1171)
> at
>
> com.sun.enterprise.web.WebContainer.createHosts(WebContainer.java:1142) >>
>
> there is no folder "docroot" for this instance...
> * when I try to restart these instances from DAS there is this message:
> DTX5016:Error in XA recovery. See logs for more details
> java.lang.RuntimeException: Connector Runtime not initialized
> at
>
> com.sun.enterprise.connectors.ConnectorRuntime.getRuntime(ConnectorRuntime.java:213)
>
> at
>
> com.sun.enterprise.connectors.util.ResourcesUtil.getRuntime(ResourcesUtil.java:122)
>
> at
>
> com.sun.enterprise.connectors.util.ResourcesUtil.getDomain(ResourcesUtil.java:115)Cannot
>
> start container web
> * when I try to deploy a simple application I get this error in the DAS
>
> log:[#|2012-02-17T11:13:36.955+0100|SEVERE|glassfish3.1.1|grizzly|_ThreadID=101;_ThreadName=Thread-2;|doSelect
>
> exception
> java.util.concurrent.RejectedExecutionException: The thread pool's
> task
> queue is full, limit: 256
> at
>
> com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
>
> at
> com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191)
> at
>
> com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
>
> Googling for this errors brings me to several Bug issues from
> glassfish
> (http://java.net/jira/browse/GLASSFISH-13837,
> http://java.net/jira/browse/GLASSFISH-6942,
> http://java.net/jira/browse/GLASSFISH-16289). But all issues have one
> problem: there are oficially fixed in my glassfish version.
>
>
>
> Do you have any Ideas or just links to fix these issues?
>
>
>
>
>
>
> --
>
> [Message sent by forum member 'lb54']
>
> View Post: http://forums.java.net/node/883607
>
>
Re: Synchronization errors in Glassfish SSH Cluster Setup
February 17, 2012 - 05:08
Sorry, but this doesn't work.
I just replaced all max-queue-size="256" with max-queue-size="16384" and copied manually the domain.xmls to the instances. After restarting the instances are again marked for restart and the log says.
[#|2012-02-17T12:43:12.370+0100|SEVERE|glassfish3.1.1|javax.enterprise.resource.jta.com.sun.enterprise.transaction|_ThreadID=73;_ThreadName=Thread-2;|DTX5016:Error in XA recovery. See logs for more details
java.lang.RuntimeException: Connector Runtime not initialized
at com.sun.enterprise.connectors.ConnectorRuntime.getRuntime(ConnectorRuntime.java:213)
at com.sun.enterprise.connectors.util.ResourcesUtil.getRuntime(ResourcesUtil.java:122)
at com.sun.enterprise.connectors.util.ResourcesUtil.getDomain(ResourcesUtil.java:115)
at com.sun.enterprise.connectors.util.ResourcesUtil.getServer(ResourcesUtil.java:129)
Then I tried to redeploy the application. This also fails.
BTW: I tried to "recover Transactions" on the DAS Web Console for one instance, here is a similar Error:
"An error occurred during replication FAILURE: Command _recover-transactions-internal failed on server instance gibson-5-instance: remote failure: recover transactions failed. Connector Runtime not initialized FAILURE: Command _recover-transactions-internal failed on server instance gibson-5.tecis.hh-instance: remote failure: recover transactions failed. Connector Runtime not initialized "
Is there maybe a problem with a connection pool? I have some connectionpools which are using Oracle as database. Because of the sychronization-problem I guess, there is no jdbc-library copied to the instances. But: Copying the libs manually does not fix this issue...
Re: Synchronization errors in Glassfish SSH Cluster Setup
February 17, 2012 - 06:02
It's supposed to fix this part:
log:[#|2012-02-17T11:13:36.955+0100|SEVERE|glassfish3.1.1|grizzly|_ThreadID=101;_ThreadName=Thread-2;|doSelect
exception
java.util.concurrent.RejectedExecutionException: The thread pool's task
queue is full, limit: 256
at
com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
at com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191)
at
com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
as for the rest - hope someone else has a solution.
Thanks.
WBR,
Alexey.
On 02/17/2012 12:56 PM, forums@java.net wrote:
> Sorry, but this doesn't work.
>
> I just replaced all max-queue-size="256" with max-queue-size="16384" and
> copied manually the domain.xmls to the instances. After restarting the
> instances are again marked for restart and the log says.
>
> [#|2012-02-17T12:43:12.370+0100|SEVERE|glassfish3.1.1|javax.enterprise.resource.jta.com.sun.enterprise.transaction|_ThreadID=73;_ThreadName=Thread-2;|DTX5016:Error
>
> in XA recovery. See logs for more details
> java.lang.RuntimeException: Connector Runtime not initialized
> at
> com.sun.enterprise.connectors.ConnectorRuntime.getRuntime(ConnectorRuntime.java:213)
>
> at
> com.sun.enterprise.connectors.util.ResourcesUtil.getRuntime(ResourcesUtil.java:122)
>
> at
> com.sun.enterprise.connectors.util.ResourcesUtil.getDomain(ResourcesUtil.java:115)
>
> at
> com.sun.enterprise.connectors.util.ResourcesUtil.getServer(ResourcesUtil.java:129)
>
>
> Then I tried to redeploy the application. This also fails.
>
> BTW: I tried to "recover Transactions" on the DAS Web Console for one
> instance, here is a similar Error:
>
> "An error occurred during replication FAILURE: Command
> _recover-transactions-internal failed on server instance
> gibson-5-instance:
> remote failure: recover transactions failed. Connector Runtime not
> initialized FAILURE: Command _recover-transactions-internal failed on
> server
> instance gibson-5.tecis.hh-instance: remote failure: recover transactions
> failed. Connector Runtime not initialized "
>
> Is there maybe a problem with a connection pool. I have some
> connectionpools
> which are using Oracle as database. Because of the
> sychronization-problem I
> guess, there is no jdbc-library copied to the instances. But: Copying the
> libs manually does not fix this issue...
>
>
> --
>
> [Message sent by forum member 'lb54']
>
> View Post: http://forums.java.net/node/883607
>
>
Re: Synchronization errors in Glassfish SSH Cluster Setup
February 17, 2012 - 09:46
I found hier the docroot Problem: http://www.java.net/forum/topic/glassfish/glassfish/deployment-problem-g...
but this solution doens't fix the startup error from my instances.
Any other hints or solutions?
Re: Synchronization errors in Glassfish SSH Cluster Setup
February 28, 2012 - 03:55
Finally it seems, that a bug in cygwin ssh client version 1.7.10.1 caused these errors.
A new Installation with a newer version (1.7.11.1) causes none of these errors.
Thank you for listening and see you soon.
BYe
The key to fixing this is to figure out why the synchronization with the
DAS is failing.
On the DAS, turn on logging for the admin tools subsystem:
asadmin set-log-levels javax.enterprise.system.tools.admin=FINE
On a node, start an instance manually with debug turned on:
set AS_DEBUG=true
asadmin start-local-instance -v gibson-5.tecis.hh-instance
In the server.log file, look for messages that mention
synchronize-files. There should be several messages for various files
and directories. If there are none, then the instance isn't connecting
to the DAS at all.
In the output of the start-local-instance command, there should be an
indication of why the synchronization is failing.
Tom