Skip to main content

Error: Skipped Loading of 400a9267146e20fc8f3d32420e91 since there is only one instance running in the cluster.

5 replies [Last post]
lprimak
Offline
Joined: 2006-08-22

Hi,

I am getting this error in my application

[#|2012-10-18T12:19:12.479-0400|INFO|glassfish3.1.2|org.shoal.ha.cache.command.load_request|_ThreadID=155;_ThreadName=Thread-2;|Skipped Loading of 400a9267146e20fc8f3d32420e91 since there is only one instance running in the cluster.|#]

There are two instances in one cluster running the application.
One of the cluster instances is spitting out this error.

What does this mean? Should I be worried about it?

This is an EJB and Web app with high availability turned on.
Asynchronous replication is also on as well as relaxVersionSemantics is on.

Thanks for your help.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
jfialli
Offline
Joined: 2003-06-16

On 10/18/12 2:16 PM, forums@java.net wrote:
> Hi, I am getting this error in my application
> [#|2012-10-18T12:19:12.479-0400|INFO|glassfish3.1.2|org.shoal.ha.cache.command.load_request|_ThreadID=155;_ThreadName=Thread-2;|Skipped
>
> Loading of 400a9267146e20fc8f3d32420e91 since there is only one instance
> running in the cluster.|#] There are two instances in one cluster
> running the
> application. One of the cluster instances is spitting out this error.
> What
> does this mean? Should I be worried about it? This is an EJB and Web
> app with
> high availability turned on. Asynchronous replication is also on as
> well as
> relaxVersionSemantics is on. Thanks for your help.
The message means that at the time of the load, only one instance in the
cluster is known to be running.
So replication is definitely not working at this time. There must be 2
or more instances running for high
availabilility to work. It is disabled and messages such as this one
are a warning for administrator to
determine why the cluster is not well formed.

There is a command "asadmin get-health " to get the health
of the cluster from the command line.
This command provides a real time health status of the current running
cluster.

Here is sample output from this command.
% asadmin get-health myCluster
instance01 started since Fri Oct 19 10:35:30 EDT 2012
instance02 started since Fri Oct 19 10:35:29 EDT 2012
instance03 started since Fri Oct 19 10:35:29 EDT 2012
Command get-health executed successfully.

The server log for the DAS and for the two instances have the health
status of instances joining and leaving
the cluster over time. This will provide you information on why the
other instance was not available.
You can search for ShoalLogger INFO messages in the DAS and clustered
instances server logs for history
of cluster members.

-Joe
>
> --
>
> [Message sent by forum member 'lprimak']
>
> View Post: http://forums.java.net/node/891569
>
>

lprimak
Offline
Joined: 2006-08-22

Thanks, Joe,

I ran the get-health command and here's the output:
bawweb3-inst started since Wed Oct 17 01:13:49 EDT 2012
bawweb4-inst started since Wed Oct 17 01:15:53 EDT 2012
Command get-health executed successfully.

There are also no logs anywhere about anything joining and leaving the cluster.
As a matter of fact, there are no messages at all besides this warning.
Is there anything else that could cause this, even though two instances are running?
Thanks

jfialli wrote:
On 10/18/12 2:16 PM, forums@java.net wrote:
> Hi, I am getting this error in my application
> [#|2012-10-18T12:19:12.479-0400|INFO|glassfish3.1.2|org.shoal.ha.cache.command.load_request|_ThreadID=155;_ThreadName=Thread-2;|Skipped
>
> Loading of 400a9267146e20fc8f3d32420e91 since there is only one instance
> running in the cluster.|#] There are two instances in one cluster
> running the
> application. One of the cluster instances is spitting out this error.
> What
> does this mean? Should I be worried about it? This is an EJB and Web
> app with
> high availability turned on. Asynchronous replication is also on as
> well as
> relaxVersionSemantics is on. Thanks for your help.
The message means that at the time of the load, only one instance in the
cluster is known to be running.
So replication is definitely not working at this time. There must be 2
or more instances running for high
availabilility to work. It is disabled and messages such as this one
are a warning for administrator to
determine why the cluster is not well formed.

There is a command "asadmin get-health " to get the health
of the cluster from the command line.
This command provides a real time health status of the current running
cluster.

Here is sample output from this command.
% asadmin get-health myCluster
instance01 started since Fri Oct 19 10:35:30 EDT 2012
instance02 started since Fri Oct 19 10:35:29 EDT 2012
instance03 started since Fri Oct 19 10:35:29 EDT 2012
Command get-health executed successfully.

The server log for the DAS and for the two instances have the health
status of instances joining and leaving
the cluster over time. This will provide you information on why the
other instance was not available.
You can search for ShoalLogger INFO messages in the DAS and clustered
instances server logs for history
of cluster members.

-Joe
>
> --
>
> [Message sent by forum member 'lprimak']
>
> View Post: http://forums.java.net/node/891569
>
>

jfialli
Offline
Joined: 2003-06-16

On 10/19/12 3:34 PM, forums@java.net wrote:
> Thanks, Joe, I ran the get-health command and here's the output:
> bawweb3-inst
> started since Wed Oct 17 01:13:49 EDT 2012 bawweb4-inst started since
> Wed Oct
> 17 01:15:53 EDT 2012 Command get-health executed successfully. There
> are also
> no logs anywhere about anything joining and leaving the cluster. As a
> matter
> of fact, there are no messages at all besides this warning. Is there
> anything
> else that could cause this, even though two instances are running?
It is not possible to diagnose what is going on without looking at the
application server log files.
Minimally, you need to run the following.

% asadmin collect-log-files --target

For documentation on where to find the zip file created with all the
server logs,
see the following:
http://docs.oracle.com/cd/E18930_01/html/821-2433/collect-log-files-1.html

You can search the log files for "ShoalLogger" messages that describe
members joining and leaving the
cluster.

Here is a sample log message from the DAS server.log that shows that all
members in a 3 member cluster are communicating
with each other.

[[#|2012-10-19T10:35:36.204-0400|INFO|44.0|ShoalLogger|_ThreadID=15;_ThreadName=GMS
ViewWindowThread Group-myCluster;|GMS1092: GMS View Change Received for
group: myCluster : Members in view for ADD_EVENT(before change analysis)
are :
1: MemberId: instance01, MemberType: CORE, Address:
10.152.23.225:9169:228.9.15.15:23933:myCluster:instance01
2: MemberId: instance02, MemberType: CORE, Address:
10.152.23.225:9106:228.9.15.15:23933:myCluster:instance02
3: MemberId: instance03, MemberType: CORE, Address:
10.152.23.225:9109:228.9.15.15:23933:myCluster:instance03
4: MemberId: server, MemberType: SPECTATOR, Address:
10.152.23.225:9149:228.9.15.15:23933:myCluster:server
|#]

DAS server.log is found in
glassfish3/glassfish/domains//logs/server*.log
If you run the asadmin collect-log-files with a target of clustername,
the DAS server log will be included.

********

If your instances are not finding each other and are running in isolated
mode, the following documentation
will help you diagnose if there is a multicast issue between cluster
members. (There is no way to know if this case applies to you
or not without looking at app server log messages collected in step above.)

http://docs.oracle.com/cd/E26576_01/doc.312/e24934/clusters.htm#gklhd

-Joe

> Thanks
>

jfialli wrote:
On 10/18/12 2:16 PM, forums@java.net wrote: > Hi, I am
> getting
> this error in my application >
> [#|2012-10-18T12:19:12.479-0400|INFO|glassfish3.1.2|org.shoal.ha.cache.command.load_request|_ThreadID=155;_ThreadName=Thread-2;|Skipped
>
> > > Loading of 400a9267146e20fc8f3d32420e91 since there is only one
> instance
> > running in the cluster.|#] There are two instances in one cluster >
> running
> the > application. One of the cluster instances is spitting out this
> error. >
> What > does this mean? Should I be worried about it? This is an EJB
> and Web >
> app with > high availability turned on. Asynchronous replication is
> also on
> as > well as > relaxVersionSemantics is on. Thanks for your help. The
> message
> means that at the time of the load, only one instance in the cluster
> is known
> to be running. So replication is definitely not working at this time.
> There
> must be 2 or more instances running for high availabilility to work.
> It is
> disabled and messages such as this one are a warning for administrator to
> determine why the cluster is not well formed. There is a command "asadmin
> get-health " to get the health of the cluster from the command line. This
> command provides a real time health status of the current running
> cluster.
> Here is sample output from this command. % asadmin get-health myCluster
> instance01 started since Fri Oct 19 10:35:30 EDT 2012 instance02 started
> since Fri Oct 19 10:35:29 EDT 2012 instance03 started since Fri Oct 19
> 10:35:29 EDT 2012 Command get-health executed successfully. The server
> log
> for the DAS and for the two instances have the health status of instances
> joining and leaving the cluster over time. This will provide you
> information
> on why the other instance was not available. You can search for
> ShoalLogger
> INFO messages in the DAS and clustered instances server logs for
> history of
> cluster members. -Joe > > -- > > [Message sent by forum member
> 'lprimak'] > >
> View Post: http://forums.java.net/node/891569 > >

>
> --
>
> [Message sent by forum member 'lprimak']
>
> View Post: http://forums.java.net/node/891569
>
>

lprimak
Offline
Joined: 2006-08-22

Looks like this happens if one of the cluster nodes fails for some reason.

jfialli
Offline
Joined: 2003-06-16

On 11/7/12 4:29 AM, forums@java.net wrote:
> Looks like this happens if one of the cluster nodes fails for some
> reason.
If there is only one instance running in a cluster, we added the warning
that you reported so the
administrator would be warned that replication of session data could not
occur since there was only
one instance in the cluster running.

In order to protect against the constraint that there must be 2 or more
instances running in a cluster
for replication to work, it is recommended that one have more than just
2 instances running in a cluster with
high availability of session data enabled. Otherwise, once one instance
fails with replication enabled in a 2 instance cluster,
there is no way for replication to work since there is no other node to
replicate to.

-Joe Fialli
>
> --
>
> [Message sent by forum member 'lprimak']
>
> View Post: http://forums.java.net/node/891569
>
>