Skip to main content

Glassfish starts, binds to ports but does not respond

18 replies [Last post]
elihusmails
Offline
Joined: 2006-05-17

I have been dealing with this problem for a few weeks on a development box and have gotten to the point that I question the stability of my setup. I have Glassfish v2ur2 and have the portal-container installed. I have the following deployed on it:

4 portlets
1 web service
1 JMS queue connection factory
1 JMS queue
1 Database resource
1 Database connection pool

From time-to-time, I will shut down and restart glassfish and when it comes back up, all the output (console/logs) appear fine, but you cannot connect to the admin pages or any webapps. I can only connect via jconsole and jconsole reports no deadlocks.

What other information should I look at in order to try and diagnose this problem? I have resorted to deleting the domain directory and untarring a working copy when it hangs to prevent re-installation 2-3 times a day.

Thank you.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Jeanfrancois Arcand

Salut,

Chris Fleischmann wrote:
> Thanks for sending the file through, it seems all of your http and https
> threads are waiting on locks
> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline for HTTP
> and com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline for HTTPS
> which looks like there waiting on other threads to finish.

Yes, this is expected. That's the Grizzly thread pool waiting for works.

Thanks

-- Jeanfrancois

>
> I wonder if after time you are running out of file descriptors. I see
> from the source code
> (http://fisheye5.cenqua.com/browse/glassfish/appserv-http-engine/src/java...)
> for GlassFish the following in relation to the "waiting" threads, in
> particular line 291:
>
> 288 if (size() - waitingThreads <= 0) { 289 try {
> 290 waitingThreads++;
> 291 wait();
>
> Showing that there are more waiting threads than there are available
> threads.
>
> When hanging are you able to also run the following commands for me;
>
> lsof -p
>
> as root : ulimit -a
>
> and
>
> strace -Ff -tt -p 2>&1 | tee .log
>
> Cheers,
>
> Chris
>
>
>
>
> glassfish@javadesktop.org wrote:
>> attached is the file.
>>
>> Thanks so much for sticking with me on this...
>> [Message sent by forum member 'elihusmails' (elihusmails)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=275387
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Mark Webb

JeanFrancois,

So the Grizzly thread pool is waiting for work? If I connect to
glassfish's admin web page, shouldn't that cause work for the threads
in the thread pool? Before I updated the settings using your blog
posting (http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2....),
I could not connect to the admin web page, JMS or Web Services.

Thanks
Mark

On Tue, May 20, 2008 at 12:39 PM, Jeanfrancois Arcand
wrote:
> Salut,
>
> Chris Fleischmann wrote:
>>
>> Thanks for sending the file through, it seems all of your http and https
>> threads are waiting on locks
>> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline for HTTP and
>> com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline for HTTPS which
>> looks like there waiting on other threads to finish.
>
> Yes, this is expected. That's the Grizzly thread pool waiting for works.
>
> Thanks
>
> -- Jeanfrancois
>
>
>
>>
>> I wonder if after time you are running out of file descriptors. I see from
>> the source code
>> (http://fisheye5.cenqua.com/browse/glassfish/appserv-http-engine/src/java...)
>> for GlassFish the following in relation to the "waiting" threads, in
>> particular line 291:
>>
>> 288 if (size() - waitingThreads <= 0) { 289 try {
>> 290 waitingThreads++;
>> 291 wait();
>>
>> Showing that there are more waiting threads than there are available
>> threads.
>>
>> When hanging are you able to also run the following commands for me;
>>
>> lsof -p
>>
>> as root : ulimit -a
>>
>> and
>>
>> strace -Ff -tt -p 2>&1 | tee .log
>>
>> Cheers,
>>
>> Chris
>>
>>
>>
>>
>> glassfish@javadesktop.org wrote:
>>>
>>> attached is the file.
>>>
>>> Thanks so much for sticking with me on this...
>>> [Message sent by forum member 'elihusmails' (elihusmails)]
>>>
>>> http://forums.java.net/jive/thread.jspa?messageID=275387
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Jeanfrancois Arcand

Salut,

Mark Webb wrote:
> JeanFrancois,
>
> So the Grizzly thread pool is waiting for work? If I connect to
> glassfish's admin web page, shouldn't that cause work for the threads
> in the thread pool?

Yes, for the duration of the request. So getting a Thread Dump showing
that is difficult unless your request takes a long time to execute.
Usually GlassFish run out of thread when there is a lock somewhere in
the app server (like the connector db connection pool). Thread gets
blocked and Grizzly starts pooling the request, waiting to threads to be
free. But if there is a lock outside Grizzly, Grizzly has no way to
unlock the threads and we ends up oin very strange situations.....

Before I updated the settings using your blog
> posting (http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2....),
> I could not connect to the admin web page, JMS or Web Services.

Have You added:

-Dcom.sun.enterprise.server.ss.ASQuickStartup=false

in domain.xml (sorry If I already asked the question, but this things
save my life most of the time :-))

Thanks

-- Jeanfrancois

>
> Thanks
> Mark
>
>
> On Tue, May 20, 2008 at 12:39 PM, Jeanfrancois Arcand
> wrote:
>> Salut,
>>
>> Chris Fleischmann wrote:
>>> Thanks for sending the file through, it seems all of your http and https
>>> threads are waiting on locks
>>> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline for HTTP and
>>> com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline for HTTPS which
>>> looks like there waiting on other threads to finish.
>> Yes, this is expected. That's the Grizzly thread pool waiting for works.
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>>
>>
>>> I wonder if after time you are running out of file descriptors. I see from
>>> the source code
>>> (http://fisheye5.cenqua.com/browse/glassfish/appserv-http-engine/src/java...)
>>> for GlassFish the following in relation to the "waiting" threads, in
>>> particular line 291:
>>>
>>> 288 if (size() - waitingThreads <= 0) { 289 try {
>>> 290 waitingThreads++;
>>> 291 wait();
>>>
>>> Showing that there are more waiting threads than there are available
>>> threads.
>>>
>>> When hanging are you able to also run the following commands for me;
>>>
>>> lsof -p
>>>
>>> as root : ulimit -a
>>>
>>> and
>>>
>>> strace -Ff -tt -p 2>&1 | tee .log
>>>
>>> Cheers,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>> glassfish@javadesktop.org wrote:
>>>> attached is the file.
>>>>
>>>> Thanks so much for sticking with me on this...
>>>> [Message sent by forum member 'elihusmails' (elihusmails)]
>>>>
>>>> http://forums.java.net/jive/thread.jspa?messageID=275387
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Mark Webb

I have read that setting that jvm option does help. Since I bumped up
the threading settings in glassfish I have had no problems. Seems
like the default settings may be too low for any sort of
testing/development deployment.

Thanks again
Mark

On Tue, May 20, 2008 at 1:29 PM, Jeanfrancois Arcand
wrote:
> Salut,
>
> Mark Webb wrote:
>>
>> JeanFrancois,
>>
>> So the Grizzly thread pool is waiting for work? If I connect to
>> glassfish's admin web page, shouldn't that cause work for the threads
>> in the thread pool?
>
> Yes, for the duration of the request. So getting a Thread Dump showing that
> is difficult unless your request takes a long time to execute. Usually
> GlassFish run out of thread when there is a lock somewhere in the app server
> (like the connector db connection pool). Thread gets blocked and Grizzly
> starts pooling the request, waiting to threads to be free. But if there is a
> lock outside Grizzly, Grizzly has no way to unlock the threads and we ends
> up oin very strange situations.....
>
>
> Before I updated the settings using your blog
>>
>> posting
>> (http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2....),
>> I could not connect to the admin web page, JMS or Web Services.
>
> Have You added:
>
> -Dcom.sun.enterprise.server.ss.ASQuickStartup=false
>
> in domain.xml (sorry If I already asked the question, but this things save
> my life most of the time :-))
>
> Thanks
>
> -- Jeanfrancois
>
>
>>
>> Thanks
>> Mark
>>
>>
>> On Tue, May 20, 2008 at 12:39 PM, Jeanfrancois Arcand
>> wrote:
>>>
>>> Salut,
>>>
>>> Chris Fleischmann wrote:
>>>>
>>>> Thanks for sending the file through, it seems all of your http and https
>>>> threads are waiting on locks
>>>> com.sun.enterprise.web.connector.grizzly.LinkedListPipeline for HTTP
>>>> and
>>>> com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline for HTTPS
>>>> which
>>>> looks like there waiting on other threads to finish.
>>>
>>> Yes, this is expected. That's the Grizzly thread pool waiting for works.
>>>
>>> Thanks
>>>
>>> -- Jeanfrancois
>>>
>>>
>>>
>>>> I wonder if after time you are running out of file descriptors. I see
>>>> from
>>>> the source code
>>>>
>>>> (http://fisheye5.cenqua.com/browse/glassfish/appserv-http-engine/src/java...)
>>>> for GlassFish the following in relation to the "waiting" threads, in
>>>> particular line 291:
>>>>
>>>> 288 if (size() - waitingThreads <= 0) { 289 try {
>>>> 290 waitingThreads++;
>>>> 291 wait();
>>>>
>>>> Showing that there are more waiting threads than there are available
>>>> threads.
>>>>
>>>> When hanging are you able to also run the following commands for me;
>>>>
>>>> lsof -p
>>>>
>>>> as root : ulimit -a
>>>>
>>>> and
>>>>
>>>> strace -Ff -tt -p 2>&1 | tee .log
>>>>
>>>> Cheers,
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>> glassfish@javadesktop.org wrote:
>>>>>
>>>>> attached is the file.
>>>>>
>>>>> Thanks so much for sticking with me on this...
>>>>> [Message sent by forum member 'elihusmails' (elihusmails)]
>>>>>
>>>>> http://forums.java.net/jive/thread.jspa?messageID=275387
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

roisinflannery
Offline
Joined: 2007-05-08

Hi elihusmails,

Have you ever considered purchasing a support contract with Sun? We have a number of different offerings and pricing schemes, to support every type of user.

With support, you receive access to fully tested and supported patches, containing fixes to recent bugs. You also will have the benefits of up to 24x7 support, meaning that you can get your questions answered all the time anytime. Please take a look at the following link for more details:
http://forums.java.net/jive/thread.jspa?messageID=276262&#276262
or else, please feel free to reply to this thread and I will provide more information.

Kind regards,
Roisin
Sustaining Engineering, GlassFish

elihusmails
Offline
Joined: 2006-05-17

Here is the result of what you asked for...
[b]
ping localhost[/b]
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.071 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.066 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.062 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.062/0.066/0.071/0.007 ms

==================================================

[b]cat /etc/hosts[/b]
127.0.0.1 localhost
127.0.0.1 markdemo

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

==================================================

[b]netstat -rn markdemo[/b]
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.214.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.214.2 0.0.0.0 UG 0 0 0 eth0

==================================================
[b]
netstat -tapn | grep 7721[/b]
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::7721 :::* LISTEN 5643/java

==================================================

[b]telnet localhost 7721[/b]
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
101 imqbroker 4.1 Patch 1
portmapper tcp PORTMAPPER 7721 [imqvarhome=/home/mark/glassfish/domains/demo/imq,imqhome=/home/mark/glassfish/imq,sessionid=3888042467979388672]
cluster_discovery tcp CLUSTER_DISCOVERY 58634
jmxrmi rmi JMX 0 [url=service:jmx:rmi://markdemo/jndi/rmi://markdemo:7821/markdemo/7721/jmxrmi]
admin tcp ADMIN 60234
jms tcp NORMAL 59350
cluster tcp CLUSTER 40359
.
Connection closed by foreign host.

elihusmails
Offline
Joined: 2006-05-17

Here are the files. As you can see, I ran glassfish with a PermGen size of 256M and observed the same results. I ran jstack and it appeared that the threads were all in a state of TIMED_WAITING, WAITING or RUNNABLE and I still could not connect.

Since I cannot connect to the admin web page, is there a way to increase the number of HTTP/HTTPS threads so that I can see if that will get me to at least connect to the server?

Thanks again!

Chris Fleischmann

Hrm.... ok, to increase the worker threads and to get Grizzly working at
its best, you could try the recommendations here:
http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2....

Interesting, I see the following error in your server.log file:

[#|2008-05-20T07:16:18.145-0400|INFO|sun-appserver9.1|javax.resourceadapter.mqjmsra.lifecycle|_ThreadID=10;_ThreadName=main;|MQJMSRA_LB1101:
Looking for Broker Running at:localhost:7721|#]

[#|2008-05-20T07:16:18.214-0400|WARNING|sun-appserver9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=ef189062-eb39-4ba3-a080-2a08d7e3d6f4;|[C4003]:
Error occurred on connection creation [localhost:7721]. - cause:
java.net.ConnectException: Connection refused|#]

A couple of quick questions, can you let me know the output of:

ping localhost

Contents of /etc/hosts file;

Results of: "netstat -rn", "hostname"

Is your JMS Broker actually running on port 7721?

-----

Cheers,

Chris

glassfish@javadesktop.org wrote:
> Here are the files. As you can see, I ran glassfish with a PermGen size of 256M and observed the same results. I ran jstack and it appeared that the threads were all in a state of TIMED_WAITING, WAITING or RUNNABLE and I still could not connect.
>
> Since I cannot connect to the admin web page, is there a way to increase the number of HTTP/HTTPS threads so that I can see if that will get me to at least connect to the server?
>
> Thanks again!
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275448
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

[att1.html]

elihusmails
Offline
Joined: 2006-05-17

attached is the file.

Thanks so much for sticking with me on this...

Chris Fleischmann

Thanks for sending the file through, it seems all of your http and https
threads are waiting on locks
com.sun.enterprise.web.connector.grizzly.LinkedListPipeline for HTTP
and com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline for HTTPS
which looks like there waiting on other threads to finish.

I wonder if after time you are running out of file descriptors. I see
from the source code
(http://fisheye5.cenqua.com/browse/glassfish/appserv-http-engine/src/java...)
for GlassFish the following in relation to the "waiting" threads, in
particular line 291:

288 if (size() - waitingThreads <= 0) {
289 try {
290 waitingThreads++;
291 wait();

Showing that there are more waiting threads than there are available
threads.

When hanging are you able to also run the following commands for me;

lsof -p

as root : ulimit -a

and

strace -Ff -tt -p 2>&1 | tee .log

Cheers,

Chris

glassfish@javadesktop.org wrote:
> attached is the file.
>
> Thanks so much for sticking with me on this...
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275387
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

elihusmails
Offline
Joined: 2006-05-17

I have attached some files..

lsof.out - output from the lsof command
strace.out - a portion of the output from the strace command.

the output from ulimit -a :
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 8192
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 8192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Chris Fleischmann

Sorry to be a pain, but can you also send me your server.log file?

glassfish@javadesktop.org wrote:
> I have attached some files..
>
> lsof.out - output from the lsof command
> strace.out - a portion of the output from the strace command.
>
> the output from ulimit -a :
> ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 8192
> max locked memory (kbytes, -l) 32
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 8192
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275409
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

[att1.html]

Chris Fleischmann

Sorry can you also send your domain.xml file for your particular
domain/instance (along with your server.log file)... The http and http
threads may well be waiting for more memory, I see from the jvm.log file
that you have 99% of your heap in use.

Heap
def new generation total 16448K, used 12406K [0x89ca0000, 0x8ae70000,
0x94740000)
eden space 14656K, 77% used [0x89ca0000, 0x8a7b50c0, 0x8aaf0000)
from space 1792K, 59% used [0x8aaf0000, 0x8abf8a98, 0x8acb0000)
to space 1792K, 0% used [0x8acb0000, 0x8acb0000, 0x8ae70000)
tenured generation total 36292K, used 22775K [0x94740000, 0x96ab1000,
0xa9ca0000)
the space 36292K, 62% used [0x94740000, 0x95d7dc08, 0x95d7de00,
0x96ab1000)
compacting perm gen total 43776K, used 43732K [0xa9ca0000, 0xac760000,
0xb5ca0000)
the space 43776K, 99% used [0xa9ca0000, 0xac7551a8, 0xac755200,
0xac760000)
No shared spaces configured.

The areas I'm thinking of exploring are:

adding a line (if not there already) to your jvm options setting your
maxperm size higher: -XX:MaxPermSize=256m

If the above doesn't fix your issues, it may well be a locking issue etc
with respect to Grizzly (we will explore further).

Regards,

Chris

Chris Fleischmann wrote:
> Sorry to be a pain, but can you also send me your server.log file?
>
> glassfish@javadesktop.org wrote:
>> I have attached some files..
>>
>> lsof.out - output from the lsof command
>> strace.out - a portion of the output from the strace command.
>>
>> the output from ulimit -a :
>> ulimit -a
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 8192
>> max locked memory (kbytes, -l) 32
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 1024
>> pipe size (512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 8192
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>> [Message sent by forum member 'elihusmails' (elihusmails)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=275409
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>

[att1.html]

elihusmails
Offline
Joined: 2006-05-17

I am using the latest java from sun on an Ubuntu 8.04 installation running inside VMware. Java full version is:

java -fullversion
java full version "1.6.0_06-b02"

java -version
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)

Chris Fleischmann

OK, another way of getting the thread-dump is to use kill -3 , can
you send in the output (ie., the jvm.log file)? See:
http://blogs.sun.com/foo/entry/how_to_get_thread_dumps for more information.

Perhaps run this command again after waiting 10 more seconds, to see
some sort of time elapsing.

Cheers,

Chris

glassfish@javadesktop.org wrote:
> I am using the latest java from sun on an Ubuntu 8.04 installation running inside VMware. Java full version is:
>
> java -fullversion
> java full version "1.6.0_06-b02"
>
> java -version
> java version "1.6.0_06"
> Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
> Java HotSpot(TM) Client VM (build 10.0-b22, mixed mode, sharing)
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275384
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

[att1.html]

elihusmails
Offline
Joined: 2006-05-17

Here is what I get printed out in the terminal, the attached file is the actual output:

Attaching to process ID 5567, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 10.0-b22
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:168)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)

Chris Fleischmann

What version of Java are you using?

Interesting, that jstack outputs: JVM version is 10.0-b22 which doesn't
seem right to me.

Can you run a :*/ java -fullversion/*

jstack will fail in some thread conditions under jdk 1.5.x, if you could
try jstack from jdk 1.6.x, that "should" work-around this issue and help
to capture the thread state.

Regards,

Chris

glassfish@javadesktop.org wrote:
> Here is what I get printed out in the terminal, the attached file is the actual output:
>
> Attaching to process ID 5567, please wait...
> Debugger attached successfully.
> Client compiler detected.
> JVM version is 10.0-b22
> java.lang.NullPointerException
> at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:224)
> at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:342)
> at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
> at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:194)
> at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:199)
> at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
> at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:168)
> at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
> at sun.tools.jstack.JStack.main(JStack.java:84)
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275378
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

[att1.html]

Chris Fleischmann

There might be no deadlocks, but there could be a "hanging" situation
where one thread (or a bunch of them) is waiting for another to finish?

The types of things I'd look in to:

1/ Do you have a LB or a Firewall between GF and your Database?

2/ Run jstack where PID is the Application Server process id, and
jstack is a executable under your /bin directory

Repeat step 2 after 10 seconds and send back to the user forum the two
jstack outputs.

Cheers,

Chris

glassfish@javadesktop.org wrote:
> I have been dealing with this problem for a few weeks on a development box and have gotten to the point that I question the stability of my setup. I have Glassfish v2ur2 and have the portal-container installed. I have the following deployed on it:
>
> 4 portlets
> 1 web service
> 1 JMS queue connection factory
> 1 JMS queue
> 1 Database resource
> 1 Database connection pool
>
> From time-to-time, I will shut down and restart glassfish and when it comes back up, all the output (console/logs) appear fine, but you cannot connect to the admin pages or any webapps. I can only connect via jconsole and jconsole reports no deadlocks.
>
> What other information should I look at in order to try and diagnose this problem? I have resorted to deleting the domain directory and untarring a working copy when it hangs to prevent re-installation 2-3 times a day.
>
> Thank you.
> [Message sent by forum member 'elihusmails' (elihusmails)]
>
> http://forums.java.net/jive/thread.jspa?messageID=275322
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

[att1.html]