Skip to main content

Number of requests per second taken care by glassfish

26 replies [Last post]
sanjeev_any
Offline
Joined: 2008-03-27

Hi,

I've setup a glassfish server and deployed an application. When one of the
tester is helping me on testing it, he observed that when he is sending 3
requests per second for about 10 times, the server hangs. Each request
takes about 4 to 5 seconds to respond.

While some background processes run during the hang, but the web pages fail
to respond.

After I changed the settings in domain.xml like increasing the acceptor-threads,
keep-alive settings etc, that problem is gone and test was successful until 4
requests per second for 10 times. However, if 5 requests per second is
requested for 10 times, the problem repeats.

When I continued increasing the above settings, 5 rps for 10 times worked fine,
but 6rps for 10 times started failing with "Too many open files" error. The ulimit
for the root user is unlimited, the file-max setting on linux machine is increased
to 655350, but still I get the same error!

The error is typically coming, I guess is, because there are too many open
connections. When I do a netstat -na | grep 8080 | wc -l, I get about 183 by
the time I receive the "Too many open files".

Is there anything I could do about this?
Kindly advice.

Kind Regards,
Sanjeev

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
sanjeev_any
Offline
Joined: 2008-03-27

Hello Hammoud,

Thank you for the references and update.

> This exception occured on my old system with
> GlassFish V1.
> Weeks of search and asking ends in the "File
> Descriptor Limit" of Linux.
>
> Currently i run GlassFish V2 many month without this
> exception using following configuration:
> http://www.i-coding.de/www/en/glassfish/too-many-open-
> files.html

I have already set the file system limit to the 655350 which
is 10 times the regular one. But still the problem appears.
For MySQL, we have a config variable like open_files_limit.
I wish there is one such setting for glassfish as well.

I shall try the other two settings given in the article:
# All files in the class directory of an web application are packed to an jar file
# Better create statistic offline (AWStats)

However, I feel, they would just minimize the problem, but
do not resolve it!? :(

> > Each request takes about 4 to 5 seconds to
> respond.
>
> Sounds like same problem i posted some days before:
> http://forums.java.net/jive/thread.jspa?messageID=2795
> 71

Incidentally, I was using almost the same h/w, s/w configuration.

Hello Jeanfrancois, Can I use the patch you have mentioned
in that post, just in case ;)

Kind Regards,
Sanjeev

sanjeev_any
Offline
Joined: 2008-03-27

Please find attached the other set of documents

sanjeev_any
Offline
Joined: 2008-03-27

I got little delayed before capturing this final netstat. Please find it attached.
So, by the time, I captured, some might've gotten closed already.

All the *03* files are before setting the QuickStart and *04* files are after that setting.

Thank you
Kind Regards,
Sanjeev

Jeanfrancois Arcand

Hi,

looks like a problem with connection pool:

> [#|2008-06-18T17:10:08.300+0000|WARNING|sun-appserver9.1|javax.enterprise.resource.resourceadapter|_ThreadID=52;_ThreadName=Timer-30;ConnectionPoolName=csp-pool;_RequestID=7b661f22-1076-4289-83be-d922c225e6ea;|A potential connection leak detected for connection pool csp-pool. The stack trace of the thread is provided below :
> com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToBusy(AbstractResourcePool.java:301)
> com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:778)
> com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652)
> com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136)
> com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594)
> com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443)
> com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:327)
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)

and all threads deadlock here:

> "httpSSLWorkerThread-8080-99" daemon prio=10 tid=0x08d1bc00 nid=0x30b3 waiting for monitor entry [0x436d9000..0x436db0b0]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at com.sun.enterprise.resource.AbstractResourcePool.stopConnectionLeakTracing(AbstractResourcePool.java:325)
> - waiting to lock <0x68c8fe48> (a java.lang.Object)

> "httpSSLWorkerThread-8080-98" daemon prio=10 tid=0x08d1a800 nid=0x30b2 waiting for monitor entry [0x4371a000..0x4371c130]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at com.sun.enterprise.resource.AbstractResourcePool.stopConnectionLeakTracing(AbstractResourcePool.java:325)
> - waiting to lock <0x68c8fe48> (a java.lang.Object)
> at com.sun.enterprise.resource.AbstractResourcePool.setResourceStateToFree(AbstractResourcePool.java:286)

> "httpSSLWorkerThread-8080-97" daemon prio=10 tid=0x08d19400 nid=0x30b1 waiting for monitor entry [0x4375b000..0x4375cdb0]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:752)
> - waiting to lock <0x68c36b58> (a com.sun.enterprise.resource.AssocWithThreadResourcePool)
> at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:652)
> at com.sun.enterprise.resource.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:136)
> at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:594)
> at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:443)

Can you file an issue here (might not be a bug, but a configuration issue):

https://glassfish.dev.java.net/servlets/ProjectIssues

and assign it to Binod?

Thanks

-- Jeanfrancois

glassfish@javadesktop.org wrote:
> I got little delayed before capturing this final netstat. Please find it attached.
> So, by the time, I captured, some might've gotten closed already.
>
> All the *03* files are before setting the QuickStart and *04* files are after that setting.
>
> Thank you
> Kind Regards,
> Sanjeev
> [Message sent by forum member 'sanjeev_any' (sanjeev_any)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281084
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

sanjeev_any
Offline
Joined: 2008-03-27

Hi,

Thank you for immediate response. Can you please guide me on which files to
attach to the issue and which problem to mention (Or should I replicate this
thread?).

Also on the Issue page, it says:
"To enter an issue, you must first be a project member and know the component
you want to report on."

Unfortunately, I'm not a project member yet! What needs to be done to become
one and file the issue!?

Kind Regards,
Sanjeev

Jeanfrancois Arcand

Hi,

glassfish@javadesktop.org wrote:
> Hi,
>
> Thank you for immediate response. Can you please guide me on which files to
> attach to the issue and which problem to mention (Or should I replicate this
> thread?).

Just paste this link, which has all the attachment:

http://forums.java.net/jive/thread.jspa?messageID=281097

>
> Also on the Issue page, it says:
> "To enter an issue, you must first be a project member and know the component
> you want to report on."
>
> Unfortunately, I'm not a project member yet! What needs to be done to become
> one and file the issue!?

I think the only way is to register for a java.net id. It is quite
simple :-)

A+

--Jeanfrancois

>
> Kind Regards,
> Sanjeev
> [Message sent by forum member 'sanjeev_any' (sanjeev_any)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281097
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

sanjeev_any
Offline
Joined: 2008-03-27

Hi,

Yep! I've filed the issue as instructed by you:
https://glassfish.dev.java.net/issues/show_bug.cgi?id=5179

> I think the only way is to register for a java.net
> id. It is quite
> simple :-)

:) yes, fortunately, it has taken the same ID as this forum.
It didn't even ask me to login.

Thank you,
Kind Regards,
Sanjeev

sanjeev_any
Offline
Joined: 2008-03-27

Hi,

I also found couple interesting errors in the log. I hope they help:

|RMI TCP Accept-8686: accept loop for ServerSocket[addr=/0.0.0.0,port=0,localport=8686] throws
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)

And also this strange error (may be due to `too many open files` error)
|"DPL8011: autodeployment failure while deploying the application : null"|#]

[#|2008-06-18T02:04:35.311+0000|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=85;_ThreadName=Timer-1;_RequestID=5c13a8bd-945b-4e81-8b16-e83c1f86df8b;|
java.lang.NullPointerException
at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)

Kind Regards,
Sanjeev

Jeanfrancois Arcand

Salut,

can you attach to this email the output of:

+ jstack PID > dumt.txt
+ netstat -an | grep <> (which is probably 8080).

Also can you try the same test, but this time adding in domain.xml:

-Dcom.sun.enterprise.server.ss.ASQuickStartup=false

It shouldn't make any difference, but just double checking.

A+

-- Jeanfrancois

glassfish@javadesktop.org wrote:
> Hi,
>
> I also found couple interesting errors in the log. I hope they help:
>
> |RMI TCP Accept-8686: accept loop for ServerSocket[addr=/0.0.0.0,port=0,localport=8686] throws
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
>
>
> And also this strange error (may be due to `too many open files` error)
> |"DPL8011: autodeployment failure while deploying the application : null"|#]
>
> [#|2008-06-18T02:04:35.311+0000|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=85;_ThreadName=Timer-1;_RequestID=5c13a8bd-945b-4e81-8b16-e83c1f86df8b;|
> java.lang.NullPointerException
> at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
>
> Kind Regards,
> Sanjeev
> [Message sent by forum member 'sanjeev_any' (sanjeev_any)]
>
> http://forums.java.net/jive/thread.jspa?messageID=280902
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

sanjeev_any
Offline
Joined: 2008-03-27

Hello Jeanfrancois,

Thank you very much for the response.
Please find attached the respective files.

Interestingly, after setting the ASQuickStart, `too many files`
error didn't occur but the server stopped responding.

Thank you for taking time.
Kind Regards,
Sanjeev

sanjeev_any
Offline
Joined: 2008-03-27

Please find the second set of attachments:

hammoud
Offline
Joined: 2005-06-20

> "Too many open files"

This exception occured on my old system with GlassFish V1.
Weeks of search and asking ends in the "File Descriptor Limit" of Linux.

Currently i run GlassFish V2 many month without this exception using following configuration:
http://www.i-coding.de/www/en/glassfish/too-many-open-files.html

> Each request takes about 4 to 5 seconds to respond.

Sounds like same problem i posted some days before:
http://forums.java.net/jive/thread.jspa?messageID=279571

Jeanfrancois Arcand

Salut

glassfish@javadesktop.org wrote:
>> "Too many open files"
>
> This exception occured on my old system with GlassFish V1.
> Weeks of search and asking ends in the "File Descriptor Limit" of Linux.
>
> Currently i run GlassFish V2 many month without this exception using following configuration:
> http://www.i-coding.de/www/en/glassfish/too-many-open-files.html
>
>
>> Each request takes about 4 to 5 seconds to respond.
>
> Sounds like same problem i posted some days before:
> http://forums.java.net/jive/thread.jspa?messageID=279571
> [Message sent by forum member 'hammoud' (hammoud)]

Well, are you using a database? Clearly the problem with this thread is
the database seems to be the bottleneck, as all threads gets blocked
trying to get a connection.

A+

- Jeanfrancois

>
> http://forums.java.net/jive/thread.jspa?messageID=281170
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

> Well, are you using a database?
yes - MySQL

> Clearly the problem with this thread is the database seems to be the bottleneck, as all threads gets blocked trying to get a connection.

I think not so - because:
1. Application uses only very simple SQL
2. MySQL slow query log does not show anything
3. Connections does not reach max
4. App-Performance log shows it occurs on first call of getParameter

Dont know if its GlassFish issue or too small OS resources (RAM or something)
But when two different persons report nearly same - then we all should try to find limits/reasons for this.

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
>> Well, are you using a database?
> yes - MySQL
>
>
>> Clearly the problem with this thread is the database seems to be the bottleneck, as all threads gets blocked trying to get a connection.
>
> I think not so - because:
> 1. Application uses only very simple SQL
> 2. MySQL slow query log does not show anything
> 3. Connections does not reach max
> 4. App-Performance log shows it occurs on first call of getParameter

How do you measure it?

>
>
> Dont know if its GlassFish issue or too small OS resources (RAM or something)
> But when two different persons report nearly same - then we all should try to find limits/reasons for this.

Agree. Except in that case really seems to be related to database. Your
issue is strange, but technically it can happens when bytes aren't
pushed fast enough from the OS/JDK. Do you think you can come with a
test case? This is quite hard for me to help unless I can see a jstack
or something that I can play with it.

Thanks

-- Jeanfrancois

> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281181
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

Hi,

> How do you measure it?

2. MySQL has option for logging long running sql
3. See it in MySQL administrator tool
4. Just timestamp compare in controller servlet

> Do you think you can come with a test case

It occurs on production server, so think it would be neccessary to build test case, which sends also much request.
Think i am not familiar enough with that things.

> Your issue is strange

Yes i know - really strange.
Perhaps you could exclude glassfish or something depending to jstack.
Would be nice, if you can view the attached file.
i dont understand the messages:)

thanx,
Hammoud

Jeanfrancois Arcand

Sa;ut,

glassfish@javadesktop.org wrote:
> Hi,
>
>
>> How do you measure it?
>
> 2. MySQL has option for logging long running sql
> 3. See it in MySQL administrator tool
> 4. Just timestamp compare in controller servlet
>
>
>> Do you think you can come with a test case
>
> It occurs on production server, so think it would be neccessary to build test case, which sends also much request.
> Think i am not familiar enough with that things.
>
>
>> Your issue is strange
>
> Yes i know - really strange.
> Perhaps you could exclude glassfish or something depending to jstack.
> Would be nice, if you can view the attached file.
> i dont understand the messages:)

This is showing exactly what I was having in mind, and a very different
problem than the database locking issue:

> "httpSSLWorkerThread-80-73" daemon prio=10 tid=0x00002aaae97fb000 nid=0x84e runnable [0x000000004aed5000..0x000000004aed6b80]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaac0ddeda8> (a sun.nio.ch.Util$1)
> - locked <0x00002aaac0dded90> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaac0ddea00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at com.sun.enterprise.web.connector.grizzly.ByteBufferInputStream.doRead(ByteBufferInputStream.java:259)
> at com.sun.enterprise.web.connector.grizzly.ByteBufferInputStream.read(ByteBufferInputStream.java:167)
> at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:743)
> at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:772)
> at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:139)
> at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:702)
> at org.apache.coyote.Request.doRead(Request.java:465)
> at org.apache.coyote.tomcat5.InputBuffer.realReadBytes(InputBuffer.java:326)
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:395)
> at org.apache.coyote.tomcat5.InputBuffer.read(InputBuffer.java:341)
> at org.apache.coyote.tomcat5.CoyoteInputStream.read(CoyoteInputStream.java:247)
> at org.apache.coyote.tomcat5.CoyoteRequest.readPostBody(CoyoteRequest.java:3008)
> at org.apache.coyote.tomcat5.CoyoteRequest.getPostBody(CoyoteRequest.java:2990)
> at com.sun.enterprise.web.connector.coyote.PwcCoyoteRequest.getPostBody(PwcCoyoteRequest.java:356)
> at org.apache.coyote.tomcat5.CoyoteRequest.parseRequestParameters(CoyoteRequest.java:2959)
> at org.apache.coyote.tomcat5.CoyoteRequest.getParameter(CoyoteRequest.java:1281)
> at org.apache.coyote.tomcat5.CoyoteRequestFacade.getParameter(CoyoteRequestFacade.java:392)
> at de.hp.core.internet.CMultiClick.(Unknown Source)

During the parsing of the http message, calling getParameters() seems to
requires more bytes that has been read so far. Hence Grizzly try to read
the missing parameters bytes and waits for the client to push those
bytes. What it means here is the client is not sending the bytes as
expected (as fast as expected). Do you have any proxy in between? Or
does the network get flooded? This doesn't look like a bug in GlassFish,
but really means that all the required bytes are send really slowly from
the client side.

Can you do a test? Can you add the following properties in domain.xml:

-Dcom.sun.enterprise.web.connector.grizzly.readTimeout=1000

and see if the getParameters() performance improve? I suspect you might
get/see IOException as the http parser will not get all the bytes it
require because it will stop waiting for all required bytes. But let's
see :-).

Thanks

-- Jeanfrancois

>
>
> thanx,
> Hammoud
> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281369
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

Hi,

> Do you have any proxy in between? Or does the network get flooded? This doesn't look like a bug in GlassFish, but really means that all the required bytes are send really slowly from

Really great that you can interpret this data:)

Before i try the new property, want to say 90% of the request are done by mobile phone (WAP).
Other application on this server, which are only connected per PC, do not show so much slow requests.

Could it be, that its just a normal case for mobile phone requests.
Requests come from all over the world, perhaps there is an gateway between which just slows it down, in some countries?

Also if the above statement could be reason for it.
Should i try your property then?

thanx for help,
Hammoud

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
> Hi,
>
>
>> Do you have any proxy in between? Or does the network get flooded? This doesn't look like a bug in GlassFish, but really means that all the required bytes are send really slowly from
>
> Really great that you can interpret this data:)
>
> Before i try the new property, want to say 90% of the request are done by mobile phone (WAP).
> Other application on this server, which are only connected per PC, do not show so much slow requests.
>

Bingo :-) This is a network related problem, and unfortunately there is
little we can do.

> Could it be, that its just a normal case for mobile phone requests.

Yes, based on the which networks they run on, it can explain why the
getParameter is taking time. What I suspect is the WAP protocol (and the
client) send the minimal bytes over the wire (which isn't including the
parameters). Grizzly parse the request and then as soon as the
application invoke getParameters(), the client then and only then send
the remaining bytes. I'm guessing here...but depending on the network,
it might takes a couple of seconds to get the remaining data. The client
is probably sending the minimal requests bytes to save bandwidth on the
first request.

> Requests come from all over the world, perhaps there is an gateway between which just slows it down, in some countries?
>

Yes that's probably the reason.

>
> Also if the above statement could be reason for it.
> Should i try your property then?

No, as it will breaks some remote client as Grizzly will close the
connection, thinking the client is doing a denial of service (sending
incomplete messages). The default time out (30 seconds) is just fine for
WAP I'm pretty sure.

>
>
> thanx for help,

You are welcome.

-- Jeanfrancois

> Hammoud
> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281470
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

> What I suspect is the WAP protocol (and the client) send the minimal bytes over the wire (which isn't including the parameters)

Extend my performance logger with url data and it confirmed that.
The long running request are mostly WAP POST request, so probably post params will be send later if the are invoked by getParameter.

> The default time out (30 seconds) is just fine for WAP I'm pretty sure.

Not shure about that, does above not mean:
1. An GlassFish thread, which does the communication with the WAP-Client, will be closed even if the request is not processed totally?
2. If there are lot of WAP-Requests, then there are also many GlassFish threads opend and waiting. Should i increase some max thread value to have always possible free?

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
>> What I suspect is the WAP protocol (and the client) send the minimal bytes over the wire (which isn't including the parameters)
>
> Extend my performance logger with url data and it confirmed that.
> The long running request are mostly WAP POST request, so probably post params will be send later if the are invoked by getParameter.
>
>
>
>> The default time out (30 seconds) is just fine for WAP I'm pretty sure.
>
> Not shure about that, does above not mean:
> 1. An GlassFish thread, which does the communication with the WAP-Client, will be closed even if the request is not processed totally?

Yes. If you change the readTimeout value, it might happens. Imagine a
hacker that send one byte every 60 seconds....all the threads will lock
trying to read this slow client and that will create a denial of
service. Hence we put a max value the client can takes to send its
bytes. In your case, reducing the value will close connection and some
of your client will be broken.

> 2. If there are lot of WAP-Requests, then there are also many GlassFish threads opend and waiting. Should i increase some max thread value to have always possible free?

Yes that will help adding more threads as client are "slowly" sending
bytes. HAving a couple more will make other request faster because they
will not be queued.

A+

-- Jeanfrancois

> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=281996
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

> In your case, reducing the value will close connection and some of your client will be broken.

Some of that long running WAP requests show processing time of >= 30s.
So i did not thought about reducing the request timeout, i thougth about increasing it, to possible do correct response to such longer request some seconds later.

> Imagine a hacker that send one byte every 60 seconds....all the threads will lock
trying to read this slow client and that will create a denial of service. Hence we put a max value the client can takes to send its bytes.

Great that such things are included:)
I will follow your advice at remain the 30s default timeout.

> Having a couple more will make other request faster because they will not be queued.

The maximum number of request processing threads is currently set to 130.
Can i see if/when i need to increase this, perhaps some exception message or so.
Or do i have to calculate this value depending on page impressions+static requests and average processsing time or something like that.

Jeanfrancois Arcand

Salut,

glassfish@javadesktop.org wrote:
>> In your case, reducing the value will close connection and some of your client will be broken.
>
> Some of that long running WAP requests show processing time of >= 30s.
> So i did not thought about reducing the request timeout, i thougth about increasing it, to possible do correct response to such longer request some seconds later.
>
>
>> Imagine a hacker that send one byte every 60 seconds....all the threads will lock
> trying to read this slow client and that will create a denial of service. Hence we put a max value the client can takes to send its bytes.
>
> Great that such things are included:)
> I will follow your advice at remain the 30s default timeout.

Yes keep it 30 seconds unless you really see that connection are closed
too fast.

>
>
>> Having a couple more will make other request faster because they will not be queued.
>
> The maximum number of request processing threads is currently set to 130.
> Can i see if/when i need to increase this, perhaps some exception message or so.

This is not simple to adjust, but I would not go over 130. I would
instead increase the request queue size so requests that are pooled
(when all the threads are used) are never expired/rejected:

receive-buffer-size-in-bytes="4096" send-buffer-size-in-bytes="8192"/>

Increase the max-pending-count to a large value (this is the size of the
request queue).

> Or do i have to calculate this value depending on page impressions+static requests and average processsing time or something like that.

Yes, that can help but with NIO, I suspect you might get better stats if
you decrease the number of threads. But I might be wrong...it is really
based on your traffic :-)

Thanks!

-- Jeanfrancois

> [Message sent by forum member 'hammoud' (hammoud)]
>
> http://forums.java.net/jive/thread.jspa?messageID=282829
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

hammoud
Offline
Joined: 2005-06-20

> Increase the max-pending-count to a large value

try now 8192.
thanx for your detailed help.

Jeanfrancois Arcand

Hi,

glassfish@javadesktop.org wrote:
> Hi,
>
> I've setup a glassfish server and deployed an application. When one of the
> tester is helping me on testing it, he observed that when he is sending 3
> requests per second for about 10 times, the server hangs. Each request
> takes about 4 to 5 seconds to respond.
>
> While some background processes run during the hang, but the web pages fail
> to respond.
>
> After I changed the settings in domain.xml like increasing the acceptor-threads,
> keep-alive settings etc, that problem is gone and test was successful until 4
> requests per second for 10 times. However, if 5 requests per second is
> requested for 10 times, the problem repeats.

Instead of changing the acceptor-threads -- leave it as 1 because there
was an issue:

https://glassfish.dev.java.net/issues/show_bug.cgi?id=4945

change the

and the problem you are experiencing will go away.

Thanks

-- Jeanfrancois

>
> When I continued increasing the above settings, 5 rps for 10 times worked fine,
> but 6rps for 10 times started failing with "Too many open files" error. The ulimit
> for the root user is unlimited, the file-max setting on linux machine is increased
> to 655350, but still I get the same error!
>
> The error is typically coming, I guess is, because there are too many open
> connections. When I do a netstat -na | grep 8080 | wc -l, I get about 183 by
> the time I receive the "Too many open files".
>
> Is there anything I could do about this?
> Kindly advice.
>
> Kind Regards,
> Sanjeev
> [Message sent by forum member 'sanjeev_any' (sanjeev_any)]
>
> http://forums.java.net/jive/thread.jspa?messageID=280809
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

sanjeev_any
Offline
Joined: 2008-03-27

Hello Jeanfrancois,

Thank you very much for the response. Its very nice of you.
I have made changes as per your suggestion. I still get the
"Too many open files error" :(, this time for 5rps 10 times.
But this time, it might not be the problem with sockets!

As mentioned earlier, the Os's max open file limit is to the
max. Is there any way to set max open file limit just for glassfish
within glassfish!?

javax.xml.ws.WebServiceException: Failed to access the WSDL at:
http://localhost:8080/Test/TestService?wsdl. It failed with:
Too many open files.
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:162)

I'd be glad if you have any pointers to share with me on my two
other posts:
http://forums.java.net/jive/thread.jspa?threadID=42563&tstart=0
http://forums.java.net/jive/thread.jspa?threadID=42564&tstart=0

Thank you very much,
Kind regards,
Sanjeev