Skip to main content

glassfish 3.1.2.2 (build 5) - default install - problem with keeping too many connections in the queue, while system usage is low

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
7 replies [Last post]
mladenadamovic
Offline
Joined: 2009-04-30

Hi guys,

I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
and I'm having problems.

The problem I'm encountering is that after a while, many connections
wait in connection queue and it even starts to refuse connections, but
system usage is low.
#uptime
20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24

If I test the website with #ab -n 1000 -c 5
http://www.numbeo.com/cost-of-living/
it gets around 980 failures out of 1000.
I'm playing with config files, but couldn't figure out where is the problem.
ulimit -n
is 1024,cannot rise it, since running in Virtuozzo enviroment.

These are parts of my domain.xml:

...

-XX:MaxPermSize=192m
-XX:PermSize=64m
-Djava.awt.headless=true
-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
-XX:+UnlockDiagnosticVMOptions
-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dosgi.shell.telnet.port=6666
-Dosgi.shell.telnet.maxconn=1
-Dosgi.shell.telnet.ip=127.0.0.1
-Dgosh.args=--nointeractive
-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
-Dfelix.fileinstall.poll=5000
-Dfelix.fileinstall.log.level=2
-Dfelix.fileinstall.bundles.new.start=true
-Dfelix.fileinstall.bundles.startTransient=true
-Dfelix.fileinstall.disableConfigSave=false
-XX:NewRatio=2
-server
-Xmx2048m
-Xms1024m
-Dproduct.name=

Anyone knows what might be the problem?

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
momaison
Offline
Joined: 2008-09-29

Hello,

From our experiments, I confirm that 1024 files is not sufficient
if your server is under load, probably because of keep-alive
connections.
We got a valuable error message in server log though : something
like 'too many open files' and the server was freezed (did not
answer to requests ; this is not what you observe however).

In case, you may watch the currently opened file descriptors with
something like lsof | grep | wc
or ls /proc/{pid}/fd | wc -l (under linux)

Increasing the ulimit -n to 4096 did not solve our problem.
We had to set the (badly named) parameter "Maximum number of
connections in keep-alive mode" to -1 (or 0 or 1 I don't remember
exactly), so that connections are always closed server-side. This
is not a big concern for our application.

Regards,

M. Maison

On 19/07/2012 20:34, Mladen Adamovic wrote:
> Hi guys,
>
> I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
> and I'm having problems.
>
> The problem I'm encountering is that after a while, many connections
> wait in connection queue and it even starts to refuse connections, but
> system usage is low.
> #uptime
> 20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24
>
> If I test the website with #ab -n 1000 -c 5
> http://www.numbeo.com/cost-of-living/
> it gets around 980 failures out of 1000.
> I'm playing with config files, but couldn't figure out where is the problem.
> ulimit -n
> is 1024,cannot rise it, since running in Virtuozzo enviroment.
>

mladenadamovic
Offline
Joined: 2009-04-30

Hi Mo,

thanks for your message. It looks like that when I turn keep alive to 1,
that many connections remain in TIME_WAIT mode.
It might be beneficial for investigation if you can write how many hits do
you have per second and average processing time of request in ms (both
value from monitoring).

Thanks and cheers

On Fri, Jul 20, 2012 at 10:24 PM, Mo Maison wrote:

>
> Hello,
>
> From our experiments, I confirm that 1024 files is not sufficient
> if your server is under load, probably because of keep-alive
> connections.
> We got a valuable error message in server log though : something
> like 'too many open files' and the server was freezed (did not
> answer to requests ; this is not what you observe however).
>
> In case, you may watch the currently opened file descriptors with
> something like lsof | grep | wc
> or ls /proc/{pid}/fd | wc -l (under linux)
>
> Increasing the ulimit -n to 4096 did not solve our problem.
> We had to set the (badly named) parameter "Maximum number of
> connections in keep-alive mode" to -1 (or 0 or 1 I don't remember
> exactly), so that connections are always closed server-side. This
> is not a big concern for our application.
>
> Regards,
>
> M. Maison
>
>
>
> On 19/07/2012 20:34, Mladen Adamovic wrote:
>
>> Hi guys,
>>
>> I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
>> and I'm having problems.
>>
>> The problem I'm encountering is that after a while, many connections
>> wait in connection queue and it even starts to refuse connections, but
>> system usage is low.
>> #uptime
>> 20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24
>>
>> If I test the website with #ab -n 1000 -c 5
>> http://www.numbeo.com/cost-of-**living/
>> it gets around 980 failures out of 1000.
>> I'm playing with config files, but couldn't figure out where is the
>> problem.
>> ulimit -n
>> is 1024,cannot rise it, since running in Virtuozzo enviroment.
>>
>>

--
Mladen Adamovic
Numbeo
Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
(Business Registration Number 62612240)
Tel. +381-66-058-595
email: mladen.adamovic@gmail.com
web: http://www.numbeo.com

oleksiys
Offline
Joined: 2006-01-25

Hi Mladen,

by "connections wait in connection queue" you mean statistics you get
using netstat with many connections in WAIT state?
If you run GF 2.1 on the same environment you don't see this?

Thanks.

WBR,
Alexey.

On 07/19/2012 08:34 PM, Mladen Adamovic wrote:
> Hi guys,
>
> I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
> and I'm having problems.
>
> The problem I'm encountering is that after a while, many connections
> wait in connection queue and it even starts to refuse connections, but
> system usage is low.
> #uptime
> 20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24
>
> If I test the website with #ab -n 1000 -c 5
> http://www.numbeo.com/cost-of-living/
> it gets around 980 failures out of 1000.
> I'm playing with config files, but couldn't figure out where is the problem.
> ulimit -n
> is 1024,cannot rise it, since running in Virtuozzo enviroment.
>
> These are parts of my domain.xml:
>
> ...
>
>
>
> default-web-module="NonWwwNumbeo" sso-cookie-http-only="false"
> network-listeners="http-listener-1">
> ww2.numbeo.com" default-web-module="LivingCost"
> sso-cookie-http-only="false"
> network-listeners="http-listener-1">
>
>
>
> max-connections="250">
>
>
>
>
> transport="tcp" name="http-listener-1"
> thread-pool="http-thread-pool">
>
> name="http-thread-pool">
>
>
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"
> system-classpath="" classpath-suffix="">
> -XX:MaxPermSize=192m
> -XX:PermSize=64m
> -Djava.awt.headless=true
> -Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
> -XX:+UnlockDiagnosticVMOptions
> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
> -DANTLR_USE_DIRECT_CLASS_LOADING=true
> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
> -Dosgi.shell.telnet.port=6666
> -Dosgi.shell.telnet.maxconn=1
> -Dosgi.shell.telnet.ip=127.0.0.1
> -Dgosh.args=--nointeractive
> -Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
> -Dfelix.fileinstall.poll=5000
> -Dfelix.fileinstall.log.level=2
> -Dfelix.fileinstall.bundles.new.start=true
> -Dfelix.fileinstall.bundles.startTransient=true
> -Dfelix.fileinstall.disableConfigSave=false
> -XX:NewRatio=2
> -server
> -Xmx2048m
> -Xms1024m
> -Dproduct.name=
>
>
> Anyone knows what might be the problem?
>
>

mladenadamovic
Offline
Joined: 2009-04-30

Hi Oleksiy,

unfortunatelly, I haven't used netstat to test that, I assumed that by
looking what is happening and by using glassfish monitoring. Perhaps I
was wrong, but I needed to describe what is going on, and I was tried
my best.

Later during the day I was curious what is different in config at that
webserver comparing to another webserver, and changed
httpthreadpool Max Thread Pool Size: 400
default value is 5.
this value of 400 threads, was choosen so that webserver hopefully
doesn't run out of 1024 network descriptors it has most.

I'd like to bring you to your attention the fishy value
Network Listeners -> http-listener-1 of 250 max connections.
The text near max connections states: Maximum number of connections in
keep-alive mode
but it is actually according to Oracle.com documentation at the
website http://docs.oracle.com/cd/E19879-01/820-4343/abefr/index.html
"Max Connections controls the number of requests that a particular
client can make over a keep-alive connection. "

So glassfish console text "Maximum number of connections in keep-alive
mode" seems incorrect, it shall be written "Maximum number of requests
per connection in keep-alive mode", which is different. Perhaps
someone can check in the source code what does that value actually do
and change the description in admin console and online doc so it's not
ambiguous?

If it's actually the number of connections in keep-alive mode,
keep-alive connections would take 250 threads and only 5 is provided
by default config.

With max threads set to 400,

after half day monitoring
Jul 19, 2012 6:08:48 PM Jul 20, 2012 9:42:38 AM

By looking into monitoring now, I can see that many connections looks queued
CountQueued15MinutesAverage 6218 count
CountQueued1MinuteAverage 294 count
CountOpenConnections 62 count
CountTotalConnections 153786 count
CurrentThreadCount 66 count

but still the load at the server is very low:
# uptime
11:43:23 up 16:06, 1 user, load average: 0.02, 0.06, 0.00

Could you explain me why so many connections have been queued in last
15 minutes and load at the server has been 0.06? (Virtuzzo VPS with
2x1.5 GHz guaranteed).
Does make sense unless OpenConnections were MaxConnections in the past
15 minutes? Or perhaps due to the fact I use only 1 acceptor thread?

Processing time is very low ProcessingTime 3 millisecond
but MaxTime is 3942 millisecond

Again, still, why so many connections hav been queued in last 15
minutes and load at the server was 0.06? Looks fishy!
Perhaps Google uses keep alive connections and max number of keep
alive connections is indeed that (not requests per connection), so
some Googlebot's simultanously keep-alive connection end up in the
queue.
CountHits 294527 count for a half day.

Regards

On Fri, Jul 20, 2012 at 10:59 AM, Oleksiy Stashok
wrote:
> Hi Mladen,
>
> by "connections wait in connection queue" you mean statistics you get using
> netstat with many connections in WAIT state?
> If you run GF 2.1 on the same environment you don't see this?
>
> Thanks.
>
> WBR,
> Alexey.
>
>
> On 07/19/2012 08:34 PM, Mladen Adamovic wrote:
>>
>> Hi guys,
>>
>> I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
>> and I'm having problems.
>>
>> The problem I'm encountering is that after a while, many connections
>> wait in connection queue and it even starts to refuse connections, but
>> system usage is low.
>> #uptime
>> 20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24
>>
>> If I test the website with #ab -n 1000 -c 5
>> http://www.numbeo.com/cost-of-living/
>> it gets around 980 failures out of 1000.
>> I'm playing with config files, but couldn't figure out where is the
>> problem.
>> ulimit -n
>> is 1024,cannot rise it, since running in Virtuozzo enviroment.
>>
>> These are parts of my domain.xml:
>>
>> ...
>>
>>
>>
>> > default-web-module="NonWwwNumbeo" sso-cookie-http-only="false"
>> network-listeners="http-listener-1">
>> > ww2.numbeo.com" default-web-module="LivingCost"
>> sso-cookie-http-only="false"
>> network-listeners="http-listener-1">
>>
>>
>>
>> > max-connections="250">
>>
>>
>>
>>
>> > transport="tcp" name="http-listener-1"
>> thread-pool="http-thread-pool">
>>
>> > name="http-thread-pool">
>>
>>
>> > -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"
>> system-classpath="" classpath-suffix="">
>> -XX:MaxPermSize=192m
>> -XX:PermSize=64m
>> -Djava.awt.headless=true
>>
>> -Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
>> -XX:+UnlockDiagnosticVMOptions
>>
>> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
>>
>> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
>>
>> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
>>
>> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
>>
>> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
>>
>> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
>>
>> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
>>
>> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>> -DANTLR_USE_DIRECT_CLASS_LOADING=true
>>
>> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>> -Dosgi.shell.telnet.port=6666
>> -Dosgi.shell.telnet.maxconn=1
>> -Dosgi.shell.telnet.ip=127.0.0.1
>> -Dgosh.args=--nointeractive
>>
>> -Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
>> -Dfelix.fileinstall.poll=5000
>> -Dfelix.fileinstall.log.level=2
>>
>> -Dfelix.fileinstall.bundles.new.start=true
>>
>> -Dfelix.fileinstall.bundles.startTransient=true
>>
>> -Dfelix.fileinstall.disableConfigSave=false
>> -XX:NewRatio=2
>> -server
>> -Xmx2048m
>> -Xms1024m
>> -Dproduct.name=
>>
>>
>> Anyone knows what might be the problem?
>>
>>

oleksiys
Offline
Joined: 2006-01-25

Hi Mladen,

On 07/20/2012 11:53 AM, Mladen Adamovic wrote:
> Later during the day I was curious what is different in config at that
> webserver comparing to another webserver, and changed
> httpthreadpool Max Thread Pool Size: 400
> default value is 5.
> this value of 400 threads, was choosen so that webserver hopefully
> doesn't run out of 1024 network descriptors it has most.
>
> I'd like to bring you to your attention the fishy value
> Network Listeners -> http-listener-1 of 250 max connections.
> The text near max connections states: Maximum number of connections in
> keep-alive mode
> but it is actually according to Oracle.com documentation at the
> website http://docs.oracle.com/cd/E19879-01/820-4343/abefr/index.html
> "Max Connections controls the number of requests that a particular
> client can make over a keep-alive connection. "
>
> So glassfish console text "Maximum number of connections in keep-alive
> mode" seems incorrect, it shall be written "Maximum number of requests
> per connection in keep-alive mode", which is different. Perhaps
> someone can check in the source code what does that value actually do
> and change the description in admin console and online doc so it's not
> ambiguous?
You're absolutely right. Can you pls. file an issue?

> If it's actually the number of connections in keep-alive mode,
> keep-alive connections would take 250 threads and only 5 is provided
> by default config.
No, it's the max number of keep-alive requests allowed on a single
connection.

> With max threads set to 400,
>
> after half day monitoring
> Jul 19, 2012 6:08:48 PM Jul 20, 2012 9:42:38 AM
>
> By looking into monitoring now, I can see that many connections looks queued
> CountQueued15MinutesAverage 6218 count
> CountQueued1MinuteAverage 294 count
> CountOpenConnections 62 count
> CountTotalConnections 153786 count
> CurrentThreadCount 66 count
>
> but still the load at the server is very low:
> # uptime
> 11:43:23 up 16:06, 1 user, load average: 0.02, 0.06, 0.00
>
> Could you explain me why so many connections have been queued in last
> 15 minutes and load at the server has been 0.06? (Virtuzzo VPS with
> 2x1.5 GHz guaranteed).
Hmm, it's probably terminology problem again. When request comes on a
connection we put connection to a queue to process its request.
So number of time connection was queued is more or less equivalent to a
number HTTP requests processed (it might not be true for HTTP pipelined
requests).

Hope it helped.

WBR,
Alexey.

> Again, still, why so many connections hav been queued in last 15
> minutes and load at the server was 0.06? Looks fishy!
> Perhaps Google uses keep alive connections and max number of keep
> alive connections is indeed that (not requests per connection), so
> some Googlebot's simultanously keep-alive connection end up in the
> queue.
> CountHits 294527 count for a half day.
>
> Regards
>
>
> On Fri, Jul 20, 2012 at 10:59 AM, Oleksiy Stashok
> wrote:
>> Hi Mladen,
>>
>> by "connections wait in connection queue" you mean statistics you get using
>> netstat with many connections in WAIT state?
>> If you run GF 2.1 on the same environment you don't see this?
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>
>> On 07/19/2012 08:34 PM, Mladen Adamovic wrote:
>>> Hi guys,
>>>
>>> I migrated some of my websites from glassfish 2.1 to glassfish 3.1.2.2
>>> and I'm having problems.
>>>
>>> The problem I'm encountering is that after a while, many connections
>>> wait in connection queue and it even starts to refuse connections, but
>>> system usage is low.
>>> #uptime
>>> 20:28:58 up 51 min, 1 user, load average: 0.09, 0.14, 0.24
>>>
>>> If I test the website with #ab -n 1000 -c 5
>>> http://www.numbeo.com/cost-of-living/
>>> it gets around 980 failures out of 1000.
>>> I'm playing with config files, but couldn't figure out where is the
>>> problem.
>>> ulimit -n
>>> is 1024,cannot rise it, since running in Virtuozzo enviroment.
>>>
>>> These are parts of my domain.xml:
>>>
>>> ...
>>>
>>>
>>>
>>> >> default-web-module="NonWwwNumbeo" sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> ww2.numbeo.com" default-web-module="LivingCost"
>>> sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>>
>>>
>>>
>>> >> max-connections="250">
>>>
>>>
>>>
>>>
>>> >> transport="tcp" name="http-listener-1"
>>> thread-pool="http-thread-pool">
>>>
>>> >> name="http-thread-pool">
>>>
>>>
>>> >> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"
>>> system-classpath="" classpath-suffix="">
>>> -XX:MaxPermSize=192m
>>> -XX:PermSize=64m
>>> -Djava.awt.headless=true
>>>
>>> -Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
>>> -XX:+UnlockDiagnosticVMOptions
>>>
>>> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
>>>
>>> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
>>>
>>> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
>>>
>>> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
>>>
>>> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
>>>
>>> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
>>>
>>> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
>>>
>>> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>>> -DANTLR_USE_DIRECT_CLASS_LOADING=true
>>>
>>> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>>> -Dosgi.shell.telnet.port=6666
>>> -Dosgi.shell.telnet.maxconn=1
>>> -Dosgi.shell.telnet.ip=127.0.0.1
>>> -Dgosh.args=--nointeractive
>>>
>>> -Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
>>> -Dfelix.fileinstall.poll=5000
>>> -Dfelix.fileinstall.log.level=2
>>>
>>> -Dfelix.fileinstall.bundles.new.start=true
>>>
>>> -Dfelix.fileinstall.bundles.startTransient=true
>>>
>>> -Dfelix.fileinstall.disableConfigSave=false
>>> -XX:NewRatio=2
>>> -server
>>> -Xmx2048m
>>> -Xms1024m
>>> -Dproduct.name=
>>>
>>>
>>> Anyone knows what might be the problem?
>>>
>>>
>>> -------------------------------------
>>> And here is the full domain.xml (in the case I missed something):
>>> >> application-root="${com.sun.aas.instanceRoot}/applications"
>>> version="5">
>>>
>>> >>
>>> location="${com.sun.aas.installRootURI}/lib/install/applications/__admingui"
>>> name="__admingui" directory-deployed="true"
>>> object-type="system-admin">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> location="${com.sun.aas.instanceRootURI}/applications/OpenRewrite/"
>>> name="OpenRewrite" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/OpenRewrite/OpenRewrite.war">
>>> >> value="false">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> location="${com.sun.aas.instanceRootURI}/applications/DecisionCrowd/"
>>> name="DecisionCrowd" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/DecisionCrowd/DecisionCrowd.war">
>>> >> value="false">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> location="${com.sun.aas.instanceRootURI}/applications/NonWwwNumbeo/"
>>> name="NonWwwNumbeo" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwNumbeo/NonWwwNumbeo.war">
>>> >> value="false">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> location="${com.sun.aas.instanceRootURI}/applications/NonWwwOpenRewrite/"
>>> name="NonWwwOpenRewrite" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwOpenRewrite/NonWwwOpenRewrite.war">
>>> >> value="false">
>>> >> value="NonWwwOpenRewrite">
>>>
>>>
>>>
>>>
>>>
>>> >>
>>> location="${com.sun.aas.instanceRootURI}/applications/NonWwwDecisionCrowd/"
>>> name="NonWwwDecisionCrowd" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwDecisionCrowd/NonWwwDecisionCrowd.war">
>>> >> value="false">
>>> >> value="NonWwwDecisionCrowd">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> location="${com.sun.aas.instanceRootURI}/applications/LivingCost/"
>>> name="LivingCost" object-type="user">
>>> >>
>>> value="${com.sun.aas.instanceRootURI}/applications/__internal/LivingCost/LivingCost.war">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> jndi-name="jdbc/__TimerPool"
>>> object-type="system-admin">
>>> >> jndi-name="jdbc/__default">
>>> >> datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource"
>>> res-type="javax.sql.XADataSource" name="__TimerPool">
>>> >> value="${com.sun.aas.instanceRoot}/lib/databases/ejbtimer">
>>> >> value=";create=true">
>>>
>>> >> datasource-classname="org.apache.derby.jdbc.ClientDataSource"
>>> res-type="javax.sql.DataSource" name="DerbyPool">
>>>
>>>
>>>
>>>
>>> >> value="sun-appserv-samples">
>>> >> value=";create=true">
>>>
>>>
>>>
>>>
>>> >> virtual-servers="__asadmin">
>>> >> virtual-servers="www.openrewrite.com">
>>> >> virtual-servers="www.decisioncrowd.com">
>>> >> virtual-servers="numbeo.com">
>>> >> virtual-servers="openrewrite.com">
>>> >> virtual-servers="decisioncrowd.com">
>>> >> virtual-servers="www.numbeo.com">
>>>
>>>
>>>
>>>
>>>
>>> >> install-dir="${com.sun.aas.productRoot}">
>>>
>>>
>>>
>>>
>>>
>>> >> network-listeners="http-listener-1,http-listener-2">
>>> >> network-listeners="admin-listener">
>>> >> default-web-module="NonWwwNumbeo" sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> ww2.numbeo.com" default-web-module="LivingCost"
>>> sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> default-web-module="NonWwwOpenRewrite" sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> hosts="www.openrewrite.com" default-web-module="OpenRewrite"
>>> sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> hosts="decisioncrowd.com" default-web-module="NonWwwDecisionCrowd"
>>> sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>> >> hosts="www.decisioncrowd.com" default-web-module="DecisionCrowd"
>>> sso-cookie-http-only="false"
>>> network-listeners="http-listener-1">
>>>
>>>
>>>
>>> >> address="0.0.0.0" lazy-init="true">
>>> >> security-enabled="true">
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as">
>>>
>>> >> address="0.0.0.0" security-enabled="true">
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as" client-auth-enabled="true">
>>>
>>>
>>> >> type="das-and-server">
>>> >> security-enabled="false" auth-realm-name="admin-realm"
>>> name="system">
>>> >> value="/admin">
>>> >>
>>> value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war">
>>> >> value="${com.sun.aas.installRoot}/..">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> session-store="${com.sun.aas.instanceRoot}/session-store">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
>>> name="admin-realm">
>>> >> value="${com.sun.aas.instanceRoot}/config/admin-keyfile">
>>>
>>>
>>> >> classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
>>> name="file">
>>> >> value="${com.sun.aas.instanceRoot}/config/keyfile">
>>>
>>>
>>> >>
>>> classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm"
>>> name="certificate">
>>> >> policy-provider="com.sun.enterprise.security.provider.PolicyWrapper"
>>> name="default"
>>> policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
>>> >> value="${com.sun.aas.instanceRoot}/generated/policy">
>>>
>>> >>
>>> policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider"
>>> name="simple"
>>> policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory">
>>> >> name="default">
>>>
>>>
>>>
>>> >> provider-id="XWS_ClientProvider"
>>> class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>> >> value="false">
>>>
>>>
>>> >> provider-id="ClientProvider"
>>> class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>> >> value="false">
>>>
>>> >>
>>> value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml">
>>>
>>> >> provider-id="XWS_ServerProvider"
>>> class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>>
>>>
>>> >> provider-id="ServerProvider"
>>> class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>>
>>> >>
>>> value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml">
>>>
>>>
>>>
>>> >> provider-id="GFConsoleAuthModule"
>>>
>>> class-name="org.glassfish.admingui.common.security.AdminConsoleAuthModule">
>>>
>>>
>>> >>
>>> value="https://localhost:${ADMIN_LISTENER_PORT}/management/sessions">
>>>
>>> >> value="/loginError.jsf">
>>>
>>>
>>> >> value="SHA-256">
>>>
>>> >> tx-log-dir="${com.sun.aas.instanceRoot}/logs">
>>> >> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"
>>> system-classpath="" classpath-suffix="">
>>> -XX:MaxPermSize=192m
>>> -XX:PermSize=64m
>>> -Djava.awt.headless=true
>>>
>>> -Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder
>>> -XX:+UnlockDiagnosticVMOptions
>>>
>>> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
>>>
>>> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
>>>
>>> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
>>>
>>> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
>>>
>>> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
>>>
>>> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
>>>
>>> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
>>>
>>> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>>> -DANTLR_USE_DIRECT_CLASS_LOADING=true
>>>
>>> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>>> -Dosgi.shell.telnet.port=6666
>>> -Dosgi.shell.telnet.maxconn=1
>>> -Dosgi.shell.telnet.ip=127.0.0.1
>>> -Dgosh.args=--nointeractive
>>>
>>> -Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
>>> -Dfelix.fileinstall.poll=5000
>>> -Dfelix.fileinstall.log.level=2
>>>
>>> -Dfelix.fileinstall.bundles.new.start=true
>>>
>>> -Dfelix.fileinstall.bundles.startTransient=true
>>>
>>> -Dfelix.fileinstall.disableConfigSave=false
>>> -XX:NewRatio=2
>>> -server
>>> -Xmx2048m
>>> -Xms1024m
>>> -Dproduct.name=
>>>
>>>
>>>
>>>
>>> >> max-connections="250">
>>>
>>>
>>>
>>>
>>> >> max-connections="250">
>>>
>>>
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as">
>>>
>>>
>>> >> default-virtual-server="__asadmin" max-connections="250"
>>> encoded-slash-enabled="true">
>>>
>>>
>>>
>>>
>>> >> encoded-slash-enabled="true">
>>>
>>>
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> name="http-finder"
>>> classname="com.sun.grizzly.config.HttpProtocolFinder">
>>> >> name="admin-http-redirect"
>>> classname="com.sun.grizzly.config.HttpProtocolFinder">
>>>
>>>
>>>
>>>
>>> >> transport="tcp" name="http-listener-1"
>>> thread-pool="http-thread-pool">
>>> >> transport="tcp" name="http-listener-2"
>>> thread-pool="http-thread-pool">
>>> >> transport="tcp" name="admin-listener"
>>> thread-pool="admin-thread-pool">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> max-thread-pool-size="50" max-queue-size="256">
>>> >> name="http-thread-pool">
>>> >> max-thread-pool-size="200">
>>>
>>>
>>> >> connector-connection-pool="HIGH" connector-service="HIGH" jvm="HIGH"
>>> thread-pool="HIGH" web-container="HIGH">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> network-listeners="http-listener-1, http-listener-2">
>>> >> value="${com.sun.aas.instanceRoot}/config/default-web.xml">
>>>
>>> >> network-listeners="admin-listener">
>>>
>>>
>>>
>>> >> id="orb-listener-1" address="0.0.0.0">
>>> >> address="0.0.0.0" security-enabled="true">
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as">
>>>
>>> >> id="SSL_MUTUALAUTH" address="0.0.0.0" security-enabled="true">
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as" client-auth-enabled="true">
>>>
>>>
>>>
>>> >> address="0.0.0.0" security-enabled="false"
>>> auth-realm-name="admin-realm" name="system">
>>> >>
>>> value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> session-store="${com.sun.aas.instanceRoot}/session-store">
>>>
>>>
>>>
>>> >> default-jms-host="default_JMS_host">
>>> >> name="default_JMS_host">
>>>
>>> >> file="${com.sun.aas.instanceRoot}/logs/server.log">
>>>
>>>
>>>
>>> >> classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
>>> name="admin-realm">
>>> >> value="${com.sun.aas.instanceRoot}/config/admin-keyfile">
>>>
>>>
>>> >> classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
>>> name="file">
>>> >> value="${com.sun.aas.instanceRoot}/config/keyfile">
>>>
>>>
>>> >>
>>> classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm"
>>> name="certificate">
>>> >> policy-provider="com.sun.enterprise.security.provider.PolicyWrapper"
>>> name="default"
>>> policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
>>> >> value="${com.sun.aas.instanceRoot}/generated/policy">
>>>
>>> >>
>>> policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider"
>>> name="simple"
>>> policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory">
>>> >> name="default">
>>>
>>>
>>>
>>> >> provider-id="XWS_ClientProvider"
>>> class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>> >> value="false">
>>>
>>>
>>> >> provider-id="ClientProvider"
>>> class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>> >> value="false">
>>>
>>> >>
>>> value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml">
>>>
>>> >> provider-id="XWS_ServerProvider"
>>> class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>>
>>>
>>> >> provider-id="ServerProvider"
>>> class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
>>>
>>>
>>> >> value="s1as">
>>>
>>>
>>> >>
>>> value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml">
>>>
>>>
>>>
>>> >> tx-log-dir="${com.sun.aas.instanceRoot}/logs"
>>> automatic-recovery="true">
>>>
>>> >>
>>> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=${JAVA_DEBUGGER_PORT}"
>>> system-classpath="" classpath-suffix="">
>>> -XX:MaxPermSize=192m
>>> -XX:PermSize=64m
>>> -server
>>> -Djava.awt.headless=true
>>> -XX:+UnlockDiagnosticVMOptions
>>>
>>> -Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed
>>>
>>> -Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy
>>>
>>> -Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf
>>>
>>> -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
>>>
>>> -Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks
>>>
>>> -Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks
>>>
>>> -Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext
>>>
>>> -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
>>> -DANTLR_USE_DIRECT_CLASS_LOADING=true
>>>
>>> -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
>>> -XX:NewRatio=2
>>> -Xmx512m
>>>
>>> -Dosgi.shell.telnet.port=${OSGI_SHELL_TELNET_PORT}
>>> -Dosgi.shell.telnet.maxconn=1
>>> -Dosgi.shell.telnet.ip=127.0.0.1
>>> -Dgosh.args=--noshutdown -c noop=true
>>>
>>> -Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/
>>> -Dfelix.fileinstall.poll=5000
>>> -Dfelix.fileinstall.log.level=3
>>>
>>> -Dfelix.fileinstall.bundles.new.start=true
>>>
>>> -Dfelix.fileinstall.bundles.startTransient=true
>>>
>>> -Dfelix.fileinstall.disableConfigSave=false
>>>
>>>
>>>
>>> >> sfsb-store-pool-name="jdbc/hastore">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="s1as">
>>>
>>>
>>> >> max-connections="250">
>>>
>>>
>>>
>>>
>>> >> encoded-slash-enabled="true">
>>>
>>>
>>> >> classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
>>> cert-nickname="glassfish-instance">
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> name="http-finder"
>>> classname="com.sun.grizzly.config.HttpProtocolFinder">
>>> >> name="admin-http-redirect"
>>> classname="com.sun.grizzly.config.HttpProtocolFinder">
>>>
>>>
>>>
>>>
>>> >> protocol="http-listener-1" transport="tcp" name="http-listener-1"
>>> thread-pool="http-thread-pool">
>>> >> protocol="http-listener-2" transport="tcp" name="http-listener-2"
>>> thread-pool="http-thread-pool">
>>> >> protocol="pu-protocol" transport="tcp" name="admin-listener"
>>> thread-pool="http-thread-pool">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> name="thread-pool-1">
>>> >> max-thread-pool-size="50" max-queue-size="256">
>>>
>>>
>>>
>>>
>>>
>>> >> value="24848">
>>> >> value="28080">
>>> >> value="28181">
>>> >> value="27676">
>>> >> value="23700">
>>> >> value="23820">
>>> >> value="23920">
>>> >> value="28686">
>>> >> value="26666">
>>> >> value="29009">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> >> special-admin-indicator="3047aff3-3214-4ac9-aa5e-a5dad78b2eea">
>>> >> Corporation,L=Santa
>>> Clara,ST=California,C=US">
>>> >> dn="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa
>>> Clara,ST=California,C=US">
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Anyone knows what might be the problem?
>>>
>>>
>>>
>>> --
>>> Mladen Adamovic
>>> Numbeo
>>> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
>>> (Business Registration Number 62612240)
>>> Tel. +381-66-058-595
>>> email: mladen.adamovic@gmail.com
>>> web: http://www.numbeo.com
>>
>
>

mladenadamovic
Offline
Joined: 2009-04-30

On Fri, Jul 20, 2012 at 3:52 PM, Oleksiy Stashok
wrote:
> You're absolutely right. Can you pls. file an issue?

I filed issue no 18926 for this: http://java.net/jira/browse/GLASSFISH-18926

> No, it's the max number of keep-alive requests allowed on a single
> connection.

> Hmm, it's probably terminology problem again. When request comes on a
> connection we put connection to a queue to process its request.
> So number of time connection was queued is more or less equivalent to a
> number HTTP requests processed (it might not be true for HTTP pipelined
> requests).

Hm, so if some bots are using keep-alive connections, than if there is
only 5 threads processing requests, they might be busy processing
keep-alive bots and visitors might have to wait.
That's why I think raising default number of threads for
httpthreadpool to Max Thread Pool Size: 400
I think default server config needs to raise this value, assuming that
512 connections server can handle, if I'm correct.
(most servers, including virtualized ones can handle 1024 connections,
but half of them might go to DB server if used).

--
Mladen Adamovic
Numbeo
Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
(Business Registration Number 62612240)
Tel. +381-66-058-595
email: mladen.adamovic@gmail.com
web: http://www.numbeo.com

oleksiys
Offline
Joined: 2006-01-25

Hi Mladen,

On 07/20/2012 08:59 PM, Mladen Adamovic wrote:
>> Hmm, it's probably terminology problem again. When request comes on a
>> connection we put connection to a queue to process its request.
>> So number of time connection was queued is more or less equivalent to a
>> number HTTP requests processed (it might not be true for HTTP pipelined
>> requests).
> Hm, so if some bots are using keep-alive connections, than if there is
> only 5 threads processing requests, they might be busy processing
> keep-alive bots and visitors might have to wait.
Most of the time (if HTTP pipelining is not used) connection is getting
queued to process just one request, then we dequeue the connection and
processing next queued connection.
So thread-pool is not stuck w/ keep-alive requests that come from one
connection.

> That's why I think raising default number of threads for
> httpthreadpool to Max Thread Pool Size: 400
> I think default server config needs to raise this value, assuming that
> 512 connections server can handle, if I'm correct.
Perf. tests we did show that 5 threads can handle thousands of
connections for simple webapps, for sure if you have a servlet, which
makes DB requests (which may take longer time) - you have to increase
the thread-pool size.

WBR,
Alexey.