Skip to main content

Using Glassfish 3 in production

7 replies [Last post]
bdushok
Offline
Joined: 2004-10-07
Points: 0

I'm currently building a server running Glassfish 3 to replace our current Glassfish 2 server. In testing the server runs great, but once online connections to our webapps seem to stall.

Using a free web site load testing site I was able to find that when somewhere between 30 to 50 simultaneous connections are made connections to the server seem to stall. Pages appear to eventually load, but I've waited as much as three minutes for the request to be processed.

I've tried several things to tune the server for production use, but haven't been successful. So far I've changed the following:

Changed/Added JVM options:
-client replaced with -server
XX:+AggressiveHeap
-Xmx1400m
-Xms1400m
-Xss128k
XX:+DisableExplicitGC

I noticed Glassfish seemed to be opening more files than the OS was allowing. I changed the OS to enable Glassfish to have up to 20000 files open (Linux file /etc/security/limits.conf).

Is there any other settings I should change to increase performance?

Thanks,
Bob

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
bdushok
Offline
Joined: 2004-10-07
Points: 0

Dominick,

Thanks for the reply.

It looks like I've done almost everything specified in the blog entry you've posted. I'll make the few additional changes which apply to my platform as well as to http-threads changes. I will reply this weekend with the results. Unfortunately, I'm restricted to making these changes during less busy weekend hours :(

I used the site www.loadimpact.com. It is a pay site, but they have a free test which will test server load with up to 50 connections.

Thanks,
Bob

Dominik Dorn

are you using the latest version of v3?

I've modified mine according to
http://weblogs.java.net/blog/jfarcand/archive/2009/11/27/putting-glassfi...

I think the http-threads make a huge difference.

Could you share the address of this load testing site? I want to try
this on my server
as well.

Thanks,
dominik

--
Dominik Dorn
http://dominikdorn.com

Eine [url=http://www.studyguru.eu]Ausarbeitung[/url] zu Deinem Seminarthema?
[url=http://www.studyguru.eu]Mitschrift[/url] von der letzten Vorlesung?
[url=http://www.studyguru.eu]Alte Prüfungen[/url]? Dass, und noch viel
mehr findest Du auf http://www.studyguru.eu ! Tausche Deine
Lernunterlagen noch heute!

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Barry van Someren

Hi,

The HTTP-threads will make a difference, you need to test this with
JMeter though.
Also, you can use +UseLargePages and setup Hugepages in Linux, but it
didn't do much for our app. (I used this blog:
http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html keep
in mind that they got the calculations wrong, use this site to
calculate the right settings http://www.peuss.de/node/67)
We also use -XX:+UseTLAB

I do recommend using the latest JVM, it did bring up performance by
another 20% (most likely the NUMA enhancements and the String concat
changes)

Regards,

Barry

On Thu, Mar 25, 2010 at 8:43 PM, Dominik Dorn wrote:
> are you using the latest version of v3?
>
> I've modified mine according to
> http://weblogs.java.net/blog/jfarcand/archive/2009/11/27/putting-glassfi...
>
> I think the http-threads make a huge difference.
>
> Could you share the address of this load testing site? I want to try
> this on my server
> as well.
>
> Thanks,
> dominik
>
>
> --
> Dominik Dorn
> http://dominikdorn.com
>
>
> Eine [url=http://www.studyguru.eu]Ausarbeitung[/url] zu Deinem Seminarthema?
> [url=http://www.studyguru.eu]Mitschrift[/url] von der letzten Vorlesung?
> [url=http://www.studyguru.eu]Alte Prüfungen[/url]? Dass, und noch viel
> mehr findest Du auf http://www.studyguru.eu ! Tausche Deine
> Lernunterlagen noch heute!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
Barry van Someren
---------------------------------------
LinkedIn: http://www.linkedin.com/in/barryvansomeren
Skype: BvsomerenSprout
Blog: http://blog.bvansomeren.com
KvK: 27317624
M: +31 (0)6-81464338
T: +31 (0)70-2500450

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

bdushok
Offline
Joined: 2004-10-07
Points: 0

Barry,

Thanks for the reply. Thanks for the tip on JMeter. I wasn't aware of that Apache project. I will download and use it.

I am already using JVM 6 U18, but I'm using the 32 bit version. I'm using a copy of Red Hat 5.4 virtualized under ESX Server 4 for this server. With the virtualized hardware I didn't consider loading Red Hat 64 bit or the 64 bit JVM. Has it increased performance for you?

Thanks,
Bob

bdushok
Offline
Joined: 2004-10-07
Points: 0

Barry,

No success. I've started clean. I installed RHEL 5.4 64 bit, the 64 bit JVM (6U18), and the 64 bit version of MySQL.

First I tried using loadimpact.com to test the site. I went through a full 50 user test without a problem, 12000 page hits and nearly 100m of data transferred. I ran a second test and got about 75% complete before the site died.

I moved to JMeter after this It seems like I can't get a good estimate of when the site dies. I've killed it running a test with 150 threads running 5 times, other times I can run a test 200 thread test 20 times before killing the server. At times the site doesn't die.

I've made modifications to the thread pool without any noticeable change. I've set the min thread count from 5 to 20 and the max from 500 to 2000. I even tried setting the max queue size to -1. There seemed to be no pattern between the load required to kill the site and the thread settings.

For an odd test I swapped our production site for this server and ran no load tests against it. In less than 30 minute the site was down. I'm sure there was traffic hitting the site during this time, but not anywhere near the amount generated by JMeter or loadimpact.

Does anyone have any suggestions on how I can troubleshoot this? Nothing regarding the server hangs appears in the logs.

BTW, on this rebuild of the server I'm using the following JVM options:
-server
-Xmx3500m
-XX:+AggressiveHeap
-Xms3500m
-Xss128k
-XX:+DisableExplicitGC
-XX:ParallelGCThreads=2
-XX:+UseParallelOldGC
-XX:+UseCompressedOops
-XX:+UseTLAB

Thanks,
Bob

Barry van Someren

Hi,

We run VMWare dev/test environments, but I never loadtest those.

The 64 Bit JVM allows larger heaps and it can improve performance as
it uses more registers from the CPU (the x86_64 extensions)
The -XX:+UseCompressedOops is used to offset the increased pointer
sizes for 64 bit usage.
To be honest, I'm unsure about the difference between 32 and 64 bit, I
believe 64 bit to be faster on paper, but I've not tested it recently.
Your settings seem good, in fact except for the garbage collector they
are pretty close to what I use (I use the CMS gc)

As for the stalling, I've seen a few mentions about this on the list;
And a patch being posted to fix this.
Not sure, but I'll see if I can dig up the email.

Currently I work mostly with Glassfish 2.1.1 so I've never seen the
stalling issue :-/

Regards,

Barry

On Sat, Mar 27, 2010 at 6:15 PM, wrote:
> Barry,
>
> No success.   I've started clean.   I installed RHEL 5.4 64 bit, the 64 bit JVM (6U18), and the 64 bit version of MySQL.
>
> First I tried using loadimpact.com to test the site.   I went through a full 50 user test without a problem, 12000 page hits and nearly 100m of data transferred.   I ran a second test and got about 75% complete before the site died.
>
> I moved to JMeter after this   It seems like I can't get a good estimate of when the site dies.   I've killed it running a test with 150 threads running 5 times, other times I can run a test 200 thread test 20 times before killing the server.   At times the site doesn't die.
>
> I've made modifications to the thread pool without any noticeable change.   I've set the min thread count from 5 to 20 and the max from 500 to 2000.   I even tried setting the max queue size to -1.   There seemed to be no pattern between the load required to kill the site and the thread settings.
>
> For an odd test I swapped our production site for this server and ran no load tests against it.   In less than 30 minute the site was down.   I'm sure there was traffic hitting the site during this time, but not anywhere near the amount generated by JMeter or loadimpact.
>
> Does anyone have any suggestions on how I can troubleshoot this?   Nothing regarding the server hangs appears in the logs.
>
> BTW, on this rebuild of the server I'm using the following JVM options:
> -server
> -Xmx3500m
> -XX:+AggressiveHeap
> -Xms3500m
> -Xss128k
> -XX:+DisableExplicitGC
> -XX:ParallelGCThreads=2
> -XX:+UseParallelOldGC
> -XX:+UseCompressedOops
> -XX:+UseTLAB
>
> Thanks,
> Bob
> [Message sent by forum member 'bdushok']
>
> http://forums.java.net/jive/thread.jspa?messageID=394034
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>

--
Barry van Someren
---------------------------------------
LinkedIn: http://www.linkedin.com/in/barryvansomeren
Skype: BvsomerenSprout
Blog: http://blog.bvansomeren.com
KvK: 27317624
M: +31 (0)6-81464338
T: +31 (0)70-2500450

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Barry van Someren

All,

Could it be related to this:
https://grizzly.dev.java.net/issues/show_bug.cgi?id=547
Try setting the linger to -1 and let us know if it helps.

Regards,

Barry

On Sat, Mar 27, 2010 at 11:27 PM, Barry van Someren
wrote:
> Hi,
>
> We run VMWare dev/test environments, but I never loadtest those.
>
> The 64 Bit JVM allows larger heaps and it can improve performance as
> it uses more registers from the CPU (the x86_64 extensions)
> The  -XX:+UseCompressedOops is used to offset the increased pointer
> sizes for 64 bit usage.
> To be honest, I'm unsure about the difference between 32 and 64 bit, I
> believe 64 bit to be faster on paper, but I've not tested it recently.
> Your settings seem good, in fact except for the garbage collector they
> are pretty close to what I use (I use the CMS gc)
>
> As for the stalling, I've seen a few mentions about this on the list;
> And a patch being posted to fix this.
> Not sure, but I'll see if I can dig up the email.
>
> Currently I work mostly with Glassfish 2.1.1 so I've never seen the
> stalling issue :-/
>
> Regards,
>
> Barry
>
>
>
> On Sat, Mar 27, 2010 at 6:15 PM,   wrote:
>> Barry,
>>
>> No success.   I've started clean.   I installed RHEL 5.4 64 bit, the 64 bit JVM (6U18), and the 64 bit version of MySQL.
>>
>> First I tried using loadimpact.com to test the site.   I went through a full 50 user test without a problem, 12000 page hits and nearly 100m of data transferred.   I ran a second test and got about 75% complete before the site died.
>>
>> I moved to JMeter after this   It seems like I can't get a good estimate of when the site dies.   I've killed it running a test with 150 threads running 5 times, other times I can run a test 200 thread test 20 times before killing the server.   At times the site doesn't die.
>>
>> I've made modifications to the thread pool without any noticeable change.   I've set the min thread count from 5 to 20 and the max from 500 to 2000.   I even tried setting the max queue size to -1.   There seemed to be no pattern between the load required to kill the site and the thread settings.
>>
>> For an odd test I swapped our production site for this server and ran no load tests against it.   In less than 30 minute the site was down.   I'm sure there was traffic hitting the site during this time, but not anywhere near the amount generated by JMeter or loadimpact.
>>
>> Does anyone have any suggestions on how I can troubleshoot this?   Nothing regarding the server hangs appears in the logs.
>>
>> BTW, on this rebuild of the server I'm using the following JVM options:
>> -server
>> -Xmx3500m
>> -XX:+AggressiveHeap
>> -Xms3500m
>> -Xss128k
>> -XX:+DisableExplicitGC
>> -XX:ParallelGCThreads=2
>> -XX:+UseParallelOldGC
>> -XX:+UseCompressedOops
>> -XX:+UseTLAB
>>
>> Thanks,
>> Bob
>> [Message sent by forum member 'bdushok']
>>
>> http://forums.java.net/jive/thread.jspa?messageID=394034
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
>> For additional commands, e-mail: users-help@glassfish.dev.java.net
>>
>>
>
>
>
> --
> Barry van Someren
> ---------------------------------------
> LinkedIn: http://www.linkedin.com/in/barryvansomeren
> Skype: BvsomerenSprout
> Blog: http://blog.bvansomeren.com
> KvK: 27317624
> M: +31 (0)6-81464338
> T: +31 (0)70-2500450
>

--
Barry van Someren
---------------------------------------
LinkedIn: http://www.linkedin.com/in/barryvansomeren
Skype: BvsomerenSprout
Blog: http://blog.bvansomeren.com
KvK: 27317624
M: +31 (0)6-81464338
T: +31 (0)70-2500450

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net