Skip to main content

Long Glassfish Pauses

1 reply [Last post]
h1d3m3
Offline
Joined: 2008-02-15
Points: 0

I have been trying to get through a pausing problem we are seeing on our application. The symptoms are, after about 2 hours in heavy load, the old gen clears itself out (CMS at around 80-90%) which takes about 10 seconds the first time, 20 the second time and so on...eventually old gen stays around 70% and the pauses become more frequent and longer (30-40 seconds long).

Conceptually, I can understand this to mean there are some tenured objects that are not getting cleared (and a good profiler will help there I guess)...but today, we had this event happen:

S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC
0.00 62.14 95.85 60.32 59.94 91 156.995 24 278.900 435.896 CMS Final Remark No GC
97.40 62.14 96.02 60.32 59.94 92 156.995 24 278.900 435.896 unknown GCCause Allocation Failure
100.00 62.14 96.02 60.52 59.94 92 156.995 24 278.900 435.896 unknown GCCause Allocation Failure
[...large snip of the same stuff...]
100.00 62.14 0.00 65.79 58.86 92 166.222 25 278.900 445.122 unknown GCCause Allocation Failure
0.00 0.00 5.13 65.79 58.86 92 166.222 25 456.792 623.014 unknown GCCause No G

Here, when eden filled up, I expected it to move to the survivor spaces as normal but instead it stayed like this for around 200 seconds. The only thing I can think of is this was a defragmentation?

Unfortunately, our app deals with a lot of small young gen stuff (http service requests) and larger tenured data (i.e. image uploads / DB work) so getting the balance right is hard.

Any help is greatly appreciated.

[ our current config ]
-Xmn9g (this seems to be working fairly well...less was going from survivor to old when this was turned up)
-XX:SurvivorRatio=3
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+DisableExplicitGC
-Xms28g
-Xmx28g
-d64
-XX:MaxPermSize=280m

[ our OS ]
32GB Ram
Name of the Operating System: SunOS
Binary Architecture name of the Operating System: sparcv9, Version: 5.10
Number of processors available on the Operating System: 64 (8 CPUs)

[ At this point I'm considering trying a different GC. ]

-XX:MaxGCPauseMillis=500
-XX:+UseParallelGC
-XX:+UseParallelOldGC
-XX:ParallelGCThreads=16
-XX:+DisableExplicitGC

[ A jmap -dump header when the app starts]
Attaching to process ID 7992, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 10.0-b22

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 30064771072 (28672.0MB)
NewSize = 9663676416 (9216.0MB)
MaxNewSize = 9663676416 (9216.0MB)
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 3
PermSize = 21757952 (20.75MB)
MaxPermSize = 293601280 (280.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 7730954240 (7372.8125MB)
used = 3362958112 (3207.166778564453MB)
free = 4367996128 (4165.645721435547MB)
43.49990968255945% used
Eden Space:
capacity = 5798232064 (5529.625MB)
used = 1623175520 (1547.9808044433594MB)
free = 4175056544 (3981.6441955566406MB)
27.994317959054356% used
From Space:
capacity = 1932722176 (1843.1875MB)
used = 1739782592 (1659.1859741210938MB)
free = 192939584 (184.00152587890625MB)
90.01721062675901% used
To Space:
capacity = 1932722176 (1843.1875MB)
used = 0 (0.0MB)
free = 1932722176 (1843.1875MB)
0.0% used
concurrent mark-sweep generation:
capacity = 20401094656 (19456.0MB)

Message was edited by: h1d3m3

Message was edited by: h1d3m3

Message was edited by: h1d3m3

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
rafaelnaskar
Offline
Joined: 2006-03-24
Points: 0

test this first:

-Xms28g
-Xmx28g
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+DisableExplicitGC
-XX:MaxPermSize=384m

else:

-Xms28g
-Xmx28g
-XX:SurvivorRatio=2
-XX:NewRatio=2
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+DisableExplicitGC
-XX:MaxPermSize=384m

Analize the BLOCKED threads that waiting the GC.

jstack -J-d64