Skip to main content

Large app, large Heap and very lage GC pauses

8 replies [Last post]
flpgdt
Offline
Joined: 2009-01-29

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
flpgdt
Offline
Joined: 2009-01-29

On last update,

I have found that with my application turned off the server has already 10gb of space taken. Well, setting up other 28gb just for my heap sure pushes it to a swap...

I monitored the application for a while and it seems that the average memory footprint is about 9gb, and so I restarted it with 10gb of fixed heap expecting no more swap.

For my surprise, when I did that 'pmap -s' I still got this:
Address Kbytes Swap Mode Mapped File
(...)
FFFFFFFF7FFC0000 24 - ----- [ anon ]
FFFFFFFF7FFF0000 64 64 rw--- [ stack ]
---------------- ---------- ----------
total Kb 11074184 10903064

Well, my application seems fine so far, but why did open this much o swap space?

(top also show this much o swap in use)
load averages: 4.09, 4.29, 3.72 10:46:59
128 processes: 123 sleeping, 1 stopped, 4 on cpu
CPU states: % idle, % user, % kernel, % iowait, % swap
Memory: 32G real, 11G free, 11G swap in use, 23G swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
22734 user 81 1 0 11G 7566M cpu/16 28:46 7.40% java

Is there any way I can check how much of the swap my app is actually using?
Again, the same question, if my heap is 10gb, why do I have this much of swap? 10gb of RAM + 10gb of swap = 20gb?

I'm still monitoring it, and it is running smooth for the last 16h, but this is usually not enough time to trigger the problem.

I'll see how it goes!

cheers

flpgdt
Offline
Joined: 2009-01-29

Updating you guys,

First of all, answering those who asked me to check memory structure and look for leeckage, I have to say that I don't have access to the code... and the documentation available is farely poor.

The good new is that it is a well tested and fairly robust system. I'm a mere implementor and I have to count on it :( Still I'll apreciate all the help and speculation!

I'm gonna fill up again the details about the platform

user@master:> java -version
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Server VM (build 1.5.0_15-b04, mixed mode)

user@master:> psrinfo -v
Status of virtual processor 0 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 4 as of: 02/02/2009 13:36:08
faulted since 12/06/2008 00:53:29.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 5 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 6 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 7 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:40:12.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 16 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 17 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 18 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 19 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 20 as of: 02/02/2009 13:36:08
faulted since 12/06/2008 00:53:29.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 21 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 22 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 23 as of: 02/02/2009 13:36:08
on-line since 12/02/2008 18:41:15.
The sparcv9 processor operates at 1500 MHz,
and has a sparcv9 floating point processor.

user@master:> prtconf -v | grep Memory
Memory size: 32768 Megabytes

user@master:> uname -a
SunOS master 5.10 Generic_118833-36 sun4u sparc SUNW,Sun-Fire-V890

Now, continuing with the research oround my flags:

It seems that the app is paging!

I have found this 'pmap -S' command, to tell me about the swapping of an app and that looks very scary! Follows:

user@master:> pmap -S 1406
1406: /revo/contrib/jdk1.5.0_15/bin/sparcv9/java -server -Xss256k -Xms28672m
Address Kbytes Swap Mode Mapped File
0000000100000000 72 - r-x-- java
0000000100110000 16 16 rwx-- java
0000000100114000 31664 31664 rwx-- [ heap ]
0000000102000000 327680 327680 rwx-- [ heap ]
(...)
FFFFFFFF7FFC0000 24 - ----- [ anon ]
FFFFFFFF7FFF0000 64 64 rw--- [ stack ]
---------------- ---------- ----------
total Kb 30696520 30461512

Then 'top':

load averages: 2.53, 2.57, 2.03 10:42:32
150 processes: 147 sleeping, 3 on cpu
CPU states: 81.8% idle, 7.8% user, 10.4% kernel, 0.0% iowait, 0.0% swap
Memory: 32G real, 502M free, 30G swap in use, 4278M swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1010 daemon 18 60 -20 2752K 784K sleep 95.3H 1.86% nfsd
1406 user 414 59 0 29G 20G sleep 509:03 1.09% java
151 daemon 16 29 0 8504K 2912K sleep 29.5H 0.45% kcfd

Ok, looks like it is paging indeed.

But now, my questions are:

Why would it swap 30GB if the max heap is 28GB? How come?

Shouldn't Java take memory only up to 28GB or throw a 'out of memory' in case it runs out?

Even so, how come it goes bad so soon? like, with 9GB of usage...

I have a hypothesis:

The memory allocation is made by a lower layer, and the JVM does not know where is the memory it is allocating. Then, lets say that only the SO is taking 15GB of memory (I know, very unlikely) and when the JVM comes up and allocates its space, it already get some stuff paged. As the JVM does not knows where its memory is, eventually it allocates one and another object paged right away. No matter which GC comes up, it gets slowed.

Now about GCs and Full GCs.

Could someone explain me whats the relation between GCs and Full GCs? I though the CMS was responsible for the Full GC execution, but I saw in the logs 'Full GC's for 'ParNew'.. hmm???

Would a full GC be a serial GC executed all over the memory in certain occasions, regardless the GC algorithm being executed? How much slower (in terms of what is it relative to) a Full GC is versus the CMS of ParNew?

peter__lawrey
Offline
Joined: 2005-11-01

> Updating you guys,
>
> First of all, answering those who asked me to check memory structure
You need to be able to see your memory banks and much is in each memory bank.

> java version "1.5.0_15"
You could try Java 6 update 11, but you might not find it is much better.
Java 7 has a new "G1" GC This promises shorter GC pause times, but in testing it was a few ms slower (in my case double :)

> It seems that the app is paging!
Which would explain long GC times.
> But now, my questions are:
>
> Why would it swap 30GB if the max heap is 28GB? How come?
Because heap is just one type of memory. There is direct memory, share libraries, perm gen, thread stack space and many more. I have seen a program use as much as 500 MB more virtual memory than the heap size.
> Shouldn't Java take memory only up to 28GB or throw a 'out of memory' in case it runs out?
You may find it throws the error before you reach 28 GB of heap size ( common complaint) but it does guarantee the total memory size.

Note: I have seen VMs will preallocate the heap space in virtual on startup. However, they don't use that memory ie.e. it doesn't become real memory, until its used.

> Now about GCs and Full GCs.
> Could someone explain me whats the relation between
> GCs and Full GCs? I though the CMS was responsible
> for the Full GC execution, but I saw in the logs
> 'Full GC's for 'ParNew'.. hmm???
It can do both, but you need to enable CMS, its not on by default.

> Would a full GC be a serial GC executed all over the
> memory in certain occasions, regardless the GC
> algorithm being executed?
yes.
> How much slower (in terms
> of what is it relative to) a Full GC is versus the
> CMS of ParNew?
About the same. CMS should mean this is less likely to get a full GC and less impact for incremental GC. However CMS consumes more CPU and it might consume more memory per object, but I haven't tested this enough, so it just a theory.

peter__lawrey
Offline
Joined: 2005-11-01

How is your memory structured?
If you have more than one memory bank, your application performance could suffer if it crosses memory banks. esp. for GC as this is a memory scan

briand
Offline
Joined: 2005-07-11

It would be useful to have some hardware, OS, and Java version information.

> argv[0]: /contrib/jdk1.5.0_15/bin/sparcv9/java
> argv[1]: -server
> argv[2]: -Xss256k
> argv[3]: -Xms28672m
> argv[4]: -Xmx28672m

That's a pretty big heap. I'll assume that you have sufficient RAM for this
size heap; if not, stop here - your heap size shouldn't be larger than the
amount of RAM in the system. If there's more than one JVM (or other
processes) running on this system, then the sum of their working set sizes
should not exceed the amount of RAM. You probably know this, but since
you didn't include hardware details, I figured I'd mention it.

> argv[5]: -Djava.awt.headless=true
> argv[6]: -Djava.net.preferIPv4Stack=true
> argv[7]: -Xloggc:/logs/gc/Gc20090129.log
> argv[8]: -XX:+UseConcMarkSweepGC
> argv[9]: -verbose:gc
> argv[10]: -XX:+PrintGCTimeStamps
> argv[11]: -XX:+PrintGCDetails
> argv[12]: -XX:+PrintTenuringDistribution
> argv[13]: -XX:NewSize=4096m
> argv[14]: -XX:MaxNewSize=4096m
> argv[15]: -XX:MaxTenuringThreshold=30

Depending on what version of Java you are using, this may be too
big. In 1.5.0_08 (IIRC) and later, the number of ages for objects in
the young gen was reduced from 32 to 16 (one bit taken away from
the per-object header for Biased Locking). Setting this to 30 in one
of those later JVM would effectively set up 'promote never' semantics.
That would force the gc algorithms to keep objects around in the
survivor spaces for many collections, as long as the survivor spaces
don't overflow. Once they do overflow, the objects will get promoted.

> argv[16]: -XX:+UseParNewGC
> argv[17]: -XX:MaxGCPauseMillis=100000

Depending on the version of HotSpot, this may not be applicable to
CMS/ParNew - however, I'd have to check that in source code to be
certain.

> argv[18]: -XX:MaxPermSize=512m
> argv[19]: -XX:CMSInitiatingOccupancyFraction=40
> argv[20]: -XX:+DisableExplicitGC
> argv[21]: -XX:+PrintCompilation
> argv[22]: -XX:SurvivorRatio=10

You might consider making this smaller (effectively making the
survivor spaces larger). If you do, you may need to increase your
young gen size a well. Note that larger survivor spaces will likely
increase minor gc times (more stuff to copy from one survivor space
to the other on every minor gc). However, it might also help in preventing
short lived objects from reaching the old gen.

In the gc logs below, your survivor ages are rather young (1, 2, occasionally
a 3 or 4 or 5) and after the minor gc's the old gen grows - so I suspect that some
of these objects are getting promoted at each minor gc. Larger survivor
spaces may help prevent that.

> argv[23]: -Dprogram.name=run.sh
>
> I'm running it in a Solaris box.
>
> The application runs fine for one or two days and
> than we start to get stuck.
>
> this is a bit of the logs to have an idea:
>
> 50.820: [GC 50.821: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 49164032 bytes, 49164032 total
> : 3495296K->48340K(3844800K), 0.1999100 secs] 3495296K->48340K(29010624K), 0.2003113 secs]
> 88.001: [GC 88.001: [ParNew
> Desired survivor size 178946048 bytes, new threshold 1 (max 30)
> - age 1: 311239176 bytes, 311239176 total
> - age 2: 44690624 bytes, 355929800 total
> : 3543636K->349504K(3844800K), 7.9404442 secs] 3543636K->1614612K(29010624K), 7.9408797 secs]

This is pretty early in your run, so this pause may be a result of first
touch to pages in either the young gen or the old gen. Try adding
-XX:+AlwaysPreTouch to your command line. This will cause the JVM
to touch every page on initialization, thus getting all the page in memory
before entering main(). Note that this will likely increase your startup time.

Somewhat relatedly, since your heap is multi-gigabytes, you might want
to consider trying larger page sizes. On Solaris, hotspot will automatically
use large pages, but it may not use the largest available page size. To
see the available page sizes, use the 'pagesize -a' command. Here's the
output from my SPARC system:

mojo> pagesize -a
8192
65536
524288
4194304

On this system, HotSpot will use the 4m page size, but on newer systems,
where 256m pages may be available, it will still use the 4m page size by
default. Switching to the larger page size may help your overall performance
and your GC performance. YYMV, cavet emptor.

To set the page size, add -XX:LargePageSizeInBytes=N. However, only use
this if you want to override the default page size. To see the page size that's
being used, run 'pmap -s
' and look for the largest anonymous mappings
(they are likely the java heap) and look at the page size indicated for that
mapping.

> 143.208: [GC 143.208: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 161321672 bytes, 161321672 total
> : 3844800K->158311K(3844800K), 2.1910047 secs] 5109908K->1768479K(29010624K), 2.1914121 secs]
> 260.333: [GC 260.334: [ParNew
> Desired survivor size 178946048 bytes, new threshold 2 (max 30)
> - age 1: 86695552 bytes, 86695552 total
> - age 2: 134834392 bytes, 221529944 total
> : 3653607K->213062K(3844800K), 0.2867113 secs] 5263775K->1823229K(29010624K), 0.2870955 secs]
> 351.400: [GC 351.402: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 84541312 bytes, 84541312 total
> - age 2: 23784944 bytes, 108326256 total
> : 3708358K->104439K(3844800K), 0.6484717 secs] 5318525K->1844007K(29010624K), 0.6489656 secs]
> 401.804: [GC 401.804: [ParNew
> Desired survivor size 178946048 bytes, new threshold 1 (max 30)
> - age 1: 272952984 bytes, 272952984 total
> - age 2: 21769184 bytes, 294722168 total
> - age 3: 22914000 bytes, 317636168 total
> : 3599735K->311722K(3844800K), 0.4198017 secs] 5339303K->2051290K(29010624K), 0.4201525 secs]
>
> ( fine until there, but later)
>
> 126826.289: [GC 126826.301: [ParNew
> Desired survivor size 178946048 bytes, new threshold 1 (max 30)
> - age 1: 216963720 bytes, 216963720 total
> - age 2: 14985192 bytes, 231948912 total
> - age 3: 26226440 bytes, 258175352 total
> - age 4: 3189112 bytes, 261364464 total
> - age 5: 1642848 bytes, 263007312 total
> : 3569589K->257367K(3844800K), 13.6111397 secs] 10939534K->7627311K(29010624K), 13.6139373 secs]

Was there any indication that a CMS cycle had started or is in progress?
It seems likely because 7627311K is less than your previous (that we can
see) ending heap size of 2051290K, so CMS must have done some work
in the mean time. Note that this is much later (time stamp of 126826.289 -
35 hours into the run). It's possible that the old gen has gotten somewhat
fragmented, and as this minor collection tries to promote objects it has to
search the free lists to find space. The more fragmented the old gen is,
the longer this is going to take. Keeping short lived objects out of the old
gen is one the best ways to prevent this fragmentation.

Fragmentation seems somewhat unlikely here, though, because there
appears to be sufficient space in the old gen; still, that can be misleading
when that space is highly fragmented. It could also be OS demand paging
due to memory shortfall.

> 128241.083: [GC 128241.095: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 151586336 bytes, 151586336 total
> : 3752663K->148243K(3844800K), 2.8472709 secs] 11122607K->7603253K(29010624K), 2.8480368 secs]
> 130516.250: [GC 130516.258: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 90793112 bytes, 90793112 total
> - age 2: 36686640 bytes, 127479752 total
> : 3643539K->124876K(3844800K), 0.3896032 secs] 11098549K->7579886K(29010624K), 0.3904482 secs]
> 133424.645: [GC 133428.565: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 52098056 bytes, 52098056 total
> - age 2: 10140640 bytes, 62238696 total
> - age 3: 32191064 bytes, 94429760 total
> : 3620172K->92534K(3844800K), 1533.2368318 secs] 11075182K->7547544K(29010624K), [b]1537.2197446[/b] secs]

This is a big one. Again, it could be fragmentation, but it could also be
contention between ParNew and CMS. It could also be a result of paging,
and with this kind of number, that's the first thing I'd look at.

> 138469.836: [GC 138469.851: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 52906800 bytes, 52906800 total
> - age 2: 1700120 bytes, 54606920 total
> - age 3: 8290112 bytes, 62897032 total
> - age 4: 32181256 bytes, 95078288 total
> : 3587830K->93039K(3844800K), 17.3476582 secs] 11042840K->7548049K(29010624K), [b]17.3551050 secs[/b]]
>
> Well, needless to say that the cluster fragments when
> this happens.
>
> By seeing this you can note that the minor GCs are
> all about 100% garbage, which if I understood, is
> bad!

To the contrary, garbage is cheap to collect from the young gen. It's
the survivors that are expensive. They need to be copied and promoted
and then CMS collected. If they are short lived, they tend to fragment the
old gen. Surely, there needs to be long lived data, but hopefully that's
mostly generated at startup and reaches some stable state.

Note, though, that it doesn't appear that all your objects are garbage.
Minor collections in the early samples are causing the old gen to grow.

>
> I know, I know, we have here 4gb for the youngs and
> that looks like a big cookie to bite, but before the
> app had only 512mb and the same crap was happening...
> unfortnatelly I don't have access to the old GC logs
> to check the usage.
>
> The majos GCs are farely happening, and they are fine
> (less than one second), but these minors are killing
> me...

I don't see and CMS log data or any signs of CMS concurrent mode
failure - they may be in your logs, but they aren't in what you showed us
here (unless I missed something).

>
> My other parallel problem is that the application
> takes like 1 hour to be restarted so people here gets
> very fedup when I have to do it (i.e. I can't afford
> try and error :( )

Ouch! Adding -XX:+AlwaysPreTouch may make this even worse.
Larger page sizes might help, but you have a large heap and it takes
time to allocate all that memory.

>
> Can anyone give me some light?

I'd try the larger page sizes, if possible, and I'd increase the size of
the survivor spaces. If you are on 1.5.0_06 or later (I think I have that
right, it may be _08 or later) I'd either drop -XX:MaxTenuringThreshold=30
change it to -XX:MaxTenuringThreshold=15 (or less). 30 is fine for earlier
release, though.

I think you should also start looking for any signs of memory leaks. I say this
because you were seeing similar symptoms when your young gen (and
overall heap size?) were smaller. If you increase the size of the heap significantly
and the problems still persist, that could be a sign of a memory leak.

HTH!
Brian

flpgdt
Offline
Joined: 2009-01-29

Hi briand!
thanks' for the answer!
unfortunately I had no time to read it before the last restart :( (after that monstrous pause, it had to be done), but hopefully we can carry on!
so.. Sorry for not giving the details about my platform :(
Its a
# uname -a
SunOS 5.10 Generic_118833-36 sun4u sparc SUNW,Sun-Fire-V890
solaris box with Java like:
#java -version
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Server VM (build 1.5.0_15-b04, mixed mode)
so its pretty new...

> That's a pretty big heap. I'll assume that you have sufficient RAM for this
> size heap; if not, stop here
yeap.. we got it all.. this is the master node.. the workers have like 40gb each :)
> Depending on what version of Java you are using, this may be too
> big. In 1.5.0_08 (IIRC) and later, the number of ages for objects in
> the young gen was reduced from 32 to 16 (one bit taken away from
> the per-object header for Biased Locking). Setting this to 30 in one
> of those later JVM would effectively set up 'promote never' semantics.
> That would force the gc algorithms to keep objects around in the
> survivor spaces for many collections, as long as the survivor spaces
> don't overflow. Once they do overflow, the objects will get promoted.

You see.. that's my big question mark now!
The logs I have are pretty much what you've seen! No full GCs going on!
like in 17h happened one or two full GCs only! and their times were ok...
Turns out that everything seemed to be happening in the eden, the tenured barely grows.
I was taking a look in other deployments of this system we have (smaller ones, but the same app) and the most common use is to make the promotion very fast!
(you will see my new set of confs)

> > argv[16]: -XX:+UseParNewGC
> > argv[17]: -XX:MaxGCPauseMillis=100000
> Depending on the version of HotSpot, this may not be applicable to
> CMS/ParNew - however, I'd have to check that in source code to be
> certain.

My JVM is meant to be fine, though the ergonomics arent really helping in my case :(

> Somewhat relatedly, since your heap is multi-gigabytes, you might want
> to consider trying larger page sizes. On Solaris, hotspot will automatically
> use large pages, but it may not use the largest available page size. To
> see the available page sizes, use the 'pagesize -a' command. Here's the
> output from my SPARC system:
> mojo> pagesize -a
> 8192
> 65536
> 524288
> 4194304

so, that's for me:
# pagesize -a
8192
65536
524288
4194304
33554432
268435456

That sounds cool, but I'm not familiar with this "-XX:LargePageSizeInBytes=N"... could give me some more info about this before I stuff it into production ? :P

> Was there any indication that a CMS cycle had started or is in progress?
> It seems likely because 7627311K is less than your previous (that we can
> see) ending heap size of 2051290K, so CMS must have done some work
> in the mean time. Note that this is much later (time stamp of 126826.289 -
> 35 hours into the run). It's possible that the old gen has gotten somewhat
> fragmented, and as this minor collection tries to promote objects it has to
> search the free lists to find space. The more fragmented the old gen is,
> the longer this is going to take. Keeping short lived objects out of the old
> gen is one the best ways to prevent this fragmentation.
>
> Fragmentation seems somewhat unlikely here, though, because there
> appears to be sufficient space in the old gen; still, that can be misleading
> when that space is highly fragmented. It could also be OS demand paging
> due to memory shortfall.

I wish I could agree straight.. but i see no sign of CMS around :( - maybe I'm missing some verbose flag monitor CMS?
What I do believe, is about the ParNew fragment or something in the Eden. How could I look for it? is there a flag for checking the fragmentation of the memory?
I know I'm sounding very newbie by now but I cant help :(

So, my new set of flags were:
argv[1]: -server
argv[2]: -Xss256k
argv[3]: -Xms28672m
argv[4]: -Xmx28672m
argv[10]: -XX:+PrintGCTimeStamps
argv[11]: -XX:+PrintGCDetails
argv[16]: -XX:+DisableExplicitGC
just the usual

argv[8]: -XX:+UseConcMarkSweepGC
I still have no reason to change it...

argv[12]: -XX:NewSize=512m
So, I removed the max new size here, which if I'm correct leaves me with the default settings for solaris with -server (Ratio=2, a third of my heap)
What I wanted to see is the Eden being resized and see how far it goes.
My guess is that it won't go very far. I think so why I've seen in this other installations a MaxNewSize of 7gb where the total heap was like 13gb...

argv[13]: -XX:MaxPermSize=512m
lol, I forgot to say, somebody left it with 5gb in one of the versions I looked at!!

argv[14]: -XX:+UseParNewGC

What about this guy? could it fragment my memory?

remamber this:

> 133424.645: [GC 133428.565: [ParNew
> Desired survivor size 178946048 bytes, new threshold 16 (max 30)
> - age 1: 52098056 bytes, 52098056 total
> - age 2: 10140640 bytes, 62238696 total
> - age 3: 32191064 bytes, 94429760 total
> : 3620172K->92534K(3844800K), 1533.2368318 secs] 11075182K->7547544K(29010624K), 1537.2197446 secs]

Thats a ParNewGC run isn´t?

argv[15]: -XX:MaxGCPauseMillis=100000

Ergonomics... placebo...

argv[17]: -XX:MaxTenuringThreshold=0
argv[18]: -XX:SurvivorRatio=128
As I mentioned before, I have found that the tenured is not moving, and no Fulls seems to be happening. I also found that this is a common configuration in other deployments.. So I gave a try.

I accidentally deleted yesterdays log from the GC (do'h!) but It will be running for a while, I'll updated if I find anything interesting in the upcoming logs!

Thanks' for the help!

filippo

Message was edited by: flpgdt

briand
Offline
Joined: 2005-07-11

> You see.. that's my big question mark now!
> The logs I have are pretty much what you've seen! No
> full GCs going on!

No signs of CMS activity? Don't look for "Full GC", look for CMS.

> # pagesize -a
> 8192
> 65536
> 524288
> 4194304
> 33554432
> 268435456
>
> That sounds cool, but I'm not familiar with this
> "-XX:LargePageSizeInBytes=N"... could give me some
> more info about this before I stuff it into
> production ? :P

Note that things we suggest, particularly when we have incomplete
information, need to be considered carefully and tested first. Don't
just drop this into your production environment before first getting
a feeling as to how it will behave there.

LargePageSizeInBytes changes the underlying page size of the
Java Heap from it's default, which on this system is 4m, to what
ever size you specify, as long as that's one of the valid page sizes.
The benefit of larger pages is that there are less virtual to physical
page translations in use, which takes pressure off of a hardware
resource know as the TLB (Transition Lookaside Buffer). There's a
fixed number of these things in the processor and larger pages
results in less pressure on this resource.

The drawback of larger pages are that they take longer to zero out.
This can be problematic when we first touch a page. To alleviate this,
you can add -XX:+AlwaysPreTouch to the command line. This shifts
this initial page touching to the startup of the JVM, but that in turn also
increases startup time.

Large pages can introduce internal fragmentation. For example, if
you are short one byte in an existing page, that byte will need to go
into a newly allocated page. If that page is 8k, then you've only
wasted 8k-1 bytes. If it's 256M, then you've wasted 256m-1 bytes.
For the java heap, though, this is not a big concern, particularly if
you are setting -Xms == -Xmx.

Using 256m pages (or even 32m pages) that are available on your
hardware can reduce the thrashing of the TLBs that can occur with
smaller pages sizes. This is particularly useful during garbage collection,
but is also a win during regular execution. This could be a source of
your long pause times -- as we promote objects to the old gen, we
start touching new pages, and those page translations must get loaded.
That may force another translation of of the TLBs, and if we need that
again soon, it will have to be reloaded.

>
> I wish I could agree straight.. but i see no sign of
> CMS around :( - maybe I'm missing some verbose flag
> monitor CMS?
> What I do believe, is about the ParNew fragment or
> something in the Eden. How could I look for it? is

The young generation doesn't fragment. It uses a different GC
policy, where objects are copied from the eden into either
the survivor space or promoted to the old gen. The copies to
the survivors have the natural result of compaction. Promotions
to the old gen, though, need to search through free lists to
find space where the object is to be copied. Once the survivors
are evacuated from the Eden space, there's nothing left in that
space but garbage, so all we do is move the Eden's top pointer
back to the bottom of Eden. Eden starts off fresh again, so there's
no fragmentation. The survivor space is always compacted at
the end of the minor gc, so no fragmentation there either.

> there a flag for checking the fragmentation of the
> memory?
> I know I'm sounding very newbie by now but I cant
> help :(

Yes, there's options for CMS fragmentation reporting, but the
output is not necessarily easy to understand. Since you say there's
no CMS activity going on here (I'd like to see the full log to verify that
claim), you'll never get the CMS fragmentation data.

>
> So, my new set of flags were:
> argv[1]: -server
> argv[2]: -Xss256k
> argv[3]: -Xms28672m
> argv[4]: -Xmx28672m
> argv[10]: -XX:+PrintGCTimeStamps
> argv[11]: -XX:+PrintGCDetails
> argv[16]: -XX:+DisableExplicitGC
> just the usual
>
> argv[8]: -XX:+UseConcMarkSweepGC
> I still have no reason to change it...
>
> argv[12]: -XX:NewSize=512m
> So, I removed the max new size here, which if I'm
> correct leaves me with the default settings for
> solaris with -server (Ratio=2, a third of my heap)
> What I wanted to see is the Eden being resized and
> see how far it goes.
> My guess is that it won't go very far. I think so why
> I've seen in this other installations a MaxNewSize of
> 7gb where the total heap was like 13gb...

With ParNew, the young gen can be larger than the old gen,
if you need it to be. So, don't be afraid of a large young gen.
The only concern with a large young gen is how long it takes
to collect. If too much of it survives, though, you may run into
promotion problems.

> argv[14]: -XX:+UseParNewGC
>
> What about this guy? could it fragment my memory?

This is the young gen collector - This is the default young gen collector
for CMS. If you dropped this argument, you'd still get +UseParNewGC.
To eliminate it, you need -XX:-UseParNewGC - but don't do that - you'll
get the Serial garbage collector instead and that would have horrible
performance consequences with a very large young gen.

There's no risk of memory fragmentation with ParNew, as I described above.

>
>
>
> remamber this:
>
> > 133424.645: [GC 133428.565: [ParNew
> > Desired survivor size 178946048 bytes, new
> threshold 16 (max 30)
> > - age 1: 52098056 bytes, 52098056 total
> > - age 2: 10140640 bytes, 62238696 total
> > - age 3: 32191064 bytes, 94429760 total
> > : 3620172K->92534K(3844800K), 1533.2368318 secs]
> 11075182K->7547544K(29010624K), 1537.2197446 secs]
>
>
>
> Thats a ParNewGC run isn´t?

Yes, but there are other dynamics involved here. OS paging could
cause this and serious fragmentation of the old gen could cause this.

> argv[17]: -XX:MaxTenuringThreshold=0
> argv[18]: -XX:SurvivorRatio=128
> As I mentioned before, I have found that the tenured
> is not moving, and no Fulls seems to be happening. I
> also found that this is a common configuration in
> other deployments.. So I gave a try.

MaxTenuringThreshold=0 is a really bad idea. As is SurvivorRation=128.
It worked for a very small subset of applications (SIP servers and other
telco based apps). It can work well as long as everything runs as expected,
but if you get a spike, it can be very bad. This config was bad enough for
most applications that we eliminated this config as the defaults for CMS.
It will result in everything getting promoted immediately to the old gen,
and CMS will have to run quite frequently. Since most of what it finds
will be garbage, it will more quickly fragment the old gen. Generally, not
a good idea to run this way.

Contact me directly at brian doherty sun com.

>
>
>
> I accidentally deleted yesterdays log from the GC
> (do'h!) but It will be running for a while, I'll
> updated if I find anything interesting in the
> upcoming logs!
>
>
>
> Thanks' for the help!
>
>
>
> filippo
>
> Message was edited by: flpgdt

dwormald
Offline
Joined: 2008-05-05

Is this an application that your developers write? You
might get more mileage out of asking the developers to
find out what java instances are being collected the most
and then have them change the code to *reuse* those
object types rather than rely on GC.
- Dave