Memory leak in G1 garbage collector
I am trying out the G1 collector to hopefully get rid of our problem of fragmentation causing CMS collector to perform full 'stop-the-world' GC from time to time. In our production environment it occurs once every 1 1/2 month or so, but is devastating since it is blocking our entire server application for 9 seconds (4GB heap) when compacting the heap due to the (not so well documented) ParNew (promotion failed) occuring. This happens when the old generation is so fragmented that CMS cannot find enough contigous space causing forcing a full heap compaction to occur.
Sadly, in our labs, when activating the G1 collector (running on JDK 1.6.0_18_ea), it incrementally expands it's memory usage until the system's resources are fully exhausted.
Reading the gc log output, it claims to use no more than the 2GB max I have assigned to it (-Xmx2g), but the OS (64-bit Windows Vista) keeps allocating more and more memory to the process until almost all 8 GB are utilized by the java process. Is this leak well known and is there a way round it? (I will download and try with JDK 1.7 ea as well and see if it exhibits same behaviour).