Skip to main content

[JAI-IMAGEIO] Leak in ImageIO.read() for pngs.

7 replies [Last post]
Anonymous

Given the following program on Linux, there is an appalling memory leak:

---------------------------------------------
import java.io.File;
import javax.imageio.ImageIO;

public class ImageTest {

public static void main(String argv[]) throws Throwable {
// don't leak file handles
ImageIO.setUseCache(false);

for(int i=0; i<1000000; i++){
// read it (leak in PNG case)
ImageIO.read(new File(argv[0]));
}
}
}
---------------------------------------------

The VM will grow unbounded (past -Xmx values) when reading a PNG file.
(I've attached a sample gif and png)

After 45 seconds:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11487 nicp 19 0 484m 237m 8976 R 96.7 23.5 0:44.35 java

After 120 seconds:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11487 nicp 16 0 887m 619m 8908 R 96.3 61.4 1:58.87 java

Eventually the VM will crash, presumably on a failed malloc.

If I use the "-Dcom.sun.media.imageio.disableCodecLib=true" switch, then
it will not grow, but things are over twice as slow.

According to:

This was fixed in the latest versions of jai, but I'm not having any
luck with this setup:

CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
LD_LIBRARY_PATH=/home/nicp/test/server/lib:

[nicp@moe server]$ ls /home/nicp/test/server/lib/
clibwrapper_jiio.jar
libclib_jiio.so
jai_imageio.jar

Which are all from the latest daily build:
jai-imageio-1_1-pre-dr-b04-lib-linux-i586-31_Aug_2006.zip

Any clues? Is my environment not picking up the new imageio libraries?

-Nic

[test.png]
[test.gif]
---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
James Cheng

Hi Nic,

> At least this all coincides with the bug already having been fixed, now
> it's just a matter of getting the new libraries working. Any help
> greatly appreciated.

You seem to have an older version of jai-imageio installed somewhere.
Can you do a 'find

-name "libclib_jiio.so"' in your file system?
It might be under jre/lib/i386/.

You might like to check the jar files under jre/lib/ext/ as well.
By default, the JVM uses CLASSPATH and LD_LIBRARY_PATH as the last
resort.

http://java.sun.com/j2se/1.5.0/docs/guide/extensions/spec.html

HTH,
-James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

James Cheng

Hi Nic,

> You seem to have an older version of jai-imageio installed somewhere.
> Can you do a 'find

-name "libclib_jiio.so"' in your file system?
> It might be under jre/lib/i386/.
>
> You might like to check the jar files under jre/lib/ext/ as well.
> By default, the JVM uses CLASSPATH and LD_LIBRARY_PATH as the last
> resort.

Forgot to mention that the following properties can be redefined for
testing/debugging purposes:

java.class.path Java class path
java.library.path List of paths to search when loading libraries
java.ext.dirs Path of extension directory or directories

Regards,
-James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

James Cheng

Hi Nic,

> Given the following program on Linux, there is an appalling memory leak:

I haven't tried it on Linux. But I will try it later, as I need to
reboot a box to Linux.

On Solaris, I saw the resident memory growing, as you observed, but
in the same way with or without using codecLib.

>
> ---------------------------------------------
> import java.io.File;
> import javax.imageio.ImageIO;
>
> public class ImageTest {
>
> public static void main(String argv[]) throws Throwable {
> // don't leak file handles
> ImageIO.setUseCache(false);
>
> for(int i=0; i<1000000; i++){
> // read it (leak in PNG case)
> ImageIO.read(new File(argv[0]));

When I inserted System.gc() here, the resident memory didn't grow after
reaching 26M. Again, it's the same with or without using codecLib.

> }
> }
> }
> ---------------------------------------------
>
> The VM will grow unbounded (past -Xmx values) when reading a PNG file.
> (I've attached a sample gif and png)
>
> After 45 seconds:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 11487 nicp 19 0 484m 237m 8976 R 96.7 23.5 0:44.35 java
>
> After 120 seconds:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 11487 nicp 16 0 887m 619m 8908 R 96.3 61.4 1:58.87 java
>
> Eventually the VM will crash, presumably on a failed malloc.
>
> If I use the "-Dcom.sun.media.imageio.disableCodecLib=true" switch, then
> it will not grow, but things are over twice as slow.
>
> According to:
>
>
> This was fixed in the latest versions of jai, but I'm not having any
> luck with this setup:
>
> CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
>
> LD_LIBRARY_PATH=/home/nicp/test/server/lib:
>
> [nicp@moe server]$ ls /home/nicp/test/server/lib/
> clibwrapper_jiio.jar
> libclib_jiio.so
> jai_imageio.jar
>
> Which are all from the latest daily build:
> jai-imageio-1_1-pre-dr-b04-lib-linux-i586-31_Aug_2006.zip
>
> Any clues? Is my environment not picking up the new imageio libraries?

You might like to use the following line of code to check the version
of the codecLib decoder:

System.out.println(com.sun.medialib.codec.png.Decoder.getVersion());

If it has something like png:0100:20060504:..., then it's the latest.

Let me know whether you still see a leak after inserting System.gc().

Thanks,
-James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

Nic Pottier

Hi James, thanks for the reply.

The bad news is that System.gc() doesn't make any difference. (it grows
more slowly but still unbounded)

The good news is it appears that I'm not picking up the new imageio libs
as I get version: png:0100:20050215:javawrapper

So that brings me back to my setup:
* I have the new imageio jars in my classpath:
CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
* I have the new SO in my LD_LIBRARY_PATH:
LD_LIBRARY_PATH=/home/nicp/test/server/lib:

[nicp@moe server]$ ls /home/nicp/test/server/lib/*.so
/home/nicp/test/server/lib/libclib_jiio.so

This is on redhat linux:
% Linux moe 2.6.11-1.1369_FC4smp #1 SMP Thu Jun 2 23:08:39 EDT 2005 i686
i686 i386 GNU/Linux

At least this all coincides with the bug already having been fixed, now
it's just a matter of getting the new libraries working. Any help
greatly appreciated.

-Nic

James Cheng wrote:
> Hi Nic,
>
>> Given the following program on Linux, there is an appalling memory leak:
>
> I haven't tried it on Linux. But I will try it later, as I need to
> reboot a box to Linux.
>
> On Solaris, I saw the resident memory growing, as you observed, but
> in the same way with or without using codecLib.
>
>>
>> ---------------------------------------------
>> import java.io.File;
>> import javax.imageio.ImageIO;
>>
>> public class ImageTest {
>>
>> public static void main(String argv[]) throws Throwable {
>> // don't leak file handles
>> ImageIO.setUseCache(false);
>>
>> for(int i=0; i<1000000; i++){
>> // read it (leak in PNG case)
>> ImageIO.read(new File(argv[0]));
>
> When I inserted System.gc() here, the resident memory didn't grow after
> reaching 26M. Again, it's the same with or without using codecLib.
>
>> }
>> }
>> }
>> ---------------------------------------------
>>
>> The VM will grow unbounded (past -Xmx values) when reading a PNG file.
>> (I've attached a sample gif and png)
>>
>> After 45 seconds:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 11487 nicp 19 0 484m 237m 8976 R 96.7 23.5 0:44.35 java
>>
>> After 120 seconds:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 11487 nicp 16 0 887m 619m 8908 R 96.3 61.4 1:58.87 java
>>
>> Eventually the VM will crash, presumably on a failed malloc.
>>
>> If I use the "-Dcom.sun.media.imageio.disableCodecLib=true" switch,
>> then it will not grow, but things are over twice as slow.
>>
>> According to:
>>
>>
>> This was fixed in the latest versions of jai, but I'm not having any
>> luck with this setup:
>>
>> CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
>>
>> LD_LIBRARY_PATH=/home/nicp/test/server/lib:
>>
>> [nicp@moe server]$ ls /home/nicp/test/server/lib/
>> clibwrapper_jiio.jar
>> libclib_jiio.so
>> jai_imageio.jar
>>
>> Which are all from the latest daily build:
>> jai-imageio-1_1-pre-dr-b04-lib-linux-i586-31_Aug_2006.zip
>>
>> Any clues? Is my environment not picking up the new imageio libraries?
>
> You might like to use the following line of code to check the version
> of the codecLib decoder:
>
> System.out.println(com.sun.medialib.codec.png.Decoder.getVersion());
>
> If it has something like png:0100:20060504:..., then it's the latest.
>
> Let me know whether you still see a leak after inserting System.gc().
>
> Thanks,
> -James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

Nic Pottier

Oh and one piece of critical missing information:

[nicp@moe server]$ java -version
java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
[nicp@moe server]$

Nic Pottier wrote:
> Hi James, thanks for the reply.
>
> The bad news is that System.gc() doesn't make any difference. (it grows
> more slowly but still unbounded)
>
> The good news is it appears that I'm not picking up the new imageio libs
> as I get version: png:0100:20050215:javawrapper
>
> So that brings me back to my setup:
> * I have the new imageio jars in my classpath:
> CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
>
> * I have the new SO in my LD_LIBRARY_PATH:
> LD_LIBRARY_PATH=/home/nicp/test/server/lib:
>
> [nicp@moe server]$ ls /home/nicp/test/server/lib/*.so
> /home/nicp/test/server/lib/libclib_jiio.so
>
> This is on redhat linux:
> % Linux moe 2.6.11-1.1369_FC4smp #1 SMP Thu Jun 2 23:08:39 EDT 2005 i686
> i686 i386 GNU/Linux
>
> At least this all coincides with the bug already having been fixed, now
> it's just a matter of getting the new libraries working. Any help
> greatly appreciated.
>
> -Nic
>
> James Cheng wrote:
>> Hi Nic,
>>
>>> Given the following program on Linux, there is an appalling memory leak:
>>
>> I haven't tried it on Linux. But I will try it later, as I need to
>> reboot a box to Linux.
>>
>> On Solaris, I saw the resident memory growing, as you observed, but
>> in the same way with or without using codecLib.
>>
>>>
>>> ---------------------------------------------
>>> import java.io.File;
>>> import javax.imageio.ImageIO;
>>>
>>> public class ImageTest {
>>>
>>> public static void main(String argv[]) throws Throwable {
>>> // don't leak file handles
>>> ImageIO.setUseCache(false);
>>>
>>> for(int i=0; i<1000000; i++){
>>> // read it (leak in PNG case)
>>> ImageIO.read(new File(argv[0]));
>>
>> When I inserted System.gc() here, the resident memory didn't grow after
>> reaching 26M. Again, it's the same with or without using codecLib.
>>
>>> }
>>> }
>>> }
>>> ---------------------------------------------
>>>
>>> The VM will grow unbounded (past -Xmx values) when reading a PNG
>>> file. (I've attached a sample gif and png)
>>>
>>> After 45 seconds:
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 11487 nicp 19 0 484m 237m 8976 R 96.7 23.5 0:44.35 java
>>>
>>> After 120 seconds:
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 11487 nicp 16 0 887m 619m 8908 R 96.3 61.4 1:58.87 java
>>>
>>> Eventually the VM will crash, presumably on a failed malloc.
>>>
>>> If I use the "-Dcom.sun.media.imageio.disableCodecLib=true" switch,
>>> then it will not grow, but things are over twice as slow.
>>>
>>> According to:
>>>
>>>
>>> This was fixed in the latest versions of jai, but I'm not having any
>>> luck with this setup:
>>>
>>> CLASSPATH=/home/nicp/test/server/lib/jai_imageio.jar:/home/nicp/test/server/lib/clibwrapper_jiio.jar:.
>>>
>>> LD_LIBRARY_PATH=/home/nicp/test/server/lib:
>>>
>>> [nicp@moe server]$ ls /home/nicp/test/server/lib/
>>> clibwrapper_jiio.jar
>>> libclib_jiio.so
>>> jai_imageio.jar
>>>
>>> Which are all from the latest daily build:
>>> jai-imageio-1_1-pre-dr-b04-lib-linux-i586-31_Aug_2006.zip
>>>
>>> Any clues? Is my environment not picking up the new imageio libraries?
>>
>> You might like to use the following line of code to check the version
>> of the codecLib decoder:
>>
>> System.out.println(com.sun.medialib.codec.png.Decoder.getVersion());
>>
>> If it has something like png:0100:20060504:..., then it's the latest.
>>
>> Let me know whether you still see a leak after inserting System.gc().
>>
>> Thanks,
>> -James
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-imageio.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

James Cheng

Hi Nic,

On 08/31/06 02:25 PM, Nic Pottier wrote:
> Oh and one piece of critical missing information:
>
> [nicp@moe server]$ java -version
> java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> [nicp@moe server]$

On Linux 2.4.21-199-smp4G, with this build of libclib_jiio.so:

png:0100:20060505:386|png:0100:20060505:386_jni

with JDK 1.4.2_04 or 1.5.0_06, and without System.gc(), I did not see
any memory leak.

Regards,
-James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

Nic Pottier

Yup, I've now verified that it is fixed in that version as well.

The crux was using the java.ext switch:

[nicp@moe server]$ java
-Djava.ext.dirs=/home/nicp/test/server/lib:$JAVA_HOME/jre/lib/ext
ImageTest source.png

I couldn't find any other way to get java to pick up my new version of
imageio over JDK 1.5's included version.

With the above it will pick up my new version of imageio without
modifying the jre or jdk libs.

Thanks much James! One less thorn in my side now.

-Nic

James Cheng wrote:
> Hi Nic,
>
> On 08/31/06 02:25 PM, Nic Pottier wrote:
>> Oh and one piece of critical missing information:
>>
>> [nicp@moe server]$ java -version
>> java version "1.5.0_04"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
>> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
>> [nicp@moe server]$
>
> On Linux 2.4.21-199-smp4G, with this build of libclib_jiio.so:
>
> png:0100:20060505:386|png:0100:20060505:386_jni
>
> with JDK 1.4.2_04 or 1.5.0_06, and without System.gc(), I did not see
> any memory leak.
>
> Regards,
> -James

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net