Skip to main content

Best way to read very large images

10 replies [Last post]
byhisdeeds
Offline
Joined: 2006-01-06

I have to read some very large png images, say 30000 x 30000 pixels, and chop them into smaller images, say 512 x 512 pieces. Can any one say what is the best method to use. Best meaning lowest memory footprint, and best performance. I would prefer not to have to use any special JAI classes as that would mean my application would need additional packages from the standard j2se installation. However if that's what I gotta do then I suppose I could require their installation.

John

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Rosati Claudio

Hi Robert,

having the same problem of John, I'm really interested in
understanding your point.

Can you post a code snippet in order to better understand what you
mean when you spoke of ROI (source region)?
Or can you give me a link to any examples/explanation?

Thanks,

Claudio

+-----------------------------------------------------------------------
-+
|
|\
| _/_/_/_/_/ _/_/_/_/ Claudio
Rosati | \
| _/ _/ _/ Advanced ComputerSystems ACS
S.p.A. | \
| _/ _/ _/ via Della Bufalotta
378 +---+
| _/ _/_/_/_/ I-00139 Roma
(RM) |
| _/ _/ _/
Italy |
| _/ _/ _/ Phone: +39 06 8709
0516 |
| _/_/_/_/_/ _/ _/ E-mail:
claudio.rosati@acsys.it |
|
|
+-----------------------------------------------------------------------
-----+

ATTENZIONE: le informazioni contenute in questo messaggio sono da
considerarsi confidenziali ed il loro utilizzo e' riservato
unicamente al destinatario sopra indicato. Chi dovesse ricevere
questo messaggio per errore e' tenuto ad informare il mittente ed a
rimuoverlo definitivamente da ogni supporto elettronico o cartaceo.

WARNING: This message contains confidential and/or proprietary
information which may be subject to privilege or immunity and which
is intended for use of its addressee only. Should you receive this
message in error, you are kindly requested to inform the sender and
to definitively remove it from any paper or electronic format.

On Dec 2, 2007, at 12:45 AM, robert engels wrote:

> The line
>
> BufferedImage img = ImageIO.read(new BufferedInputStream(new
> FileInputStream(args[0])));
>
> causes the entire image to be read into memory.
>
> The line
>
> BufferedImage out = img.getSubimage(100,100,100,100);
>
> keeps a reference to the original image.
>
> Using the source region is the proper way (which is what I said to
> use - the ROI).
>
>
> On Dec 1, 2007, at 11:54 AM, jai-imageio@javadesktop.org wrote:
>
>> OK. I was a little hasty. When I try the following code to open a
>> 100M PNG to crop a 100x100 pixel section I get out of memory
>> unless I set the heap above 1.4G.
>>
>> BufferedImage img = ImageIO.read(new BufferedInputStream(new
>> FileInputStream(args[0])));
>> BufferedImage out = img.getSubimage(100,100,100,100);
>> ImageIO.write(out, "png", new File(args[1]));
>>
>> However when I try the following code It works with about 4M of
>> heap memory.
>>
>> ImageReader reader = ImageIO.getImageReadersByFormatName
>> ("PNG").next();
>> ImageInputStream iis = ImageIO.createImageInputStream(new
>> FileInputStream(args[0]));
>> reader.setInput(iis);
>> ImageReadParam param = reader.getDefaultReadParam();
>> param.setSourceRegion(new Rectangle(100,100,100,100));
>> BufferedImage out = reader.read(0, param);
>> ImageIO.write(detail, "png", new File(args[1]));
>>
>> Can anybody tell me why this would be. Both methods take about the
>> same time to load and save the image (~37sec).
>> John
>> [Message sent by forum member 'byhisdeeds' (byhisdeeds)]
>>
>> http://forums.java.net/jive/thread.jspa?messageID=248117
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
>> For additional commands, e-mail: interest-help@jai-
>> imageio.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-
> imageio.dev.java.net
>
>

----------------------------------------------------------------------
ATTENZIONE: le informazioni contenute in questo messaggio sono da considerarsi confidenziali ed il loro utilizzo e' riservato unicamente al destinatario sopra indicato. Chi dovesse ricevere questo messaggio per errore e' tenuto ad informare il mittente ed a rimuoverlo definitivamente da ogni supporto elettronico o cartaceo.

WARNING: This message contains confidential and/or proprietary information which may be subject to privilege or immunity and which is intended for use of its addressee only. Should you receive this message in error, you are kindly requested to inform the sender and to definitively remove it from any paper or electronic format.
----------------------------------------------------------------------
[att1.html]

robert engels

It is in the message below... the code using 'source region' works.

On Dec 3, 2007, at 3:06 AM, Rosati Claudio wrote:

> Hi Robert,
>
> having the same problem of John, I'm really interested in
> understanding your point.
>
> Can you post a code snippet in order to better understand what you
> mean when you spoke of ROI (source region)?
> Or can you give me a link to any examples/explanation?
>
> Thanks,
>
> Claudio
>
>
> +---------------------------------------------------------------------
> ---+
> |
> |\
> | _/_/_/_/_/ _/_/_/_/ Claudio
> Rosati | \
> | _/ _/ _/ Advanced ComputerSystems ACS
> S.p.A. | \
> | _/ _/ _/ via Della Bufalotta
> 378 +---+
> | _/ _/_/_/_/ I-00139 Roma
> (RM) |
> | _/ _/ _/
> Italy |
> | _/ _/ _/ Phone: +39 06 8709
> 0516 |
> | _/_/_/_/_/ _/ _/ E-mail:
> claudio.rosati@acsys.it |
> |
> |
> +---------------------------------------------------------------------
> -------+
>
>
> ATTENZIONE: le informazioni contenute in questo messaggio sono da
> considerarsi confidenziali ed il loro utilizzo e' riservato
> unicamente al destinatario sopra indicato. Chi dovesse ricevere
> questo messaggio per errore e' tenuto ad informare il mittente ed a
> rimuoverlo definitivamente da ogni supporto elettronico o cartaceo.
>
> WARNING: This message contains confidential and/or proprietary
> information which may be subject to privilege or immunity and which
> is intended for use of its addressee only. Should you receive this
> message in error, you are kindly requested to inform the sender and
> to definitively remove it from any paper or electronic format.
>
>
> On Dec 2, 2007, at 12:45 AM, robert engels wrote:
>
>> The line
>>
>> BufferedImage img = ImageIO.read(new BufferedInputStream(new
>> FileInputStream(args[0])));
>>
>> causes the entire image to be read into memory.
>>
>> The line
>>
>> BufferedImage out = img.getSubimage(100,100,100,100);
>>
>> keeps a reference to the original image.
>>
>> Using the source region is the proper way (which is what I said to
>> use - the ROI).
>>
>>
>> On Dec 1, 2007, at 11:54 AM, jai-imageio@javadesktop.org wrote:
>>
>>> OK. I was a little hasty. When I try the following code to open a
>>> 100M PNG to crop a 100x100 pixel section I get out of memory
>>> unless I set the heap above 1.4G.
>>>
>>> BufferedImage img = ImageIO.read(new BufferedInputStream(new
>>> FileInputStream(args[0])));
>>> BufferedImage out = img.getSubimage(100,100,100,100);
>>> ImageIO.write(out, "png", new File(args[1]));
>>>
>>> However when I try the following code It works with about 4M of
>>> heap memory.
>>>
>>> ImageReader reader = ImageIO.getImageReadersByFormatName
>>> ("PNG").next();
>>> ImageInputStream iis = ImageIO.createImageInputStream(new
>>> FileInputStream(args[0]));
>>> reader.setInput(iis);
>>> ImageReadParam param = reader.getDefaultReadParam();
>>> param.setSourceRegion(new Rectangle(100,100,100,100));
>>> BufferedImage out = reader.read(0, param);
>>> ImageIO.write(detail, "png", new File(args[1]));
>>>
>>> Can anybody tell me why this would be. Both methods take about
>>> the same time to load and save the image (~37sec).
>>> John
>>> [Message sent by forum member 'byhisdeeds' (byhisdeeds)]
>>>
>>> http://forums.java.net/jive/thread.jspa?messageID=248117
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: interest-unsubscribe@jai-
>>> imageio.dev.java.net
>>> For additional commands, e-mail: interest-help@jai-
>>> imageio.dev.java.net
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
>> For additional commands, e-mail: interest-help@jai-
>> imageio.dev.java.net
>>
>>
>
>
>
> ATTENZIONE: le informazioni contenute in questo messaggio sono da
> considerarsi confidenziali ed il loro utilizzo e' riservato
> unicamente al destinatario sopra indicato. Chi dovesse ricevere
> questo messaggio per errore e' tenuto ad informare il mittente ed a
> rimuoverlo definitivamente da ogni supporto elettronico o cartaceo.
>
> WARNING: This message contains confidential and/or proprietary
> information which may be subject to privilege or immunity and which
> is intended for use of its addressee only. Should you receive this
> message in error, you are kindly requested to inform the sender and
> to definitively remove it from any paper or electronic format.
>

[att1.html]

bpb
Offline
Joined: 2004-06-23

John,

Be forewarned that the setSourceRegion() approach will not get you memory reduction in all cases. It's a function of the format, the image layout, and in some cases the implementation.

[b]PNG[/b]

You've already verified that setSourceRegion() works for the Java SE PNG reader but I believe it will not solve the problem if the JAI Image I/O Tools PNG reader is used as the entire image will be loaded anyway in that case and merely copied to the target buffer.

[b]JPEG[/b]

I don't believe that either implementation of the JPEG reader is able to selectively read a region without decompressing the entire image. This is partly an artifact of the implementations and partly of the nature of the format and compression methodology itself.

[b]TIFF[/b]

Whether a region may be efficiently extracted depends on the compression type and the tiling/strip layout of the data. If the data are not compressed then an arbitrary region may be extracted efficiently regardless of the strip/tiling layout. If the data are compressed then each strip or tile overlapping the region is decompressed in its entirety. This will cause increased memory usage which is dependent on the strip/tile layout.

I would also like to point out this method

http://java.sun.com/j2se/1.5.0/docs/api/javax/imageio/ImageReader.html#isRandomAccessEasy(int)

which ideally returns "true" if and only if extracting an arbitrary region is efficient.

Brian

> Thanks. I will try a small test program to see if the
> memory of the originalimage is a factor.
>
> I do not have a choice of the format of the input
> images. Some maybe png, others maybe jpg, or tiff. So
> I have to be able to deal with all.
>
> John

bpb
Offline
Joined: 2004-06-23

> It may not help though. Internally the imageio PNG
> reader uses a
> MediaLibImage, and I am uncertain if that loads
> everything into
> memory. Sun folks?

I believe that to be the case.

> My cursory examination of the code shosw that the
> pure Java J2SE
> imageio PNG plugin appears to only read what is
> requested.

The subsequent testing using setSourceRegion() appears to validate this which is also as I thought.

Brian

robert engels

It does not appear that PNG supports tiling.

SO, I think you are going to have to set the region of interest (ROI)
on the read. PNG does not support tiling, but by using ROI you can
read just a portion into memory, and write the other images.

It may not help though. Internally the imageio PNG reader uses a
MediaLibImage, and I am uncertain if that loads everything into
memory. Sun folks?

My cursory examination of the code shosw that the pure Java J2SE
imageio PNG plugin appears to only read what is requested.

BTW, if you have a choice, the source image should be a tiled TIFF,
not a PNG.

On Nov 30, 2007, at 8:49 PM, jai-imageio@javadesktop.org wrote:

> I have to read some very large png images, say 30000 x 30000
> pixels, and chop them into smaller images, say 512 x 512 pieces.
> Can any one say what is the best method to use. Best meaning lowest
> memory footprint, and best performance. I would prefer not to have
> to use any special JAI classes as that would mean my application
> would need additional packages from the standard j2se installation.
> However if that's what I gotta do then I suppose I could require
> their installation.
>
> John
> [Message sent by forum member 'byhisdeeds' (byhisdeeds)]
>
> http://forums.java.net/jive/thread.jspa?messageID=248083
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-
> imageio.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

byhisdeeds
Offline
Joined: 2006-01-06

Thanks. I will try a small test program to see if the memory of the originalimage is a factor.

I do not have a choice of the format of the input images. Some maybe png, others maybe jpg, or tiff. So I have to be able to deal with all.

John

byhisdeeds
Offline
Joined: 2006-01-06

When I try to open a 100M png file using ImageIO and getSubImage(..) I need over 1G of heap space. Guess I'll try JAI

John

byhisdeeds
Offline
Joined: 2006-01-06

OK. I was a little hasty. When I try the following code to open a 100M PNG to crop a 100x100 pixel section I get out of memory unless I set the heap above 1.4G.

BufferedImage img = ImageIO.read(new BufferedInputStream(new FileInputStream(args[0])));
BufferedImage out = img.getSubimage(100,100,100,100);
ImageIO.write(out, "png", new File(args[1]));

However when I try the following code It works with about 4M of heap memory.

ImageReader reader = ImageIO.getImageReadersByFormatName("PNG").next();
ImageInputStream iis = ImageIO.createImageInputStream(new FileInputStream(args[0]));
reader.setInput(iis);
ImageReadParam param = reader.getDefaultReadParam();
param.setSourceRegion(new Rectangle(100,100,100,100));
BufferedImage out = reader.read(0, param);
ImageIO.write(detail, "png", new File(args[1]));

Can anybody tell me why this would be. Both methods take about the same time to load and save the image (~37sec).
John

robert engels

The line

BufferedImage img = ImageIO.read(new BufferedInputStream(new
FileInputStream(args[0])));

causes the entire image to be read into memory.

The line

BufferedImage out = img.getSubimage(100,100,100,100);

keeps a reference to the original image.

Using the source region is the proper way (which is what I said to
use - the ROI).

On Dec 1, 2007, at 11:54 AM, jai-imageio@javadesktop.org wrote:

> OK. I was a little hasty. When I try the following code to open a
> 100M PNG to crop a 100x100 pixel section I get out of memory unless
> I set the heap above 1.4G.
>
> BufferedImage img = ImageIO.read(new BufferedInputStream(new
> FileInputStream(args[0])));
> BufferedImage out = img.getSubimage(100,100,100,100);
> ImageIO.write(out, "png", new File(args[1]));
>
> However when I try the following code It works with about 4M of
> heap memory.
>
> ImageReader reader = ImageIO.getImageReadersByFormatName("PNG").next
> ();
> ImageInputStream iis = ImageIO.createImageInputStream(new
> FileInputStream(args[0]));
> reader.setInput(iis);
> ImageReadParam param = reader.getDefaultReadParam();
> param.setSourceRegion(new Rectangle(100,100,100,100));
> BufferedImage out = reader.read(0, param);
> ImageIO.write(detail, "png", new File(args[1]));
>
> Can anybody tell me why this would be. Both methods take about the
> same time to load and save the image (~37sec).
> John
> [Message sent by forum member 'byhisdeeds' (byhisdeeds)]
>
> http://forums.java.net/jive/thread.jspa?messageID=248117
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-
> imageio.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

byhisdeeds
Offline
Joined: 2006-01-06

Sorry I didn't associate ROI with source region. I'm doing a number of things at the same time. I see the issue now.

Because I have to cut up the large image into smaller images, the first method of loading the whole image into memory initially, and then saving subimages, gives the best performance (speed).

However what I'm gonna look at is loading strips of regions from the source image, then saving the sub images from this region, then loading the next strip, and so on.

It will allow me some tradeoff between fastest response and memory requirements.

John