Skip to main content

Reading huge files without having to load everything in memory

8 replies [Last post]
felipequintella
Offline
Joined: 2009-05-20
Points: 0

Hello everyone,

I am new to JAI development, and I'm working on a GIS project. My objective is to read huge files (up to 2gb or more) and process them, without having to get everything in memory, so one could use a heap space of 64mb.

I know I have to tile the image, and read the tiles as I need, but I'm having some problems with this.

I used the setSourceRegion() method to define my region of interest, and passed the corresponding ImageReadParam to the ImageReader, finally to call the readAsRenderedImage method. But the resulting RenderedImage is actually the whole image, and not only the region I defined.

Do you know how can I achieve this?

Here the code I'm using:

ImageReadParam imageReadParam = new TIFFImageReadParam();
RenderedOp resultOp = ImageReadDescriptor.create(new FileImageInputStream(new File("radar.tif")),0,true,false,true,null,null, imageReadParam,null,null);

ImageReader imageReader = (ImageReader)resultOp.getProperty("JAI.ImageReader");

imageReadParam.setSourceRegion(new Rectangle(40,50,90,110));
imageReadParam.setSourceSubsampling(1, 1, 0, 0);
RenderedImage image = imageReader.readAsRenderedImage(0, imageReadParam);
RasterMetadata rasterMetadata = new RasterMetadata(0,0,0,0,0,0);
GridCoverageFactory.createGridCoverage(image, rasterMetadata).show(); //This is the method that shows the image, for debuging purposes only. The image shown should be only the region I defined, but loads as the whole image (consuming all my memory, if i use large files)

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Andrey Kuznetsov

As Simone already said you should use ImageRead operation.

Here is code:

public static RenderedImage readTiled(File f, int tileWidth, int
tileHeight) {

ImageInputStream iis = ImageIO.createImageInputStream(f);
ParameterBlockJAI pbj = new ParameterBlockJAI("ImageRead");

ImageLayout layout = new ImageLayout();
layout.setTileWidth(tileWidth);
layout.setTileHeight(tileHeight);
RenderingHints hints = new RenderingHints(JAI.KEY_IMAGE_LAYOUT,
layout);

pbj.setParameter("Input", iis);
return JAI.create("ImageRead", pbj, hints);
}

Andrey

jai-imageio@javadesktop.org schrieb:
> Thank you for your answer, Simone.
>
> I've already looked at the geotools project.
>
> However, I'm still having some problems. I've managed to set up a image input, tile it, and read the tiles as I need them, but it seems as the TileCache is not working, because if I loop through all the tiles, eventually i get out of memory.
>
> Normally the tileCache should flush the old tiles to prevent a outofmemory exception, or am I wrong?
> [i]
> RenderedOp resultOp = ImageReadDescriptor.create(
> new FileImageInputStream(
> new File("/home/felipe/Bureau/Nantes/Nantes_est.tif")),0,true,false,true,null,null,null,null,null);
> ImageReader imageReader = (ImageReader)resultOp.getProperty("JAI.ImageReader");
>
> ImageReadParam imageReadParam = imageReader.getDefaultReadParam();
> TiledImage image = new TiledImage(imageReader.readAsRenderedImage(0, imageReadParam),300,300);
>
> for (int i=0;i > for (int j=0;j > {
> image.getTile(i, j);
> System.out.println(i+" "+j);
> }[/i]
>
> And the console output:
> ....
> 16 92
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> So, it was able to compute nearly 1500 tiles before running out of memory.
> Also, the access time to each uncomputed tile is surprisingly big, 1sec to get each tile, what brings me to 30min just to access all the tiles in my image (i'm talking about a 1,4GB, 30000x15000 pixels image)
> What am I missing?
> [Message sent by forum member 'felipequintella' (felipequintella)]
>
> http://forums.java.net/jive/thread.jspa?messageID=347565
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-imageio.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

felipequintella
Offline
Joined: 2009-05-20
Points: 0

Andrey,

Thank you for the example code in how to use ImageRead operation.

I implemented this operation, and everything runs as it should now. My only complain is that the operations take a really long time (for my big images), but I think that's something I can't change.

Thank you again Andrey and Simone for your help.

Felipe

Simone Giannecchini

I'd recommend you to have a look at the geotools project.

Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Wed, May 20, 2009 at 3:44 PM, wrote:
> Hello everyone,
>
> I am new to JAI development, and I'm working on a GIS project. My objective is to read huge files (up to 2gb or more) an process them, without having to get everything in memory, so one could use a heap space of 64mb.
>
> I know I have to tile the image, and read the tiles as I need, but I'm having some problems with this.
>
> I used the setSourceRegion() method to define my region of interest, and passed the corresponding ImageReadParam to the ImageReader, finally to call the readAsRenderedImage method. But the resulting RenderedImage is actually the whole image, and not only the region I defined.
>
> Do you know how can I achieve this?
>
> Here the code I'm using:
> [i]
> ImageReadParam imageReadParam = new TIFFImageReadParam();
> RenderedOp resultOp = ImageReadDescriptor.create(new FileImageInputStream(new File("radar.tif")),0,true,false,true,null,null, imageReadParam,null,null);
>
> ImageReader imageReader = (ImageReader)resultOp.getProperty("JAI.ImageReader");
>
> imageReadParam.setSourceRegion(new Rectangle(40,50,90,110));
> imageReadParam.setSourceSubsampling(1, 1, 0, 0);
> RenderedImage image = imageReader.readAsRenderedImage(0, imageReadParam);
> RasterMetadata rasterMetadata = new RasterMetadata(0,0,0,0,0,0);
> GridCoverageFactory.createGridCoverage(image, rasterMetadata).show(); //This is the method that shows the image, for debuging purposes only. The image shown should be only the region I defined, but loads as the whole image (consuming all my memory, if i use large files)[/i]
> [Message sent by forum member 'felipequintella' (felipequintella)]
>
> http://forums.java.net/jive/thread.jspa?messageID=346987
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-imageio.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

felipequintella
Offline
Joined: 2009-05-20
Points: 0

Thank you for your answer, Simone.

I've already looked at the geotools project.

However, I'm still having some problems. I've managed to set up a image input, tile it, and read the tiles as I need them, but it seems as the TileCache is not working, because if I loop through all the tiles, eventually i get out of memory.

Normally the tileCache should flush the old tiles to prevent a outofmemory exception, or am I wrong?
[i]
RenderedOp resultOp = ImageReadDescriptor.create(
new FileImageInputStream(
new File("/home/felipe/Bureau/Nantes/Nantes_est.tif")),0,true,false,true,null,null,null,null,null);
ImageReader imageReader = (ImageReader)resultOp.getProperty("JAI.ImageReader");

ImageReadParam imageReadParam = imageReader.getDefaultReadParam();
TiledImage image = new TiledImage(imageReader.readAsRenderedImage(0, imageReadParam),300,300);

for (int i=0;i for (int j=0;j {
image.getTile(i, j);
System.out.println(i+" "+j);
}[/i]

And the console output:
....
16 92
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

So, it was able to compute nearly 1500 tiles before running out of memory.
Also, the access time to each uncomputed tile is surprisingly big, 1sec to get each tile, what brings me to 30min just to access all the tiles in my image (i'm talking about a 1,4GB, 30000x15000 pixels image)
What am I missing?

Simone Giannecchini

Ciao Felipe,
I understand you have looked at geotools, I'd suggest to look again :-).

You are making a few errors here, above all:

1> you are not using the JAI imageread operation (this is not an
error, but let's say aperfectible approach)
2> you are loading your image in meory entirely by trying to create a
TiledImage with it.

In geotools parts of these issues are handled under the hood, at least
for gis data, that's why I suggested you to try it again and post a
question there.

Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Mon, May 25, 2009 at 5:32 PM, wrote:
> Thank you for your answer, Simone.
>
> I've already looked at the geotools project.
>
> However, I'm still having some problems. I've managed to set up a image input, tile it, and read the tiles as I need them, but it seems as the TileCache is not working, because if I loop through all the tiles, eventually i get out of memory.
>
> Normally the tileCache should flush the old tiles to prevent a outofmemory exception, or am I wrong?
> [i]
> RenderedOp resultOp = ImageReadDescriptor.create(
>                                new FileImageInputStream(
>                                                new File("/home/felipe/Bureau/Nantes/Nantes_est.tif")),0,true,false,true,null,null,null,null,null);
> ImageReader imageReader = (ImageReader)resultOp.getProperty("JAI.ImageReader");
>
>                ImageReadParam imageReadParam = imageReader.getDefaultReadParam();
>                TiledImage image = new TiledImage(imageReader.readAsRenderedImage(0, imageReadParam),300,300);
>
>                for (int i=0;i >                        for (int j=0;j >                        {
>                                image.getTile(i, j);
>                                System.out.println(i+" "+j);
>                        }[/i]
>
> And the console output:
> ....
> 16 92
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> So, it was able to compute nearly 1500 tiles before running out of memory.
> Also, the access time to each uncomputed tile is surprisingly big, 1sec to get each tile, what brings me to 30min just to access all the tiles in my image (i'm talking about a 1,4GB, 30000x15000 pixels image)
> What am I missing?
> [Message sent by forum member 'felipequintella' (felipequintella)]
>
> http://forums.java.net/jive/thread.jspa?messageID=347565
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-imageio.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

felipequintella
Offline
Joined: 2009-05-20
Points: 0

Thank you again, Simone.

I'll care to take another look at Geotools and see if I can find something there too.

About the ImageRead operation, I tought that using the ImageReadDescriptor was the same thing, with another approach, but I must have overseen something. Also, I tought that a TiledImage computed its tiles just when needed. But, again, I'll take another look at Geotools.

I'll get back with a feedback.

Meanwhile, if you have any more tips about how to correctly use the ImageRead operation, I would be glad to learn.

Thanks again,
Felipe.

Simone Giannecchini

Ciao Felipe,
see below...
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Mon, May 25, 2009 at 8:50 PM, wrote:
> Thank you again, Simone.
>
> I'll care to take another look at Geotools and see if I can find something there too.
>
> About the ImageRead operation, I tought that using the ImageReadDescriptor was the same thing, with another approach, but I must have overseen something.

In my understanding, there is not guarantee that you won't get back
simply a BufferedImage, while with JAI ImageRead you get a RenderedOp
backed by an ImageReadOp which let you control tiling, multithreading
and so on.

> Also, I tought that a TiledImage computed its tiles just when needed. But, again, I'll take another look at Geotools.

Yhea, that's tue, but it keep an hard reference to them, this means
that if you iterate over your TiledImage, you'd orce the loading for
the underline tiles and you'd keep them in memory.

Simone.

>
> I'll get back with a feedback.
>
> Meanwhile, if you have any more tips about how to correctly use the ImageRead operation, I would be glad to learn.
>
> Thanks again,
> Felipe.
> [Message sent by forum member 'felipequintella' (felipequintella)]
>
> http://forums.java.net/jive/thread.jspa?messageID=347596
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
> For additional commands, e-mail: interest-help@jai-imageio.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai-imageio.dev.java.net
For additional commands, e-mail: interest-help@jai-imageio.dev.java.net

felipequintella
Offline
Joined: 2009-05-20
Points: 0

Simone,
Thank you for the explanations. I'm starting to understand a little more how all this works. I was a little confused in why there's the ImageReadDescriptor, if we use directly the ImageRead operation with JAI.create().

I also didn't know that a TiledImage kept a hard reference to all its computed tiles. That's why I kept running out of memory.

I will continue to search for more information at Geotools and the web.

Thank you for your help,
Felipe.