Skip to main content

Processing large images

9 replies [Last post]
nsch
Offline
Joined: 2009-04-19

Hi,

I tried to load, process and store some large images (93 MB, 32-bit floating TIFF) with JAI. In the processing chain I tried to smooth the image 6 times with the "convolve" operator and after that subtract the smoothed image from the original image with teh "subtract" operator.
All this takes so much time that I stopped the process. Is it possible to do such a processing chain with JAI or do I have to find another way?

best regards,
Natalie

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
felipequintella
Offline
Joined: 2009-05-20

Hello everyone,

I'm having a very similar issue myself.

I am trying to load/process/save a 200mb and a 1,3gb TIFF file, and I'm also experiencing incredibly long execution times.

For the 200mb file, the process to open, convolve a gaussian filter and save took 40min. I haven't tried to save the 1,3gb file yet, but appling a Extrema operator on this file took nothing less than 25 minutes.

I hope there's a way to make things faster.

Have you solved your problem in some way?

Thanks,
Felipe

Simone Giannecchini

Felipe,
much of this slowdowns may depend on the structure of yuor tiff files.
If you have a large striped tiff as an instance performances are
usually bad, hence you'd better retile the raster on read via an
ImageLayout.

Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Tue, May 26, 2009 at 9:15 PM, wrote:
> Hello everyone,
>
> I'm having a very similar issue myself.
>
> I am trying to load/process/save a 200mb and a 1,3gb TIFF file, and I'm also experiencing incredibly long execution times.
>
> For the 200mb file, the process to open, convolve a gaussian filter and save took 40min. I haven't tried to save the 1,3gb file yet, but appling a Extrema operator on this file took nothing less than 25 minutes.
>
> I hope there's a way to make things faster.
>
> Have you solved your problem in some way?
> [Message sent by forum member 'felipequintella' (felipequintella)]
>
> http://forums.java.net/jive/thread.jspa?messageID=347733
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
> For additional commands, e-mail: interest-help@jai.dev.java.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
For additional commands, e-mail: interest-help@jai.dev.java.net

Felipe Quintella Correia

Simone,
I think I'm working with stripped tiffs, but I'm not sure (have this tag on
one of them: ).

My reading procedure is this one:

ImageInputStream iis = ImageIO.createImageInputStream(new
File(fileName));

ParameterBlockJAI pbj = new ParameterBlockJAI("ImageRead");

pbj.setParameter("Input", iis);

ImageLayout layout = new ImageLayout();
layout.setTileWidth(tileWidth);
layout.setTileHeight(tileHeight);
RenderingHints hints = new
RenderingHints(JAI.KEY_IMAGE_LAYOUT,layout);

renderedImage = JAI.create("ImageRead", pbj, hints);

If by retile the raster on read, you mean this, them I'm pretty sure that's
what I'm doing. One thing I noticed is that performances were incredibly
boosted once I increased TileCached size (from default to 1024*1024*300) and
used a heap size of 512M.

I don't know if this is the correct answer to my problem, but it seem to be
working. Do you second this?

Thank you again,
Felipe Quintella

2009/5/27 Simone Giannecchini

> Felipe,
> much of this slowdowns may depend on the structure of yuor tiff files.
> If you have a large striped tiff as an instance performances are
> usually bad, hence you'd better retile the raster on read via an
> ImageLayout.
>
> Simone.
> -------------------------------------------------------
> Ing. Simone Giannecchini
> GeoSolutions S.A.S.
> Owner - Software Engineer
> Via Carignoni 51
> 55041 Camaiore (LU)
> Italy
>
> phone: +39 0584983027
> fax: +39 0584983027
> mob: +39 333 8128928
>
>
> http://www.geo-solutions.it
> http://simboss.blogspot.com/
> http://www.linkedin.com/in/simonegiannecchini
>
> -------------------------------------------------------
>
>
>
> On Tue, May 26, 2009 at 9:15 PM, wrote:
> > Hello everyone,
> >
> > I'm having a very similar issue myself.
> >
> > I am trying to load/process/save a 200mb and a 1,3gb TIFF file, and I'm
> also experiencing incredibly long execution times.
> >
> > For the 200mb file, the process to open, convolve a gaussian filter and
> save took 40min. I haven't tried to save the 1,3gb file yet, but appling a
> Extrema operator on this file took nothing less than 25 minutes.
> >
> > I hope there's a way to make things faster.
> >
> > Have you solved your problem in some way?
> > [Message sent by forum member 'felipequintella' (felipequintella)]
> >
> > http://forums.java.net/jive/thread.jspa?messageID=347733
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
> > For additional commands, e-mail: interest-help@jai.dev.java.net
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
> For additional commands, e-mail: interest-help@jai.dev.java.net
>
>
[att1.html]

Simone Giannecchini

Ciao Felipe,
yeah retiling on read + increasing tile cache should made the trick
with the type or processing you are doing (kernel etc...) since it
allows tile cache to effectively cache tiles close to one another
which are using for this "area" based processing.

Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://simboss.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

On Wed, May 27, 2009 at 9:31 AM, Felipe Quintella Correia
wrote:
> Simone,
> I think I'm working with stripped tiffs, but I'm not sure (have this tag on
> one of them:  ).
>
> My reading procedure is this one:
>
>         ImageInputStream iis = ImageIO.createImageInputStream(new
> File(fileName));
>
>         ParameterBlockJAI pbj = new ParameterBlockJAI("ImageRead");
>
>         pbj.setParameter("Input", iis);
>
>         ImageLayout layout = new ImageLayout();
>         layout.setTileWidth(tileWidth);
>         layout.setTileHeight(tileHeight);
>         RenderingHints hints = new
> RenderingHints(JAI.KEY_IMAGE_LAYOUT,layout);
>
>         renderedImage = JAI.create("ImageRead", pbj, hints);
>
>
> If by retile the raster on read, you mean this, them I'm pretty sure that's
> what I'm doing. One thing I noticed is that performances were incredibly
> boosted once I increased TileCached size (from default to 1024*1024*300) and
> used a heap size of 512M.
>
> I don't know if this is the correct answer to my problem, but it seem to be
> working. Do you second this?
>
> Thank you again,
> Felipe Quintella
>
>
> 2009/5/27 Simone Giannecchini
>>
>> Felipe,
>> much of this slowdowns may depend on the structure of yuor tiff files.
>> If you have a large striped tiff as an instance performances are
>> usually bad, hence you'd better retile the raster on read via an
>> ImageLayout.
>>
>> Simone.
>> -------------------------------------------------------
>> Ing. Simone Giannecchini
>> GeoSolutions S.A.S.
>> Owner - Software Engineer
>> Via Carignoni 51
>> 55041  Camaiore (LU)
>> Italy
>>
>> phone: +39 0584983027
>> fax:      +39 0584983027
>> mob:    +39 333 8128928
>>
>>
>> http://www.geo-solutions.it
>> http://simboss.blogspot.com/
>> http://www.linkedin.com/in/simonegiannecchini
>>
>> -------------------------------------------------------
>>
>>
>>
>> On Tue, May 26, 2009 at 9:15 PM,   wrote:
>> > Hello everyone,
>> >
>> > I'm having a very similar issue myself.
>> >
>> > I am trying to load/process/save a 200mb and a 1,3gb TIFF file, and I'm
>> > also experiencing incredibly long execution times.
>> >
>> > For the 200mb file, the process to open, convolve a gaussian filter and
>> > save took 40min. I haven't tried to save the 1,3gb file yet, but appling a
>> > Extrema operator on this file took nothing less than 25 minutes.
>> >
>> > I hope there's a way to make things faster.
>> >
>> > Have you solved your problem in some way?
>> > [Message sent by forum member 'felipequintella' (felipequintella)]
>> >
>> > http://forums.java.net/jive/thread.jspa?messageID=347733
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
>> > For additional commands, e-mail: interest-help@jai.dev.java.net
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
>> For additional commands, e-mail: interest-help@jai.dev.java.net
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
For additional commands, e-mail: interest-help@jai.dev.java.net

chinmay_info
Offline
Joined: 2009-03-27

Hi,

I am facing a similar problem:-
I have a Tiff Image of pixel size 33000 X 33000 and Physical file size = 1GB.

My operations are as follows:-
1) Read the image.
2) Scale the image to half from both sides i.e. scalex = 0.5 and scaley=0.5
3) Store the Scaled Image to file.

I am using for scaling JAI's Scale operator and for Storing,JAI's FileStore Operator.

My tile cache is DefaultTileCache
Size specifed is 1024 X 1024X 1024 bytes = 1GB

Code for Tile Cache setting :-
---------------------------------------------------------------------------------
final JAI jaiDefaultInstance = JAI.getDefaultInstance ();
final TileCache tileCache = jaiDefaultInstance.getTileCache ();
tileCache.setMemoryCapacity (1024 * 1024 * 1024);
tileCache.setMemoryThreshold (0.25f);
--------------------------------------------------------------------------------------

With the Tiff Image of 1GB, the operation takes approx 2-3 hrs at times and sometimes it throws out of memory exception.

Is their a similar reason for my problem what is above mentioned????

How can we avoid this problem??? Really not getiing anything on this issue

Help expected immediately

nsch
Offline
Joined: 2009-04-19

Now I need your help again. How do I compute a combined kernel?

Bob Deen

One problem may be the 6x convolve. The tiling system works best when
the source tiles are the same size as the destination tiles. Convolves
take more space, requiring a fetch of 9 source tiles to compute one
destination tile. But, each of those tiles must be completely computed,
which then requires another layer of tiles be fetched from the next
level up. Let's say you have a 1k x 1k image, tile size 100x100. To
compute the first layer, 9 tiles are required, meaning a 300x300 area.
To completely compute *those* (because the entire tile is computed, even
if only a tiny portion along the edge is needed), it requires an extra
tile on either side, for 500x500. The third level takes 700x700. The
fourth, 900x900. By the time you get to the fifth iteration, you're
fetching the entire image in order to compute one small tile. That's
not really a problem in most cases, because the tile cache takes care of
it for you. But for large images, the tile cache is not big enough to
hold the entire image, especially not 6 copies of it, so it thrashes to
death. For 6 iterations you need 13^2+11^2+9^2+7^2+5^2+3^2 = 454 source
tiles to compute one destination tile!! In addition, to work
efficiently, the tile cache has to maintain all tiles necessary for a
full line's worth of destinations (since tiles are typically computed in
raster order and those tiles are reused), meaning 48 x (# of tiles per
line) must be in the cache. At a minimum!

I think it's a minor flaw in the design of JAI; there should be a way to
say "I only need a small bit of this tile". But, it's what we have.

Rather than performing the convolution 6 times, I think you'd be far
better off computing a combined kernel so you can do the entire
convolution in one operation. Then you only need 9 tiles to compute one
source tile (assuming the kernel is smaller than a tile still), which
should fit in the cache fine. Even with the (much) larger kernel needed
by this method, you should come out ahead due to the way JAI works with
tiles.

Hope that helps...

-Bob

Bob Deen @ NASA-JPL Multimission Image Processing Lab
Bob.Deen@jpl.nasa.gov

jai-interest@javadesktop.org wrote:
> Hi,
>
> I tried to load, process and store some large images (93 MB, 32-bit floating TIFF) with JAI. In the processing chain I tried to smooth the image 6 times with the "convolve" operator and after that subtract the smoothed image from the original image with teh "subtract" operator.
> All this takes so much time that I stopped the process. Is it possible to do such a processing chain with JAI or do I have to find another way?
>
> best regards,
> Natalie
> [Message sent by forum member 'nsch' (nsch)]
>
> http://forums.java.net/jive/thread.jspa?messageID=342691
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
> For additional commands, e-mail: interest-help@jai.dev.java.net
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
For additional commands, e-mail: interest-help@jai.dev.java.net

cafeanimal
Offline
Joined: 2008-04-29

> I think it's a minor flaw in the design of JAI; there
> should be a way to
> say "I only need a small bit of this tile". But,
> it's what we have.

I'm a little confused by that Bob - most likely because I don't understand correctly. When Convolve is working on a specific tile it cobbles the sources such that it only needs to load those portions of the adjacent tiles that hold the data for the kernel's overlap. Is that not correct ?

> Rather than performing the convolution 6 times, I
> think you'd be far
> better off computing a combined kernel so you can do
> the entire
> convolution in one operation.

Once again I'm confused :) I would have thought it wasn't possible to create a combined kernel equivalent to repeating a convolution N times because of the way the changes in the intermediate layers propagate over the N passes. In other words, it's algorithmic isn't it ?

Yours seeking to learn :)
Michael

Bob Deen

jai-interest@javadesktop.org wrote:
>> I think it's a minor flaw in the design of JAI; there
>> should be a way to
>> say "I only need a small bit of this tile". But,
>> it's what we have.
>
> I'm a little confused by that Bob - most likely because I don't understand correctly. When Convolve is working on a specific tile it cobbles the sources such that it only needs to load those portions of the adjacent tiles that hold the data for the kernel's overlap. Is that not correct ?

That's what *should* happen... but I believe (95% certainty ;-) ) that
it only requests whole tiles from the source. So even if you just need
a piece of the edge for a border, the entire tile has to get computed...
which causes this kind of problem when you chain area ops.

I'd love to be proven wrong on this... it's something I've been
concerned about since JAI first started. However, in practice it has
not been a problem; such cases are rare.

>> Rather than performing the convolution 6 times, I
>> think you'd be far
>> better off computing a combined kernel so you can do
>> the entire
>> convolution in one operation.
>
> Once again I'm confused :) I would have thought it wasn't possible to create a combined kernel equivalent to repeating a convolution N times because of the way the changes in the intermediate layers propagate over the N passes. In other words, it's algorithmic isn't it ?

Nope... a convolution is simply the sum of each pixel in the area
multiplied by a weight. So chaining them is a linear mathematical
process which can be simplified to produce a single kernel that achieves
the same result in one step.

For example, say you have a kernel like this: [0.5 1.0 0.5]
then pixel x is computed as P(x-1)/2 + P(x) + P(x+1)/2. If you chain
them, then P(x-1) is computed as Q(x-2)/2 + Q(x-1) + Q(x)/2. P(x) and
P(x+1) are computed similarly. If you combine those, you get a kernel
that looks like:

[ Q(x-2)/2 + Q(x-1) + Q(x)/2 ] / 2
+ Q(x-1)/2 + Q(x) + Q(x+1)/2
+ [ Q(x)/2 + Q(x+1) + Q(x+2)/2 ] / 2

=

Q(x-2)/4 + Q(x-1)/2 + Q(x)/4
+ Q(x-1)/2 + Q(x) + Q(x+1)/2
+ Q(x)/4 + Q(x+1)/2 + Q(x+2)/4

or a combined kernel of: [ 0.25 1.0 1.5 1.0 0.25 ]

So if you apply that to Q (the original image) directly, you get the
same answer as if you applied both kernels in succession. Try it and
see. ;-)

This same concept extends to a 2-D kernel, and there are fairly simple
methods of computing the combined kernel... I'm sure you can find a
reference online.

Hope that helps...

-Bob

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@jai.dev.java.net
For additional commands, e-mail: interest-help@jai.dev.java.net