Skip to main content

Accessing a large number of bufferedimages

9 replies [Last post]
byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

I am sampling 128 x 128 points distributed across about 500 jpeg images (each 600x600) to create an image. This means that I must open these 500 files and read some RGB values at some x,y location with image.getRGB(x,y). I have a cache of the bufferedimage objects so that I don't need to re-create them too regularly. However the time to iterate through all the xy values is too long (>2 minutes) and I am running into memory problems when the cache holds more than 200 entries (~200MB used memory).

My current logic is:
1) lookup bufferedimage in cache. If found return bufferedimage.
2) create new bufferedimage for disk file.
3) add new bufferedimage to cache.
4) if number of entries in cache > max_cache_size remove least accessed cache entry.
5) return bufferedimage

My question is, can anyone tell me whether it would be faster and probably less memory exhaustive for me to replace the raster data in the bufferedimage object rather than create a new bufferedimage and ask the garbage collector reclaim the previous one. A sort of pool of bufferedimages to reuse. Also I am using ImageIO.read(...) to create the bufferedimages. Is this the fastest and best method.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
kschneid
Offline
Joined: 2003-06-10
Points: 0

> I'm already tiling, that's why I have only images
> 600x600. I could try and break them into smaller
> units but I'm wondering how much of a spped bump I
> would get.

Do you mean that you had a single, original image that you broke into the 600x600 images, or are those 600x600 images independent of each other? Whenever someone talks about tile-based image processing, the first thing I think of is JAI.

You may also want to investigate if it's possible to introduce some parallelism into your loading/processing.

campbell
Offline
Joined: 2003-06-24
Points: 0

> FYI. I'm using b70 mustang and I've noticed that
> ImageIO.read(file) takes twice as long as
> JPEGCodec.createJPEGDecode(file_stream).decodeAsBuffer
> edImage().

We're aware of some performance discrepancies between the different decoders and we're trying to address them. I'd suggest trying the JAI Image I/O Tools JPEG plugin, as it is natively accelerated with vector instructions:
https://jai-imageio.dev.java.net/binary-builds.html#Daily_builds_1.1

On Solaris/SPARC I've seen performance of the JAI IIO Tools JPEG reader can be 2-3x faster than the core IIO JPEG reader, but on Windows there is currently an open bug to be aware of (I'm told this is being investigated):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5094376

As someone else has mentioned, it would be worth looking into JAI since it has mechanisms in place for dealing with large images without keeping them all in memory, etc.

Thanks,
Chris
Java 2D Team

linuxhippy
Offline
Joined: 2004-01-07
Points: 0

As far as I know the slowest part is most likely decoding the jpeg files.

1.) The first thing I would try is the latest mustang build, since a lot of optimizations have been done lately by chris.

2.) I guess loading the images is far more expensive than compressing/decompressing its data in-memory.
I would implement a two-level cache, the first should hold normal bufferedimages, the second cache should hold the zipped image data. Therefore using BufferedImages with a byte-based raster would be more efficient, since you could grab the raster (take care not to make a copy but instead access the raster-data directly).

1.) if the image is in image-cache fetch it
2.) if not look at the cache which holds the compressed raster
3.) if its zipped cache, unzip it create a bufferedimage and copy it to first cache. I would look at the cache at the number of accesses and the last access time (to not throw out newly decompressed images).

The zip-classes are Inflater/Deflater.

good luck, lg Clemens

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

Thanks. I hadn't thought of zipping. I'll try it.

FYI. I'm using b70 mustang and I've noticed that ImageIO.read(file) takes twice as long as JPEGCodec.createJPEGDecode(file_stream).decodeAsBufferedImage().

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

I tried the compressing, but it takes much, much longer. I think it probably due to the fact that the raster data is of type DataBufferInt and not byte. Therefore I must copy it into byte arrays for compression , and back into int arrays after de-compression.

andrew
Offline
Joined: 2004-03-04
Points: 0

You could write a custom DataBuffer (TYPE_INT) that uses one or more byte arrays to store the image data.
This would allow you to save 25% memory by storing each pixel as a 3 byte RGB value and it would be much easier to do compression.
(note however: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6353518)

Another solution if you can afford to do some preprocessing would be tiling your images.
I.e. load each image, split it up in say 25x25 subimages and save each subimage (as png for example). Then only load those subimages you need for sampling.

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

I'm already tiling, that's why I have only images 600x600. I could try and break them into smaller units but I'm wondering how much of a spped bump I would get.

I'll try the custom DataBuffer as that would definately save on memory usage.

Thanks.

andrew
Offline
Joined: 2004-03-04
Points: 0

> I'm already tiling, that's why I have only images
> 600x600. I could try and break them into smaller
> units but I'm wondering how much of a spped bump I
> would get.

If you can make each 50x50 tile 2.8 KB on average you could load all files into memory (raw data, not decoded images).
This would require around 196 MB plus overhead.
Then just decode each tile as needed using JPEGImageDecoder. With a recent CPU you should be able to decode 1200 tiles per second. 128*128/1200 = 13.7 secs

linuxhippy
Offline
Joined: 2004-01-07
Points: 0

I would if possible choose a BYTE type for BufferedImage, since this would allow you to compress on-the-fly without type conversations.

I would not worry about Inflater/Deflater performance, since it performs usually quite good, I think even in MBs/s.

I don't know wether tiling is really a good idea, every time you read a tiled map, you have to decompress all the other image data you don't really need, just because the tile you want is on the image.

lg Clemens