A Navigable Image Panel

The digital camera revolution leaves us with many images to handle. I bought my Canon A95 camera over two years ago and have taken some 13,000 photos since then. With such a deluge of images, we need good software for quick and easy image viewing. What software do you use to browse through images? Are you happy with the way it works, and how it zooms an image and navigates around the image to see enlarged parts of it?

My camera came with ZoomBrowser EX software that uses a simple image viewer. Zooming only starts when a pull-down list with zoom level values gains focus. The browser uses scroll bars for image navigation, which is cumbersome and slow. It has arbitrary lower and upper zoom limits and uses a "nearest neighbor" interpolation that creates pixelation effects for large zoom factors. Finally, the area of an image you are zooming in most often disappears from the view, meaning you have to use scroll bars in order to find it.

Wow, that is a long list of complaints! Wouldn't it be nice to have an image viewer with fast, predictable zooming so that magnified details of the image stay on the screen rather than disappear from view, a viewer with good quality rendering, and easy navigation around the zoomed image? The Navigable Image Panel presented in this article, is my attempt at this.

Navigation

When an image is larger than its container's display area, a scroll pane with scroll bars is commonly used to allow the user to move the image around the container's view. Scroll bars also give rough indication about the zoom level and how far away the displayed area of the image is from the top, bottom, left, and right edges of the image.

Scroll bars do not work well with zoomed images, especially at large zoom levels. In most cases, the user needs to use both the horizontal and vertical scroll bars to bring various areas of the image into the view. Scroll bars are also of little value when it comes to "having a larger picture": they say nothing about the areas adjacent to the area currently in the view.

When I was looking for alternatives to scroll bars, I recalled a different approach implemented in my Canon A95 digital camera. The camera has four buttons to move the zoomed image around the view of the LCD display, and the zoom lever for zooming in and out. It shows a semi-transparent rectangle in the lower right-hand corner of the LCD display, which represents the whole image (Figure 1).

Figure 1. Canon A95 navigation rectangle

There is also a smaller, solid rectangle inside the semi-transparent rectangle, which represents the part of the image currently displayed on the screen. The size of the solid rectangle and its location within the semi-transparent rectangle are proportional to the size and location of the displayed area of the image. The smaller the visible, zoomed area of the image, the smaller the solid rectangle. If we move the image towards the left edge, the solid rectangle moves closer to the left edge of the semi-transparent rectangle. Inspired by this solution, I dumped scroll bars in favor of what I call a "navigation image."

The idea behind the proposed navigation is quite simple: we display a smaller version of the image in the corner of the panel and click it to show which part of the image should be displayed (Figure 2).

Figure 2. Navigation image

Clicking the mouse anywhere within the navigation image causes the panel to display that area of the image at the current zoom level and centered around that point of the image where the mouse click happened. In this way, we can quickly move around the image by pointing with the mouse to the areas of interest, which is simpler and faster than using scroll bars. In order to keep track of which part of the image we are currently looking at, we draw a white rectangle around that part of the navigation image.

Initially, the navigation image has the width of 15 percent of the panel's width, but this can be easily changed. Move the mouse over the navigation image and turn the mouse wheel in order to increase/decrease the size of the navigation image.

The method of navigation described above is not only fast, but also quite precise. And if this is not precise enough, you can also drag the image with the mouse to adjust its exact position in the view.

The Navigable Image Panel can also be navigated programmatically, allowing the user to come up with a new, custom GUI for navigation. In order to implement a custom navigation, turn off the navigation image with the setNavigationImageEnabled() method, and then use the following methods to move the image around the panel: getImageOrigin(), setImageOrigin(Point), and setImageOrigin(int, int). The image origin is the upper left corner of the scaled image in the panel coordinate system. The reader might want to check the Coordinate Systems section below to find out more about the various coordinate systems used in Navigable Image Panel.

Zooming

Commonly, image viewers assume the center of the displayed image is the zooming center. The zooming center is the point of an image that remains stationary during zooming. Such a point stays at the same location on the screen and other points move radially away from it (zooming in) or towards it (zooming out). The further a given point is from the zooming center, the faster it moves. As a result, the more peripheral a detail of an image is, the greater the chance that it will disappear off the screen before we can see it at the desired magnification.

Navigable Image Panel is different in this regard. The zooming center is not statically bound to the center of the panel. Instead, it moves with the mouse pointer, or, in other words, is bound to the mouse position. Navigable Image Panel assumes that the zooming center is the point where the mouse pointer is. The user moves the mouse pointer to the part of the image that he/she is interested in, zooms in, and voila! that part remains stationary, no matter how peripheral it is to the center of the panel (Figures 3 and 4).

Figure 3. Mouse-bound zooming center: before zooming in

Figure 4. Mouse-bound zooming center: after zooming in

When an image is loaded into the panel, it is displayed in its entirety with its aspect ratio preserved. The image stretches from the top to bottom or left to right boundaries of the panel, depending on its size, orientation, and the size of the panel. This is defined as 100 percent of the image size and its corresponding zoom level is 1.0 (Figure 5).

Figure 5. Initial image size and location

The default zooming device is the mouse scroll wheel. Turning the wheel by one position zooms the image by a zoom increment (the default is 20 percent). Whether zooming is in or out depends on the mouse wheel's direction. If the mouse does not have the scroll wheel, it is easy to use two mouse buttons as a zooming device:

panel.setZoomDevice(ZoomDevice.MOUSE_BUTTON);

The left button zooms in and the right button zooms out.

Zooming can also be controlled programmatically, allowing implementation of a custom method. In this case the user needs to set the zoom device to "none" in order to disable both the mouse wheel and buttons for zooming purposes:

panel.setZoomDevice(ZoomDevice.NONE);

Then, the user can use the setZoom() method to change the zoom level. This method accepts a new zoom level as its parameter. It is assumed that the zooming center is the center of the panel. In order to be able to specify a different zooming center there is an overloaded setZoom() method that accepts a new zooming center as the second parameter:

setZoom(double newZoomLevel, Point newZoomingCenter)

For all zooming methods, the zoom increment value can be changed with the setZoomIncrement() method.

High Quality Rendering

Before an image is rendered in the panel, it needs to be scaled. The size of the image is different than the size of the panel and some sort of interpolation/decimation is required to increase/decrease the number of pixels to display the image for a given zoom level. There are three possible interpolation algorithms available in Java 5, each requiring different computational times and producing results of different quality. The default interpolation is nearest neighbor. Whenever an image is drawn on the screen using Graphics.drawImage() method, the nearest neighbor interpolation is applied by default, which produces the fastest rendering and acceptable quality. The quality is quite good up to a given zoom level. Beyond that level, pixelation effects become visible and lines and boundaries become jagged (Fig. 6).

Figure 6. Nearest neighbor interpolation

Figure 7. Bilinear interpolation

Bilinear interpolation is more time intensive, but produces better results (Figure 7). Bicubic interpolation requires even more computational time and its rendering quality is the best of the three interpolation methods.

How fast is bilinear interpolation? It takes half a second to render a 6-megapixel image on a PC with Windows XP, 1GB RAM, and a 3.2GHz Pentium 4 CPU, using Java 5. Rendering time increases with image resolution and reaches two seconds for 16-megapixel images. These times are clearly too long for quick zooming in Navigable Image Panel.

But wait a minute! Rendering time is proportional to the number of pixels to be interpolated. When we zoom in, we decrease the number of pixels in the original image that need to be interpolated in order to fill up the screen. We could start with the nearest neighbor interpolation initially, and then switch to bilinear interpolation when the number of pixels to be processed is small enough to ensure an acceptable speed of rendering. Let us try it out by modifying the paintComponent() method in the following way:

protected void paintComponent(Graphics gr) {
    super.paintComponent(gr);
    if (isHighQualityRendering()) {
        Graphics2D gr2 = (Graphics2D)gr;
        gr2.setRenderingHint(RenderingHints.KEY_INTERPOLATION, 
            RenderingHints.VALUE_INTERPOLATION_BILINEAR);
    }
    gr.drawImage(image, originX, originY, getScreenImageWidth(), 
        getScreenImageHeight(), null);
}

The isHighQualityRendering() method is defined as follows:

private boolean isHighQualityRendering() {
    return (highQualityRenderingEnabled 
        && scale > HIGH_QUALITY_RENDERING_SCALE_THRESHOLD);
}

When the scaled image is smaller than the original image (scale < 1.0) the Graphics.drawImage() method uses the default, fast nearest neighbor interpolation. As the scaled image becomes as large as the original image, we set a rendering hint to instruct the rendering engine to use the bilinear interpolation instead. When we run the code with the modified paintComponent() method, everything works fine initially. The moment the bilinear interpolation should start, rendering is slow and does not show any signs of improvement, even if we keep zooming in. This should come as no surprise, since Java uses the immediate mode image buffer model. This model requires processing of complete images, so interpolation is applied to the whole image rather than to the part of it rendered on the screen. What we need to do is to create a separate image that contains only the pixels that will be used for interpolation and pass it to drawImage(). The BufferedImage class has a getSubimage() method that will do the job:

public void paintComponent(Graphics gr) {
    super.paintComponent();
    if (isHighQualityRendering()) {
        Rectangle rect = getImageClipBounds();
        BufferedImage subimage = image.getSubimage(rect.x, 
            rect.y, rect.width, rect.height);
        Graphics2D gr2 = (Graphics2D)gr;
        gr2.setRenderingHint(
            RenderingHints.KEY_INTERPOLATION, 
            RenderingHints.VALUE_INTERPOLATION_BILINEAR);
        gr.drawImage(subimage, ...);
    }
}

Zooming now works well with bilinear interpolation in place. Slightly slower image dragging at high zoom levels is the price we pay for better rendering. The reader might want to experiment with different threshold values for high quality rendering (the HIGH_QUALITY_RENDERING_SCALE_THRESHOLD constant) in order to get the best trade-off between responsiveness and rendering quality. Those users who prefer top responsiveness over better image quality for large zoom levels can turn off the high quality rendering with the setHighQualityRenderingEnabled() method. Alternatively, they can upgrade to Java 6 and set the INTERPOLATION_TYPE variable in the code to bicubic interpolation. In this case, slight delay during image zooming and dragging is similar when bilinear interpolation is set in Java 5.

Coordinate Systems

Throughout this article three different coordinate systems are used (Figure 8).

Figure 8. Coordinate systems

Image coordinates refer to the original image. The image coordinate origin is the IO point.

Screen image coordinates apply to a scaled image displayed in the screen. The original image needs to be scaled before it is rendered in the panel. Its width and height are multiplied by the current zoom value (getScreenImageWidth() and getScreenImageHeight()). The screen image coordinate origin is the same IO point.

Panel coordinates refer to the rendering area of the panel. Their origin (the originX and originY variables in the image coordinates system) is the upper left corner of the panel (the PO point).

All coordinate systems have x values increasing to the right and y values increasing downward.

A number of methods translate coordinates from one coordinate system to another. These are used by the zooming and image dragging methods, making implementation easier and simpler. In order to retain high accuracy when performing coordinate transformation calculations, a custom Coords class is used instead of Point, with double values rather than integers.

Memory and CPU Usage Considerations

NIP has been tested with a number of JPEG files, both large and small, to assess memory and CPU usage. Most often a component like NIP will be used in an application for browsing through a number of JPEG files. At a minimum, two JPEG files will be loaded into memory at any given point in time: one currently displayed in the image panel, and the other to be read from a file (or any other source), waiting to replace the first one. Therefore, all testing of NIP has been done with two images in memory.

The most common digital cameras on the market today have resolutions of five to seven megapixels. They create 2-3MB JPEG files when the highest quality is selected. NIP requires 33MB of RAM to run a simple test program that displays two consecutive photos of this size. It needs to be stressed that NIP itself requires around 22MB, leaving 10MB for the two BufferedImages. With larger images, this proportion changes and for two 16-megapixel images of 13 and 14 MB, NIP requires 105MB of RAM. The largest image [10] I have found is 25MB image file from a 22-megapixel camera, and two of these requires 200MB of RAM. The amounts of RAM above were declared using the -Xmx option of the Java launcher.

Testing the largest 25MB image was interesting from a performance point of view. Nearest neighbor interpolation was fast, but when bilinear interpolation kicked in at a higher zoom level, the performance ground to a halt. It took 25 seconds for the Graphics.drawImage() to complete. I pressed Ctrl-Break and analyzed a thread dump. It turned out that the test image I was dealing with was not using the standard RGB color space, but another one, and conversion to the standard RGB color space was taking a long time. When I opened that image in GIMP [11] and save it (GIMP can read images with different color spaces and saves images in the standard RGB color space), all the sluggish performance disappeared. In order to test whether an image uses the standard RGB color space, the isStandardRGBImage() method has been added to Navigable Image Panel.

Conclusion

In this article I have presented an image panel that uses a navigation image rather than scroll bars; sports powerful zooming with a dynamic zooming center; and dynamic interpolation, providing good quality rendering and satisfactory responsiveness. All features of Navigable Image Panel can be controlled programmatically, allowing the user to come up with new ways of image zooming and navigation. The panel works well even with very large images.

Resources

• Source code [12] for this article
• Romain Guy's ZoomableImagePanel [13] was my starting point to write NavigableImagePanel
• Programmer's guide [14] to the Java 2D API

Slav Boleslawski [1] designs database systems at an Australian law enforcement agency.