Skip to main content

opengl pipeline, BufferedImage and binlinear interpolation slowness

4 replies [Last post]
wmeissner
Offline
Joined: 2005-10-09
Points: 0

I'm writing a video renderer using java2d, and it works fine with either the opengl or the standard pipeline when doing nearest-neighbor scaling. However, when I use the OpenGL pipeline and set the interpolation to bilinear, it slows down to about 1fps.

However, If I draw the BufferedImage into a VolatileImage and then draw that to screen, speed is excellent (about the same speed as opengl with nearest-neighbour interpolation).

Here is how I'm rendering:
1) new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB)
2) load video into the BufferedImage:
int[] pixels = ((DataBufferInt) bImage.getRaster().getDataBuffer()).getData();
... read video data from native memory into the pixel array ...
3) call repaint() on the JComponent
4) in paintComponent(), draw the image using:
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BILINEAR);
g.drawImage(bImage, 0, 0, getWidth(), getHeight(), null);

Knowing that getting the DataBuffer from the BufferedImage unmanages it, I tried getting a BufferedImage via GraphicsConfiguration.createCompatibleImage(), drawing the video image into that without scaling, and using the compatible image in paintComponent(), but it runs just as slow.

When not scaling, the java2d trace during paintComponent() looks like:
sun.java2d.opengl.OGLSwToSurfaceBlit::Blit(IntRgb, AnyAlpha, "OpenGL Surface")

When scaling, it becomes hundreds of repititions of:
sun.java2d.loops.Blit::Blit(IntArgbPre, SrcNoEa, IntArgb)
sun.java2d.opengl.OGLMaskBlit::MaskBlit(IntArgb, SrcOver, "OpenGL Surface")
sun.java2d.opengl.OGLGeneralBlit::Blit(Any, AnyAlpha, "OpenGL Surface")
sun.java2d.loops.Blit::Blit(IntArgb, SrcNoEa, IntArgbPre)
sun.java2d.loops.MaskBlit$General::MaskBlit(IntArgbPre, SrcOverNoEa, "OpenGL Surface (render-to-texture)")

So the question is: Is that expected behaviour, and its just a trap for the unwary or am I doing something wrong somewhere?

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Chris Campbell

java2d@javadesktop.org wrote:
> Thanks Chris.
>
> I knew I'd forget something from the original post.
> Graphics card: nvidia geforce 6600
> Driver: nvidia 97.55
> OS: Linux 2.6.20.
> java version "1.6.0_02-ea" (happens on 1.6.0-b105 as well).
> Image: 720x576, 32bpp, rgb format. - chosen to test dvd resolution rendering.
>

Thanks for the info.

> Yep, this is exactly how I'm painting.
> - copy video data into (unmanaged) BufferedImage
> - copy BufferedImage into VolatileImage
> - scale VolatileImage to the destination (with BILINEAR enabled)
>
> And that works really well. It just seemed a bit non-obvious that I would have to go through the extra steps.
>

It's probably the best you can do for now (with the OpenGL-based
pipeline in JDK 6). Even if we figured out a better way to implement it
internally, it probably wouldn't be a whole lot faster than what I
outlined above.

However, it kind of pains me to see so much extra copying of pixels
going on, which is usually a killer for video applications. What I
outlined above is about the best you can do with our current Java2D
architecture, but if you're serious about performance, you might want to
think about having an alternate codepath that uses JOGL when available.
If using OpenGL directly, you can bypass a lot of the copying involved
in the approach you're using above. For example, I'd recommend:
- use pixel buffer object (PBO) extension to lock video buffer memory
- decode video frame directly into that memory chunk
- upload PBO to texture memory using glTexSubImage2D()
- copy texture to hardware backbuffer (if using a heavyweight
component) or the Swing backbuffer (via GLJPanel, for example)

This approach cuts out at least three different copy operations; not
sure what your performance requirements are though, and whether it's
necessary. This is getting a bit off-topic, but if you're interested,
you could probably get some help from the folks on the JOGL forum over
at javagaming.org...

> i.e. I naively thought that java2d would have figured out to do the equivalent of BufferedImage->VolatileImage before scaling the image, if the image was unmanaged, and the desination was managed.

We unfortunately don't have those smarts in our image management code at
the moment, but I'll make a note of it for a possible enhancement at a
later date. The tricky part is to know how much VRAM to spend for
intermediate use (for isolated cases like this, it's not such a big
deal, but for more complicated cases, we have to make sure we don't
start creating lots of intermediate VRAM surfaces haphazardly). Anyway,
it's something to think about.

Thanks,
Chris

> [Message sent by forum member 'wmeissner' (wmeissner)]
>
> http://forums.java.net/jive/thread.jspa?messageID=216710
>
> ===========================================================================
> To unsubscribe, send email to listserv@java.sun.com and include in the body
> of the message "signoff JAVA2D-INTEREST". For general help, send email to
> listserv@java.sun.com and include in the body of the message "help".

===========================================================================
To unsubscribe, send email to listserv@java.sun.com and include in the body
of the message "signoff JAVA2D-INTEREST". For general help, send email to
listserv@java.sun.com and include in the body of the message "help".

Chris Campbell

If you want to ask a question about the OGL pipeline, you'll need to
at least provide:
- graphics card
- driver version
- OS version

Also, it would help to know the size in pixels of the original
BufferedImage. As you suspect, calling DataBuffer.getData() will
defeat image management, so you'll be getting sysmem->VRAM blits,
which are slow (and bilinear ones are even slower, although they
might get a bit faster if we fix 4841762 someday). Image management
doesn't provide any benefit in the case of video applications though
anyway, because the image content is changing on every frame and
therefore we don't get a chance to cache it in VRAM.

In any case, you won't get any benefit from the OGL pipeline by
scaling a (sysmem-based) BufferedImage with bilinear filtering
because OpenGL doesn't really offer a direct route for that. I think
the alternate approach you describe is about the best you can do:
- copy video data into (unmanaged) BufferedImage
- copy BufferedImage into VolatileImage
- scale VolatileImage to the destination (with BILINEAR enabled)

Assuming you have a modern GPU with FBO support, then that should be
reasonably fast.

Chris

On May 13, 2007, at 8:43 PM, java2d@javadesktop.org wrote:
> I'm writing a video renderer using java2d, and it works fine with
> either the opengl or the standard pipeline when doing nearest-
> neighbor scaling. However, when I use the OpenGL pipeline and set
> the interpolation to bilinear, it slows down to about 1fps.
>
> However, If I draw the BufferedImage into a VolatileImage and then
> draw that to screen, speed is excellent (about the same speed as
> opengl with nearest-neighbour interpolation).
>
> Here is how I'm rendering:
> 1) new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB)
> 2) load video into the BufferedImage:
> int[] pixels = ((DataBufferInt) bImage.getRaster().getDataBuffer
> ()).getData();
> ... read video data from native memory into the pixel array ...
> 3) call repaint() on the JComponent
> 4) in paintComponent(), draw the image using:
> g.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
> RenderingHints.VALUE_INTERPOLATION_BILINEAR);
> g.drawImage(bImage, 0, 0, getWidth(), getHeight(), null);
>
> Knowing that getting the DataBuffer from the BufferedImage
> unmanages it, I tried getting a BufferedImage via
> GraphicsConfiguration.createCompatibleImage(), drawing the video
> image into that without scaling, and using the compatible image in
> paintComponent(), but it runs just as slow.
>
> When not scaling, the java2d trace during paintComponent() looks like:
> sun.java2d.opengl.OGLSwToSurfaceBlit::Blit(IntRgb, AnyAlpha,
> "OpenGL Surface")
>
> When scaling, it becomes hundreds of repititions of:
> sun.java2d.loops.Blit::Blit(IntArgbPre, SrcNoEa, IntArgb)
> sun.java2d.opengl.OGLMaskBlit::MaskBlit(IntArgb, SrcOver, "OpenGL
> Surface")
> sun.java2d.opengl.OGLGeneralBlit::Blit(Any, AnyAlpha, "OpenGL
> Surface")
> sun.java2d.loops.Blit::Blit(IntArgb, SrcNoEa, IntArgbPre)
> sun.java2d.loops.MaskBlit$General::MaskBlit(IntArgbPre,
> SrcOverNoEa, "OpenGL Surface (render-to-texture)")
>
> So the question is: Is that expected behaviour, and its just a trap
> for the unwary or am I doing something wrong somewhere?
> [Message sent by forum member 'wmeissner' (wmeissner)]
>
> http://forums.java.net/jive/thread.jspa?messageID=216689
>
> ======================================================================
> =====
> To unsubscribe, send email to listserv@java.sun.com and include in
> the body
> of the message "signoff JAVA2D-INTEREST". For general help, send
> email to
> listserv@java.sun.com and include in the body of the message "help".

===========================================================================
To unsubscribe, send email to listserv@java.sun.com and include in the body
of the message "signoff JAVA2D-INTEREST". For general help, send email to
listserv@java.sun.com and include in the body of the message "help".

wmeissner
Offline
Joined: 2005-10-09
Points: 0

Thanks Chris.

I knew I'd forget something from the original post.
Graphics card: nvidia geforce 6600
Driver: nvidia 97.55
OS: Linux 2.6.20.
java version "1.6.0_02-ea" (happens on 1.6.0-b105 as well).
Image: 720x576, 32bpp, rgb format. - chosen to test dvd resolution rendering.

Yep, this is exactly how I'm painting.
- copy video data into (unmanaged) BufferedImage
- copy BufferedImage into VolatileImage
- scale VolatileImage to the destination (with BILINEAR enabled)

And that works really well. It just seemed a bit non-obvious that I would have to go through the extra steps.

i.e. I naively thought that java2d would have figured out to do the equivalent of BufferedImage->VolatileImage before scaling the image, if the image was unmanaged, and the desination was managed.

wmeissner
Offline
Joined: 2005-10-09
Points: 0

More info about the video card: 128M ram, and according to glxinfo, it supports GL_EXT_framebuffer_object.