Skip to main content

Using media player

10 replies [Last post]
the_ether
Offline
Joined: 2007-12-04

I want to write an app. to decode streamed video. However I want to use the RTP protocol and then use my own software to check for any lost packets. This means I need to feed the video to the decoder one frame at a time.

I will be using MPEG with no B frames, so the media player will not need to buffer anything.

Two questions:

1- How do I feed just one frame at a time to the media player in an efficient, low-latency way? Ideally, I should be able to call player.prefetch once and then start / stop for each frame then write a new frame to the file and then start / stop again.

The system needs to work with low latency so as soon as the frame is decompressed it should be displayed on the screen, not buffered somewhere.

2- How can I find out capabilities of the MPEG decoder supported on the phone, for example what level / profile it supports? I'm thinking of developing primarily for the LG KU990 "Viewty".

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
terrencebarr
Offline
Joined: 2004-03-04

Graham,

Sorry for the slow reply. Here is the answer from our engineer.

Best regards,

-- Terrence

> I am assuming that the phones that can support 30fps must be using hardware acceleration but that since there if no JSR for interfacing that, we would need to get support from the manufacturer and write our own JNI to interface with the rest of our Java program.
>
> Correct?

Yes, this is correct. You will have to introduce your own JNI interface to access the native video decoder. Some support from the manufacturer will be needed as well in case there's no native API to access the decoder.

terrencebarr
Offline
Joined: 2004-03-04

Graham,

It took a little while but our engineer looked into this topic some more now. Here is his answer:

It is quite a complicated scenario that goes beyond the scope of JavaME, and its Multimedia API (JSR 135) respectively. MMAPI doesn't offer a way to feed one frame at a time to a media player. Fiddling with prefetch/start/stop and using a dynamic file, the content of which will be updated with each frame's arrival, does seem to be a workaround here, however it will be definitely way too slow in real life and it may be totally impossible with some MMAPI implementations. So doing it in a "low-latency" way doesn't seem to me to be feasible. Furthermore, MMAPI doesn't provide you with any means to access the native decoder directly. In fact, the API is designed to be simple to use - i.e. if you have a video resource somewhere (file / http / rtp / ...), you just pass it to a player, realize the player and start it. Then you can, of course, control its location, size, stream position, audio volume, visibility, full screen mode, etc.
Using RTP shouldn't be a problem as far as your device supports it (please note that there's no support for RTP in the Wireless Toolkit). However combining it with some 3rd party software that checks for lost packets will be very painful if not impossible at all (in Java).

A workaround here might be the way that for instance MobiTV uses to broadcast "mobile TV" - instead of using RTP, they use HTTP to download chunks of the video as single JPEG images and display them in a loop. I saw the application running with some 3.3 fps and it seemed acceptable to me.

So, if you can't afford to use the workarounds mentioned above, I'd suggest to:
- either decode the video yourself and display the decoded frames as images in a loop one after another (in Java)
- or create a KNI (JNI) wrapper to access the native decoder and have it decode individual frames for you
- or implement the application entirely in native code

I know both these approaches are very costly. Another way is to wait for some of the future JSRs that consider such scenarios.

As for 2)
I guess you will have to check the capabilities of a particular device's decoder with the device's manufacturer.
There are two methods in the MMAPI's Manager class that you can use to query capabilities of a particular MMAPI implementation:
a)
public static java.lang.String[] getSupportedContentTypes(java.lang.String protocol)

- returns the list of supported content types for a given protocol

b)
public static java.lang.String[] getSupportedProtocols(java.lang.String content_type)

- returns a list of supported protocols given a content type

Hope that helps,

-- Terrence

the_ether
Offline
Joined: 2007-12-04

Your post was extremely helpful. Thank you.

> As for 2)
> I guess you will have to check the capabilities of a
> particular device's decoder with the device's
> manufacturer.
> There are two methods in the MMAPI's Manager class
> that you can use to query capabilities of a
> particular MMAPI implementation:
> a)
> public static java.lang.String[]
> getSupportedContentTypes(java.lang.String protocol)
>
> - returns the list of supported content types for a
> given protocol
>
> b)
> public static java.lang.String[]
> getSupportedProtocols(java.lang.String content_type)
>
> - returns a list of supported protocols given a
> content type

We need a low-latency system at 30fps so native Java or moving JPEG is not viable for us. That would mean JNI / KNI.

So are you saying that we'd need support from the device manufacturer, or that we could learn enough from the above java calls and write our own JNI / KNI interfaces?

I am assuming that the phones that can support 30fps must be using hardware acceleration but that since there if no JSR for interfacing that, we would need to get support from the manufacturer and write our own JNI to interface with the rest of our Java program.

Correct?

It's not so much the capabilities, but the method of interfacing them.

This sounds like what PacketVideo has done: spoken to several manufacturers and produced their own, proprietary Java interface.

Regards

Graham

terrencebarr
Offline
Joined: 2004-03-04

Hi,

Chiming in late here. I'm not a media expert but my impression is the MMAPI was designed to play back successive frames of media from files or URLs, not individual frames one-at-a-time.

I know there are 3rd party video decoding libraries ... you might want to look into those if your device is powerful enough to do decoding in Java (as opposed to a native layer engine).

Also, I've asked one of our media experts to look into your question. He is currently traveling and will need a few days to get you an answer.

Hope this helps,

-- Terrence

the_ether
Offline
Joined: 2007-12-04

Hello

> I know there are 3rd party video decoding libraries
> ... you might want to look into those if your device
> is powerful enough to do decoding in Java (as opposed
> to a native layer engine).

Oh. I only knew of the IBM Alphaworks one. I doubt the device could decode in pure Java but I'd be interested to learn of the other libraries.

FYI, we are looking at developing for the LG KU990 "Viewty" phone. It claims to be able to decode DivX at 30fps VGA. The phone doesn't use one of the popular OSes so we will need to use J2ME. I am simply assuming that its VM will know to tap into whatever decode accelerator is available.

> Also, I've asked one of our media experts to look
> into your question. He is currently traveling and
> will need a few days to get you an answer.

Very kind of you, thank you.

We are also going to try and do some experiments and will post our results. I'm hoping that simply connecting to a URL using the built-in RTP/RCP will work for us since we will be controlling how fast the data is produced on the server, but I am skeptical we will be able to get it to work without buffering on the client side.

Regards

Graham

sfitzjava
Offline
Joined: 2003-06-15

Well it's been a day or two since I delved into MMAPI.
My first question is why do you need to know about the lost packets? The protocol (RTP) implementation on the phone should handle those situations as much as possible.

You can setup a input stream that reads your RTP protocol and place that into the Manager.createPlayer(InputStrea,String). This will allow you to look at the data as it comes in, and buffer some of it up. However it sounds like you are expecting the data to be decoded by the time you look at it. Well due to JavaME security you can not hook into those features at the point you describe. The media player hooks the decoders and displays so that it can provide good performance. Most of the video playback services are in native code, and as such are not available to hook into.

For the starting and stopping of the frame playback, look at the StopTimeControl or the FramePositioningControl. You can get access to them using the Player.getControl("").

To find out what a device can support use the Manager.getSupportedProtocols(), then you will have to parse though the results to find MPEG then examine that protocol string for the profile information you are interested in (the javadocs describe this, as well as some good example at SonyEricssons developer site).

Regards
-Shawn

the_ether
Offline
Joined: 2007-12-04

> My first question is why do you need to know about
> the lost packets? The protocol (RTP) implementation
> on the phone should handle those situations as much
> as possible.

RTP does nothing to handle lost packets except to allow us to know that one has been lost. RTP is not TCP, it simply adds a header with a sequence number, sample time and the code of the source device. To take an action as a result of discovering you have lost a packet you need RTCP to send a message back to the server and that has many flavours.

But the issue of RTP and RTCP is not fundmental to my questions. It was simply an easier way of explaining why I want to decode 1 frame at a time.

>
> You can setup a input stream that reads your RTP
> protocol and place that into the
> Manager.createPlayer(InputStrea,String). This will
> allow you to look at the data as it comes in, and
> buffer some of it up. However it sounds like you are
> expecting the data to be decoded by the time you look
> at it.

No. We want to receive just one frame, check for any missing RTP packets, send a signal back to the server with the ID of the mising packet so that the server can repair the hole by using I-blocks in the next frame, then we want the client to decode the frame. We will be using slices (aka GOB: Group of Blocks) so if a RTP packet is lost, we lose the one slice it contains. The decoder should be able to decode the remaining slices.

This is a very low latency application so we cannot have any buffering of frames nor can we request any data to be re-sent from the server.

So I was expecting to receive a frame and check the RTP packets then pass the file name that stores the checked data to Manager.createPlayer(InputStrea,String). The problem is whether I can play just that one frame like that.

> For the starting and stopping of the frame playback,
> look at the StopTimeControl or the
> FramePositioningControl. You can get access to them
> using the Player.getControl("").

Thanks, but this doesn't answer my question I'm afraid. I need to be able to feed just one frame at a time. As I see it that would mean creating and destroying a player for each frame. AFAIK, Once I create a player with a specific filename and then run prefetch, I won't be able to write the next frame's data into that file without first closing the player. But maybe (hopefully) I am wrong.

In our current system we simply use the XviD decoder where we can pass the memory address of where we write each frame to and then call the decode function.

Our video is not pre-recorded, it is created and encoded in real-time, so I don't think I'll simply be able to point the Java player at a URL.

Furthermore, there is no mention of buffering in the Java documentation. I therefore assume it automatically buffers frames in the player. That's no good to us as we need very low latency with zero buffering.

>
> To find out what a device can support use the
> Manager.getSupportedProtocols(), then you will have
> to parse though the results to find MPEG then examine
> that protocol string for the profile information you
> are interested in (the javadocs describe this, as
> well as some good example at SonyEricssons developer
> site).

That's useful; thanks.

Regards

Graham

sfitzjava
Offline
Joined: 2003-06-15

>RTP does nothing to handle lost packets except to allow us to know that one has been lost. >RTP is not TCP

Yes I know that, I meant that the phone implementation for a RTP type should be smart enough to realize a packet is out of sequence or lost, carry on and not crash.

However from the sound of it you are wanting way more control that JavaME is going to provide. It sounds like you need to write your system in a native language for the device such as Symbian, or WinMobile, UIQ, Palm. JavaME does not allow the overrides and hooks that standard JavaSE allows. Especially when JavaME starts to hit peripherals on the device.

>Our video is not pre-recorded, it is created and encoded in real-time, so I don't think I'll simply be able to point the Java player at a URL.

Well actually I think you can, here are some links that should help.
http://discussion.forum.nokia.com/forum/showthread.php?t=68772
http://developer.sonyericsson.com/thread.jspa?=&start=0&threadID=26419
http://discussion.forum.nokia.com/forum/showthread.php?t=93215
http://today.java.net/pub/a/today/2006/08/22/experiments-in-streaming-ja...

I found it works best if you encode your video h.263 w/ 128k arm and not h.264 as 263 is more commonly supported at 128k than 264, and with better quality than h.264 @64k amr

Best wishes,
-Shawn

the_ether
Offline
Joined: 2007-12-04

Thank you Shawn for so much information. You have been most kind.

I'll go through all the references and see what happens. My initial target phone is the LG KU990 "Viewty" which claims to be able to decompress MPEG4 at 30fps at QVGA. My intention is to try it out over HSDPA at 700kbps.

Regards

Graham

sfitzjava
Offline
Joined: 2003-06-15

I would be interested in seeing what you finally come up with, or hear about the paths/hurdles/successes etc.. that you run into along the way.

The truly hard problems yield the best information.

-Shawn