Skip to main content

Benchmark

10 replies [Last post]
john_silver
Offline
Joined: 2006-11-13
Points: 0

Hi all, how are we all doing today ?
I have a simple question that seems to have no simple answer. What is sun using to benchmark it's JVM, or phoneme to stay on topic ? I have tried several Benchmark suites that I have found laying around the net. Tried most of http://www.epcc.ed.ac.uk/javagrande/links.html. More precisely I want to validate that optimising the JVM for let say a coprocessor, like iwmmxt on arm/xscale is worth it. And to see the performance differences of the JVM across different hardware and/or compile flags. I do not want applets, middlets ... just a simple test that can be ran from command line and that measures overall performance and/or specific issues. I am coding my test suite right now and it seems that for sure something else exist. Hinkmod, you had a modified version of SPECJVM98 for cdc, is it because there is nothing now, or that there was nothing available then ?

Thank you for your time, and any input is always appreciated.

John

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
terrencebarr
Offline
Joined: 2004-03-04
Points: 0

Link correction. The Performance and Benchmarking topic is at:

http://wiki.java.net/bin/view/Mobileandembedded/PerformanceBenchmarking

terrencebarr
Offline
Joined: 2004-03-04
Points: 0

John,

Sorry for the slow reply, I have not forgotten you. I am discussing an issue with EEMBC regarding GrinderBench and hope to be able to give you the info on running the command-line version very shortly.

In the meantime, I have created a Twiki entry around benchmarking and performance info:

http://wiki.java.net/bin/view/Mobileandembedded/Benchmarking

Thanks for your patience.

-- Terrence

Terrence Barr
Evangelist, Java Mobile & Embedded Community
Sun Microsystems, Germany
www.mobileandembedded.org

terrencebarr
Offline
Joined: 2004-03-04
Points: 0

John,

Yes, sorry, you need to use some undocumented command-line features to invoke the non-graphical version. I'll get back to you tomorrow.

-- Terrence

strim
Offline
Joined: 2006-10-20
Points: 0

Hi John,

While Terrence is trying to help you with command-line access to lovely
EEMBC benchmarks, I recommend to check:

SciMark 2.0
http://math.nist.gov/scimark2/download_java.html

I have no information whether it can be started against cldc/cdc, but
I believe it can be useful. Especially regarding the sources availability.

Regards,
Strim

john_silver
Offline
Joined: 2006-11-13
Points: 0

Hi all, pushing a bit with GrinderBench :
-Intalled wap plugin for firefox
-Downloaded the .jad
-Opened the .jad
-wget the .jar specified in the .jad ...

When running the several benchmarks I get a class not found on javax/microedition/lcdui/CommandListener or javax.microedition.midlet.MIDlet...

Then I opened the jar to see other classes I could call like: com.sun.mep.bench.Chess.ChessBenchmark
but to no avail... interesting the capital *C* in the package name *Chess*

So I guess it does not meet my requirements, being on cdc, cldc with foundation classlib. So the hunt continues for an embedded, command line only, no ui, benchmark suite.

Thanks all, and please if you think of something else, don't hesitate. As always time is a great factor on software dev, I though I could save some, now it is back to the schedule and see how doing our own benchmark suite will fit in all of that :)

Laterz

John

john_silver
Offline
Joined: 2006-11-13
Points: 0

Hello!
Thanks for all your inputs.
Yes, I have been looking at EEMBC web site, sounds great. But I was put off by the fact that it seems oriented phone specific. Althought the site describes other features and platforms it seems it is only available through a wap browser. I think that if I am to get the *free non graphical binaries*, I will have to contact a sales rep... ? Or maybe I haven't look hard enough ? Maybe somebody could enlight me as to where there is a download link for the free binaries to run on something else than a wap browser ? If not then I guess I will try to download the thing with my phone and start from there and try some stuff...

Also thank you Danila for the xscale info, althought I was already aware of it, being the first one to post on the *feature* forum regarding an xscale related problem :) I just want to see if we can tweak it just a bit more... :)

And I would like to note again, what a great thing sun has done, releasing multiple projects to the community. I have been using JAVA since 1997, and at some point in history, I was cursing at some decisions made. I even went for other technologies, dropping sun's JAVA for a couple of years... But now it is all good! And I am sure that with the help of the community (world) JAVA will take an even bigger place in this industry.

Thank you.

John

danila
Offline
Joined: 2006-09-25
Points: 0

Hi John,

there are two different VMs in the phoneME project:
- phoneME Advanced VM (sources in phoneme/components/cdc)
- phoneME Feature VM (sources in phoneme/components/cldc)

The two VMs are quite different in design and implementation.

Your first post on xscale actually refers to the phoneME Advanced VM. Both VMs include XScale-specific optimizations, feel free to play with them and tweak them, and let us know the results :)

Thanks,
Danila

danila
Offline
Joined: 2006-09-25
Points: 0

Hi John,

the phoneME Feature VM does contain some XScale-specific optimizations. You can experiment with linux_bvd configuration.

Thanks,
Danila

terrencebarr
Offline
Joined: 2004-03-04
Points: 0

John,

EEMBC GrinderBench (www.grinderbench.com) is what you want. You can download the binaries for free and it is a very good application-level non-graphical benchmark. We use it on a regular basis for performance testing.

I happen to be the chairman of the EEMBC Java committee so I'd be happy to answer any questions you have.

Cheers,

-- Terrence

Terrence Barr
Evangelist, Java Mobile & Embedded Community
Sun Microsystems, Germany
www.mobileandembedded.org

akz003
Offline
Joined: 2006-10-13
Points: 0

Hi, John

thanks for your interest.
I can say that we were using so far the most popular and widely used in Java ME industry benchmarks - mainly for VM side it is EEMBC and in general for the whole stack Jbenchmark(s), Amark, Morphmark, others (please forgive me if I forgot some other names), also the benchmarks that go inside Java Device Test Suite.

It would be really interesting to have you complete your own suite and measure independently the results for various Java ME stack implementations.

I would also ask Sergey (aka Strim) to step in and give his advises regarding benchmarks.

Regards,
Kostya