Posted by spericas
on November 4, 2005 at 4:06 PM PST
Earlier in this blog I introduced Japex, a simple yet powerful tool to write micro-benchmarks in Java. At the time, I've only touched on the basic features by briefly explaining how to write a Japex driver and a Japex configuration file. Today, I'd like to talk about a brand new feature: scatter plots for displaying benchmark's results.
Up until now only bar charts were supported in Japex. The result values displayed in these charts are controlled by the parameters japex.resultValue and japex.resultUnit (c.f., manual for more information). An example of such a chart is the following,
Displaying throughput in transactions per second (by setting japex.resultUnit to "tps") or latency in milliseconds (by setting japex.resultUnit to "ms") as shown above may be sufficient for some benchmarks, but for others, a better result unit is one that combines processing speed with the size of the input being processed. For that purpose, Japex also supports megabits per second (by setting japex.resultUnit to "mbps" and japex.inputFile). This unit is particularly useful to compare the processing speed of a certain application with respect to the speed of a network card, a system's bus, etc.
Now, there are cases in which a pair (time and space or throughput and space) instead of a single scalar is the easiest way to compare the drivers' performance. Scatter plots are ideal for this, and here is a an example of plot comparing FI and XML performance in 2Ds as it is now supported in Japex,
The global parameter japex.chartType (which can be set to 'barchart' or 'scatterchart') controls the type of the chart generated by Japex. Whenever japex.chartType is set to 'scatterchart', each driver is responsible for producing a extra value for the "x" axis. I.e., in addition to setting japex.resultValue (or let Japex compute this automatically) a driver needs to set japex.resultValueX, and the best place to do this is in the finish() method.
Scatter plots were integrated into Japex only a few days ago, and we're still learning how to make best use of them (e.g., we've also added the parameter japex.resultAxis[X] which can be set to either 'normal' or 'logarithmic') so we would love to hear from you, or better yet, have you as a contributor to improve this framework even more. Happy benchmarking!