Posted by alexeyp
on November 27, 2006 at 1:21 AM PST
Java ME conformance and quality are verified by testsuites that consist from thousands of tests. The article is gives some tips and tricks on how to run large test suite on micro device.
This blog is based on and is about my and my colleagues'
experience in JavaTM ME testing tools and test
It is inspired by SUN open-sourcing
href="https://phoneme.dev.java.net/">Java ME software stack and
Java testing tools .
Implementation Testing vs. Application Testing
Java ME developers perform their
routinely with unit tests written as part of their
work well for this: all you need to do is
to package the test framework and all your tests with your application
provide an entry point to start test execution. These frameworks run
your tests in a single MIDlet run and display results on the device
This is all what the most of the application developers need.
Things change when the object of your testing is a Java ME
implementation on a mobile device (that is the entire Java ME software
stack). We will talk below about the specific challenges that
complicate the testing process.
Some of these are specific to Java ME platform, others steam from the
size and complexity of the technology.
Implementation Testing Tools
A number of tools were developed over the years to address
the implementation testing needs. Some URLs:
Dealing with Large Test Suites
The first issue to deal with is the test suite scalability.
Due to the variety and complexity of Java ME technologies,
the number of tests is large.
Let's say we wrote 10000 tests for the CLDC VM specification.
If we tried to execute them in a JUnit-like test framework,
we would quickly find out that
- 10000 tests do not fit into a single application, you have
to split them in smaller chunks.
- some tests crash the VM. If we tried to execute all tests
in a single MIDlet.startApp call, we would never see the result of any
test (even those which don't crash). You need to be able to detect VM
exits and catch any unexpected exceptions.
- some tests crash the VM on purpose. These are negative
tests which verify that VM handles certain conditions properly (e.g.
- device runs out of memory because all class files of the
test suite do not fit into it (garbage collector frees unused data from
the memory, but doesn't free classes: these are garbage-collected only
together with their class loader).
- tests running in the same VM interfere with each other
causing failures which are hard to explain or debug. Or tests misbehave
and lock up the VM (this is where you learn that shared resources and
race conditions are a big deal in the testing world).
- you need an accurate report once the test run is complete:
if 100 tests out of 10000 failed, you need detailed information on
every failure and you don't want it displayed on the device screen.
There should be a way to load test results from the device onto your
server or desktop machine. In a perfect world test results for every
test cycle are stored in an easily accessible database.
Since tests are typically run on a device with unstable
stack, execution of a large test suite there is an additional
challenge. This is equally true for much more capable platforms like
Java SE. One should expect all sorts of things to go wrong:
- Implementation bugs causing failures and crashes.
- Resource leaks resulting in memory overflow, unavailable
file handlers and sockets.
- Incomplete implementation lacking networking or user
There are many other problems to be solved.
For example, how do you automate testing of your Java-enabled Blu-ray
Let's talk about these and other problems on specific examples.
CDC and Java SE Testing Solutions
Java SE platform is where Java technology originated. Not
surprisingly, this is where the testing solutions were initially worked
CDC + Foundation Profile is a platform similar in many ways to Java SE
(but without the myriad of recent extensions and features), the
approach, described below, came from Java SE and works for CDC+FP.
CDC platform is richer compared with CLDC. The most important
benefits we have is reflection APIs and user-defined class loaders.
This makes many testing tasks easier.
The most robust testing set-up proved to be a client-server
for the test harness: the main harness application runs in a stable
Java SE environment and a small agent is deployed on the device under
The main harness application manages test suite configuration and
as well as the test result collection and storage.
The agent connects to the harness application over the network. All the
agent does is repeatedly download and execute tests (a separate class
loader instance is created for each test download) as well
as send test results back to the main application.
This effectively solves most of the problems outlined above:
- Small memory requirements. All you need is enough memory
to hold the agent application and the largest test. Test classes are
garbage-collected each time test class loader is garbage-collected, so
you are not limited by the size of a test suite.
- Test interference is minimized. Since tests are loaded
one-by-one, they do not affect each other and their resources are freed
once execution is complete.
- No multiple application downloads during the test cycle.
There is only one small
agent application to download. The rest is handled automatically.
- No test results on device. You are not dependent on file
system, you don't do detective work if the device dies. By the way,
file system independence has the added benefit of nicely handling the
security-constrained environments such as an applet.
There are several critical dependencies in this model: you
need a (stable) network, class loader and a few extra security
permissions. These may not be available everywhere by default, but this
is topic for different discussion.
CLDC and MIDP Testing Solutions
Things get more difficult on a CLDC/MIDP stack.
There is no class loader to rely on (and no reflection either).
The only universal way to load test code on the device is via
application download. While AMS (Application Management System) does
not support automation
by default, a very simple extension called "autotest" satisfies
most testing needs.
The autotest feature is a special AMS mode which iteratively
downloads, installs, runs and removes applications from a specified
URL. Test harness makes new test applications available at this URL
and the AMS picks them up.
10000 tests will be packaged into a number of test
Each application will include a small execution agent (to report test
result back to the test harness) and the test classes.
In a trivial case we will have 10000 different applications.
Shortening the Test Cycle
Test applications are downloaded over the network.
The good news for MIDP is that all devices support HTTP download.
The bad news is that some of these connections are as slow
as 9 kbit/s. It will take 2.5 hours to download 10 MB of test classes.
And there are many other things which will affect the test run
Even though execution agent is small, it is bundled with every
application, wasting a lot of network traffic!
Each test bundle download requires new a network connection; these will
slow down the execution.
Test data and test results are sent back and forth between the harness
and the test application; extra network bandwidth is consumed.
Each test needs time to execute: some tests do extensive computations
and will take a few minutes to run.
Since the system is distributed, one must be minimally taking care of
proper task scheduling. For example, test applications must
already be packaged and signed when request for the next application
comes - if you start packaging task only on getting request
the next test application download, it may have surprisingly
effect on execution time.
In the early days of CLDC TCK 1.0 (test suite, used for
certification of implementations of CLDC 1.0 specification) test
execution was taking up to 2
weeks in some environments!
With a few improvements, one can pass CLDC TCK in a few hours:
- Pre-install shared agent code on the device (romize
classes of the test agent).
- Package several tests per downloaded application to reduce
the number of network connections and AMS operations. Each device has a
different maximum application size so the total number of applications
- Use multiple devices in parallel. Let the harness assemble
and sort test results.
Described test execution models have certain restrictions. As example:
using Java ME application to run tests does not allow to test AMS
behaviour, that is primary focus for some Java ME specs. This specific
problem has more then one solution already, as well as there are
solutions that allow to run interactive, distributed tests for Java ME.
Comments are welcome.