Posted by robogeek
on July 19, 2005 at 11:08 AM PDT
This is prompted by a feature request I saw on the javadesktop.org JDIC bulletin board: javadesktop.org/forums/thread.jspa?messageID=99951
As I mentioned earlier, I've spent more than a few brain cycles thinking about GUI test automation. It's been so much that at times I've been worried about my sanity - how many people do you know daydream about the minute details of GUI interactions, looking for ways to drive the interaction by a program? At least I don't talk to myself ... too often.
But I digress.
The JDIC thread is a feature request where the author asks: More generally, it'd be good to be able to interface a bit more with other applications running on the same machine, grabbing their windows as graphics, activating menus (that would let us write macro scripting applications).
Yeah - I'd like those abilities too. A large issue in my team (the JSE SQE Team) is testing the Java Plugin and JWS features. Because they relate with web browsers, a lot of the test scenarios involving Plugin and JWS involve browser interaction. And, because our product (Java) is cross-platform we have to test those browser interactions across platform (well, Windows, Linux and Solaris anyway). It ends up costing us a lot of manual labor due to manual testing due to the lack of automation tools to cover the interactions required in these scenarios.
Hheck, yeah, we really want to be able to "interface more" with web browsers from a Java test.
By way of offering my proposed answer, let me point to an article I read long ago: Scripting the Unscriptable in Mac OS X
The "unscriptable" in this case is any GUI application ... Since the article is about how to "script" GUI interactions on Mac OS X, the author first describes the past. In the past they would use the QuicKeys application to "hack" the system to simulate a "ghost" user who would press keys and move/click the mouse. Sounds a lot like the Robot class but with a nice GUI around it to help the user do the work.
On OS X the QuicKeys approach doesn't work because OS X is more secure. At least that's what the article says.
But, enter Uncle Sam and the Section 508 requirements that are what's driving the push to Accessibility. Those requirements have turned into this scenario being played out with every desktop GUI "vendor": Given any application on Mac OS X, it needs to be possible for some other application to discover what interface elements it is displaying. The other application needs to be able to "read" these elements ("There's a button that says OK") and it needs to be able to access them (click the OK button).
What that means is that Apple developed a programming interface to support Accessibility "agent" software in making OS X accessible to the disabled. That programming interface allows an accessibility agent to query and discover the location and characteristics of other applications running on the computer screen. And, further, it was not just Apple who developed these programming interfaces ... no ... Microsoft developed an accessibility tool API ... and so has Sun. Sun has engineers working on accessibility for both GNOME and Mozilla, donating code to support accessibility to both of those projects.
The next step of the "unscriptable" article is to put 2+2 together. Namely, Apple provides with OS X the AppleScript tool. To "script" the "unscriptable" one merely needed a way to make calls to the accessibility support from AppleScript, and that's what the SystemEvents "application" does. It allows AppleScript programs to query the accessibility support and then interact with (a.k.a. "script") any GUI application, and do the interaction as if it's a real user with a keyboard and mouse.
Unfortunately that solution is limited to one platform: Mac OS X.
Fortunately it points to a solution for Java testers. One merely needs to make the same architecture. Namely: write an interface in JNI that exposes accessibility "agent" support to Java applications. With that support exposed to Java, then we can use it with Robot and with the tools such as the ones listed here that build on Robot, then be able to automate interactions with any GUI application. And we would be able to do so on *ANY* platform (due to it being in Java).