Posted by giovanisalvador
on January 4, 2009 at 1:44 PM PST
When a developer needs to create applications based on a framework (Struts, for instance), what does he do? Does he read the entire documentation to understand how to benefit from the framework's extension points? What if the documentation is not fully available? What if he has a tool to guide him on the proper steps of using a framework? Does it help? So that's what this post is about.
I have achieved one important milestone for my personal and professional life, which is masterâ€™s degree graduation. My master degree's thesis was related to software reuse, more specifically, about a language projected during a doctorâ€™s degree of a professor. This language, called Reuse Description Language (RDL), has been projected to assist developers during what is called framework instantiation, which is to explore the extension points (AKA variability points) of frameworks and to drive developers to a correct sequence of steps on software reuse activities. Using RDL, developers wouldn't need to understand all the framework's variability points. They would rely on a RDL script created to explore those points to help the developer to create applications based on that framework.
RDL was written in Java and works basically this way:
There is an engine that parses the RDL script and the XMI representation of the framework being instantiated.
The script contains commands to explore the variability points of the framework and interact with the developer, asking for classes, operations and attributes to be created or modified.
The engine has as input both the RDL script and the XMI representation of the framework.
The output is another XMI, with all the new assets that were created.
Here is a simple RDL script.
instantiation cookbook Simple;
simple : class;
newSimple : class;
classImpl : class;
simple := get_class("Simple");
newSimple := class_extension(simple, ? );
myintf := get_interface("MyInt");
classImpl := interface_implementation(myintf, ? );
new_method(classImpl, ? );
new_method(classImpl, ? );
All the "get" commands are retrieving from the XMI (loaded in memory by the RDL engine) a reference to the classes and interfaces of the framework so it is possible to perform commands such as extension and implementation commands. All the question marks are points where the developer (or anybody running the script) is asked to provide input like names of classes and methods.
Also, there are conditional commands (now shown here for simplicity) that asks the developer to choose between a hierarchy of classes which one he/she would like to extend.
After running the entire script, a new XMI file is created with the original framework plus all the assets that were created. Then all we need to do is to load the XMI in any XML tool in order to generate the classes that were created. After that, load the entire new classes into any IDE to code the necessary business rules for the application.
So the questions are:
How productive are those steps as opposed to what I called the "manual" instantiation (no tool assisting the instantiation, just load the IDE and code)?
Do we have more or less dependency on the framework documentation, to understand its variability points, using RDL?
Does this technique lead to more or less mistakes as I have an engine assisting me?
To answer these questions, as part of my thesis I performed an experiment (I also modified RDL code a litle to prepare it for this experiment) where a set of developers coded a specific use case using Java with Eclipse and another set of developers coded the same use case but using RDL to assist the creation of the necessary classes plus the coding activity also using Java with Eclipse.
All developers coded the same use case using the same framework, especially built for this experiment.
How did I collect the data to analyze it? If you see the questions above again you will notice that they are related to time (productivity), dependency on a documentation (number of accesses to the frameworkâ€™s documentation) and errors (both business logic as well as misuse of the framework). The Time was related to the total time of development. Number of accesses to the documentation were counted when developer stopped doing his/her activity to check the documentation. And finaly, errors were logged and counted using automated test classes (JUnit classes).
After running the experiment and collected all the data, I inserted the data into SPSS (see references) tool, which is a very useful tool to make statistical analysis.
You basically plot lots of charts and graphs to better analyze the information gathered as well as to assess the hypothesis stated.
The Conclusion of the Experiment
All developers in the experiment are senior Java developers.
More than 50% of them own some Java certification.
All of them perform reuse activities on a daily basis.
Average of 6.5 years developing complex applications.
Here I summarize the results of the experiment as well as some important information:
In this experiment RDL didnâ€™t add any value from productivity, dependency or errors perspective. That means the developers that were assisted by RDL were not more productive then the ones without it. Also, the developers that were assisted by RDL didnâ€™t reduce their dependency on a good documentation. Finally, errors were not reduced. The experiment shown they were very similar.
The only difference was from time perspective. The developers who coded using RDL had their total development time very similar to each other while the ones who coded without RDL had more dispersion between them. But from statistical perspective we couldnâ€™t state there was significant difference between the 2 sets of developers.
After the statistical analysis we also made the developers to answer a survey. The result is that most of them like the idea on having a tool to assist them to create the necessary assets of an application but they also said they want good documentation to read when they need to maintain the code or provide new functionality. They said the tool would help them to skip the boring process of class creation but they would need to understand the framework in its details.
Despite I could not generalize this experiment to other tools with the same purpose as RDL I have questions in my mind: Do developers need tools to help or to guide them in the correct sequence of steps of software reuse? Or all we need is good documentation and good communication between team members so we can build reliable code, achieving business needs? From what this experiment showed me (for RDL in the context studied) developers don't need those tools.
but maybe other experiments can be created to provide more general answers and show different information. If you know of any other experiment, please, let me know. If you need to know more details about my experiment and/or empirical software engineering, let me know.