Fitnesse Testing for Fast-Paced Agile Web Development

One of the goals of an agile web software development team is to deliver quality software in a timely fashion. For most web applications the server-side software uses a database and the final products are HTML and/or XML. HTML is rendered by a browser and XML is either processed by a browser (AJAX) or exchanged with a partner web application (business-to-business integration). Fitnesse [12] tests can communicate with the database and can make assertions against HTML and XML. Agile web teams can use Fitnesse to create comprehensive test suites that make system-level assertions. Once these test suites exist, the team can release updates to their production web site as often as every day, confident that all of the system-level requirements (both old and new) are being met.

The purpose of this article is to illustrate the use of Fitnesse by agile web development teams. It first will discuss some of the reasons that agile web development teams would want to adopt a rapid release schedule. Next it will suggest that Fitnesse is a viable testing tool for use with this schedule. Then it will provide a number of examples to demonstrate the simplicity and power of Fitnesse. Finally, it will provide tips regarding Fitnesse test writing.

In this article, the agile term "whole team" refers to the collective group of software developers, DBAs, technical testers, managers, and customers. Agile software development is a discipline. For more information regarding agile software methodologies, see Ron Jeffries's XP web site [13].

Why Would a Development Team Release Software Updates Every Day?

• Short answer: Good for the business.
• The company's product is an e-product. Super-short releases allow the marketing department to obtain rapid customer feedback and to respond to that feedback quickly. They can test modifications to existing products and release new products sooner and more often. The agile concept of rapid feedback works well for the marketing team, too!
• It's good for search engine optimization. Frequently updated web sites perform better in organic search engine results. This enables the business to attract customers to its web site without paying for sponsored links.
• To respond to changing customer needs. Businesses that support diverse customer niches demand more agility from their products.
• For contractual agreements with e-partners. Contractual agreements involving business-to-business web site integration can dictate when the software is released.

The rapid release schedule accommodates a wide range of sizes and types of projects, because each day the whole team has the opportunity to release software updates, regardless of how long the updates took to develop. Although the whole team could probably get by without releasing its software quite as often, once a rapid release schedule is adopted, the team members will wonder how they ever managed without it.

Simple web sites can be managed effectively by a content management system and require little developer, DBA, and technical tester work. However, more complex web sites require software customizations to provide tracking services for their e-products. This tracking can include page impressions, clicked links, dynamic advertising, and links to partner sites, which can then be customized for each e-product. These types of requirements are dynamic enough to keep patterns in the software design from forming, thereby forcing the whole team to release new products more often.

Why Fitnesse?

• It's easy to use. Web developers, DBAs, technical testers, managers, and customers can all learn how to use wiki markup to make test assertions.
• It's easy to install and lightweight to run. Get started with Fitnesse by running it on a developer workstation.
• It performs assertions against HTML, XML, Java, and the database, all from a single tool. The complexity of your requirements will not outgrow Fitnesse's capabilities, because it can be extended to test anything testable with Java. Test writing is limited only by the whole team's imagination.
• It's open source and well documented.

Agile web developers write Java software to test their Java objects, web pages, databases, and XML documents. JUnit [14] is a popular testing framework for regression testing Java technologies. Fitnesse is analogous to JUnit in that it is a testing engine built using Java technologies. However, Fitnesse is different because its user interface is a web application with test suites created and managed using wiki markup. The key difference is that JUnit is primarily used by developers, whereas Fitnesse's user interface is friendly to non-developers, too. The developer extends Fitnesse's testing engine to expose new assertion methods. Then any team member (technical or not) can use the wiki markup to populate and run these new assertion methods. In the end, the developers use both JUnit and Fitnesse; each for different purposes.

Fitnesse describes itself as a "fully integrated standalone wiki, and acceptance testing framework" that runs test assertions using wiki markup. The wiki markup is used to create test tables whose assertions are run inside of the Java web FitServer. Upon completion of the test table, the FitServer displays a green-colored cell for each correct assertion and a red-colored cell for each failed assertion.

A fixture [15] is a Java class that parses a wiki test table and then connects the test table to methods in the Java class. Fixtures support input parameters and connect a method's output to the wiki's display. For teams that want to quickly get started, there are HTML and database fixtures [16] that can be installed and used without customization. For teams that need more functionality, the complexity of the assertions is limited only by the whole team's time and imagination, because Fitnesse's Java test fixtures are meant to be extended and customized. For instance, the developer team can write Java software that queries a database and then uses that data to assert that a web page is rendering that data correctly in HTML or XML.

HtmlFixture Asserting Text and Links Are Correct

Project requirements:

• Create a home page with title, meta-keywords, and a meta-description.
• Create a "Daily News" page with title, meta-keywords, and a meta-description.
• Create a link from the home page to the "Daily News" page.
• Use the mock-up in Figure 1 to create the display.

Figure 1. Mock-up of horizontal navigation including the "Daily News" link

The HtmlFixture test table below loads a web page, asserts that the title and meta tags are correct, asserts that the "Daily News" display text and URL are correct, and then clicks on the link to prove that it works. Use Fitnesse markup variables [17] to define the HtmlFixture's URL. This will allow you to quickly toggle the environment (develop, test, production) being tested. The domain markup variable is defined in the root suite so that the test runner can control which environment all of the HTML tests will run against.

!define domain {www.jhound.com}

Also, give each tested HTML element a unique id tag so that the Element Focus command can be used to find and focus on the element.

<a id="rssMenuLink" href="/rss/index.do">Daily News</a>

Here is the example wiki source code. The pipe (|) character starts and ends cells and rows.

!|com.jbergin.HtmlFixture|
|http://${domain}/|
|Element Focus|jhoundTitle|title|
|Text|www.JHound.com JavaHound- Welcome to 
 JavaHound|
|Element Focus|Description|meta|
|Attribute|name|Description|
|Attribute|content|Find Java information: a hub 
 for information regarding Java, Extreme 
  Programming, Web Services, and Java 
  certifications.|
|Element Focus|Keywords|meta|
|Attribute|name|Keywords|
|Attribute|content|java j2ee, java junit, java 
 hibernate, java web services, java extreme 
 programming, java testing, java web applications, 
 java struts, java servlets, java jsp, java xml, 
 java certification, java certification quiz, java 
 object/relational mapping, javaone 2004|
|Element Focus|rssMenuLink|a|
|Attribute|href|/rss/index.do|
|Text|Daily News|
|Click|nowOnDailyNewsPage|
|Element Focus|jhoundTitle|title|
|Text|www.JHound.com JavaHound- JavaHound RSS|
|Element Focus|Description|meta|
|Attribute|name|Description|
|Attribute|content|Read daily news articles|
|Element Focus|Keywords|meta|
|Attribute|name|Keywords|
|Attribute|content|java technology news, sports 
 news, world news, us news, political news, java 
 news, extreme programming news, j2ee news|

Here is the line-by-line description:

1. Defines the test table's fixture type as HtmlFixture.
2. HTMLUnit requests a web page and parses its HTML with an XML parser.
3. Finds and focuses on a HTML title element with an id="jhoundTitle".
4. Asserts that its text is correct.
5. Finds and focuses on a HTML meta element with an id="Description".
6. Asserts that its name attribute is Description.
7. Asserts that its content attribute's text is correct.
8. Finds and focuses on a HTML meta element with an id="Keywords".
9. Asserts that its name attribute is Keywords.
10. Asserts that its content attribute's text is correct.
11. Finds and focuses on a HTML anchor element with an id="rssMenuLink".
12. Asserts that its href attribute is correct.
13. Asserts that its text is correct.
14. Clicks on the link; HTMLUnit then parses the response HTML.
15. (15-22) Proves that clicking the link worked by asserting that the "Daily News" page's title and meta tags are correct.

Figure 2 shows what the HtmlFixture test table looks like after it has finished running with all of its assertions passing.

Figure 3 shows what the HtmlFixture test table looks like after it has finished running, with one of its assertions failing.

Steps needed to run this example:

1. Download and install [20] Fitnesse.
2. Download and install [21] the HtmlFixture into Fitnesse.
3. Create a Fitnesse test page with the aforementioned wiki source code.

[22]HtmlFixture Asserting Form Values Are Correct Before and After a Form Submission

Project requirements:

• Create a quiz question with four possible answers.
• The second answer option (or "B") is the correct answer.
• When the user selects the correct answer, it should be highlighted in green and the submit button text should change to "Correct, Next Question."
• Use the mock-ups in Figures 4 and 5 to create the display.

Figure 4. Mock-up of a quiz question before the user has made a selection

Figure 5. Mock-up of a quiz question after the user has made a selection

The HtmlFixture test table loads a web page, submits its form, and then asserts that the submission result displays the "Correct Answer" page. Here is the example wiki source code:

!|com.jbergin.HtmlFixture|
|http://${domain}/java/quiz.do|
|Element Focus|radio_answerB|input|
|Set Value|checked|
|Element Focus|form_quiz|form|
|Submit|CorrectAnswerPage|
|Element Focus|submitButton_correct|input|
|Attribute|value|Correct, Next Question|
|Attribute|type|submit|

Here is the line-by-line description:

1. Defines the test table's fixture type as HtmlFixture.
2. HTMLUnit requests a web page and parses its HTML with an XML parser.
3. Finds and focuses on a HTML input element with an id="answerB".
4. Sets the second radio button (or answer B) to checked.
5. Finds and focuses on the HTML form element with an id="form_quiz".
6. Submits the form; HTMLUnit then parses the response HTML.
7. Finds and focuses on a HTML input element with an id="submitButton_correct".
8. Asserts that its value attribute is Correct, Next Question.
9. Asserts that its type attribute is submit.

Figure 6 shows what the HtmlFixture test table looks like after it has finished running with all of its assertions passing.

[23] Figure 6. HtmlFixture asserting form values are correct before and after a form submission--click image for full-size screen shot

Steps needed to run this example:

1. Download and install [20] Fitnesse.
2. Download and install [21] the HtmlFixture into Fitnesse.
3. Create a Fitnesse test page with the aforementioned wiki source code.

HtmlFixture with preprocessing to enable dynamic assertions

Project requirements:

• Update the quiz question text and dollar values from Figure 4 [24] to be database-driven.

The domain and database_level markup variables [17] are defined so that the test runner can control which environment the test will run against:

!define database_level {prod}
!define domain {www.jhound.com}

This example is leveraging the Symbol command in HtmlFixture. The Symbol command is used within a HtmlFixture test table to create symbol/value pairs. These symbol/value pairs can be referenced later in the same test table or, as in this example, subsequent test tables. Instead of using a static HtmlFixture test table to create the symbol/value pairs, the first test table is a customized ColumnFixture [25] that dynamically preloads HtmlFixture's symbol key/value pairs. Here is the example wiki source code for the first test table:

!|com.jhound.fitnesse.DynamicSymbolSettingFixture|
|domain|database_level|loadDynamicData?|
|${domain}|${database_level}|symbols successfully 
 loaded|

Here is the line-by-line description:

1. Defines the test table's fixture type as DynamicSymbolSettingFixture.
2. Sets DynamicSymbolSettingFixture's publicly defined properties domain and database_level to the values defined in line 3. Calls DynamicSymbolSettingFixture's loadDynamicData() method.
3. Displays the values for domain and database_level. Asserts that DynamicSymbolSettingFixture's loadDynamicData() method returns symbols successfully loaded.

Here is DynamicSymbolSettingFixture's Java source code. This fixture code processes the first test table and loads the symbol values at runtime from a database:

package com.jhound.fitnesse;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import com.jbergin.HtmlFixture;
import fit.ColumnFixture;

public class DynamicSymbolSettingFixture
      extends ColumnFixture {
  /*
   * domain and database_level need to be public
   * so that the Fitnesse engine can set their 
   * values.
   */
  public String domain; 
  public String database_level;

  public String loadDynamicData() {
    String returnMessage = null;
    String questionLevel = null;
    String questionText = null;
    Connection c = null;
    try {
      c = ConnectionManager.getConnection(
                           this.database_level);
      PreparedStatement pStmt = c
        .prepareStatement(
         "select question_level, question_text "
       + "from quiz_questions "
       + "where question_level = 100 ");
      ResultSet rs = pStmt.executeQuery();
      if (rs.next()) {
        questionLevel = rs
              .getString("question_level");
        questionText = rs
              .getString("question_text");
      } else {
        returnMessage = 
          "Error: did not find any data!";
      }
      rs.close();
      pStmt.close();
    } catch (SQLException e) {
      returnMessage = "Error: " + e;
    } 
    finally {
      if (c != null) {
        try {
          c.close();
        } catch (SQLException e1) {
          // do nothing
        }
      }
    }
    /*
     * Set symbol key/value pairs that will be 
     * accessible from within the HtmlFixture test
     * table.
     */
    HtmlFixture.setSymbol("url", "http://"
          + this.domain + "/java/quiz.do");
    HtmlFixture.setSymbol("questionValue", "$"
          + questionLevel + " Question: ");
    HtmlFixture.setSymbol("questionText",
          questionText);

    return (returnMessage == null) 
          ? "symbols successfully loaded"
          : returnMessage;
  }
}

The second test table is a HtmlFixture. The static URLs and static assertion values, as seen in previous examples, are replaced by the symbols url, questionValue, and questionText. The symbols are used to create this HtmlFixture test table at runtime. Here is the example wiki source code for the second test table:

!|com.jbergin.HtmlFixture|
|symbol url|
|Element Focus|span_scoreValue|span|
|Text|symbol questionValue|
|Element Focus|span_question|span|
|Text|symbol questionText|

Here is the line-by-line description:

1. Defines the test table's fixture type as HtmlFixture.
2. HTMLUnit requests a web page using the url symbol and parses its HTML with an XML parser.
3. Finds and focuses on a HTML span element with an id="span_scoreValue".
4. Asserts that its text is correct using the questionValue symbol.
5. Finds and focuses on a HTML span element with an id="span_question".
6. Asserts that its text is correct using the questionText symbol.

This test will remain valid because the assertions are connected to the database. The test remains valid from day to day and in different environments (develop, test, production) because the test dynamically accounts for any changes. Figure 7 shows what the DynamicSymbolSettingFixture and HtmlFixture test tables look like after they have finished running with all of their assertions passing.

[26] Figure 7. HtmlFixture with preprocessing to enable dynamic assertions--click image for full-size screen shot

Steps needed to run this example:

1. Download and install [20] Fitnesse.
2. Download and install [21] the HtmlFixture into Fitnesse.
3. Create a customized ColumnFixture [25] that preloads HtmlFixture's symbol key/value pairs.
4. Create a Fitnesse test page with the aforementioned wiki source code.

[22]XMLHTTPFixture Asserting XML Output Is Correct

Project requirements:

• Allow users of the web site to download the quiz's questions and answers in XML.

Figure 8 shows the XML document that the XMLHTTPFixture test table is going to test.

[27] Figure 8. XML output being tested--click image for full-size screen shot

This fixture depends upon XMLUnit 1.0 [28] and www.JHound.com Fixtures [11]. This test table is similar to the HtmlFixture test tables in Figures 2, 3, and 6. The difference is that it loads an XML (not HTML) document over HTTP and then uses XPath expressions to assert that the document is both correctly formed and that it contains the correct data. The XMLHTTPFixture is usable without modification and currently has three assertion commands:

1. Exists: assert that a XML element or attribute exists. Use @attributeName for attributes.
2. Not Exists: assert that a XML element does not exist.
3. Value: assert that a XML element or attribute has a particular text value. Use @attributeName for attributes.

Here is the example wiki source code:

!|com.jhound.fitnesse.XMLHTTPFixture|
|http://${domain}/java/quiz-xml.do|
|Exists|//www.jhound.com|
|Exists|//www.jhound.com/certification-millionaire
 |
|Exists|//www.jhound.com/certification-millionaire
 /questions|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/@value|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/@value|$100|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/problem|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/problem|Which of the 
 following correctly declares and initializes an 
 array of Strings?|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[1]|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[1]|String[] array =
 {"hello"; "bye";"testing"};|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[2]|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[2]|String[] array =
 {"hello", "bye","testing"};|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[3]|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[3]|String[3] array 
 = {"hello", "bye","testing"};|
|Exists|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[4]|
|Value|//www.jhound.com/certification-millionaire
 /questions/question[1]/answer[4]|String[] array =
 new String[3] {"hello", "bye","testing"};|
|Not Exists|//www.jhound.com/certification-
 millionaire/questions/question[1]/answer[5]|

Here is the line-by-line description:

1. Defines the test table's fixture type as XMLHTTPFixture.
2. Opens a java.net.URLConnection and loads the requested URL into a org.w3c.dom.Document object.
3. (3-19) XMLUnit's SimpleXpathEngine is used to assert that the Document object's expected elements and attributes exist and that they contain the correct the data.

Figure 9 shows what the XMLHTTPFixture test table looks like after it has finished running with all of its assertions passing.

[29] Figure 9. XMLHTTPFixture asserting XML output is correct--click image for full-size screen shot

Figure 10 shows what the XMLHTTPFixture test table looks like after it has finished running with a couple of its assertions failing.

[30] Figure 10. XMLHTTPFixture asserting XML output is incorrect--click image for full-size screen shot

With a little more work to the XMLHTTPFixture, these test tables could be made dynamic like the test tables in Figure 7 [31].

Steps needed to run this example:

1. Download and install [20] Fitnesse.
2. Download and install [28] XMLUnit 1.0 into Fitnesse.
3. Download and install [32] the XMLHTTPFixture into Fitnesse.
4. Create a Fitnesse test page with the aforementioned wiki source code.

Test Writing Tips

• Use Fitnesse to create system-level test suites for web applications. Test the HTML output with HtmlFixture. Start with HtmlFixture because it is easy to use, and then graduate to more dynamic tests.
• Employ developers to write the Fitnesse tests. Developers should apply the test-first development [33] technique, used in writing unit-level software, to their system-level test writing. Test-first development's rapid feedback is an important part of agile software development. When developers write the Fitnesse tests, they are forced to understand the problem at the system level, because they must think about how they are going to integrate their unit-level code into the larger system and then test that the larger system works. Once a developer has passed the system-level test, they know the task is complete. Technical testers then have the freedom to expand system-level tests.
• Run the tests as part of the whole team's continuous integration. Run the test suites before and after every promotion. Set up the tests to run on a scheduled basis. As the test suite grows it will take longer and longer to run. Therefore, running the suite on a scheduled basis allows the whole team to check the results without having to wait for them to finish.
• Make the tests dynamic from both user interface and data perspectives. Click on links and submit forms to prove that the page flow integrates successfully. Also, connect assertions to the database so that the tests remain valid from day to day and in different environments (development, testing, production). If the data being tested can change, then the test should dynamically account for those changes.
• Do small releases. It is easier to maintain the correctness of Fitnesse tests when, at any given time, there are only a small number of tests failing. While a project is actively being developed, its Fitnesse tests will fail in the production environment. Therefore, longer-running projects will cause a lot of the Fitnesse tests to fail against the production environment for an extended period of time. This will inhibit the whole team's ability to measure the correctness of the system.
• Put all Fitnesse test tables into one Fitnesse test page when testing the same requirement on multiple web pages. Putting all of the Fitnesse test tables into one Fitnesse test page ensures that the tests will only needed to be updated in one place when the requirement changes.
• Group the tests into suites by product or component. Ideally, all of the tests would be run each time any software update is made. However, as the test suite grows and takes longer to run, the whole team is more likely to take the easy road and not run all of the tests. Therefore, it is important to be able to run a subset of tests for a single product or component.

Conclusion

This article demonstrated the power of Fitnesse as a system-level testing tool for web applications. Developing comprehensive system-level tests will provide rapid feedback to the whole team, thus preventing mistakes in integration and promotion. The tests track the correctness of the system and will eventually become as important as the web software they test. Having this test safety net allows the whole team to maintain a rapid release schedule and deliver smaller projects simultaneously with larger ones. In conclusion, Fitnesse is good for the whole team (web developers, DBAs, technical testers, managers, and customers) because everyone can get involved in Fitnesse test writing and running.

What Next?

The next advancement in your Fitnesse test writing is to further integrate the tests with the database. For example, what if a web application needs to record the number of times a particular link has been clicked? The answer is to write and run a custom ColumnFixture to check the current number of clicks, then run a HtmlFixture test table to click the link, and then re-run the custom ColumnFixture test table again to confirm that the number of clicks has increased by one.

Resources

• Java fixture code [34] for this article, including www.JHound.com [35] fixtures
• Wiki markup [36] for this article
• Fitnesse [12]
• Fitnesse for HTML [37]
• HTMLUnit [38]
• JUnit [14]
• XMLUnit [28]
• XP/Agile methodologies [13] as defined by Ron Jeffries

The author would like to thank Gary Brown for teaching him the agile development methodology. He would also like to thank Kelli Moran-Miller and Travis Fritts for reviewing this article and providing invaluable feedback.

Robert J. Miller [1] is an agile web developer at CARFAX, Inc. [39], an MBA student at the University of Missouri-Columbia [40], and an entrepreneurial scholar at the Missouri Innovation Center [41].