Thoughts about functional testing ObjectLedge web applications.
See also notes about Parallel testing
Goals
- end-to-end regression testing of web applications
- acceptance testing of use cases (tracking project progress)
- parallel testing (functional comparison of application versions across a major refactoring)
Testing framework requirements
- tests must be fully automated ie runnable without human interaction
- testing needs to be integrated into maven build process, so that test could be run on the integration machine (poor bastard)
- an option to run single test/subset of tests/full suite of tests from withing eclipse would be nice, but is not critical
- container should be started once for the complete batch of tests executed, and stopped afterwards
- single test should initiate a session with the web application and then execute a number of get/post requests to the application, and check assertions on the responses.
to create more involed tests strings extracted from the responses 1 .. n-1 need to be available for use as get/post parameters to the request n
Test design guidelines
- test should be independent of one another : failure of one test should not disrupt other tests
- test should be repeatable : test should restore the system to the state from before running the test
- weaker version of the above would be : test should restore the system to stage from before running the test provided that the test completed successfully
Application design guidelines
- for every operation that creates resources there should be a removal operation, or alternatively creating multiple resources with identical characteristics should be allowed.
- when new resources are created, their identifiers should be printed inside HTML comments, or HTML form hidden inputs, to so that test can easily extract them for future use. Otherwise tests have to resort to complex & fragile extraction techniques.
Tools
Tools that we consider using for implementing the functional testing process:
MaxQ
Home page: http://maxq.tigris.org/
A testing framework that provides a proxy server that can record interaction of the user with the application - which is nice for drafting testcases. The test cases have the forms of Python scripts executed using http://www.jython.org/.
Does not concern itself with container startig / stopping.- The documentation does not give any information about runing the tests from an JUnit test runner. However the tests itself extends org.junit.TestCase so it should be possible to create a TestSuite that would discover them and pass necessary references to test runner.
Jython scripts give a great deal of flexibility. Common base class of the tests can be augmented with custom functionality- A number of standard Python libraries can be used, as well as arbitrary Java libraries.
- The test case sources would be much more compact than Cactus test cases.
Cactus
Home page: http://jakarta.apache.org/cactus/
A testing framework that implements automated servlet container startup / shutdown.
- Starts/stops container as necessary but the most recent version of Cactus (1.7) supports Tomcat up to version 5.0 - and we are using 5.5
- Good integration with Maven
- Eclipse plugin is provieded but broken.
- Can HttpUnit can be used to perform assertions on web application resposes.
- Cactus is mainly concerned with server-side unit testing, but can be used for end-to-end testing as well.
- Cactus test cases require some redunant typing compared to MaxQ test cases.
See End to end testing with Cactus for more in-depth analysys.
HttpUnit
Home page: http://httpunit.sourceforge.net/
HttpUnit emulates the relevant portions of browser behavior, including form submission, JavaScript, basic http authentication, cookies and automatic page redirection, and allows Java test code to examine returned pages either as text, an XML DOM, or containers of forms, tables, and links. When combined with a framework such as JUnit, it is fairly easy to write tests that very quickly verify the functioning of a web site.
The same techniques used to test web sites can be used to test and develop servlets without a servlet container using ServletUnit, included in the download.
- does not concern itself with container starting / stopping
- provides a number of powerful assertion mechanisms. Extracting values & passing them to following requests should be easy.
- HttpUnit is much better suited for performing end-to-end testing with Cactus.
- Thanks to high level representation of the received resonses, esp. HTML forms test cases can be quite compact and easy to read.
ServletUnit provides an alternative for starting / stopping the containter, although it's geared towards unit testing servlet behaviour (which is done easier using mock objects) and not end-to-end testing.
Latka
Home page: http://jakarta.apache.org/commons/latka/
Latka is a functional (end-to-end) testing tool. It is implemented in Java, and uses an XML syntax to define a series of HTTP (or HTTPS) requests and a set of validations used to verify that the request was processed correctly.
- quick look at the docs & samples suggests that there is no way to extract values to be passed to following requests, which disqualifies this tool.
JTidy
Home page: http://jtidy.sourceforge.net/
Can be used for validating HTML pages. Common test base class could have an option to validate all received reposnses for HTML/XHTML correctness, possibly controlled by vm property or similar.
- it should be easy to integrate this tool with any type of Java or Jython based tests.
Container starting/stopping
Since most end-to-end testing tools do not handle container startup / shutdown, and Cactus 1.7 does no support Tomcat 5.5 integration, it seems that we'll need to implement our own server startup/shutdown goals as a part of maven build process. ledge-maven-plugin would be an apropriate place to do that.