Parallel testing

The goal of parallel testing is determinig if two application instances behave in the same way.

We are facing this situation with Cyklotron right now. Old Cyklotron aka Cyklotoron 1 or CykloKlon is being ported from the old toolking to ObjectLedge / Coral. The ported application works for a large part, but 1) there are some things not ported yet, and 2) some things might have been broken during porting. We are now looking for a way of making a parallel test of the two systems to weed out both types of these issues.

Here are some possible scenarios:

Interactively recorded test of C1 transposed for C2

The test would be prepared as follows:

• Known state of C1 database would be stored for reference
• C1 testcases would be recorded using MaxQ
  • Store intermediate database state
  • Perfrorm a series of interactions
  • If you are happy with the testcase save the script
  • Otherwise restore database state and re-record
• the a copy of resulting MaxQ scripts would be machine processed to covert request URLs to C2 format
• C1 initial state database would be transformed to C2 format using ARL import tool (+possibly some Cyklotron specific RMLs or programmatic tools)
• processed MaxQ scripts would be run against a C2 format until all diferences are eliminated.

Notes:

• testcases must be executed in the proper order - each one depends on all the previous testcases because of ARL/Coral entity identifiers.
• C1 & C2 need to be augmented with machine-friendly error reporting functionality, so that stacktraces & error results can be easily noticed by MaxQ assertions.

Pros:

• Test can be created quickly.
• Little changes to applications themselves required - error reporting is centralised.

Cons:

• Tests are very fragile - if C1 functionality changes in some place, esp. in terms of number of resources that are created the tests and all the following tests will break an will have to be regenerated more or less from source materials (scenarios)
• If C2 implements some functionality by creating different number of resources there will be trouble. And if it's something low level, this method may fail alltogether.

Manually created Cyklotron functional tests

Test preparation:

• for each test scenario HttpUnit test case would be created by hand
• request URL would be generated with a LinkTool that would be able to generate C1 and C2 links depending on test configuration
• data would be extracted from responses and passed to following requests

Notes:

• C1 & C2 would have to augmented with some machine friendly status reporting, plus screens should be modified to emit information useful for tests in HTML comments, to make extraction simpler

Pros:

• Tests would be far less fragile
• Tests would be independent from one antother - you could run just one, not the whole lot each time
• Tests would stay correct independent of ongoing development

Cons:

• More modifications to C1 and C2 codebase necessary
• Test are written by hand, not generated.
• Maybe we could use MaxQ with custom template to generate Java unit tests skeletons, and fill in assertions / extration / parametrization?