Tuesday, February 19, 2008

Automated SOA Regression Testing

In testing SOA environments, I found it necessary to come up with a simply, automated means of creating reusable tests to quickly verify that services in a SOA deployment were still working as expected across version upgrades. I use a product from Crosscheck Networks called SOAPSonar which provides automated regression testing and automated regression baseline creation. Some key features that were required for an automated solution that this product provides include:

  1. Comparison to baseline measurements includes XML diff capability to allow you to select which portions of the response documents to isolate and diff services characteristics while also allowing only portions of the responses to be used for the diff checking. SOAPSonar allows node fragment, element value, and attribute value exclusions to be defined such that values that tend to differ in each response (timestamps, etc) can be ignored so as to not skew the diff results. Plus it is simple drag drop in a tree view format which makes the comparison easy.

  1. Regression baselines can be created automatically. The ability to send the entire test sequence to the service and store the behavior as the baseline. SOAPSonar does this and automatically creates the xml diff profile which can then be modified as needed for the verification characteristics (often no modification is required).

  1. Regression baselines can be easily shared. SOAPSonar stores the baseline policies within the test project file which can then be shared with other teams. This allows the regression testing to be integrated directly into build automation or other existing scheduling harnesses.

The steps below provide the actual step by step instructions for how to baseline your services. To run these steps you will first need to download an eval of SOAPSonar Enterprise Edition from http://www.crosschecknet.com.

Creating a SOA Regression Baseline

Load up your WSDL in the SOAPSonar interface and create a set of inputs for your back-end services. This can be a single service or multiple services.

Go to the test suite view and press the “Create Regression Baseline” icon to have SOAPSonar run the tests and store the results in a baseline. This will result in the baseline regression policy editor appearing which allows you to specify exclusions if necessary on the document diff rules.

Generating a new baseline will run all test cases and test iterations in the test suite and store all the responses in sequence to an internal baseline regression data set. Before this baseline set of responses is generated, you will be presented with a dialog that allows you to choose how to initialize the success criteria functions for each baseline response. The options include an XML diff of the entire document, copying existing success criteria settings defined on each referenced test case, updating existing settings, or leaving the criteria empty by default where it can be edited and modified manually.

Regression Baseline Criteria Editor

Once the Baseline Regression data set has been created and stored, you are then presented with the Baseline Criteria editor where you can selectively choose for each test iteration how to determine success or failure. The criteria can be XML diff based where you choose to always diff the current test response against this stored baseline response and detect any variances.

To exclude items from the diff, simply click on the item and select the Exclude icon from the toolbar, or right-click on the node and select Exclude.

Running a SOA Baseline Regression Test

To run a baseline regression test, simply choose the “Use Saved Regression Baseline Success Criteria” option and press the Run Test button.

Generating an HTML Baseline SOA Regression Graphical Diff Report

After running the baseline test you can then generate and export an HTML report which highlights the Pass/Fail results of the test and also provides graphical diff reports for each test iteration. In this report, you can view the diff of the request vs. baseline request, response vs. baseline response, and each diff (with applied exclusions) for each XPath expression.

To generate the report, click on the log file and choose the report labeled “[HTML] Baseline Regression XML Diff Report”. You will be prompted for the directory to store the results. The results will be written to the selected directory with the name of the log file with a .htm extension and a subfolder of the test data created with the same name. The test data written with the report includes all actual XML requests and responses, as well as the baseline requests and responses.

The resulting report provides a summary and detailed view of the test results and also provides links to view the graphical diffs of:

1) Stored Baseline Request vs Currently Sent Request

2) Stored Baseline Response vs Currently Received Response

3) XML Fragment Diff of each diff rule defined.

The HTML report is contained in it’s own subdirectory and can be published to a central server.


Creating a baseline of expectation for service behavior in an automated way can significantly reduce testing time for testing version updates of services and detect functional breakage. Given the complexity of version management among SOA components, a baseline strategy will pay big dividends in time savings of the life of the deployment.

1 comment:

DevRabbit IT Solutions Inc. said...

The Article on Automated SOA Regression Testing is nice give detail information about it.Thanks for Sharing the information about Regression Testing . Software Testing Services