Wednesday, June 24, 2009

Limits of Open Source SOA Testing Tools

In this article, we will discuss the limits of adopting an Open Source SOA testing tool for SOA and Web Services projects. Open Source has become an essential and popular resources for many tools and platforms used in SOA deployments. From operating systems such as Linux, to databases such as MySQL, and browsers such as Firefox, Open Source has a proven track record for cost-effective applications and tools.

SOA testing involves the ability to test SOAP, XML, and REST based messaging against a service endpoint in order to assess the robustness, reliability, and resilience of the service. Comprehensive testing of a service endpoint involves 4 primary focus areas: Functional, Performance, Interoperability, and Security.

Functional testing provides the ability to verify the proper behavior of services and build regression test suites to automate testing and baseline expected behavior of services to quickly assess and validate functionality through the lifecycle of service revisions.

Performance testing provides a concurrent, simultaneous loading agent framework which can determine throughput and capacity statistics of the back-end service across the range of input and client load variances to validate Service Level Agreement rates and well as identify bottlenecks and potential architectural weaknesses and performance dependencies.

Interoperability testing maximizes interoperability by measuring both the design characteristics of a service as well as the runtime adherence to standards and best-practices. Isolating potential interoperability issues early on in the lifecyle can significantly optimize efforts of integration when exposing the service to trading partners and clients which may be build on a varying array of disparate web services technologies and platforms.

Security testing assesses the risk posture and robustness of a service with regard to vulnerability, data-leakage, data privacy, and data integrity. Each web service is unique based on the schema which defines the input and response message structure of how to communicate with the service. Using the WSDL schema as the source, security tests can be built to create boundary condition tests for the service which then identify the robustness of the service handling inputs outside the range of expectation. Further, the various security and identity specifications set forth by the W3C and OASIS provide a framework to test for the level of data integrity, data privacy, and access control on the service transactions and endpoint itself

Open Source SOA Testing
The Open Source tools available today for SOA testing focus primarily on the Functional testing of a service. Since functional testing is often first in the SOA development lifecycle, and adopted early-on in the development and implementation phase, it thus becomes widely adopted by development teams both because it is free, and also because the use cases are often limited to simple unit testing of service messaging.

However, as services mature and move down on the SOA lifecyle to system testing, integration, pre-production analysis and validation, the other perspectives of SOA testing need to play a role in the comprehensive assessment of the quality, robustness, and capabilities of a published service. It is in these areas where the Open Source XML/REST testing tools fall short.

Open Source Limitations - SOA Functional Testing
Generally the functional testing capabilites of an Open Source testing tool are adequate for the simple type of SOA deployments which do not have complex WSDLs, Schemas, or message patterns. Once the deployments gain more complexity however, the challenges of functional testing move from single request-response testing to scenario testing where the functional behavior is measured not by one request-response, but rather several transactions each dependent on the other as a business functional unit. Having the ability to test these types of functional scenarios effectively requires the ability to maintain state between one test result and the next.

Open Source Limitations - SOA Performance Testing
While there are several Open Source performance testing products on the market, these are primarily tools used in the static web testing paradigms. When dealing with SOAP and XML -based transactions, the static data testing behaviors of these performance harnesses does not allow for the unique-wire signature requirements of the message patterns as is indicative of actual service transactions. Thus, when running performance tests with a web-based testing platform, the results are that the end-points become inundated with static messages which are not characteristic of actual traffic patterns. In fact, in many cases, the service endpoint itself is supposed to reject these static messages as replay-attacks on the service.

Another consideration of performance testing is the level of security and identity provisions that messages may be required to carry in order to access the service. The static Open Source performance testing harnesses do not provide solutions for message security and message identity requirements.

Open Source Limitations -SOA Interoperability Testing
The promise of SOA provides a open, reusable architecture which lowers cost through the reuse ROI factor. The challenge of SOA however is the ability to widely interoperate with other technologies that also communicate via SOAP, XML, and REST. Interoperability testing involves both design-time analysis of service characteristics, such as WSDL and schema, as well as run-time assessment of a service robustness in terms of consuming and handling message patterns that may fall outside the expected structures. Open Source toolkits leverage the available WS-I analysis framework to provide a means to assess the design-time characteristics of a WSDL and Schema according to published profiles, and also provides some run-time analysis reporting of message patterns. However, the Open Source toolkits do not provide the ability to generate messages that fall outside these expected patterns, which is the key to measuring the actual posture of the run-time service. In fact, it is testing of messages that are not expected where the true measure of a service posture can be determined.

Open Source Limitations - SOA Security Testing.
Security testing falls across many areas. From a threat perspective, security testing involves integrity and structure of messages with injection attacks at the parameter and data structure levels in order to assess the behavior and resilience of the service endpoint when faced with data values and message structures outside of the expected format.

From a trust perspective, security testing involves PKI with encryption, signatures, and identity tokens. This requires testing frameworks that understand the various emerging standards from W3C and OASIS in order to support the wide range of security message formats and also requires the means to retrieve and utility X509 and Private Keys from a variety of sources including Windows Keystore, Java Keystore, SmartCards, PKCS12 files, etc.

The Open Source tools are designed for general testing and message creation, but lack the in-depth security and identity features to be considered viable for this type of testing.

Adopting an Open Source tool for SOA testing seems the simplest, most cost effective choice for developers and testers early on. However, you should plan and consider the implications of a longer term strategy with an Open Source testing tool. The many other aspects of service testing which contribute to a comprehensive testing solution across the entire SOA lifecyle that go beyond simple functional testing.

Companies who specialize in SOA and Web Services testing focus their products on specific customer use-cases and testing needs, rather than the needs of the Open Source community.

You may find that paying for a testing solution ends up costing less than not paying for one.


Yaron Naveh said...

You use "testers" and "developers" interchangebly here. In many cases developers only seek a quick invocation tool and hence their attraction to the free stuff.

PJ said...

It is the fact that often times "testers" and/or "developers" can not see the forest through the trees. They are focused on their specific job, rightfully so, but to gain agility, organizations need to implement enterprise test automation solutions, rather than individual teams using point solutions, that are not re-usable across the enterprise. It is management that must see the forest and move to an enterprise solution that is complete and collaborative across the testing continuum.

jack said...

The tools have different strengths. I really like ITKO Lisa if you have to go deep into the JAVA components that expose the services. And PushToTest is very good at also testing rich internet clients.

Software testing services

Jay Philips said...

This is a good introduction to the world of SOA testing and the things to keep in mind while doing. You can check out the SOA testing tool SOArite from It provides SOA functional and workflow testing of WS, REST and Http Services. In addition it also provides testing for JMS, JDBC and TCP for connectivity to legacy systems

Devika said...

Thanks a lot for all your valuable article! We are really happy about the your...
SEO Training in Chennai

Qualitia Soft Test Automation Tool said...

Great article, there are many other open source software testing tools available. Qualitia is again a Selenium based testing automation tool for QTP/UFT.

Savithaa Nirmal said...

Great efforts put it to find the list of articles useful for Testers.Definitely will share the same to other forums.We are also one of the best sources to learn Selnium -Selenium Training Institute in Chennai

Krishna Veni said...

Good and nice post, thanks for sharing your post... keep rocks///

PHP Training in chennai | Android Training in chennai