Home About Me Project List Mail me
Last update: 2013-01-10

X.C.A.T. - XML Comparison and Testing

Abstract

Several years ago I needed to compare XML documents for the purpose of testing. However, a simple file compare  did not suffice because the XML document produced by the program to be tested contained dynamic data. The expected document I planned to compare with was static. As a result testing failed. Hence, I helped myself by writing this little Java based XML comparison and testing framework.
For example, the documents I was creating were always the same given the test input. But they had a date element which included the current date. (Think of booking date for example.) Plain file comparison failed because the date elements deviated. This is where this framework can help with.

Introduction

Assume you develop an application which produces XML documents. In the course of development you will create various stages of tests such as unit tests, assembly test etc. Since your program produces XML you will have to do XML testing - i.e. you need to compare actual and expected XML documents. Such documents may contain invariant and variant (dynamic) elements.  The variant elements make it challenging to pre-produce and store expected documents for later comparison. An typical example for dynamic elements that may be excluded from comparison are timestamps. The X.C.A.T. framework allows you to mark the value of (variant) XML tags as "don't care". As an result such data is not taken into account when comparing actual and expected result. Furthermore it brings a kind of server which allows to post an XML request and then return an XML response. Document content can be made dynamic. The framework currently uses BeanShell for that purpose. Other scripting libraries can be added if desired because the invokation of BeanShell is encapsulated by an interface. So there is no direct dependency between test framework and the actual scripting solution.

XML documents are transformed into a DOM tree. That means that the complete document is read and helt in memory. Given today's memory sizes this should not be a problem unless your XML is getting very large. The DOM approach might become a limitation when deploying this framework.

I wrote this framework several years ago. I guess it was written in Java 1.3. Recently I needed it again. So I was porting it to Java 1.6. For example, it now uses generic types, for each loops etc.

XML Comparison

  1. Create an XML document that you expect as a result of your code you want to test. Keep it in memory or store as file.
  2. Think about which elements are dynamic and should be excluded from the comparison. Mark those elements with $$ in the expected result XML.
  3. Create a test case that invokes your code.
  4. To that test case add framework code in order to compare actual XML with expected XML.

Suppose your program your are about to test would be expected to produce the following XML document. Note the booking date. It changes from test execution to test execution. It makes straight forward XML document comparison challenging.

<bookingList> <bookingDate>2013-03-17 17:52</bookingDate> <positions> <position> <article>iMac 27"</article><price>1.600 EUR</price> </position> <position> <article>MacBook Pro 13"</article><price>1.100 EUR</price> </position> </positions> </bookingList>
Actual XML document to be verified (actual.xml)
Suppose further that you do not care for the booking date but for the program to pick the correct price for the goods ordered. Therefore prepare an document for comparison as follows. Note that booking date has a value of $$ in order to indicate that this field is fynamic and we do not care for.
<bookingList> <bookingDate>$$</bookingDate> <positions> <position> <article>iMac 27"</article><price>1.600 EUR</price> </position> <position> <article>MacBook Pro 13"</article><price>1.100 EUR</price> </position> </positions> </bookingList>
Expected XML document for comparison (expected.xml)
Now invoke the framework for comparing actual result and expected result. This is how your test code could look like. You will receive a fairly readable error message in case the test fails.
class MyTest { ... @Test public void testMyProgram() throws FileNotFoundException, IOException, SAXException { ... Your code that produces "actual.xml". (You already have prepared "expected.xml" in advance.) ... ComparisonStatus l_status = XmlDomFileComparator.areEqual( ".../expected.xml", ".../actual.xml" ); assertEquals( ComparisonStatus.OK, l_status ); ... } ... }
Possible test code (file based)
When unit testing I actually would not like to read/write files. The framework also allows to compare XML strings. So, you can directly compare your result with an XML document that you set up in your unit test code.
class MyTest { ... @Test public void testMyProgram() throws FileNotFoundException, IOException, SAXException { ... Your code that produces the actual XML document. Let's say, that the variable actualXmString holds the string representation of that document while expectedXmlString holds the string representation of what you expect. ... XmlDomNode actual = XmlDomFileReader.readDocument( actualXmlString ); XmlDomNode expected = XmlDomFileReader.readDocument( expectedXmlString ); ComparisonStatus l_status = expected.isEquivalent( actual ); assertEquals( ComparisonStatus.OK, l_status ); ... } ... }
Possible test code (string based)

Request/Response Server

How It Works

Seldom does your application exist alone. More probably your application interchanges information with other application. Information interchange today is widely based on XML. This framework provides an XML server simulation in order to simulate another server your application depends on.
Basically the server maintains a map with XML request documents and XML response documents. When you send an actual request it tries to figure out which of the stored requests fits. Then it returns the response associated with that request. Use $$ for request elements that do not matter. (Otherwise a lookup in the map might fail.) To make the responses dynamic use scripts within $ symbols. These scripts will be evaluated by the server before returning the response.

In unit test I typicaly prefer to have everything in the code. However, with XML it might be more handy to work with XML documents in the filesystem rather that using verbose Java code to produce an XML document. Therefore I will now explain how to use files. (Even though it is also possible to prepare the map programmatically.)

  1. First you chose a directory where to store the prepared XML documents. For example this might be ./testdata.
  2. In that folder create a subfolder requests. Store all request XML documents in there.
  3. Create another subfolder responses. Store all response XML documents in there.
  4. Two corresponding request and response documents need to share the same file name. Using the corresponding constructor will cause the framework to load all request/response pairs from the given directory structure.
  5. In your test code add the following lines as depicted below.
  6. Further programming might be required for the server to be integrated into your software. For example: It might be required to implement certain interfaces.
// Setting up the server XmlDomRequestResponseMap l_server = new XmlDomRequestResponseMap(); l_server.addRequestsFromFilesystem( new File( "./testdata" ) ); ... // Now you can request data from the server, providing an XML request document XmlDomNode l_response = l_server.getResponse( ...request XML goes here... );
Example for a test using the request/response server simulation.
Be advised that server simulation here does not actually mean that data is sent via the network. The framework only simulates a request/respons communication pattern. For unit testing I actually do not want to depend on network, database and other resources to be available.

How Scripting Works

You need to implement interface IInterpreterWrapper in order to integrate a scripting language. The framework comes with built-in support for BeanShell. In other words BeanShellInterpreter is such an implementation. To make a long story short, what implementations of IInterpreterWrapper have to provide is:
  1. A way to pass values from the server to the scripts. This is achieved via passVariable(). Currently two variables are known by the framework:
    1. request - The request document (input) for use by the scripts.
    2. response - The response document (output) the scripts are embedded into. Access to the static elements can be used by the scripts to refer to certain values.
  2. A way to release variables. To be implemented in freeVariable().
  3. A way to execute a script. This happens inside evaluate().

Prerequisites

Version History

Version Date Change
1.0.0 2008-09-11 Initial version.
1.1.0 2013-01-10 Made code fit for Java 1.6 (generics, for each loop etc.).

Copyright Statement

X.C.A.T. Copyright (C) 2008-2013 Holger Zahnleiter

This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. The program and its source code are published under the GNU General Public License (GPL). See http://www.gnu.org/licenses/gpl-3.0.txt for details.

References/Links

http://www.beanshell.org BeanShell - Lightweight Scripting for Java

Downloads

xcat-1.1.0-src.zip


Disclaimer

The information on this web site and the documents downloadable from here have been carefully reviewed and is believed to be reliable, but I do not assume any liability arising out of the application or use of any documents, programs or circuit described herein. Furthermore I want to declare that I'm not responsible in any way for the content of other web pages, books and other sources I'm refering to.