Service Regression Testing Toolkit User's Guide
Server Side : testing facet generation
Looking inside the testing facet
Server Side: testing facet execution
Creating the monitoring DataBase
The Service Regression Testing toolkit uses test cases as a way to stipulate contracts between a service provider and service users. Test suites are published by the service provider as a XML-encoded facet. When a user acquires a service, s/he can use such test suites to check whether the service behaves as desired. If no deviation from the expected behaviour is noticed, the contract is stipulated, and the test suite specifies the service behaviour required to fulfil the contract. Then, the user can periodically run the test suite to discover if changes made to the service implementation entail the violation of any of the initial assumptions and expectations.
The toolkit is composed of two different tools:
Server Side : testing facet generation
To run the Testing Facet Generator tool execute startserver.bat from a shell or click on startserver.bat file icon from File Manager.
The tool can be invoked with different optional parameters:
Syntax : startserver.bat [-sd] [-o testcasefilename] [-f facetfilename]
the arguments work as follows:
-sd : avoid to produce the informative
part of the facet, i.e. it only produces the <FacetSpecificationData>
-o : testcasefilename : indicates the file where the test cases will be
stored (i.e. second level of the facet)
-f : facetfilename indicates where the facet will be stored
Launching startserver.bat the following interface is shown.
The Preferences menu permits to set proxy configuration and the parameters related to the monitoring database (major details will be found in the following sections).
Clicking on “Generate” button the Testing Facet Generator XML–encodes a unit test suite provided with the service. To generate the facet, the user can specify Owner, type and a brief description of the test suite, the type of assertions (and eventually the deltaValue needed to adjust the measures done and the service URL, this two parameters will explained later) to generate and Policy parameters (frequency and max number of execution). Owner and TestDescription aren't mandatory.
Note that the information provided here is used only for informative purpose. Moreover the user can specify the facet and test path.
The test suite encoding can involve functional and/or non functional behavior. The tester can choose the type of XML test suite to generate (Assertions Type in the following figure).
- Functional : the tester chooses “Functional” if he wants to generate a test suite considering only service functional behaviour : the generated test suite will contain only functional information (the generated test suite considers only the expected output for the given inputs).
- NonFuncional: the tester chooses “NonFunctional” if he wants to generate a test suite considering only service non-functional behavior: the generated test suite will contain only non-functional information (the generated test suite considers only the expected response time and throughput for the inputs contained in the JUnit assertions).
The measured values will be adjusted by the delta parameter specified by the tester. For example, if the user specifies a Delta of 23% and the measured value of the service response time is 100 ms, the generated facet will report a response time constraint of 123 ms (less than).
- FunctionalANDNonFunctional: the tester chooses “FunctionalANDNonFunctional” if he wants to generate a test suite considering service functional and non-functional behaviour.
Each assertion will contain both functional and non-functional information.
If the tester chooses to generate a test suite considering also service NonFunctional behaviour, he must specify if the service under test is deployed.
- If the service isn't deployed Non Functional behavior will evaluated invoking the effective local service implementation.
- If the service under test is deployed, the Non functional behaviour will be evaluated invoking the service.
Next, the service provider indicates to the tool the service class and the JUnit source file containing the test suite for the service–under–test .
Then, the tool starts analyzing both the service and the JUnit class and shows to the user the list of test cases contained in the test suite ("Choose test case" window).
1. automatic test case transformation: the tool automatically translates any expression, but service operation invocations, in literals and generates the XML–encoded test suite;
2.selective translation: the user can select which method invocations, corresponding to service operations (see the Method list window), should be evaluated and which should be left symbolic in the testing facet.
In a JUnit assertion like
assertEqual(expected, actual)
The actual parameter can be an expression composed of multiple invocations:
operation(operation(operation(javaStatic(operation(….)))))
Clicking on the "Next" button, the facet is generated. The figure below summarizes facet information.
Clicking on View the tester can see the generated test suite.
In the following figures we can see an example of a facet and of the associated test suite containing only functional aspects.
-The facet
- the XML encoded test suite
In the following figure we can see an example of a facet and related test suite containing functional and non functional assertions.
-The facet
- the test
JUnit lets the developer
test software code units by making assertions that the intended requirements are
met. Test cases written for JUnit can be re-used (or written from scratch) to
test a web service operation. A simple JUnit test case is shown below:
import junit.framework.*;
public class FindLocationTest extends TestCase {
public FindLocation locator = new FindLocation();
...
public void testGetAirport() {
Location expectedValue = locator.getLocation("Benevento");
assertEquals(expectedValue.nearestAirport,"Napoli Capodichino");
}
}
Test cases can be grouped in test suites. Click
here to get more information
about JUnit
Looking inside the testing facet
In the example of the figure below, the first assertion checks whether the
operation getRestaurantID returns a correct ID, i.e., a sequence of four digits.
The upper part the figure
shows the JUnit test case, while the lower part show how the three different
assertions are XML–encoded. Note that each functional assertion is followed by a
QoS–assertion. As shown, some parameters of the assertions appear in the XML as
literals. For the first assertion, they were already literal in the JUnit test
suite. However, it can happen that a literal value contained in the XML test
suite results from the evaluation of an expression contained in the JUnit test
case. Whilst, there are some parameters that appears as symbolic. These are Java
static methods, e.g. Pattern.matches, that can be invoked from the regression
testing tool without the need for accessing the service implementation, and
service operations, e.g. getRestaurantID. The second assertion checks whether
the checkRestaurantAvailability returns a value equal to true or false,
indicating the availability of the requested number of seats. Similarly, the
third assertion (that fails when the service evolves) checks whether, in case
the number of seats is not specified, the result is an error message.
Server Side: testing facet execution
From the main window, clicking on the button “Exec”, we can execute a test facet. The following window will be shown.
The user can choose among three types of execution:
· Functional: only Functional behaviour will be considered during regression testing execution: the test will report if the service response is compliant with the expected values in the testing facet.
· Non Functional: : only Functional behavior will be considered during regression testing execution: the tool will evaluate if the contraints on the QoS parameters defined in the testing facet will be satisfied..
· Functional and Non Functional: testing execution will consider both functional and non-functional behaviour.
The tester can specify the URL of the service-to-test or a file containing service implementation:
The two snapshots show a possible facet Functional execution: the shown percentage of passed/failed test cases is related only to the functional behaviour exhibited by the service.
The progressBar shows at runtime the current number of passed/failed test cases.
Clicking on ViewTable, the result of the execution is shown in a table. Each row of the table contains the following information:
- the test case name.
- the generation date.
- the execution date.
- the result of the test ( functional and/or non functional).
- a button to view the XML log related to the chosen test case.
A possible execution is shown in the following figure.
Clicking on "ViewTable" button, the user can see a table summarizing the result of the execution:
Clicking on "Expand" the user can see the XMLLog related to the test case. As we can see in the following figure, in this case, the log contains the results of the execution of Functional and NonFunctional execution.
Information contained in the shown XML file has the following format:
<test name=”testName”>
[<reusedInvocation> reusedInvocation </reusedInvocation>] *
<Assertion>
<Functional>
<invocation >
java invocation
</invocation>
[<expected > expected Object </expected>]**
<measured > measured Object </measured>
[<result > functional test result </result>]**
</Functional>
[<QoSAttribute>
<attribute name =”responseTime” threshold =”thresholdValue” >
<measured > measured Responsetime </measured>
<result > Passed if measuredReponseTime < threshold </result>
</attribute>
<attribute name =”throughput” threshold =”thresholdValue” >
<measured > measured throughput </measured>
<result > Passed if measured throughput > threshold </result>
</attribute>
</QOSAttribute>]***
</Assertion>
< Assertion >
…….
</ Assertion >
</test>
(*) (if monitoring is used)
(**) (if functional or functionalANDNonFunctional is chosen)
(***) (if Nonfunctional or functionalANDNonFunctional is chosen)
In order to minimize the number of service invocations during the execution of the regression testing, it's possible to reuse the results of previous invocations having the same inputs; in this case rather than invoke explicitly the service, we can simulate the invocation using data monitored by a specific module. This component, provided for each Axis installation, is able to monitor service invocations and to store data in a database. If the tool uses these data, the service invocation is "simulated": the monitor verifies if a previous invocation with the same input is stored in the database. In this case, the monitor returns the related output without invoking the service. Otherwise, an invocation is needed.
To enablee monitoring, the tester has to set the parameters ( IP Address, user, password ) of the database containing the monitoring data.
Clicking on Preferences/Monitoring (or just typing CTRL+M), the following window will be shown.
If the tester wants to execute the test using monitoring data, he must specify the information needed to access to the monitoring database:
- IP Address
- the user name
- the related password
If the tester chooses to execute the testing facet using monitoring data after the execution, he can see how many invocations were reused.
In the resulting XML log will be reported all the reused invocations (nel tag reusedInvocation in figura):
In order to monitor the invocations of a service is necessary to add a module able to do this. To enable the monitoring of a service, just copy SOAPMonitor.class and SOAPMonitorConfig.txt (available in work/monitoring) under the folder WEB-INF/classes of your axis installation folder.
If you choose a different name for your axis folder or you have multiple instances of axis to monitor, just copy the specified files in all the axis folders, and execute the following step for each of them.
In the file server-config.wsdd located under axis_home/WEB-INF you must insert, after the tag </globalConfiguration>, the following line:
<handler name="monitor" type="java:SOAPMonitor"/>
Moreover, you must insert within the tag <global Configuration> the following lines :
<global Configuration>
...
<responseFlow>
<handler type="monitor"/>
</responseFlow>
...
</global Configuration>
Then copy the file mysql-connector-java-3.1.11-bin.jar under TOMCAT_HOME\common\lib.
The added class is able to log all the service invocations.
As specified in section Installation, the installation of the RegressionTestingToolkit requires that a DBMS MySQL is installed; use the provided soapmonitoring.sql file to create the database necessary both to the monitoring class (SOAPMonitor ) and to the RegressionTestingToolkit.
The file SOAPMonitorConfig.txt contains the information used by the monitoring class (username, password and IP Address of the DBMS). Modify it as needed.
To test services accessible using a proxy, from the main window the tester must choose Preferences/LANSettings (or choosing CTRL+S); the following window will be shown:
If the tester choose to use the proxy he must specify its IP Address and the used port.
To run the Regression Testing Tool , launch startclient.bat from a shell or click on startclient.bat file icon from File Manager.
From Preferences menu, the tester can set a proxy to use or, to reduce the number of invocations against a service, the parameters related to a monitoring database (see Lan Settings and Monitoring sections).
From the main window, the user can choose to download and save a test facet. Clicking on “Download facet” button the following window is shown:
The user inserts the facet URL, and, chosen the path of the downloaded facet, the download starts.
The tester can choose to execute test
1. from a facet URL (clicking on “Execute from Remote Facet” button from the main window)
The following figure shows a possible execution
2 from a
previously downloaded facet (clicking on “Execute from Local Facet” button from
the main window). The following figure shows a possible execution
Note that the URL of the service to test must always be specified. See section Server Side: testing facet execution for further details.
If the service Testing Facet doesn't contain non-functional assertion, the tester can generate the non-functional assertion using the Pre-execution capability of the tool. This feature allows to calculate a mean value for QoS attributes over a number of executions (only response time is measured at moment). The Pre-execution can also update values contained in a downloaded facet file.
To start the pre-execution click on Pre-Execution button
To execute the pre-execution is necessary to set “Constraint Delta” to specify the adjusting value for the measures (in percentage), and the number of invocations to have an average value for the measures. At the end of the pre-execution ,a new XML test suite is available in the folder shown in the shell, containing the new QoS Assertions.
The specified facet can be addressed specifying a local path (if it's a local facet) or an URL.
At the end of the pre-execution, the button “View Preexecuted Test” allows to see the new pre-executed test suite file.
An useful feature of the regression testing tool is the possibility to perform capture/replay. During the service usage, I/O data are "captured" and stored within a monitoring database. When a service evolves, the tester can decide to re-test the service by "replaying" the inputs. Test case pass or fail can be determined doing an exact match between previously monitored outputs and actual outputs. This feature is available clicking on “Capture / Replay”.
The following window is shown:
The user must specify the starting date and the end date of the range:
Due to the range set, the possible URLs (invoked during the specified days) are shown and the user has to choose which URL he wants to consider:
The possible operations the user can re-execute can be selected:
The figure below shows a scenario of capture/replay results:
The information reported in the progressBar can be seen in more details in a table clicking on "View Table".
Each row of the table is related to an operation reporting the number of passed/failed executions. The XML log reporting the results can be shown clicking on "Expand"
The XML file shows the information in the following format
<operation name=”operationName”>
<invoke>
<SOAPIN>
soapIn message
</SOAPIN>
<EXPECTEDSOAPOUT>
soapOut monitored in a previous invocation
</EXPECTEDSOAPOUT>
<MEASUREDSOAPOUT>
soapOUT measured
<MEASUREDSOAPOUT>
<result> true or false depending on measuredSOAPOut is equal or not to expectedSOAPOUT
</result>
</invoke>
<invoke>
…….
</invoke>
</operation>
The regression tool, e.g. to perform statistical computations over periodical execution, accepts command line invocations to execute the facet against a web service identified by URL:
-exec URI FacetPath
(e.g. startserver.bat -exec serviceURI facetPath)
After the executions, you obtain a log file named log_CurrentDate.xml.
The current version works under Windows and Unixes operating systems.
The tool requires:
Simply create a folder onto local hard disk and unzip the .zip file in it.
Four new folders will be created:
Copy the whole content of work/webapps in your <Tomcat Installation folder>/webapps folder. Copying this folder two copy of Axis will be available, with some simple service (as will explained later). For each service monitoring is enabled (the monitoring class is included and the related file server-config.wsdd is modified accordingly).
from <MySQL Installation folder>/bin
type:
1) mysqladmin drop soapmonitoring -u root -p
2) mysqladmin create soapmonitoring -u root -p
3) mysql soapmonitoring -u root -p < soapmonitoring.sql
or simply copy
the whole content of work/DB/data in <MySQL Installation folder>/data
In ServerSide folder and in ClientSide folder you find a configuration file named config.txt.
In ServerSide, this file contains the following lines:
FacetFolder: XMLfile
ServerPort: 8080
The first line (Facet Folder) specifies the web folder path where the files facet.xml and test.xml will be copied. Note that the folder (for default XMLfile) must be a web folder (the default web folder is <tomcatInstallationDir>/webapps/XMLFile). You need to create that folder and create in it a directory named WEB-INF.
Note that when you copy the whole content of work/webapps in your <Tomcat Installation folder>/webapps folder, even the XMLfile web folder is copied.
The second line specifies the port used by the web server (Tomcat in this case; 8080 is the default value).
In ClientSide, the file config.txt contains the following lines
FacetFolder: XMLfile
ServerPort: 8080
IP: 127.0.0.1
User: root
Password:
Lines 3,4,and 5 represent database parameters containging capture/replay monitored data (supposing to have a database with the user root having no password ).
The third line contains the IP address of the DBMS; the forth and the fifth lines contain the user name and the password of the database user.
The work/webapps folder also contains some simple service useful to test the tool. These services are available in the ServerSide/webapps/axisV1 and axisV2 folders just copied. These folders contain two different versions of the services.
axisV1 and axisV2 contain two different verisons of the same services.
To test the correctness of the installation:
1. start Tomcat (Win32: launch <Tomcat Installation folder>/bin/tomcat5.exe from command prompt);
2. supposing that your local Tomcat application server runs on the default port 8080) type in your browser http://localhost:8080/axisV1/Calculator/Calculator.jws?wsdl . You should see the service wsdl in your browser;
3. You can also try:
http://localhost:8080/axisV1/Calculator/Calculator.jws?wsdl
http://localhost:8080/axisV2/Calculator/Calculator.jws?wsdl
http://localhost:8080/axisV1/services/FindLocation?wsdl
http://localhost:8080/axisV2/services/FindLocation?wsdl
http://localhost:8080/axisV1/SocialSecurityNumber/SocialSecurityNumber.jws?wsdl
http://localhost:8080/axisV2/SocialSecurityNumber/SocialSecurityNumber.jws?wsdl
assertTrue parameter must be a service operation invocation that returns a boolean.
All classes of the service to test must be in the same package.