|
|
KitCAT is a JUnit [7] based testing framework to facilitate functional testing of converged telecom applications. A KitCAT test case uses the KitCAT API (Java) to interact with the system under test (SUT) using SIP/RTP protocols. In conjunction with other web testing APIs like that offered by HtmlUnit [1], users can write functional tests for converged applications. A single KitCAT test case can create, control and coordinate multiple SIP/RTP as well as browser endpoints involved in functional testing.
Figure 1 shows how KitCAT is typically used for functional testing of converged applications that have both web as well as VoIP functions. KitCAT uses the JAIN SIP [6] compliant NIST SIP stack [2] underneath for SIP interaction with the system under test (SUT). A test agent in KitCAT is used to simulate a SIP/RTP end user of the SUT and thus responsible for sending and receiving SIP/RTP messages. A test case creates agents which are then used to communicate with the SUT. An agent also maintains a state which is advanced appropriately as SIP messages are sent and received by that agent. A test case can control and coordinate execution of agents as well as assert the execution states of agents based on messages sent/received by them. An agent can act as either a caller initiating SIP messages to the SUT or a callee receiving SIP messages from the SUT. A KitCAT test case hosts both caller and callee agents within a single process thus facilitating the coordination among the callers and callees.
KitCAT offers the following main features for testing converged applications:
The prerequisites for using KitCAT are -
The command line instructions described in this manual are intended for Unix environments, but are also applicable in a Windows environment. Appropriate batch command files are provided where necessary. Note: For the batch files to work properly in Windows, you should enable the delayed environment variable expansion - add /V:ON parameter to your command prompt window executable (cmd.exe /V:ON)
The following step-by-step instructions can be used to guide you through the installation process.
The above test sets up a simple test (see TwoWay.java(method=simpletest under examples directory) in the KitCAT distribution) call between two agents, exchanges media and hangs up. The test case also makes appropriate assertions to ensure that the call was successful. The output log messages are those from the test case, KitCAT framework as well the NIST SIP stack. If the installation is successful, then you should see the test run terminate as shown above with the message “Install test succeeded”. A batch file has been provided to run the above test on a Windows platform. You should see similar log output on Windows as well.
KitCAT is a Java library and hence any tools that are used for Java based developement and testing can be used to build and run KitCAT test cases. The jar files in the KitCAT distribution (under lib directory) should be in the classpath for building and running KitCAT test cases. Moreover, the log4j settings should be available to KitCAT during runtime. This can be done by including the path to log4j.properties in the classpath.
To get started, we have provided simple command line tools with KitCAT to generate a skeleton test case and associated files for building and running the test case. Before following these instructions, an environment variable KITCAT_HOME should be set to the KitCAT installation directory. For example,
Switch to a directory where the test case and associated files need to be generated. Run the test generation utility. A sample interaction is shown below.
The following directory structure is created from the above run.
The above utility generates a simple test case skeleton and the associated files to build and run the test case. Now we can work with this test project either from a command line environment or eclipse environment.
To build and run the test project from a command-line, the generated ant based build scripts can be used. Switch to the newly generated test directory and do the following:
The generated test case uses a test.properties file to get test parameters such as the location of the SIP application server, HTTP app server and the listen port for the SIP stack. You should fill in the parameters appropriately. To begin with you must fill in the SipStackListenPort parameter. If you forget this step you’ll get an exception when running the generated test case.
Now that we have built the test case, we illustrate two ways to run the test case from the command line. Both these methods use a JUnit test runner to run the test. The first method provides the tester with a shell script that uses a command line based JUnit runner directly and the second method uses an ant script which uses a JUnit runner provided by ant (version 1.7.x or higher). The ant based JUnit runner outputs results in a web page which is easier to view and navigate in a unit testing scenario. The non-ant based JUnit runner is useful in scenarios where a KitCAT test case is embedded as part of a larger system test scenario which may involve a number of other non-KitCAT tests. Moreover, this run method can be used with ant versions prior to 1.7.x, that don’t have support for JUnit ant tasks.
First, set the SipStackListenPort property in etc/test.properties to a valid port number (eg. 12345). From the command line, the test case can be run as follows:
The output from the run includes the output from KitCAT framework as well as the SIP messages log output from the NIST SIP stack.
We have also provided an ant script for running a test case using the JUnit task in ant. First, set the SipStackListenPort property in etc/test.properties to a valid port number (eg. 12345).
A report directory is generated under which the test results can be found in html format. For example, the above run generated the web page in Figure 2.
 
|
To build and run the test project from an Eclipse environment, the generated Eclipse related files (.classpath and .project) can be used. The following instructions enable you to get started with KitCAT test cases quickly in an Eclipse environment.
A KitCAT test case is a JUnit test case that uses the KitCAT library to simulate SIP/RTP endpoints. Each test agent in KitCAT contains a SIP and RTP endpoint. The behavior of a test agent is governed by a state machine within KitCAT. A KitCAT test case hosts multiple such agents whose execution is controlled and coordinated by the test case. A test case has three different types of statements that either query or affect the state of test agents:
|
|
User Commands: A user command causes a test agent to send a SIP message or schedule RTP packets for transmission. Table 1 shows some of the user commands currently available in KitCAT.
processSIP primitive: A KitCAT test case presents the test writer with a sequential execution model to execute a test case. There are no call-back functions to handle the asynchronously arriving SIP messages. Instead, the KitCAT runtime framework handles these asynchronous SIP messages in background threads and buffers them in internal KitCAT buffers. It then provides a synchronous (blocking) interface for the test case to process these messages using the processSIP API. This primitive causes all buffered messages for all agents to be processed by the agents until a specified time expiry occurs. If all messages are processed by all agents, then this primitive causes the test case execution to block until either a newly arrived message is buffered for processing or a timeout occurs whichever is earlier. Processing of messages involve execution of the agent state machines causing their states to possibly change. Sometimes, this could cause agents to send further SIP messages in response to the processed message. For example, by default an agent sends an ACK in response to a 200 OK to an INVITE. Whenever there is a need to introduce a delay in the test case, it is highly recommended to use the processSIP primitive instead of Thread.sleep(). This ensures that SIP messages are still being processed during the delay. processSIP is the only primitive where arriving SIP messages are processed.
Assertions: Assertions in KitCAT query agents’ states to check whether the test case execution caused the expected state changes in agents. An assertion statement itself does not cause any side-effects on an agent. During our implementation, we tried providing specific assertion primitives for carrying out specific types of assertions. For example, there could assertions based on states of agents (assertState), relating the states of multiple agents (assertAgentsConnected), about messages exchanged by agents (assertMsgRecvdByAgent), and so on. As you can see, the problem with this approach is that we end up with many assertion primitives and a new type of assertion would require a change to the KitCAT framework. We soon realized the need for a more flexible and reusable assertion mechanism that will allow us to create different types of assertions easily. Assertions in KitCAT are designed to use the assertThat primitive that is part of JUnit 4.4. For more information about the assertThat primitive, see [5, 4, 7].
The logic of the assertThat primitive is roughly as follows -
It takes in a subject which is of type java.lang.Object and a matcher object of type org.hamcrest.Matcher, which checks for specific properties on the subject. This simple primitive allows us to create a variety of matchers for specific purposes. For example, the following assertion checks that two agents are connected by making sure that they are in the appropriate states and also that they have exchanged SDPs.
|
|
Here, connectedTo is a static factory method that creates a matcher to check whether the two agents are connected. This assertion mechanism helps with the readability of a test case, and also provides a facility for the test writer to write new matchers that are not already available as part of KitCAT. KitCAT provides convenient-to-use matcher factory methods (see Table 2) to create appropriate matcher objects.
A simple test case is shown below. This example along with other example test cases are available in the examples directory as part of the KitCAT distribution (see examples/src/TwoWay.java). These examples illustrate usage of the various primitives and assertion conditions possible in KitCAT. For detailed discussion about the execution model and semantics of KitCAT primitives see [9].
This test case creates two SIP endpoints (test agents) and tests whether a call can be established between the two agents. The test case also checks media interaction between the two agents.
It must be noted that sending of SIP messages occur as soon as a user command (e.g., . call, end, info) is invoked on a test agent. For example, when the statement alice.call(bob) is executed, it immediately results in a SIP INVITE being sent to Bob without processSIP being called. But, agent Bob processes this INVITE only when a processSIP primitive is called. Assertion statements don’t have any side-effects and hence do not affect the state of the test agents.
In the above test case, there is a user-defined method to check media connectivity between two agents.
Such methods are reusable across multiple test cases. The extract below illustrates use of some of the media related primitives in KitCAT. The playAudio primitive queues RTP packets for transmission by a background KitCAT thread. KitCAT uses G.711 μ-law codec with 20ms for RTP packetization. Each KitCAT agent stores received RTP packets to a disk file (testmethodname_agentname.audio.raw) under the out directory - e.g., simpletest_Alice.audio.raw contains the audio stream received by Alice during this test run. The clearMediaBuffer primitive is used to clear internal agent data structures that keep track of RTP packets received.
Currently KitCAT does not process audio files with data format headers in the media file (e.g., .wav file). Any media file should be converted to a raw format (header-less data only) using a utility like sox [3]. An example interaction is shown below.
In Section 5.1, we discussed a test case, which dealt with SIP/RTP test endpoints. We now illustrate how we can integrate web testing in a KitCAT test case. Even though we show a very simple example, the concepts can be and have been applied to write test cases to do functional testing of non-trivial converged applications.
Although KitCAT is not tied to any web testing frameworks, we have been successful using HtmlUnit [1] for functional testing of web applications and highly recommend using it. An easy way to get started would be to download ( http://htmlunit.sourceforge.net/) the HtmlUnit related jar files on to the lib directory in the KitCAT installation. An alternative way would be to create a User Library in Eclipse for HtmlUnit (using the procedure described for creating KitCAT User Library in Section 3.3) and then including this user library in the Java build path for this project.
The following example shows a test case for unit testing a simple call forwarding application. This call forwarding application has an extremely simplified web interface that has been developed for purposes of unit testing functionality. In this test, we set up call forwarding for a subscriber through the web interface. The call is set to be forwarded to another agent in the same test case. The test case then makes sure that a call to the subscriber is indeed forwarded to the new location. As part of the call forwarding application, there is a simple jsp page that is used to setup the forwarding parameters. The setForwardingUser method uses HtmlUnit to invoke this jsp page ( http://hudson:28501/ucfTest/set.jsp?fwdUser=forwardedTo&fwdHost=dart&fwdPort=12345 ) provided by the application. This directs the application to forward all calls to the subscriber to the SIP address sip:forwardedTo@dart:12345 which is the address of the test agent forwardedTo. The test case now initiates a call to the subscriber and asserts that the agent designated to receive the forwarded calls gets it instead of the subscriber.
More sophisticated test cases can be achieved using the concepts shown above. You can obtain the code for a ECharts For SIP Servlet(E4SS) [8] based implementation of the call forwarding example described here along with this test case from the E4SS development kit that can be downloaded from http://echarts.org.
KitCAT uses log4j for logging messages. There are 4 loggers that test writers should be aware of -
| Log4j Logger name | Used by | Default Level |
| KitCAT.Tester | test cases as well KitCAT | INFO |
| KitCAT.SipStack | SIP stack | INFO |
| KitCAT.E4JS | KitCAT subsystem | WARN |
| KitCAT.ECharts | KitCAT subsystem | WARN |
The first logger (KitCAT.Tester) is obtained by invoking the getLogger primitive within a test case (see the generated test case in 3.1) and can be used by the test case to log messages. The stack (KitCAT.SipStack) logger is turned on by default (log level is INFO) so that the SIP messages that are sent or received are logged by the stack. To turn stack logging off, you can change the level of KitCAT.SipStack logger to WARN. Typically the subsystem loggers are turned on (changing their level to DEBUG) only for KitCAT framework debugging purposes.
Running a KitCAT test case produces atleast two directories - out and logs (Using ant based test run produces a report directory also). The out directory contains the files containing received RTP streams for each agent. Under the logs directory, typically there is a run.log file. The name of this file and the logs directory can be changed in the log4j.properties file. With the default log4j settings, the run.log file contains logs from the test case as well as helpful messages from the KitCAT framework. This file also contains the logs produced by the SIP stack when messages are sent and received.
[1] HtmlUnit. http://htmlunit.sourceforge.net/.
[2] NIST JAIN SIP Stack. https://jain-sip.dev.java.net/.
[3] SoX - Sound eXchange. http://sox.sourceforge.net/.
[4] Hamcrest. Available from: http://code.google.com/p/hamcrest.
[5] Hamcrest Tutorial. Available from: http://code.google.com/p/hamcrest/wiki/Tutorial/.
[6] JAIN(tm) SIP Specification. Java Community Process, 2003. Available from: http://jcp.org/aboutJava/communityprocess/final/jsr032/.
[7] JUnit. Available from: http://www.junit.org/.
[8] T. M. Smith and G. W. Bond. Echarts for sip servlets: a state-machine programming environment for voip applications. In IPTComm ’07: Proceedings of the 1st international conference on Principles, systems and applications of IP telecommunications, pages 89–98, New York, NY, USA, 2007. ACM. Available from http://echarts.org.
[9] Venkita Subramonian. KitCAT - A Framework for Converged Application Testing. Technical Report TD-7DHQYT, AT&T Labs Research, 2007. Available from http://echarts.org.
