The following testing terminology is used by UniSoft frequently and is included here for purposes of clarification:
A test description details an individual unit of functionality or behavior derived from statements contained in the API specification to be tested. Test descriptions are produced in native language and describe the state required prior to the aspect of the API call being tested, the parameters supplied to the call and the expected return value, error condition and/or events generated. (Reference: IEEE Std 2003).
A test strategy description details, in a pseudo-code style, the way in which a thorough test of the corresponding test description can be implemented. (Reference: IEEE Std 2003 for description of "thorough testing")
The test case software implements an automated test for the test strategy description associated with each test description. For the DVB-MHP APIs, the test case software will be implemented in Java code and each test case will be provided in a form that can be broadcast to a conforming receiver. The execution of each test case on the conforming receiver will result in a single result for each test description indicating the receiver's correct or incorrect handling of the described unit of functionality.
UniSoft uses a standard style for test suite development where possible, and particularly for application programmer interfaces (API) tests which use an assertion based testing technique.UniSoft's approach, which is designed to allow customer review and feedback at each of the critical stages in the development cycle, consists of the following stages:
As part of the contract discussions a Statement of Work (SOW) is drawn up by UniSoft and agreed with the customer. The SOW defines the work to be undertaken and the quality metrics and acceptance criteria which are applicable to the project.
All project deliverables and dependencies with their associated time scales are included in the SOW.
During this stage the major structural issues relating to the test suite are discussed and documented. This stage is a pre-cursor to any development and enables the customer to understand and agree the basic structure of the test suite. It is expected that the major aspects of this phase will be to decide on the functionality required for the test suite driver and to determine the limitations that are appropriate for the production of individual test cases. It is also expected that some discussion of specific test case issues will be necessary where there are clear choices to be made about specific testing strategies.
The output from this phase will be a specification of the basic test suite structure and driver software and a series of documented strategies for the more complex test cases. The test plan will also refine the time scales associated with the remainder of the project life cycle.
For interface testing, the assertion generation stage will produce, for each interface to be tested, a set of unitary statements describing the aspect of the interface to be tested by a particular test case. These statements will be generated from the API specification and will provide traceability between the test cases and the behavior described in the specification.
The output from this phase will be a set of documents, which will detail the test cases to be written for the API. The documents will also detail any statements in the specification which are considered untestable without recourse to the internal behavior of the implementation and any statements in the API documentation which are open to more than one interpretation by an application programmer. The documents will be subject to review by the customer to ensure that both accuracy and coverage are provided by the assertions.
For any assertions generated in the previous stage, a test case strategy will be developed which will be used in coding the test case. The test case strategies will be reviewed for commonality. Areas, in which common routines should be developed during the Test Code Generation phase, will be identified. The test case strategies will be subjected to peer review within the development team at UniSoft and, where particular difficulties are identified, assistance may be sought from the customer to identify the most appropriate strategy for particular tests.
From the detailed strategies and specifications developed in the previous activities, test code will be developed for the test driver, test case libraries and the test cases. The code will be checked to ensure that only the agreed interfaces are used and that the code will be portable to different processors.
The test code produced during the development phase will be subjected to peer review independently of any testing of the code. This helps to ensure that the code is accurate even in areas which will not be exercised during the testing process.
The coding methods used by UniSoft include the use of techniques to ensure that before a test is considered to have passed the expected path through the code has been executed. This technique is also used to provide trace information relating to execution of a particular test.
UniSoft will ensure that the test code is executed on different platforms provided by the customer and will analyze the results to ensure that faults in the test code are identified and corrected.
Where practical, UniSoft will modify the test code to simulate failures and will ensure that these are correctly reported in the execution of the tests.
The test code will be both tested as individual units and in conjunction with the installation and execution procedures to be used with the final test suite.
At reasonable intervals during the development process UniSoft will deliver a snapshot of the tests that have been successfully completed at that point in time. The snapshot will not include full test suite documentation and reporting capabilities, but will include Release Notes sufficient to allow the customer to execute the tests and to analyze the results. This technique allows client's early access to the technology and to provide feedback to UniSoft regarding any problems that are identified.
UniSoft operates an Alpha, Beta and General Release cycle with each of these releases containing all of the software and documentation necessary to execute the test suites. This cycle provides ample opportunity for the customer to review and comment on the product prior to final acceptance and General Release.
Following General Release, the customer may enter into a support and maintenance contract with UniSoft for the developed test suite.