The Testing Matrix
A variety of different forms or levels of testing must be carried out before going live. This will involve technical staff and end users at various times.
System testing is invariably an iterative procedure as more often than not expected outcomes are not fully achieved, and some re-configuration is required before repeating the test procedures until ultimately the desired results are gained. However it is important that time and resources are not wasted on testing aspects of the system that are not yet ready, and certainly in larger-scale systems it is possible that different levels and volumes of testing may be carried out simultaneously across different parts of the system. In this situation, it is important to ensure that outcomes of one testing type are not affecting the status of other tests. There may be a need to separate testing instances to allow isolated tests of different levels on the same product.
Another important point to bear in mind is that you may need to test areas/modules of the system that are not going to be used for their functionality but which have configuration/set-up implications for the functional areas you will be using. This is particularly common in systems that are components of an enterprise-wide solution.
System testing should be performed by testers who are trained to plan, execute, and report on application and system code. They should be aware of scenarios that might not occur to the casual end user, like testing for null, negative, and format inconsistent values. A tester should be able to repeat the steps that caused an error. We have adapted the template included in the Project Management infoKit to give you a starting point for defining the Roles and Responsibilities for staff involved in system testing.
There are many types of testing and a variety of labels for each element. Testing models cover 'vertical' and 'horizontal' aspects to create a testing 'matrix' of approaches that can be applied as appropriate. Perhaps the easiest way to consider these is to divide testing into 'phases' and 'types'.
Testing phases cover:
- Development Testing (pre-Implementation)
- Preliminary Testing (involving only the core team)
- User Acceptance Testing (involving end users)
- Final Testing
Types of testing within these phases include:
- Unit/Module Testing
- System/Functionality Testing
- Interface/Integration Testing
- Volume/Performance/Capacity Testing
Different combinations of these types and phases can be represented using a matrix-like approach, an example of which is given below. Each of the types can have a number of sub-categories which may have a different slant or be more important depending on the phase e.g. An iteration of Unit/Module testing in the Development phase may be wholly concerned with testing the success of data migration scripts, while Interface/Integration testing in the Development phase will more likely have a stronger 'technical' slant than the end-user operation slant it might get in the Preliminary and Acceptance phases.
All types and phases are likely to be iterative and also bear in mind that the outcomes of a later phase may re-trigger components of an earlier phase - for example unsatisfactory outcomes of the Acceptance Phase may lead to development work that is required to go through all of the first three phases again before being re-submitted to the Acceptance Phase testers. It is crucial to allow for these iterations in your planning.
We have attempted here to cross-reference the more well-used labels for testing types and phases but these can be very interchangeable and similar types of testing are often referred to by different labels depending on the methodology used. An added complication is that tests can be referred in relation to the procedure, approach or model being used - e.g. 'exception testing' and 'black box testing', more often within the development and preliminary phases.
