ISO/IEC DIS 14598-5 Information Technology Evaluation of software product Part 5: Process for evaluators

This international standard provides guidance for the practical implementation of software product evaluation when several parties need to understand, accept and trust evaluation results.

In particular, it may be used to apply the concepts described in ISO/IEC 9126.

It defines and describes the activities needed to analyse evaluation requirements, to specify, design and perform evaluation actions and to conclude the evaluation of any kind of software product.

This standard can be used by:

• testing laboratory evaluators, when providing software product evaluation services
• software suppliers, when planning evaluation of their products, including evaluation to be carried out by independent testing services
• software buyers, when requesting evaluation information from a supplier or testing service
• software users, when evaluating products, including the use of test reports provided by testing laboratories
• certification bodies, in establishing new schemes for software products.

The evaluation process is described in ISO 14598-5.2 as a step-wise procedure allowing the definition of evaluation requirements in terms of quality characteristics as stated in ISO 9126.

The evaluation takes into account various documents that can be considered as a part of the software product (e.g. software requirements and design, test and validation reports, etc.) It is recommended that the evaluator uses a library of evaluation modules that define evaluation methods. These modules could be standardised, although no provision for that is proposed in the standard.

The evaluation leads to the production by the evaluator of an evaluation report.

Evaluation process characteristics

The standard defines the main characteristics of the evaluation process (repeatability, reproducibility, impartiality, objectivity) and describes the specific steps and relationships characterising its activities:

• analysis of evaluation requirements
  analysis of the requester's requirements in order to identify the actual requirements for the evaluation;
• evaluation specification
  produced on the basis of the evaluation requirements and the description of the software product provided by the requester;
• evaluation design and definition of the evaluation plan
  on the basis of the evaluation specification, the components of the software product to be evaluated and the evaluation methods proposed by the evaluator;
• execution of the evaluation plan
  which consist of inspecting, modelling, measuring, and testing components of the product according to the evaluation plan; these actions can be performed using software tools (usually provided by the evaluator); the actions performed by the evaluator are recorded and the results obtained put in the draft evaluation report;
• evaluation conclusion
  which consist of the delivery of the evaluation report and the disposal by the evaluator of the product components evaluated.

Evaluation process inputs

The requester provides the following input to the evaluation process:

• an initial version of the evaluation requirements
• a description of the software product to be evaluated which identifies the components of the product which are submitted to the evaluation
• the product

The evaluator provides the following input to the evaluation process:

• pre-defined evaluation specifications
• evaluation methods
• evaluation tools.

Evaluation process outputs

During the evaluation process, the following information is produced:

• evaluation requirements
  which describe the objectives of the evaluation; in particular, the criticality required for the product is described;
• evaluation specification
  which defines all the analysis and measurement to be performed and all components of the product that will be analysed and measured;
• evaluation plan
  that describes operational procedures needed to implement the evaluation specification; in particular all the methods and tools to be used in the evaluation are described;
• evaluation records
  that consist of the evaluation plan and a detailed account of actions performed by the evaluator while executing the evaluation plan; these records are kept by the evaluator;
• evaluation report
  which contains evaluation requirements, the evaluation specification, results from measurements and analysis performed and any other information necessary to be able to repeat or reproduce the evaluation; the evaluation report is delivered to the requester.

The Evaluation Process

	Services
	Overview