Open Systems Dependability

9.1 OVERVIEW

This chapter begins the discussion of a framework for evaluating

Open Systems Dependability (OSD), with the emphasis on the

evaluation of the whole lifecycle of a system. This is in addition to

the traditional one on the system itself using dependability metrics

(cf., ISO/IEC 9126-1:2001 and ISO/IEC 25000:2005). The evaluation

determines not only how much but also how well dependability is

achieved by examining the description of what the lifecycle of the

system accomplishes as well as how and why. The description to be

examined is an assurance case; in particular, it is a D-Case when the

lifecycle is the DEOS process.

The proposed evaluation framework divides its focus between

the substance of the lifecycle processes of a system and the quality

of what they produce; the latter includes not only the system but

also assurance cases at their various stages of existence. This split is

similar to that between ‘process arguments’ and ‘product arguments’

commonly found in assurance cases. There, a process argument may

show that, e.g., processes generate trustworthy evidence, while a

product argument may directly show from the evidence that, e.g.,

residual risks for the product are acceptably low. The two aspects

have further connections and importance in the evaluation of Open

Systems Dependability: development and operation processes are

a part of a ‘system (product)’ that is required to achieve OSD as a

whole; D-Cases (assurance cases) produced in the processes are

TOWARD AN OPEN SYSTEMS DEPENDABILITY

TOWARD AN OPEN SYSTEMS DEPENDABILITY

EVALUATION FRAMEWORK

EVALUATION FRAMEWORK

9

9

Toward an Open Systems Dependability Evaluation Framework

Toward an Open Systems Dependability Evaluation Framework 125

incorporated in the systems and controls parts of their operation

processes.

Evaluation of open aspects of a system should be an integral part

of that of all other aspects. The framework delineates certain focuses

of OSD evaluation so that it combines with other types of evaluation

without undue duplication of work. OSD evaluations may tell what

traditional benchmarking is justifi ably suffi cient; risk evaluations

at a higher level may tell what part of systems can be taken to be

closed. The framework is to be refi ned with such interfacing issues

with existing types of evaluations, but this chapter concentrate on

key aspects of OSD evaluation itself.

Section 9.2 organizes the evaluation of the lifecycle processes of

a system into that of four ‘process views’:

1. consensus building,

2. accountability achievement,

3. failure response,

4. change accommodation.

Figure 3-1 of the DEOS process depicts those four items variously

as key phases or cycles: it shows what the main concerns of activities

are when the activities are organized in the sequences and cycles.

However, those concerns cut across processes employed across the

lifecycle; i.e., they are addressed neither by a single process nor at

a single phase. Thus, a process view for each concern is required

to be defi ned, which gathers all relevant activities in the lifecycle

of a system, cf., (ISO/IEC 15288:2008, ISO/IEC 12207:2008). The

evaluation is on how well these process views achieve their desired

outcomes.

Section 9.3 specifi es four aspects of assurance case documents

as the focuses of evaluation:

1. internal consistency,

2. inter-system consistency,

3. soundness in the real world,

4. adequacy of degree confi dence.

Here, assurance cases are taken to contain all relevant descriptions

on artefacts produced throughout the lifecycle of a system, including

the ones on the lifecycle itself. The evaluation here determines the