Validation should answer stakeholders' questions
What does the word Validation mean? It is certainly the ‘buzz-word’ of the moment and is often being used as a replacement for the words demonstration and evaluation. But are we able to simply change the word that is used to describe R&D Simulation work and consider that sufficient or is there something more that R&D should be doing in order to use the term ‘validation’ reasonably?
A definition from the Oxford English dictionary
validate
verb (-ting) make valid, ratify. validation noun.
authenticate, authorize, certify, endorse, legalize, legitimize, ratify
If we check on the definition of each of these qualifying words they all tend to have a common thread of ‘approval’ in some sort of an official sense. Thus the word ‘validation’ has strong formal usage.
Remaining in the context of R&D and not referring to any other service provision or regulatory functions are we in ATM R&D in a position to apply the word ‘Validated’ to our work and expect everyone to accept this? There is no official validation authority in ATM R&D. But can we assume some form of unofficial authority as a result of proven excellence? If we look at the success rate of concepts implemented from ATM R&D into real usage the evidence does not seem to provide R&D with this unofficial yet peer-group recognised authority. It would appear that change is necessary to gain this authority.
To foster change it is necessary to have;
Validation as a process.
Validation can be viewed as a process of providing key stakeholders (‘decision makers’) with various types of information that together help them to make appropriate decisions about eventual implementation of the proposed ATM concept. Thus, to be effective, those involved in validation should be aware of the information needs and decision-making criteria of these key decision makers.
The type of ‘information’ that is required to support the ‘validation’ process depends on the type of stakeholder. System operators (controllers and pilots) will require contextual and often dynamic evidence to support understanding of potential behaviours. Thus real-time simulations and various forms of ‘mock-up’ or prototype are useful especially if they promote exploration of expected behaviours. Investors will require business cases and regulators will require safety cases.
It is also important to understand what validation will not do. It will not test the product or procedure against requirements nor will it be a strict end test with pass/fail criteria. The process of ‘verifying’ that requirements have been satisfied is sometimes referred to in engineering circles as ‘validation’. In the V model validation can be seen as a layer above the technical and operational requirements (see diag.)
What validation sets out to do is to allow ATM stakeholders (principally those who make or influence decisions on implementing new systems) to judge the potential value of a new procedure or supporting process. Validation is thus a process of supplying information of expected performance and capabilities to those stakeholders who influence the progression of a prototype into a product or procedure that is implemented into the live ATM system.
Validation – the need to improve rigour
Historically speaking validation of changes to ATM systems has been a subjective task carried out by controllers who determine whether the proposed change is acceptable. As ATM Systems become more complex it becomes more difficult for controllers to understand all the potential behaviours (unwanted as well as wanted) of those systems. Without a more analytical approach to testing, the process of validation becomes less complete and thus less reliable. As an example, safety aspects become more difficult to validate in these new, more complex ATM systems because controllers are not able to predict the new classes of error that could occur. Thus we (in ATM R&D) need to identify suitable test and evaluation techniques that provide more complete information about how risk can be identified and managed in these new semi-automated systems. Stakeholders can be supported in their decision making by using scenarios based around heavy traffic, complex traffic, extreme weather, key failures and potential error modes.
A final point to encapsulate the main aspect of validation. Above all the validation process should be attempting to show if a proposed change to the ATM system is useful and usable. But the final decisions will be taken by stakeholders not by ATM R&D. It is stakeholders who will pose the questions about proposed changes and thus they will determine if those changes are valid or not.
For further information or discussion of this brief overview contact
Nigel Makins
Links
MAEVA web-site including the Validation Guidelines Handbook
Action Plan 5 of EUROCONTROL/FAA R&D Committee - Joint Validation Strategy July 2002
The FAA plan for upgrading their system - useful links.
Some airline views about ATM - understanding stakeholders
A controller’s view of human factors in ATM