The Investigation Process Research Resource Site
A Pro Bono site with hundreds of resources for Investigation Investigators
Home Site Guide FAQs Tutorials Old News Contribute Forums

. . . .. . . . . . . . last updated 4/28/07
INVESTIGATING
INVESTIGATIONS

to advance the
State-of-the-Art of
investigations, through
investigation process
research.



Research Resources:

Search site for::


Launched Aug 26 1996.

 

Introduction by Webmaster:

This report of research describes results of an investigation into New Zealand investigations involving "pilot errors" and offers application of a model to help investigators with analyses in accident investigations. Although it is not the main focus of this site, the paper also describes application of the taxonomy to historical data, with some interesting results.

The report is posted here to stimulate discussion of the methodology and findings and how they relate to the improvement of accident investigation processes.

This document was contributed by Dmitri Zotov, and was formatted by your webmaster for publication here.

COMMENTS ABOUT THE PAPER, for publication at this site are invited. See guidance for submitting comments.

NOTE:

This paper offers a model for investigation derived from reports of accidents. Its focus is investigation, rather than accident causation and thus it is offered here as an example of the kind of report analysis that qualifies as investigation process research. for the purposes of this site.


How To Prang Your Aircraft In Six Different Ways

(Or, for the academically inclined, Cognitive Failure Analysis of Pilot Error)

by Dmitri Zotov
Massey University

TABLE OF CONTENTS


Introduction

The question of why human error accidents occur has been addressed by Reason (1990, 1991) with his concept of systemic failures. Rasmussen (1980, 1982) looked at how such errors occurred, with his classification of errors as deficient skill, misapplication of rules, and lack of knowledge of the operating system. But as Rasmussen pointed out, this is a different concept from what the error was. O'Hare, Wiggins, Batt and Morrison (1994) took the view that the nature of the errors made by pilots was still poorly understood; without such understanding we would be limited in the actions we could take to avert human error accidents.

Until recently, most systems of classifying errors listed faulty performance of the task: ‘Failed to maintain airspeed’, or ‘Did not avoid obstacle’, or ‘Misjudged altitude’. These classification schemes were very cumbersome, often comprising hundreds of categories. Indeed, they were so complex that coding errors were commonplace, and so the validity of the database as a whole was open to question. Besides, these taxonomies were really only lists of what happened. They were of little value in trying to decide what remedial action should be taken.

Rasmussen argued that what was needed was not a classification by tasks, but by the task as performed by the human. He devised such a scheme after studying errors made by operators of nuclear power plants. On first sight, you might think that this had little to do with aircraft accidents, but this work was adapted by O'Hare et al., who devised a six-element taxonomy to classify pilot errors (Figure 1). This taxonomy has been extensively tested (Wiegmann & Shappell, 1997; Zotov, 1997) and Zotov found that every reported accident in the New Zealand database between 1965 and 1990, which had been subject to official investigation, was able to be classified by this system.

The process has been used both to determine the distribution of types of errors in large numbers of accidents, and to analyse the pilot's actions in individual accidents as a guide to 'solving' the accident.

The analytical process

To classify the pilot errors which led to an accident, we start by examining whether the necessary information was available to the pilot, in order to take corrective action. If not, there was nothing he could have done. However, information may have been available, but he may not have sought or detected it. For example, an airliner with a ‘glass cockpit’ display was being flown by a crew who were used to conventional instruments. A partial engine failure led to high vibration. The engine instruments correctly displayed which engine was vibrating, but the inconspicuous display was not noticed by the crew, who shut down the wrong engine. Such an error is coded as an Information Error.

Next, the crew may receive information, but their appreciation of the situation may be at variance with reality. A Boeing 737 had a major fire on take-off as a result of uncontained turbine disk failure. The passengers, and everyone at the airport, saw the flame streaming behind the aircraft like the tail of a comet. The pilot’s subsequent RT call

“We appear to have a fire in No. 1 engine”

has to have been the understatement of the year. Such faulty appreciation is classified as a Diagnosis Error.

Having correctly diagnosed the state of play, the pilot may have to choose between competing goals, for example, to divert to an alternate, or to hold until the weather improves. A frequent Goal Error in General Aviation is deciding that it will be possible to proceed towards the destination in the face of deteriorating weather.

FIGURE 1 Taxonomic algorithm

There may be a number of ways to achieve a goal. These are termed ‘Strategies’. For example, to achieve the goal of ‘Diverting to an Alternate’, the pilot might decide to climb only to minimum safe altitude, perhaps minimising traffic delays, or to climb until the descent path is intercepted, often the most economical way. The choice of strategy may be less than optimum.

Procedures are groups of tasks by means of which strategies are achieved. Thus, the strategy of diverting to an alternate may commence with an overshoot: raise the nose, apply climb power, raise the undercarriage, accelerate to flap raising speed and raise the flaps, and so on. Performing the wrong procedure, or the correct procedure wrongly, is classified as a Procedural Error.

Of course, any action in a procedure can be performed incorrectly. An example of such an Action Error would be raising the undercarriage instead of the flaps, at the end of the landing roll. [1]



Comparison with GEMS

This form of analysis has a parallel in Reason’s Generic Error-modelling System of Slips, Lapses, Mistakes and Violations. While there may be academic debate over the relative merits of these systems, neither is ‘right’ or ‘wrong’. It is a question of which approach is helpful in a particular application. Cognitive failure analysis has a number of virtues for accident investigation. In the overall picture, it can provide guidance as to where resources can best be directed to reduce the accident rate. While an accident is being investigated, it can enable the investigator to understand what the pilot may have been trying to achieve – a first step to seeing what corrective action might be possible for the future.

Error distributions: major and minor accidents

We can analyse what we might term the ‘triggering’ error: the first departure from accepted and prudent practice. Of course, there are likely to be other errors after the initial event, as the pressure builds up, but it is the triggering error which gives the pilot the opportunity to make the subsequent errors.

Take as an example an accident to a motor-glider which suffered an engine failure at 400 feet in the circuit after take-off. Prima facie, an engine failure at 400 feet in the circuit should not result in an accident. However, the pilot flew round the circuit with the engine extended, and at an excessive speed, which resulted in rapid loss of height. In pulling up over a tree on the approach, the pilot stalled the aircraft, and it dived into a swamp short of the airfield. Stalling the aircraft was an Action Error, but was preceded by the Procedural Errors of not retracting the engine, and flying with excessive speed. However, there was more to it than that. The engine stopped because the fuel was exhausted. The pilot was aware that the contents shown by the fuel gauge indicated a lower consumption during the previous flight than he had expected. It would have been possible (though not easy) to have dipped the tank before continuing the flight. Had the pilot been aware of the fuel state he could have topped up the fuel, so the engine would not have stopped from fuel exhaustion, and the pilot would not then have been put in the position which led to the subsequent errors. There was therefore an Information Error, in that the pilot did not seek available information which would have averted the accident.

O'Hare et al (1994) analysed the errors in quite a number of accident reports. It was hoped that the use of official reports would avoid the limitations of self-reporting, such as self-presentation, which are likely to affect other sources such as confidential reporting systems.

The results of their study of accidents involving powered, fixed-wing aircraft are shown in Fig 2. The bar graphs show the distribution of the triggering errors, in the order in which those errors appear in the algorithm. (Subsequent errors were not analysed statistically, because of difficulty with dependencies).

Figure 2 Comparison of Error

Figure 2.Comparison of errors - major and minor accidents

One feature of these results is a marked difference in the types of errors which have triggered major or minor accidents. It appears that major accidents usually stem from faulty decisions, while faulty implementation tends to result in minor accidents.

This is a little surprising. Any investigator has seen accidents in which the pilot's survival appears miraculous. For example, in one accident an agricultural aircraft plunged 300 feet vertically, but struck terrain which was almost as steep, and covered with trees sufficiently large as to progressively retard the fall but not so large as to cause catastrophic disruption.

On the other hand, there have been accidents where death has resulted from such trivial reasons as a resilient seat cushion, which both amplified the impact forces and permitted the pilot's harness to slacken. It has therefore become customary to regard major accidents, minor accidents and incidents as one and the same thing, the outcome being due to chance (Billings & Reynard, 1981).

There are many more minor than major accidents (the ratio is of the order of 10: 1), and still more incidents, so there has been continuing pressure to investigate minor accidents and incidents in detail, and so learn the lessons whereby major accidents could be averted. This has recently culminated in an ICAO Requirement to this effect (ICAO 1994).

The results found by O'Hare et al (1994) were therefore not only counter-intuitive, but could have important implications for policy on resource allocation in accident investigations. If minor accidents are different in kind from major accidents, a fortiori incidents (where no accident occurred but there was the potential for one to happen) might bear little resemblance to major accidents in their causal factors. The possibility that the errors in serious and minor accidents were different in character needed to be re-examined.

One possible reason for the observed difference between the distributions for major and minor accidents lay in the nature of the database. About 75% of the minor accidents reported had not been investigated, but were merely transcriptions of the pilots' accident reports. These were subject to all the usual limitations of self-reporting. It was often not evident, on the face of the reports, which accidents had been investigated. To determine this, it was necessary to review the investigation files.

Review of the New Zealand accident database

To overcome the limitations of the earlier investigation, the present study (Zotov, 1997) was restricted to fatal accidents, and those non-fatal accidents which had been subject to investigation.

In order to get a clear-cut division, the boundary between 'major' and 'minor' accidents was redrawn, using 'Fatal' and 'Non-fatal' categories. Where direct comparisons were made with results obtained by O'Hare et al (1994), adjustments were made to compensate for this change.

The files (some 2500) on all accidents officially reported in New Zealand between 1965 and 1990 were examined. All accidents which had been subject to some degree of field investigation (so that the Inspector had had a chance to form his own opinions) were coded - some 800 accidents altogether.

The large database permitted the accidents to be studied by aircraft class (fixed-wing, helicopter etc), and by pilot experience.


Results
Figure 3 Fixed Wing

Figure 3. Fatal and Non-Fatal Accidents Fixed Wing

The largest aircraft class was that for fixed-wing powered aircraft not engaged in agricultural operations. The graphs showing the frequencies of the different sorts of pilot errors are shown in Fig 3, both for fatal and non-fatal accidents. They are quite similar, showing a marked preponderance of goal-selection errors. Although the difference is statistically significant, it is of little practical importance. It was concluded that the marked difference between major and minor accidents found by O'Hare et al (1994) arose from the inclusion, in their data for minor accidents, of many self-reported accidents.


By far the commonest triggering event is faulty goal selection. This is understandable. If you make a bad choice of goal, such as deciding to press on into adverse weather because you ‘have to’ get home, the pressure will build up as the weather gets worse. You will be less able to think clearly, you may make poor choices of strategies, you may perform procedures poorly, and your aircraft handling will suffer. The aircraft is likely to strike the terrain with considerable energy, with undesirable consequences.

By contrast, if you make an action error – say, applying insufficient power to counter a wind-gradient on approach – the result is likely to be a red face, rather than a disaster.

The high proportion of goal selection errors provides a theoretical underpinning for the work that has gone into improving decision-making by pilots – CRM and its General Aviation equivalent, Aeronautical Decision Making. If we need any justification for these moves, it is here.

At the same time, the small proportion of accidents which arise as a result of action errors suggests that not too much effort should be put into trying to improve training in aircraft handling, because this is already well taught. The considerable work which went into setting standards for instruction and training was, to a large extent, misdirected.

Helicopter accidents follow much the same pattern, but the emphasis is on strategy errors in the fatal accidents distribution.

However, Fig 4 - fixed-wing agricultural aircraft - shows something quite different. Alone among all classes of aircraft, here we find action errors are a major factor. Perhaps it is the unforgiving environment in which they operate - but if this is the case, why don't venison recovery helicopters follow this pattern? Is it that the pilots are risk-seekers? Again, the comparison with venison recovery helicopters casts doubt on that proposition. Can it reflect training deficiencies? It may be that these results suggest a lack of foresight in the Government actions which led to the break-up of the large firms capable of providing adequate training and supervision. Finding the cause of the discrepancy could lead to a marked reduction in the accident rate for agricultural aircraft.

Figure 4. Error Classification - Agricultural Aircraft

Figure 4. Error Classification - Agricultural Aircraft

As to pilot experience, there appears to be no effect at all: pilots with a few thousand hours experience make the same sorts of errors - predominantly goal selection - as do pilots with a few hundred hours. Of course, student pilots make action errors, like not raising the nose before touchdown, but such uninvestigated accidents were excluded from the database.

So far we have talked about pilot errors, but what about mechanical failures? While fixed-wing powered aircraft conform to the general rule that about three-quarters of all accidents are due to pilot error, this premise becomes less valid as the inherent hazards of the job increase. This is particularly the case with helicopter venison recovery, but it applies to helicopters generally (Table 1).

Table 1 Percentage of fatal accidents arising from pilot error
Fixed wing power *
Fixed wing agricultural
Helicopter **
Helicopter, deer recovery
Light sporting
ݬ80%
74%
65%
52%
50%
NB: * Excluding agricultural aircraft
** Excluding deer recovery helicopters

Fig 5. Helicopter fatal accidents: mechanical c.f. human factors

We are not going to make too big a dent in the helicopter accident rate, by improved pilot training. (Fig 5).

It is probably a function of the way we use helicopters in New Zealand. If we persist in operating single-engine helicopters over unlandable terrain, or operating in the Dead Man's Curve, any mechanical failure is likely to be disastrous. By necessary implication, if we say we ‘have to’ operate helicopters like this, then we accept the present accident rate and put up with it.

Accident investigation

The analytical procedure has been used in the course of accident investigations, to try to discover what the pilot was trying to do (as opposed to what we think he ought to have been trying to do).

In one very puzzling accident, a number of witnesses had seen or heard the aircraft following an apparently random flight path among hills at night in poor weather. Had the pilot really been wandering aimlessly among the hills at night, while telling the controller that he was making an instrument approach? The problem was compounded because one witness was unable to tie down the time of the occurrence to better than half an hour or so. He might have heard the aircraft early on in the accident sequence, or towards the end. With one uncertainty in the data, two possible flight paths could be found which fitted the observations and the aircraft performance, but neither of these seemed logical.

The problem was discussed with Dr O'Hare, and was resolved by considering what the pilot could have been trying to achieve at each stage of the flight. One flight path could be explained in terms of accumulating errors and a progressively shorter ‘look-ahead’ time, while the other required him to have made a wild mental leap at one point, but subsequently to have acted rationally. The former explanation was preferred.

Once we know what the errors were that the pilot made, we are well-placed to look at how those errors came about. For example, an action error may betoken a skill deficiency; an incorrect procedure may indicate inadequate knowledge of rules. And having worked out how, we can then look for the systemic failings that will tell us why the errors that led to the accident were made.

Conclusion

The results of this study (Zotov, 1997) suggest that there are some statistically significant differences between the triggering errors for fatal and non-fatal accidents. For practical purposes, however, the distributions are broadly similar, and they do not detract from Billings and Reynard's hypothesis, that incidents and accidents come from a common population. It would appear to be worthwhile to review a confidential incident database, to see to what extent self-reporting has coloured the information available.

The distribution of types of errors shows the validity of the recent emphasis on decision-making during pilot training. There is not much to be gained by improvements in the teaching or testing of aircraft handling: we already do a good job in that area.

However, the errors by agricultural pilots appear to be different in kind from those of all other pilots, and discovery of the reasons for this could well be rewarding in bringing about a reduction of the agricultural accident rate.

In the case of helicopters, a change in the approach to helicopter usage in New Zealand seems the most likely way to bring down the very high accident rate.

In any event, the analytical process devised by O'Hare et al (1994) has the potential to be a useful investigation tool.


References:

tBillings C. E. & Reynard, W. D. (1981). Dimensions of the information transfer problem. In C. E. Billings and E. S. Cheaney (Eds.) Information Transfer Problems in the Aviation System . Moffet Field, CA: NASA Ames. Research Centre, pp. 9 - 14, NASA-TP-1875. Cited in Rouse and Rouse (1983), and in Nagel (1988).

International Civil Aviation Organization. (1994). International standards and recommended practices: aircraft accident and incident investigation. (8th ed.). Ontario: Author.

O'Hare, D., Wiggins, M., Batt, R., & Morrison, D. (1994). Cognitive failure analysis for aircraft accident investigation. Ergonomics. 37 (11). 1855-1870.

Rasmussen, J. (1980). What can be learned from human error reports? In: K. D. Duncan, M. Gruneberg & D. Wallis (Eds.). Changes in working life . New York: Wiley.

Rasmussen, J. (1982). Human errors: a taxonomy for describing human malfunction in industrial installations. Journal of Occupational Accidents. 4. 311-335.

Reason, J. (1990). Human error. Cambridge: Cambridge University Press.

Reason, J. (1991). Identifying the latent causes of aircraft accidents before and after the event. Proceedings of the 22nd International Seminar of the International Society of Air Safety Investigators. (pp. 39 - 45). Sterling, Virginia, USA: ISASI.

Wiegmann, D. A., & Shappell, S. A. (1997). Human factors analysis of post-accident data: applying theoretical taxonomies of human error . International Journal of Aviation Psychology. 7 (pp. 67-81).

Zotov, D. V. (1997). Pilot error: cognitive failure analysis. ttUnpublished Master's thesis. Palmerston North: Massey University.


Annex A

ERROR CLASSIFICATION PROCEDURE

Section 1

1.1 Was the information necessary to recover from the situation or minimise the damage to the aircraft or its occupants, available to effect a timely intervention?

Yes Next question

No Go to Section 2

No Information Go to Section 2

1.2 Did the pilot observe any of the information which would have allowed the recovery of the situation or the minimisation of the damage to the aircraft or its occupants?

Yes Go to Section 2

No Information Error Code 1

No Information Go to Section 2

Section 2

2.1 Did the pilot attempt to diagnose the state of the system on the basis of the information available?

Yes Next Question

No Failed to Diagnose system state Code 2/1

No Information Go to Section 3

2.2 Did the pilot complete the diagnosis of the system state?

Yes Next Question

No Incomplete Diagnosis Code 2/2

No Information Go to Section 3

2.3 Did the pilot diagnose accurately the information available concerning the state of the aircraft?

Yes Go to Section 3

No Next Question

No Information Go to Section 3

2.4 Was this a major contributing factor in the subsequent accident/incident?

Yes Misdiagnosis of system state Code 2/3

No Go to Section 3

Section 3

3.1 Did the pilot have a goal in mind when the accident/incident occurred?

Yes Next Question

No Failed to formulate Goal Code 3/1

No Information Go to Section 4

3.2 Was the goal reasonable under the circumstances (e.g. weather, experience, etc)

Yes Go to Section 4

No Incorrect Goal Code 3/2

No Information Go to Section 4

Section 4

4.1 Did the pilot choose a strategy* with which to achieve the goal?

Yes Next Question

No Failed to adopt strategy Code 4/1

No Information Go to Section 5

4.2 Was the strategy chosen the most appropriate one under the circumstances?

Yes Go to Section 5

No Incorrect Strategy Code 4/2

No Information Go to Section 5

Section 5

5.1 Did the pilot attempt to execute a procedure** consistent with the strategy?

Yes Next Question

No Failed to execute procedure Code 5/1

No Information Go to Section 6

5.2 Was the procedure the most appropriate under the circumstances?

Yes Next question

No Incorrect Procedure Code 5/2

No Information Go to Section 6

Section 6

6.1 Was the procedure carried out as intended?

Yes No Errors Code 0

No Action Error Code 6

* Strategy: finding the means to satisfy a goal

** Procedure: a specification for conducting a set of predetermined subtasks or actions that are components of a higher-level task

(Note: the ‘fine grain’ coding, which subdivides the categories, gives extra detail which may be useful in subsequent studies).


[1] ttA detailed description of the process, which has been found useful for successful analysis, is at Annex A

s