Improving expert forecasts in reliability. Application and evidence for structured elicitation protocols

Abstract

Quantitative expert judgementsare used in reliability assessmentsto informcritically important decisions. Structured elicitation protocols have been advocated to improveexpert judgements, yet their application in reliability ischallenged by a lack of examples or evidence that they improve judgements. This paper aims to overcome these barriers. We present a case study where two world-leading protocols, the IDEA protocol and the Classical Model were combined and applied by the Australian Department of Defence for a reliability assessment. We assess the practicality of the methods, and the extent to which they improve judgements. The average expert was extremely overconfident, with 90% credible intervals containing the true realisation 36% of the time. However,steps contained inthe protocols substantially improvedjudgements. In particular, an equal weighted aggregation of individual judgements, and the inclusion ofa discussion phase and revised estimate helped to improve calibration, statistical accuracy and the Classical Model score. Further improvements in precision and information were made via performance weighted aggregation. This paper provides useful insights into the application of structured elicitation protocols for reliability andthe extent to which judgements are improved. The findings raise concerns about existing practices for utilising experts in reliability assessments and suggest greater adoption of structured protocols is warranted. We encourage the reliability community to further develop examples and insights.