Better together: reliable application of the post-9/11 and post-Iraq US intelligence tradecraft standards requires collective analysis

File Description SizeFormat 
fpsyg-09-02634.pdfPublished version1.06 MBAdobe PDFView/Open
Title: Better together: reliable application of the post-9/11 and post-Iraq US intelligence tradecraft standards requires collective analysis
Authors: Marcoci, A
Burgman, M
Kruger, A
Silver, E
McBride, M
Thorn, F
Fraser, H
Wintle, B
Fidler, F
Vercammen, A
Item Type: Journal Article
Abstract: Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.
Issue Date: 7-Jan-2019
Date of Acceptance: 7-Dec-2018
ISSN: 1664-1078
Publisher: Frontiers Media
Journal / Book Title: Frontiers in Psychology
Volume: 9
Copyright Statement: © 2019 Marcoci, Burgman, Kruger, Silver, McBride, Singleton Thorn, Fraser, Wintle, Fidler and Vercammen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Sponsor/Funder: Office of the Director of National Intelligence (US Agency)
Funder's Grant Number: TBC
Keywords: Social Sciences
Psychology, Multidisciplinary
intelligence analysis
intelligence failures
intelligence reform
tradecraft standards
inter-rater reliability
1701 Psychology
Publication Status: Published
Article Number: ARTN 2634
Appears in Collections:Centre for Environmental Policy
Faculty of Natural Sciences

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commonsx