Improving analytic reasoning via crowdsourcing and structured analytic techniques

Abstract

How might analytic reasoning in intelligence reports be substantially improved? One conjecture is that this can be achieved through a combination of crowdsourcing and structured analytic techniques (SATs). To explore this conjecture, we developed a new crowdsourcing platform supporting groups in collaborative reasoning and intelligence report drafting using a novel SAT we call “Contending Analyses.” In this paper we present findings from a large study designed to assess whether groups of professional analysts working on the platform produce better-reasoned reports than those analysts produce when using methods and tools normally used in their organizations. Secondary questions were whether professional analysts working on the platform produce better reasoning than the general public working on the platform; and how usable the platform is. Our main finding is a large effect size (Cohen’s d = 1.37) in favor of working on platform. This provides early support for the general conjecture. We discuss limitations of our study, implications for intelligence organizations, and future directions for the work as a whole.