Some statistical and data challenges when building early-stage digital experimentation and measurement capabilities
File(s)
Author(s)
Liu, Chak Hin Bryan
Type
Thesis
Abstract
Digital experimentation and measurement (DEM) capabilities – the knowledge and tools necessary to run experiments with digital products, services, or experiences and measure their impact – are fast becoming part of the standard toolkit of digital/data-driven organisations in guiding business decisions. Many large technology companies report having mature DEM capabilities, and several businesses have been established purely to manage experiments for others. Given the growing evidence that data-driven organisations tend to outperform their non-data-driven counterparts, there has never been a greater need for organisations to build/acquire DEM capabilities to thrive in the current digital era. This thesis presents several novel approaches to statistical and data challenges for organisations building DEM capabilities. We focus on the fundamentals associated with building DEM capabilities, which lead to a richer understanding of the underlying assumptions and thus enable us to develop more appropriate capabilities. We address why one should engage in DEM by quantifying the benefits and risks of acquiring DEM capabilities. This is done using a ranking under lower uncertainty model, enabling one to construct a business case. We also examine what ingredients are necessary to run digital experiments. In addition to clarifying the existing literature around statistical tests, datasets, and methods in experimental design and causal inference, we construct an additional dataset and detailed case studies on applying state-of-the-art methods. Finally, we investigate when a digital experiment design would outperform another, leading to an evaluation framework that compares competing designs’ data efficiency. These approaches aim to enable one to run experiments that produce less biased estimates more quickly and adapt to various business constraints. As we maintain the theoretical rigour, we also emphasise applied use cases, interpretable processes/results, and practical tradeoffs, all to ensure that the contributions are accessible to researchers and practitioners from diverse scientific backgrounds.
Version
Open Access
Date Issued
2023-10
Date Awarded
2024-03
Copyright Statement
Creative Commons Attribution Licence
License URL
Advisor
McCoy, Emma
Sponsor
Engineering and Physical Sciences Research Council
ASOS
Grant Number
EP/S023151/1
Publisher Department
Mathematics
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)