Improving the evaluation of digital mental health interventions in randomised controlled trials by understanding and incorporating user engagement
File(s)
Author(s)
Elkes, Jack
Type
Thesis
Abstract
Introduction
Self-guided Digital mental health interventions (DMHI) help more individuals access mental health support. However, engagement with DMHIs is heterogeneous so treatment policy Estimands cannot answer the patient-centred question of how DMHIs efficacy changes with engagement. Alternative Estimands can target this question but first the engagement profile with DMHIs must be understood.
Aim
I aim to find an optimal strategy to find groups of users from available engagement measures, which can then be used in the alternative Estimands to assess how changes in engagement impact the efficacy.
Methods
First, a systematic review assessed current practice for describing engagement in trials evaluating DMHIs. Next, data from three previous clinical trials was used to find the optimal unsupervised cluster algorithm to identify groups of users (the engagement profile). Then using two different Estimands (principal stratum and hypothetical) the change in intervention efficacy across the groups was assessed. Finally, a methodological framework was created for other trialists to implement this analysis.
Results
My systematic review (publications 2016-2021) found over 70% of the 184 trials identified included a summary of engagement data but only 10% (n=19) reported estimates of intervention efficacy changes with engagement. From the previous three case studies, K-means was identified as optimal algorithm (with PCA when many engagement measures). Estimates of the intervention efficacy across the groups changed. The optimal methods were then summarised as 8 steps in the framework.
Conclusions
Distinct user groups can be identified from engagement data using K-means, and this performs well across different case studies with different engagement measures. The intervention efficacy does change across the groups, highlighting that researchers must have access to methods to assess the impact of engagement. Finally, the methodological framework created ensure these methods are accessible and available to be used by other trialists.
Self-guided Digital mental health interventions (DMHI) help more individuals access mental health support. However, engagement with DMHIs is heterogeneous so treatment policy Estimands cannot answer the patient-centred question of how DMHIs efficacy changes with engagement. Alternative Estimands can target this question but first the engagement profile with DMHIs must be understood.
Aim
I aim to find an optimal strategy to find groups of users from available engagement measures, which can then be used in the alternative Estimands to assess how changes in engagement impact the efficacy.
Methods
First, a systematic review assessed current practice for describing engagement in trials evaluating DMHIs. Next, data from three previous clinical trials was used to find the optimal unsupervised cluster algorithm to identify groups of users (the engagement profile). Then using two different Estimands (principal stratum and hypothetical) the change in intervention efficacy across the groups was assessed. Finally, a methodological framework was created for other trialists to implement this analysis.
Results
My systematic review (publications 2016-2021) found over 70% of the 184 trials identified included a summary of engagement data but only 10% (n=19) reported estimates of intervention efficacy changes with engagement. From the previous three case studies, K-means was identified as optimal algorithm (with PCA when many engagement measures). Estimates of the intervention efficacy across the groups changed. The optimal methods were then summarised as 8 steps in the framework.
Conclusions
Distinct user groups can be identified from engagement data using K-means, and this performs well across different case studies with different engagement measures. The intervention efficacy does change across the groups, highlighting that researchers must have access to methods to assess the impact of engagement. Finally, the methodological framework created ensure these methods are accessible and available to be used by other trialists.
Version
Open Access
Date Issued
2025-09-21
Date Awarded
2026-03-01
Copyright Statement
Attribution 4.0 International Licence (CC BY)
License URL
Advisor
Cornelius, Victoria
Cro, Suzie
Sin, Jacqueline
Sponsor
National Institute for Health Research (Great Britain)
Grant Number
NIHR301810
Publisher Department
School of Public Health
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)