Evaluating the clarity of the questions being addressed in randomised trials: a systematic review of estimands
File(s)
Published version
Author(s)
Type
Journal Article
Abstract
Objective: To evaluate how often the precise question being addressed about an intervention (the estimand) is stated or can be worked out from reported methods, and to identify what types of questions are addressed in phase II-IV randomised trials.
Design: A systematic review of the clarity of research questions addressed in randomised trials in 2020 in six leading general medical journals.
Eligibility criteria: Phase II-IV randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, cross-over, non-inferiority, and equivalence trials were excluded.
Data source: A search of PubMed was performed in February 2021.
Main outcome measures: The number of trials which stated the precise primary question being addressed about an intervention (the primary estimand) or for which this could be unambiguously worked out from the reported methods using statistical knowledge. The strategies being used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional medications (termed intercurrent events), and the corresponding types of questions being addressed.
Results: A total of 255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117/255 (46%) trials the primary question addressed could be worked out from the reported methods. Intercurrent events occurred in 95% of trials; but the handling of these could only be determined in 125/255 (49%) trials. Most trials which provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and addressed the effect of the intervention regardless (96/125, 76%) i.e. as if introduced into routine practice (treatment policy strategy). 4/99 (4%) trials with treatment non-adherence due to adverse events estimated the treatment effect in a hypothetical setting (the effect as if participants continued treatment despite adverse events) and 19/24 (79%) trials where some patients died estimated the treatment effect in a hypothetical setting (the effect as if participants did not die).
Conclusions: It is unclear in most trials what precise research question is being addressed. The main driver for this is lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.
Design: A systematic review of the clarity of research questions addressed in randomised trials in 2020 in six leading general medical journals.
Eligibility criteria: Phase II-IV randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, cross-over, non-inferiority, and equivalence trials were excluded.
Data source: A search of PubMed was performed in February 2021.
Main outcome measures: The number of trials which stated the precise primary question being addressed about an intervention (the primary estimand) or for which this could be unambiguously worked out from the reported methods using statistical knowledge. The strategies being used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional medications (termed intercurrent events), and the corresponding types of questions being addressed.
Results: A total of 255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117/255 (46%) trials the primary question addressed could be worked out from the reported methods. Intercurrent events occurred in 95% of trials; but the handling of these could only be determined in 125/255 (49%) trials. Most trials which provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and addressed the effect of the intervention regardless (96/125, 76%) i.e. as if introduced into routine practice (treatment policy strategy). 4/99 (4%) trials with treatment non-adherence due to adverse events estimated the treatment effect in a hypothetical setting (the effect as if participants continued treatment despite adverse events) and 19/24 (79%) trials where some patients died estimated the treatment effect in a hypothetical setting (the effect as if participants did not die).
Conclusions: It is unclear in most trials what precise research question is being addressed. The main driver for this is lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.
Date Issued
2022-08-23
Date Acceptance
2022-06-21
Citation
BMJ: British Medical Journal, 2022, 378, pp.1-12
ISSN
0959-535X
Publisher
BMJ Publishing Group
Start Page
1
End Page
12
Journal / Book Title
BMJ: British Medical Journal
Volume
378
Copyright Statement
© 2022 The Author(s). This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
License URL
Sponsor
National Institute for Health Research
Identifier
https://www.bmj.com/content/378/bmj-2022-070146
Grant Number
NIHR300118
Subjects
General & Internal Medicine
1103 Clinical Sciences
1117 Public Health and Health Services
Publication Status
Published
Date Publish Online
2022-08-23