Public availability and adherence to pre-specified statistical analysis approaches was low in published randomised trials.
File(s)
Accepted version
Author(s)
Kahan, Brennan C
Ahmad, Tahania
Forbes, Gordon
Cro, Suzie
Type
Journal Article
Abstract
OBJECTIVE: Pre-specification of statistical methods in clinical trial protocols and Statistical Analysis Plans (SAPs) can help to deter bias from p-hacking, but is only effective if the pre-specified approach is made available. STUDY DESIGN AND SETTING: For 100 randomised trials published in 2018 and indexed in PubMed we evaluated how often a pre-specified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available pre-specified analysis, we compared this to the trial publication to identify whether there were unexplained discrepancies. RESULTS: Only 11 of 100 trials (11%) had a publicly available pre-specified analysis approach for their primary outcome; this document was dated before recruitment began for only one trial. Of the 11 trials with an available pre-specified analysis approach, 10 (91%) had one or more unexplained discrepancies. Only 4/100 trials (4%) stated that the statistician was blinded until the SAP was signed off and only 10/100 (10%) stated the statistician was blinded until the database was locked. CONCLUSION: For most published trials, there is insufficient information available to determine whether results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the pre-specified and final analysis methods.
Date Issued
2020-12-01
Date Acceptance
2020-07-23
Citation
Journal of Clinical Epidemiology, 2020, 128, pp.29-34
ISSN
0895-4356
Publisher
Elsevier
Start Page
29
End Page
34
Journal / Book Title
Journal of Clinical Epidemiology
Volume
128
Copyright Statement
© 2020 Elsevier Ltd. All rights reserved. This manuscript is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence http://creativecommons.org/licenses/by-nc-nd/4.0/
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/32730852
PII: S0895-4356(20)30197-9
Subjects
bias
p-hacking
pre-specification
protocol
randomised trial
statistical analysis plan
transparency
Publication Status
Published
Coverage Spatial
United States
Date Publish Online
2020-07-28