Why even more clinical research studies may be false: effect of asymmetrical handling of clinically unexpected values.
Author(s)
Shun-Shin, Matthew James
Francis, Darrel P
Type
Journal Article
Abstract
BACKGROUND: In medical practice, clinically unexpected measurements might be quite properly handled by the remeasurement, removal, or reclassification of patients. If these habits are not prevented during clinical research, how much of each is needed to sway an entire study? METHODS AND RESULTS: Believing there is a difference between groups, a well-intentioned clinician researcher addresses unexpected values. We tested how much removal, remeasurement, or reclassification of patients would be needed in most cases to turn an otherwise-neutral study positive. Remeasurement of 19 patients out of 200 per group was required to make most studies positive. Removal was more powerful: just 9 out of 200 was enough. Reclassification was most powerful, with 5 out of 200 enough. The larger the study, the smaller the proportion of patients needing to be manipulated to make the study positive: the percentages needed to be remeasured, removed, or reclassified fell from 45%, 20%, and 10% respectively for a 20 patient-per-group study, to 4%, 2%, and 1% for an 800 patient-per-group study. Dot-plots, but not bar-charts, make the perhaps-inadvertent manipulations visible. Detection is possible using statistical methods such as the Tadpole test. CONCLUSIONS: Behaviours necessary for clinical practice are destructive to clinical research. Even small amounts of selective remeasurement, removal, or reclassification can produce false positive results. Size matters: larger studies are proportionately more vulnerable. If observational studies permit selective unblinded enrolment, malleable classification, or selective remeasurement, then results are not credible. Clinical research is very vulnerable to "remeasurement, removal, and reclassification", the 3 evil R's.
Date Issued
2013-06-25
Date Acceptance
2013-04-23
Citation
PLoS One, 2013, 8 (6), pp.e65323-e65323
ISSN
1932-6203
Publisher
Public Library of Science
Start Page
e65323
End Page
e65323
Journal / Book Title
PLoS One
Volume
8
Issue
6
Copyright Statement
© 2013 Shun-Shin, Francis. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
License URL
Sponsor
British Heart Foundation
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/23825524
PII: PONE-D-13-02018
Grant Number
FS/10/38/28268
Subjects
Biomedical Research
Humans
Publication Status
Published online
Coverage Spatial
United States
Article Number
e65323