Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction
File(s)
Published version
Author(s)
Robinson, Matthew C
Glen, Robert C
Lee, Alpha A
Type
Journal Article
Abstract
Machine learning methods may have the potential to significantly accelerate drug discovery. However, the increasing rate of new methodological approaches being published in the literature raises the fundamental question of how models should be benchmarked and validated. We reanalyze the data generated by a recently published large-scale comparison of machine learning models for bioactivity prediction and arrive at a somewhat different conclusion. We show that the performance of support vector machines is competitive with that of deep learning methods. Additionally, using a series of numerical experiments, we question the relevance of area under the receiver operating characteristic curve as a metric in virtual screening. We further suggest that area under the precision-recall curve should be used in conjunction with the receiver operating characteristic curve. Our numerical experiments also highlight challenges in estimating the uncertainty in model performance via scaffold-split nested cross validation.
Date Issued
2020-07-01
Date Acceptance
2019-12-22
Citation
Journal of Computer-Aided Molecular Design, 2020, 34, pp.717-730
ISSN
0920-654X
Publisher
Springer (part of Springer Nature)
Start Page
717
End Page
730
Journal / Book Title
Journal of Computer-Aided Molecular Design
Volume
34
Copyright Statement
© The Author(s) 2020. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/31960253
PII: 10.1007/s10822-019-00274-0
Subjects
Medicinal & Biomolecular Chemistry
0304 Medicinal and Biomolecular Chemistry
0307 Theoretical and Computational Chemistry
Publication Status
Published
Coverage Spatial
Netherlands
Date Publish Online
2020-01-20