Motivation:
Recognition of biomedical entities from scientific text is a critica
l component of natural
language processing and automated information extraction platfo
rms. Modern named entity recognition
approaches rely heavily on supervised machine learning tech
niques, which are critically dependent on
annotated training corpora. These approaches have been shown to
perform well when trained and tested
on the same source. However, in such scenario, the performance
and evaluation of these models may be
optimistic, as such models may not necessarily generalize to in
dependent corpora, resulting in potential
non-optimal entity recognition for large-scale tagging of widel
y diverse articles in databases such as
PubMed.
Results:
Here we aggregated published corpora for the recognition of bio
molecular entities (such as
genes, RNA, proteins, variants, drugs, and metabolites), identi
fied entity class overlap and performed
leave-corpus-out cross validation strategy to test the efficiency o
f existing models. We demonstrate
that accuracies of models trained on individual corpora decre
ase substantially for recognition of the
same biomolecular entity classes in independent corpora. Thi
s behavior is possibly due to limited
generalizability of entity-class-related features captured by i
ndividual corpora (model “overtraining”) which
we investigated further at the orthographic level, as well as potenti
al annotation standard differences.
We show that the combined use of multi-source training corpora re
sults in overall more generalizable
models for named entity recognition, while achieving comparab
le individual performance. By performing
learning-curve-based power analysis we further identified that
performance is often not limited by the
quantity of the annotated data.