Unsupervised quality estimation for neural machine translation
File(s)
Working paper
Author(s)
Type
Working Paper
Abstract
Quality Estimation (QE) is an important component in making Machine
Translation (MT) useful in real-world applications, as it is aimed to inform
the user on the quality of the MT output at test time. Existing approaches
require large amounts of expert annotated data, computation and time for
training. As an alternative, we devise an unsupervised approach to QE where no
training or access to additional resources besides the MT system itself is
required. Different from most of the current work that treats the MT system as
a black box, we explore useful information that can be extracted from the MT
system as a by-product of translation. By employing methods for uncertainty
quantification, we achieve very good correlation with human judgments of
quality, rivalling state-of-the-art supervised QE models. To evaluate our
approach we collect the first dataset that enables work on both black-box and
glass-box approaches to QE.
Translation (MT) useful in real-world applications, as it is aimed to inform
the user on the quality of the MT output at test time. Existing approaches
require large amounts of expert annotated data, computation and time for
training. As an alternative, we devise an unsupervised approach to QE where no
training or access to additional resources besides the MT system itself is
required. Different from most of the current work that treats the MT system as
a black box, we explore useful information that can be extracted from the MT
system as a by-product of translation. By employing methods for uncertainty
quantification, we achieve very good correlation with human judgments of
quality, rivalling state-of-the-art supervised QE models. To evaluate our
approach we collect the first dataset that enables work on both black-box and
glass-box approaches to QE.
Date Issued
2020-05-21
Citation
2020
Publisher
arXiv
Copyright Statement
© 2020 The Author(s)
Identifier
http://arxiv.org/abs/2005.10608v1
Subjects
cs.CL
cs.CL
Notes
Accepted for publication in TACL
Publication Status
Published