Curious case of language generation evaluation metrics: a cautionary tale
File(s)
Working paper
Author(s)
Caglayan, Ozan
Madhyastha, Pranava
Specia, Lucia
Type
Working Paper
Abstract
Automatic evaluation of language generation systems is a well-studied problem
in Natural Language Processing. While novel metrics are proposed every year, a
few popular metrics remain as the de facto metrics to evaluate tasks such as
image captioning and machine translation, despite their known limitations. This
is partly due to ease of use, and partly because researchers expect to see them
and know how to interpret them. In this paper, we urge the community for more
careful consideration of how they automatically evaluate their models by
demonstrating important failure cases on multiple datasets, language pairs and
tasks. Our experiments show that metrics (i) usually prefer system outputs to
human-authored texts, (ii) can be insensitive to correct translations of rare
words, (iii) can yield surprisingly high scores when given a single sentence as
system output for the entire test set.
in Natural Language Processing. While novel metrics are proposed every year, a
few popular metrics remain as the de facto metrics to evaluate tasks such as
image captioning and machine translation, despite their known limitations. This
is partly due to ease of use, and partly because researchers expect to see them
and know how to interpret them. In this paper, we urge the community for more
careful consideration of how they automatically evaluate their models by
demonstrating important failure cases on multiple datasets, language pairs and
tasks. Our experiments show that metrics (i) usually prefer system outputs to
human-authored texts, (ii) can be insensitive to correct translations of rare
words, (iii) can yield surprisingly high scores when given a single sentence as
system output for the entire test set.
Date Issued
2020-10-26
Citation
2020
Publisher
arXiv
Copyright Statement
© 2020 The Author(s).
Sponsor
Commission of the European Communities
Identifier
http://arxiv.org/abs/2010.13588v1
Grant Number
678017
Subjects
cs.CL
cs.CL
Notes
7 pages, accepted to COLING 2020
Publication Status
Published