Simultaneous machine translation with visual context
Author(s)
Type
Journal Article
Abstract
Simultaneous machine translation (SiMT) aims to translate a continuous input
text stream into another language with the lowest latency and highest quality
possible. The translation thus has to start with an incomplete source text,
which is read progressively, creating the need for anticipation. In this paper,
we seek to understand whether the addition of visual information can compensate
for the missing source context. To this end, we analyse the impact of different
multimodal approaches and visual features on state-of-the-art SiMT frameworks.
Our results show that visual context is helpful and that visually-grounded
models based on explicit object region information are much better than
commonly used global features, reaching up to 3 BLEU points improvement under
low latency scenarios. Our qualitative analysis illustrates cases where only
the multimodal systems are able to translate correctly from English into
gender-marked languages, as well as deal with differences in word order, such
as adjective-noun placement between English and French.
text stream into another language with the lowest latency and highest quality
possible. The translation thus has to start with an incomplete source text,
which is read progressively, creating the need for anticipation. In this paper,
we seek to understand whether the addition of visual information can compensate
for the missing source context. To this end, we analyse the impact of different
multimodal approaches and visual features on state-of-the-art SiMT frameworks.
Our results show that visual context is helpful and that visually-grounded
models based on explicit object region information are much better than
commonly used global features, reaching up to 3 BLEU points improvement under
low latency scenarios. Our qualitative analysis illustrates cases where only
the multimodal systems are able to translate correctly from English into
gender-marked languages, as well as deal with differences in word order, such
as adjective-noun placement between English and French.
Date Issued
2020-09-01
Date Acceptance
2020-09-01
Citation
Transactions of the Association for Computational Linguistics, 2020, 8, pp.539-555
ISSN
2307-387X
Publisher
Massachusetts Institute of Technology Press
Start Page
539
End Page
555
Journal / Book Title
Transactions of the Association for Computational Linguistics
Volume
8
Copyright Statement
© 2020 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license https://creativecommons.org/licenses/by/4.0/.
Sponsor
Commission of the European Communities
British Council (Turkey)
Identifier
http://arxiv.org/abs/2009.07310v3
Grant Number
678017
352343575 - 154082
Subjects
cs.CL
cs.CL
Notes
Long paper accepted to EMNLP 2020, Camera-ready version
Publication Status
Published
Date Publish Online
2020-09-01