IRUS Total

Probing the need for visual context in multimodal machine translation

File Description SizeFormat 
Ozan_Probing the need for visual context.pdfPublished version2.52 MBAdobe PDFView/Open
Title: Probing the need for visual context in multimodal machine translation
Authors: Caglayan, O
Madhyastha, P
Specia, L
Barrault, L
Item Type: Conference Paper
Abstract: Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in order to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially deprive the models from source-side textual context. Our results show that under limited textual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model.
Issue Date: 2-Jun-2019
Date of Acceptance: 22-Feb-2019
URI: http://hdl.handle.net/10044/1/71002
Publisher: Association for Computational Linguistics
Start Page: 4159
End Page: 4170
Copyright Statement: ©2019 Association for Computational Linguistics. ACL materials are Copyright © 1963–2019 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
Sponsor/Funder: Commission of the European Communities
British Council (Turkey)
Funder's Grant Number: 678017
352343575 (Lucia Specia)
Conference Name: Conference of the North American Chapter of the Association for Computational Linguistics
Keywords: cs.CL
Notes: Accepted to NAACL-HLT 2019, reviewer comments addressed, camera-ready
Publication Status: Published
Start Date: 2019-06-02
Finish Date: 2019-06-07
Conference Place: Minneapolis, MN, USA
Open Access location: https://www.aclweb.org/anthology/N19-1422
Appears in Collections:Computing