BreakingNews: article annotation by image and text processing

File Description SizeFormat 
top.pdfAccepted version4.59 MBAdobe PDFView/Open
Title: BreakingNews: article annotation by image and text processing
Authors: Ramisa, A
Yan, F
Moreno-Noguer, F
Mikolajczyk, K
Item Type: Journal Article
Abstract: Current approaches lying in the intersection of computer vision and NLP have achieved unprecedented breakthroughs in tasks like automatic captioning or image retrieval. Most of these methods, though, rely on training sets of images associated with annotations that specifically describe the visual content. This paper proposes going a step further and explores more complex cases where textual descriptions are loosely related to images. We focus on the particular domain of News. We introduce new deep learning methods that address source and popularity prediction, article illustration, and article geolocation. An adaptive CNN is proposed, that shares most of the structure for all tasks, and is suitable for multitask and transfer learning. Deep CCA is deployed for article illustration, and a new loss function based on Great Circle Distance is proposed for geolocation. Furthermore, we present BreakingNews, a novel dataset with approximately 100K news articles including images, text, captions, and enriched with heterogeneous meta-data. BreakingNews allows exploring all aforementioned problems, for which we provide baseline performances using various CNN architectures, and different representations of the textual and visual features. We report promising results and bring to light several limitations of current state-of-the-art, which we hope will help spur progress in the field.
Issue Date: 30-Jun-2017
Date of Acceptance: 30-Jun-2017
ISSN: 0162-8828
Publisher: Institute of Electrical and Electronics Engineers
Start Page: 1072
End Page: 1085
Journal / Book Title: IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume: 40
Issue: 5
Copyright Statement: © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor/Funder: Engineering & Physical Science Research Council (E
Funder's Grant Number: EP/N007743/1
Keywords: Science & Technology
Computer Science, Artificial Intelligence
Engineering, Electrical & Electronic
Computer Science
News dataset
story illustration
caption generation
vision and text
0801 Artificial Intelligence And Image Processing
0806 Information Systems
0906 Electrical And Electronic Engineering
Artificial Intelligence & Image Processing
Publication Status: Published
Appears in Collections:Faculty of Engineering
Electrical and Electronic Engineering

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commons