Multimodal sentiment analysis to explore the structure of emotions

File Description SizeFormat 
1805.10205.pdfAccepted version2.09 MBAdobe PDFView/Open
Title: Multimodal sentiment analysis to explore the structure of emotions
Authors: Hu, A
Flaxman, S
Item Type: Conference Paper
Abstract: We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual analysis and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as "self-reported emotions." We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model's results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of images - both photographs and memes - on social networks.
Issue Date: 19-Jul-2018
Date of Acceptance: 7-May-2018
ISBN: 978-1-4503-5552-0
Publisher: ACM
Start Page: 350
End Page: 358
Journal / Book Title: KDD '18 Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Copyright Statement: © 2018 Copyright held by the owner/author(s). Publication rights licensed to the Association for Computing Machinery. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
Conference Name: KDD 2018
Publication Status: Published
Start Date: 2018-08-19
Finish Date: 2018-08-23
Conference Place: London, UK
Open Access location:
Appears in Collections:Mathematics

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commonsx