Deep structured learning for facial action unit intensity estimation
File(s)
Accepted version
Author(s)
Walecki, Robert
Rudovic, Ognjen Oggi
Pavlovic, Vladimir
Schuller, Bjoern
Pantic, Maja
Type
Conference Paper
Abstract
We consider the task of automated estimation of facial expression intensity. This involves estimation of multiple output variables (facial action units - AUs) that are structurally dependent. Their structure arises from statistically induced co-occurrence patterns of AU intensity levels. Modeling this structure is critical for improving the estimation performance, however, this performance is bounded by the quality of the input features extracted from face images. The goal of this paper is to model these structures and estimate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning. To this end, we propose a novel Copula CNN deep learning approach for modeling multivariate ordinal variables. Our model accounts for ordinal structure in output variables and their non-linear dependencies via copula functions modeled as cliques of a CRF. These are jointly optimized with deep CNN feature encoding layers using a newly introduced balanced batch iterative training algorithm. We demonstrate the effectiveness of our approach on the task of AU intensity estimation on two benchmark datasets. We show that joint learning of the deep features and the target output structure results in significant performance gains compared to existing structured deep models and deep models for analysis of facial expressions.
Date Issued
2017-11
Date Acceptance
2017-11-01
Citation
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 2017, pp.5709-5718
ISSN
1063-6919
Publisher
Institute of Electrical and Electronics Engineers
Start Page
5709
End Page
5718
Journal / Book Title
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Volume
2017
Copyright Statement
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor
Commission of the European Communities
Commission of the European Communities
Identifier
http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000418371405084&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=1ba7043ffcc86c417c072aa74d649202
Grant Number
645094
688835
Source
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Subjects
Science & Technology
Technology
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
Engineering, Electrical & Electronic
Computer Science
Engineering
Publication Status
Published
Start Date
2017-07-21
Coverage Spatial
Honolulu, HI
Date Publish Online
2017-11-09