Audio-visual object localization and separation using low-rank and sparsity
File(s)
Accepted version
Author(s)
Pu, J
Panagakis, Y
Petridis, S
Pantic, M
Type
Conference Paper
Abstract
The ability to localize visual objects that are associated with an audio source and at the same time seperate the audio signal is a corner stone in several audio-visual signal processing applications. Past efforts usually focused on localizing only the visual objects, without audio separation abilities. Besides, they often rely computational expensive pre-processing steps to segment images pixels into object regions before applying localization approaches. We aim to address the problem of audio-visual source localization and separation in an unsupervised manner. The proposed approach employs low-rank in order to model the background visual and audio information and sparsity in order to extract the sparsely correlated components between the audio and visual modalities. In particular, this model decomposes each dataset into a sum of two terms: the low-rank matrices capturing the background uncorrelated information, while the sparse correlated components modelling the sound source in visual modality and the associated sound in audio modality. To this end a novel optimization problem, involving the minimization of nuclear norms and matrix ℓ 1 -norms is solved. We evaluated the proposed method in 1) visual localization and audio separation and 2) visual-assisted audio denoising. The experimental results demonstrate the effectiveness of the proposed method.
Date Issued
2017-06-19
Date Acceptance
2017-03-05
Citation
Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, 2017, pp.2901-2905
ISBN
9781509041176
ISSN
2379-190X
Publisher
IEEE
Start Page
2901
End Page
2905
Journal / Book Title
Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on
Copyright Statement
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor
Commission of the European Communities
Commission of the European Communities
Grant Number
645094
688835
Source
ICASSP 2017
Publication Status
Published
Start Date
2017-03-05
Finish Date
2017-03-09
Coverage Spatial
New Orleans, LA, USA