Multimodal representation models for prediction and control from partial information
OA Location
Author(s)
Zambelli, Martina
Cully, Antoine
Demiris, Yiannis
Type
Journal Article
Abstract
Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a multimodal variational autoencoder is proposed to enable an iCub humanoid robot to learn representations of its sensorimotor capabilities from different sensor modalities. The proposed model is able to (1) reconstruct missing sensory modalities, (2) predict the sensorimotor state of self and the visual trajectories of other agents actions, and (3) control the agent to imitate an observed visual trajectory. Also, the proposed multimodal variational autoencoder can capture the kinematic redundancy of the robot motion through the learned probability distribution. Training multimodal models is not trivial due to the combinatorial complexity given by the possibility of missing modalities. We propose a strategy to train multimodal models, which successfully achieves improved performance of different reconstruction models. Finally, extensive experiments have been carried out using an iCub humanoid robot, showing high performance in multiple reconstruction, prediction and imitation tasks.
Date Issued
2020-01
Date Acceptance
2019-09-30
Citation
Robotics and Autonomous Systems, 2020, 123
ISSN
0921-8890
Publisher
Elsevier
Journal / Book Title
Robotics and Autonomous Systems
Volume
123
Sponsor
Commission of the European Communities
Commission of the European Communities
Grant Number
612139
643783
Subjects
Science & Technology
Technology
Automation & Control Systems
Computer Science, Artificial Intelligence
Robotics
Computer Science
Multimodal learning
Autonomous learning
Variational autoencoder
MOTOR
IMITATION
ROBOTS
cs.RO
cs.RO
0801 Artificial Intelligence and Image Processing
0906 Electrical and Electronic Engineering
0913 Mechanical Engineering
Industrial Engineering & Automation
Publication Status
Published online
Article Number
103312
Date Publish Online
2019-10-18