Multimodal representation models for prediction and control from partial information

Publication available at:
Title: Multimodal representation models for prediction and control from partial information
Authors: Zambelli, M
Cully, A
Demiris, Y
Item Type: Journal Article
Abstract: Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a multimodal variational autoencoder is proposed to enable an iCub humanoid robot to learn representations of its sensorimotor capabilities from different sensor modalities. The proposed model is able to (1) reconstruct missing sensory modalities, (2) predict the sensorimotor state of self and the visual trajectories of other agents actions, and (3) control the agent to imitate an observed visual trajectory. Also, the proposed multimodal variational autoencoder can capture the kinematic redundancy of the robot motion through the learned probability distribution. Training multimodal models is not trivial due to the combinatorial complexity given by the possibility of missing modalities. We propose a strategy to train multimodal models, which successfully achieves improved performance of different reconstruction models. Finally, extensive experiments have been carried out using an iCub humanoid robot, showing high performance in multiple reconstruction, prediction and imitation tasks.
Issue Date: Jan-2020
Date of Acceptance: 30-Sep-2019
DOI: 10.1016/j.robot.2019.103312
ISSN: 0921-8890
Publisher: Elsevier
Journal / Book Title: Robotics and Autonomous Systems
Volume: 123
Sponsor/Funder: Commission of the European Communities
Commission of the European Communities
Funder's Grant Number: 612139
Keywords: Science & Technology
Automation & Control Systems
Computer Science, Artificial Intelligence
Computer Science
Multimodal learning
Autonomous learning
Variational autoencoder
0801 Artificial Intelligence and Image Processing
0906 Electrical and Electronic Engineering
0913 Mechanical Engineering
Industrial Engineering & Automation
Publication Status: Published online
Open Access location:
Article Number: 103312
Online Publication Date: 2019-10-18
Appears in Collections:Computing
Electrical and Electronic Engineering
Faculty of Engineering

Unless otherwise indicated, items in Spiral are protected by copyright and are licensed under a Creative Commons Attribution NonCommercial NoDerivatives License.

Creative Commons