Predictive visuo-tactile object perception via robotic manipulation
File(s)
Author(s)
Dutta, Anirvan
Type
Thesis
Abstract
Exploring the properties of unknown objects, such as shape, inertia, and stiffness, is essential for autonomous robotic systems to operate safely and efficiently in unstructured environments. Precise estimation of these attributes is important for ensuring stable and well controlled manipulation as well as for anticipating the outcomes of prehensile and non-prehensile actions. Despite advances in robotics, current systems remain limited in their ability to perceive intrinsic object properties while performing contact-rich manipulation. In contrast, humans exhibit remarkable perception capabilities while grasping and manipulating objects, dynamically adjust their grip force and movement trajectory, compensating for sensorimotor delays through a seamless integration of vision and touch. Even before manipulation, our brains form detailed expectations about its weight, texture, and structural rigidity based on visual cues and experience.
A fundamental principle which explains such perceptual capability in humans is that of predictive processing, where it is hypothesised that humans learn internal models of their environment and continuously refine their top-down percepts (expectations) with real-time sensory feedback. This process enables inferring invariant causal environmental properties, such as consistent geometric shapes, edges, textures, along with associated uncertainties from high-dimensional visual and tactile observations despite variations in interaction.
Inspired by these insights, this thesis introduces a predictive perception framework for estimating key object properties through interactive robotic manipulation. The approach models object–robot interaction as a probabilistic Markov process and applies Bayesian inference for structured, robust estimation. Within this framework, three novel methods are proposed to tackle the perception of object shape, inertial parameters, and complex non-rigid attributes such as stiffness and surface friction. By explicitly modelling modality-specific noise, the framework integrates visual and tactile sensing coherently, enabling reliable estimation of object properties, as well as out-of-distribution prediction and active exploration. These capabilities represent a step towards general purpose robotic perception for downstream manipulation tasks.
A fundamental principle which explains such perceptual capability in humans is that of predictive processing, where it is hypothesised that humans learn internal models of their environment and continuously refine their top-down percepts (expectations) with real-time sensory feedback. This process enables inferring invariant causal environmental properties, such as consistent geometric shapes, edges, textures, along with associated uncertainties from high-dimensional visual and tactile observations despite variations in interaction.
Inspired by these insights, this thesis introduces a predictive perception framework for estimating key object properties through interactive robotic manipulation. The approach models object–robot interaction as a probabilistic Markov process and applies Bayesian inference for structured, robust estimation. Within this framework, three novel methods are proposed to tackle the perception of object shape, inertial parameters, and complex non-rigid attributes such as stiffness and surface friction. By explicitly modelling modality-specific noise, the framework integrates visual and tactile sensing coherently, enabling reliable estimation of object properties, as well as out-of-distribution prediction and active exploration. These capabilities represent a step towards general purpose robotic perception for downstream manipulation tasks.
Version
Open Access
Date Issued
2025-04-30
Date Awarded
2026-03-01
Copyright Statement
Attribution-Non Commercial-No Derivatives 4.0 International Licence (CC BY-NC-ND)
Advisor
Burdet, Etienne
Sponsor
European Union
BMW Group AG
Grant Number
INTUITIVE GA861166
PHASTRAC GA101092096
SmartNets GA860949
Publisher Department
Department of Bioengineering
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)