Design and benchmarking of a multimodality sensor for robotic manipulation with GAN-based cross-modality interpretation
File(s)
Accepted version
Author(s)
Type
Journal Article
Abstract
In this article, we present the design and benchmark of an innovative sensor, ViTacTip, which fulfills the demand for advanced multimodal sensing in a compact design. A notable feature of ViTacTip is its transparent skin, which incorporates a “see-through-skin” mechanism. This mechanism aims at capturing detailed object features upon contact, significantly improving both vision-based and proximity perception capabilities. In parallel, the biomimetic tips embedded in the sensor's skin are designed to amplify contact details, thus substantially augmenting tactile and derived force perception abilities. To demonstrate the multimodal capabilities of ViTacTip, we developed a multitask learning model that enables simultaneous recognition of hardness, material, and textures. To assess the functionality and validate the versatility of ViTacTip, we conducted extensive benchmarking experiments, including object recognition, contact point detection, pose regression, and grating identification. To facilitate seamless switching between various sensing modalities, we employed a generative adversarial network (GAN)-based approach. This method enhances the applicability of the ViTacTip sensor across diverse environments by enabling cross-modality interpretation.
Date Issued
2025-01-01
Date Acceptance
2024-12-01
Citation
IEEE Transactions on Robotics, 2025, 41, pp.1278-1295
ISSN
1552-3098
Publisher
Institute of Electrical and Electronics Engineers
Start Page
1278
End Page
1295
Journal / Book Title
IEEE Transactions on Robotics
Volume
41
Copyright Statement
Copyright © 2025 IEEE. This is the author’s accepted manuscript made available under a CC-BY licence in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy)
License URL
Publication Status
Published
Date Publish Online
2025-01-07