Coarse-to-fine learning for multi-pipette localisation in robot-assisted in vivo patch-clamp
File(s)
Accepted version
Author(s)
Type
Conference Paper
Abstract
In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios. To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assissted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved >98% accuracy within 10μm, and > 89% accuracy within 5μm for the localisation of multi-pipette tips. The average MSE is 2.52 μm.
Date Issued
2025-11-27
Date Acceptance
2025-10-01
Citation
2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025, pp.20877-20884
ISBN
979-8-3315-4393-8
ISSN
2153-0858
Publisher
IEEE
Start Page
20877
End Page
20884
Journal / Book Title
2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Copyright Statement
© 2025 IEEE. This is the author’s accepted manuscript made available under a CC-BY licence in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy)
License URL
Source
2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Publication Status
Published
Start Date
2025-10-19
Finish Date
2025-10-25
Coverage Spatial
Hangzhou, China