Integrating computer vision with neuromuscular interfacing for the semi-autonomous control of robotic limbs
File(s)
Author(s)
Hasbani, Milia Helena
Type
Thesis
Abstract
Grasping with an active prosthetic hand is a complex process leading to traditionally un-intuitive control where users cycle through grasp types using buttons, co-contraction, or thresholding. To overcome such limitations of state-of-the-art, we introduce a shared control strategy based on a Human Intention Detection system with assistance from a Computer Vision (CV) system. High-density surface EMG (HDsEMG) is used to control a prosthetic wrist and hand while the CV system analyses visual inputs to select a grasp type which is applied upon hand closure. Kinematic prediction Models were trained from concurrent HDsEMG and motion capture recordings to predict kinematics of wrist movements and closing and opening of the hand. We systematically investigated the complexity of the training movement sets used to train the models, resulting in successful predictions of kinematics when training with a reduced training set (r2 = 0.70, RMSE = 0.13 for a 3-Degree of Freedom (DoF) system). The cinematic Prediction Models were extended to an online system and validated by individuals with limb loss who were able to control a prosthesis. Participants successfully completed 100% of all target-reaching tasks with a cursor on a screen for 2-DoF movements. In parallel, a CV system was developed and trained to predict one of four grasp types (general, spherical, pinch, tripod) from the image of an object. The novel CV algorithm correctly classified 88% of grasps from unseen images in an offline setting and demonstrated an accuracy of 79% when used in real-time reaching
and grasping experiments with an eye-in-hand camera. The two systems were integrated into a realtime shared control framework for the semi-automatic control of a prosthetic hand. The CV system complements the user’s direct control and reduces the cognitive effort required to grasp objects, while still allowing the user to feel in natural control of the prosthesis.
and grasping experiments with an eye-in-hand camera. The two systems were integrated into a realtime shared control framework for the semi-automatic control of a prosthetic hand. The CV system complements the user’s direct control and reduces the cognitive effort required to grasp objects, while still allowing the user to feel in natural control of the prosthesis.
Version
Open Access
Date Issued
2023-02-08
Date Awarded
01/07/2023
Advisor
Farina, Dario
Sponsor
Engineering and Physical Sciences Research Council
Grant Number
EP/R513052/1
Publisher Department
Bioengineering
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)