Deep learning for force decoding form multimodal non-invasive brain signals
File(s)
Author(s)
Ortega San Miguel, Pablo
Type
Thesis
Abstract
Brain-Computer Interfaces (BCIs) bypass the nerve-muscle communication pathways.
They offer people with disabilities new opportunities to assist, rehabilitate and restore their force control.
Such control allows us to stabilise our manipulations, making them precise and safe.
However, non-invasive BCIs, a surgery-free technology, have only made moderate progress in force decoding compared to invasive approaches.
In this thesis, we explore and develop deep learning (DL) force decoding methods for electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS): two signals reflecting complementary brain function information.
We first release HYGRiP, a dataset including ready-to-use multimodal brain (fNIRS and EEG) and unilateral hand signals (force and electromyography, EMG).
We show that fast force generation introduces non-stationarities that remarkably increase fNIRS variability.
Hence, we developed hemCNN, a convolutional neural network (CNN) method for fNIRS signals decoding and neuroimaging that outperforms conventional decoding approaches and reveals hand-specific cortical activations.
We continue generating HYGRiiP.
Here, both hands contract simultaneously but generate different dynamic forces.
With this multimodal dataset, we aim to answer two longstanding questions in non-invasive BCI: the possibility to decode dynamic forces and two simultaneous degrees of freedom (DoF).
We show that multimodal non-invasive brain signals reconstruct bimanual force better than either of the signals alone.
Our deep-fusion method further increases force reconstruction and better exploits signals with different physiological and physical origins.
We show that most of the reconstruction is due to the robust detection of hand-specific contraction, which outperforms previous linear and EEG methods.
Although we identified a limited hand-specific reconstruction of dynamic forces,
our deep-fusion method revealed traces of hand-specific force modulation in EEG and fNIRS, suggesting that force modulated signals are accessible without surgery.
However, current sensing systems are still too noisy to detect them on a single-trial basis.
Our results show that appropriately designed DL methods can improve fNIRS and multimodal signal processing for force decoding.
They offer people with disabilities new opportunities to assist, rehabilitate and restore their force control.
Such control allows us to stabilise our manipulations, making them precise and safe.
However, non-invasive BCIs, a surgery-free technology, have only made moderate progress in force decoding compared to invasive approaches.
In this thesis, we explore and develop deep learning (DL) force decoding methods for electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS): two signals reflecting complementary brain function information.
We first release HYGRiP, a dataset including ready-to-use multimodal brain (fNIRS and EEG) and unilateral hand signals (force and electromyography, EMG).
We show that fast force generation introduces non-stationarities that remarkably increase fNIRS variability.
Hence, we developed hemCNN, a convolutional neural network (CNN) method for fNIRS signals decoding and neuroimaging that outperforms conventional decoding approaches and reveals hand-specific cortical activations.
We continue generating HYGRiiP.
Here, both hands contract simultaneously but generate different dynamic forces.
With this multimodal dataset, we aim to answer two longstanding questions in non-invasive BCI: the possibility to decode dynamic forces and two simultaneous degrees of freedom (DoF).
We show that multimodal non-invasive brain signals reconstruct bimanual force better than either of the signals alone.
Our deep-fusion method further increases force reconstruction and better exploits signals with different physiological and physical origins.
We show that most of the reconstruction is due to the robust detection of hand-specific contraction, which outperforms previous linear and EEG methods.
Although we identified a limited hand-specific reconstruction of dynamic forces,
our deep-fusion method revealed traces of hand-specific force modulation in EEG and fNIRS, suggesting that force modulated signals are accessible without surgery.
However, current sensing systems are still too noisy to detect them on a single-trial basis.
Our results show that appropriately designed DL methods can improve fNIRS and multimodal signal processing for force decoding.
Version
Open Access
Date Issued
2021-03
Date Awarded
2021-10
Copyright Statement
Creative Commons Attribution NonCommercial Licence
License URL
Advisor
Faisal, Aldo
Sponsor
Engineering and Physical Sciences Research Council
Grant Number
EP/L016796/1
Publisher Department
Computing
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)