Deep Learning multimodal fNIRS & EEG signals for bimanual grip force decoding
File(s)Ortega_2021_J._Neural_Eng._18_0460e6.pdf (5.91 MB)
Published version
Author(s)
Ortega, Pablo
Faisal, Aldo
Type
Journal Article
Abstract
Objective Non-invasive BMI offer an alternative, safe and accessible way to interact with the environment. To enable meaningful and stable physical interactions, BMIs need to decode forces. Although previously addressed in the unimanual case, controlling forces from both hands would enable BMI-users to perform a greater range of interactions. We here investigate the decoding of hand-specific forces. Approach We maximise cortical information by using EEG and fNIRS and developing a deep-learning architecture with attention and residual layers (cnnatt) to improve their fusion. Our task required participants to generate hand-specific force profiles on which we trained and tested our deep-learning and linear decoders. Main results The use of EEG and fNIRS improved the decoding of bimanual force and the deep-learning models outperformed the linear model. In both cases, the greatest gain in performance was due to the detection of force generation. In particular, the detection of forces was hand-specific and better for the right dominant hand and cnnatt was better at fusing EEG and fNIRS. Consequently, the study of cnnatt revealed that forces from each hand were differently encoded at the cortical level. cnnatt also revealed traces of the cortical activity being modulated by the level of force which was not previously found using linear models. Significance Our results can be applied to avoid hand-cross talk during hand force decoding to increase the robustness of BMI robotic devices. In particular, we improve the fusion of EEG and fNIRS signals and offer hand-specific interpretability of the encoded forces which are valuable during motor rehabilitation assessment.
Date Issued
2021-08-31
Date Acceptance
2021-08-04
Citation
Journal of Neural Engineering, 2021, 18, pp.1-21
ISSN
1741-2560
Publisher
IOP Publishing
Start Page
1
End Page
21
Journal / Book Title
Journal of Neural Engineering
Volume
18
Copyright Statement
© 2021 The Author(s). Published by IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
License URL
Sponsor
UK Research and Innovation
Identifier
https://iopscience.iop.org/article/10.1088/1741-2552/ac1ab3/meta
Grant Number
EP/V025449/1
Subjects
Science & Technology
Technology
Life Sciences & Biomedicine
Engineering, Biomedical
Neurosciences
Engineering
Neurosciences & Neurology
fNIRS
EEG
brain-machine interface
deep learning
bimanual force decoding
BRAIN-MACHINE INTERFACE
CORTICAL CONTROL
MOTOR
CORTEX
ACTIVATION
MOVEMENT
DYNAMICS
REPRESENTATION
OSCILLATIONS
HANDEDNESS
EEG
bimanual force decoding
brain-machine interface
deep learning
fNIRS
Biomedical Engineering
0903 Biomedical Engineering
1103 Clinical Sciences
1109 Neurosciences
Publication Status
Published
Date Publish Online
2021-08-31