Deep learning multimodal fNIRS and EEG signals for bimanual grip force decoding

Abstract Objective. Non-invasive brain-machine interfaces (BMIs) offer an alternative, safe and accessible way to interact with the environment. To enable meaningful stable physical interactions, BMIs need decode forces. Although previously addressed in unimanual case, controlling forces from both hands would BMI-users perform a greater range of interactions. We here investigate decoding hand-specific Approach. maximise cortical information by using electroencephalography (EEG) functional near-infrared spectroscopy (fNIRS) developing deep-learning architecture attention residual layers ( cnnatt ) improve their fusion. Our task required participants generate force profiles on which we trained tested our linear decoders. Main results. The use EEG fNIRS improved bimanual models outperformed model. In cases, greatest gain performance was due detection generation. particular, better for right dominant hand at fusing fNIRS. Consequently, study revealed that each were differently encoded level. Cnnatt also traces activity being modulated level not found models. Significance. results can be applied avoid hand-cross talk during robustness BMI robotic devices. fusion signals interpretability are valuable motor rehabilitation assessment.