HYBRID MULTIMODAL FUSION FRAMEWORK INTEGRATING EMG AND FORCE SIGNALS FOR ENHANCED HAND MOVEMENT PREDICTION

Abstract

Accurate prediction of hand movements is important to improve human–machine interaction, especially in rehabilitation and assistive applications. However, the nonlinearity of the electromyography (EMG) signal often limits the reliability of motion classification. It causes the sensor fusion to be unstable and not intelligent enough to continuously predict the user hand movements. To address this, we propose a hybrid multimodal fusion framework that integrates EMG and force signals to improve prediction accuracy and robustness. The framework investigates the relationship between forearm EMG signals, various grasping tasks, and finger/wrist joint angles. It goes beyond discrete classification to allow continuous motion intention prediction of wearable hand control. The proposed hybrid multimodal fusion framework has two levels: feature-level fusion and decision-level fusion. Canonical Correlation Analysis (CCA) is used to extract highly correlated features across modalities at the feature level. Linear Discriminant Analysis (LDA), Support Vector Machines (SVM), and Artificial Neural Networks (ANN) are used to classify these features to identify six different hand movements. To strengthening the decision consistency, majority voting is used at the classifier level. The performance of the system is evaluated based on the confusion matrices, accuracy, and F1-scores. Results show the proposed framework is significantly better than unimodal approaches, with the highest accuracy of 97.86% being achieved by the Waveform Length. Through experiments using data from 10 healthy subjects, it was established that multimodal fusion is effective in addressing the nonlinearity of the EMG signal, which results in more accurate hand gesture recognition. The findings assist in developing an efficient control scheme for wearable hand devices that provide smooth, user-intent-driven motions. By enhancing accuracy and responsiveness, the proposed approach improves support for activities of daily living (ADL), reducing the possibility of user dissatisfaction and the discontinuation of the device.