SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

RGB-D-based technology combines the advantages of RGB and depth sequences which can effectively recognize human actions in different environments. However, spatio-temporal information between modalities is difficult to learn from each other. To enhance exchange modalities, we introduce a SlowFast multimodality compensation block (SFMCB) designed extract features. Concretely, SFMCB fuses features two independent pathways with frame rates into single convolutional neural network achieve performance gains for model. Furthermore, explore fusion schemes combine feature rates. facilitate learning multiple pathways, loss functions are utilized joint optimization. evaluate effectiveness our proposed architecture, conducted experiments on four challenging datasets: NTU RGB+D 60, 120, THU-READ, PKU-MMD. Experimental results demonstrate model, utilizes mechanism capture complementary multimodal inputs.