Multitask Learning From Augmented Auxiliary Data for Improving Speech Emotion Recognition

Despite the recent progress in speech emotion recognition (SER), state-of-the-art systems lack generalisation across different conditions. A key underlying reason for poor is scarcity of datasets, which a significant roadblock to designing robust machine learning (ML) models. Recent works SER focus on utilising multitask (MTL) methods improve by shared representations. However, most these studies propose MTL solutions with requirement meta labels auxiliary tasks, limits training systems. This paper proposes an framework (MTL-AUG) that learns generalised representations from augmented data. We utilise augmentation-type classification and unsupervised reconstruction as allow data without requiring any tasks. The semi-supervised nature MTL-AUG allows exploitation abundant unlabelled further boost performance SER. comprehensively evaluate proposed following settings: (1) within corpus, (2) cross-corpus cross-language, (3) noisy speech, (4) adversarial attacks. Our evaluations using widely used IEMOCAP, MSP-IMPROV, EMODB datasets show improved results compared existing methods.