SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing

Paralinguistic speech processing is important in addressing many issues, such as sentiment and neurocognitive disorder analyses. Recently, Transformer has achieved remarkable success the natural language field demonstrated its adaptation to speech. However, previous works on have not incorporated properties of speech, leaving full potential unexplored. In this paper, we consider characteristics propose a general structure-based framework, called SpeechFormer++, for paralinguistic processing. More concretely, following component relationship signal, design unit encoder model intra- inter-unit information (i.e., frames, phones, words) efficiently. According hierarchical relationship, utilize merging blocks generate features at different granularities, which consistent with structural pattern signal. Moreover, word introduced integrate word-grained into each encoder, effectively balances fine-grained coarse-grained information. SpeechFormer++ evaluated emotion recognition (IEMOCAP & MELD), depression classification (DAIC-WOZ) Alzheimer's disease detection (Pitt) tasks. The results show that outperforms standard while greatly reducing computational cost. Furthermore, it delivers superior compared state-of-the-art approaches.