WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other processing tasks. As signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., universal representations all tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream WavLM jointly learns masked prediction and denoising pre-training. By this means, does not only keep content modeling capability by prediction, but also improves potential non-ASR denoising. In addition, employs gated relative position bias Transformer structure better capture sequence ordering of input speech. We scale up training dataset from 60k hours 94k hours. Large state-of-the-art performance on SUPERB benchmark, brings significant improvements various their representative benchmarks. The code models are available at https://aka.ms/wavlm.