Language-Adversarial Transfer Learning for Low-Resource Speech Recognition

The acoustic model trained using the knowledge from the shared hidden layer (SHL) model outperforms the model trained only by using the target language, especially under low resource conditions. However, the shared features may contain some unnecessary language dependent information. It will degrade the performance of the target model. Therefore, this paper proposes language-adversarial transfer learning to alleviate this problem. Adversarial learning is used to ensure that the shared layers of the SHL-model can learn more language invariant features. Experiments are conducted on IARPA Babel datasets. The results show that the target model trained using the knowledge transferred from the adversarial SHL-model achieves up to 10.1% relative word error rate reduction when compared with the target model trained using the knowledge transferred from the SHL-model.