Major-Minor Long Short-Term Memory for Word-Level Language Model

Language model (LM) plays an important role in natural language processing (NLP) systems, such as machine translation, speech recognition, learning token embeddings, natural language generation, and text classification. Recently, the multilayer long short-term memory (LSTM) models have been demonstrated to achieve promising performance on word-level language modeling. For each LSTM layer, larger hidden size usually means more diverse semantic features, which enables the LM to perform better. However, we have observed that when a certain LSTM layer reaches a sufficiently large scale, the promotion of overall effect will slow down, as its hidden size increases. In this article, we analyze that an important factor leading to this phenomenon is the high correlation between the newly extended hidden states and the original hidden states, which hinders diverse feature expression of the LSTM. As a result, when the scale is large enough, simply lengthening the LSTM hidden states will cost tremendous extra parameters but has little effect. We propose a simple yet effective improvement on each LSTM layer consisting of a large-scale Major LSTM and a small-scale Minor LSTM to break the high correlation between the two parts of hidden states, which we call Major-Minor LSTMs (MMLSTMs). In experiments, we demonstrate the LM with MMLSTMs surpasses the existing state-of-the-art model on Penn Treebank (PTB) and WikiText-2 (WT2) data sets and outperforms the baseline by 3.3 points in perplexity on WikiText-103 data set without increasing model parameter counts.