Attention-aided partial bidirectional RNN-based nonlinear equalizer in coherent optical systems

We leverage the attention mechanism to investigate and comprehend the contribution of each input symbol of the input sequence and their hidden representations for predicting the received symbol in the bidirectional recurrent neural network (BRNN)-based nonlinear equalizer. In this paper, we propose an attention-aided novel design of a partial BRNN-based nonlinear equalizer, and evaluate with both LSTM and GRU units in a single-channel DP-64QAM 30Gbaud coherent optical communication systems of 20 x 50 km standard single-mode fiber (SSMF) spans. Our approach maintains the Q-factor performance of the baseline equalizer with a significant complexity reduction of similar to 56.16% in the number of real multiplications required to equalize per symbol (RMpS). In comparison of the performance under similar complexity, our approach outperforms the baseline by similar to 0.2dB to similar to 0.25dB at the optimal transmit power, and similar to 0.3dB to similar to 0.45dB towards the more nonlinear region. Published by Optica Publishing Group under the terms of the Creative Commons Attribution 4.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Automated summary

In this paper, we utilize the attention mechanism to investigate the contribution of each input symbol and their hidden representations in predicting the received symbol in a bidirectional recurrent neural network-based nonlinear equalizer. We propose an attention-aided partial BRNN-based nonlinear equalizer design that achieves a significant complexity reduction while maintaining the performance of the baseline equalizer.