Performance and Complexity Analysis of Bi-Directional Recurrent Neural Network Models Versus Volterra Nonlinear Equalizers in Digital Coherent Systems

We investigate the complexity and performance of recurrent neural network (RNN) models as post-processing units for the compensation of fibre nonlinearities in digital coherent systems carrying polarization multiplexed 16-QAM and 32-QAM signals. We evaluate three bi-directional RNN models, namely the bi-LSTM, bi-GRU and bi-Vanilla-RNN and show that all of them are promising nonlinearity compensators especially in dispersion unmanaged systems. Alpha s far as inference is concerned, omicron ur simulations show that the three models provide similar compensation performance, therefore, in real-life systems, the simplest scheme based on Vanilla-RNN units should be preferred. We compare bi-Vanilla-RNN in its many-to-many form with Volterra nonlinear equalizers and exhibit its superiority both in terms of performance and complexity, thus highlighting that RNN processing is a very promising pathway for the upgrade of long-haul optical communication systems utilizing coherent detection.

Automated summary

The study investigates the complexity and performance of RNN models as post-processing units for compensating fiber nonlinearities in digital coherent systems. The results show that Vanilla-RNN units are the preferred choice due to their simplicity. Bi-directional Vanilla-RNN outperforms Volterra nonlinear equalizers in terms of both performance and complexity, indicating that RNN processing is a promising pathway for upgrading long-haul optical communication systems.