Complex-Valued Neural Network Design for Mitigation of Signal Distortions in Optical Links

Nonlinearity compensation is considered as a key enabler to increase channel transmission rates in the installed optical communication systems. Recently, data-driven approaches - motivated by modern machine learning techniques - have been proposed for optical communications in place of traditional model-based counterparts. In particular, the application of neural networks (NN) allows improving the performance of complex modern fiber-optic systems without relying on any a priori knowledge of their specific parameters. In this work, we introduce a novel design of complex-valued NN for optical systems and examine its performance in standard single mode fiber (SSMF) and large effective-area fiber (LEAF) links operating in relatively high nonlinear regime. First, we present a methodology to design a new type of NN based on the assumption that the channel model is more accurate in the nonlinear regime. Second, we implement a Bayesian optimizer to jointly adapt the size of the NN and its number of input taps depending on the different fiber properties and total length. Finally, the proposed NN is numerically and experimentally validated showing an improvement of 1.7 dB in the linear regime, 2.04 dB at the optimal optical power and 2.61 at the max available power on Q-factor when transmitting a WDM 30 x 200G DP-16QAM signal over a 612 km SSMF legacy link. The results highlight that the NN is able to mitigate not only part of the nonlinear impairments caused by optical fiber propagation but also imperfections resulting from using low-cost legacy transceiver components, such as digital-to-analog converter (DAC) and Mach-Zehnder modulator.

Automated summary

Nonlinearity compensation is crucial for increasing channel transmission rates in optical communication systems. Data-driven approaches using neural networks have shown to improve the performance of complex fiber-optic systems without prior knowledge of specific parameters. The proposed neural network design, optimized using a Bayesian optimizer, has successfully demonstrated improved performance in both linear and nonlinear regimes of fiber-optic communication.