Research Article - (2026) Volume 3, Issue 2
Model Based Maximum Likelihood Detector for Optical Communications Systems Employing the Nonlinear Fourier Transform
Received Date: Mar 09, 2026 / Accepted Date: Apr 06, 2026 / Published Date: Apr 10, 2026
Copyright: ©2026 Aluf, O. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Citation: Aluf, O. (2026). Model Based Maximum Likelihood Detector for Optical Communications Systems Employing the Nonlinear Fourier Transform. Space Sci J, 3(2), 01-06
Abstract
Optical communication systems are implemented in many systems and employ the nonlinear fourier transform (NFT). The use of the model is based on a maximum likelihood detector. The fiber communication system is described by the NFT scheme and by Digital Signal Processing (DSP) architecture. We use the previously developed analytical model for the nonlinear frequency division multiplexing channel to develop a maximum likelihood detection scheme for both symbol-wise detection and (if possi- ble) sequence detection via the Viterbi algorithm with quantized observed samples.
Keywords
Maximum Likelihood (ML), Optical Communication, Nonlinear Fouriertransform (NFT), Digital Signal Processing (DSP), Symbol-Wise Detection, NPDE, NFDM, Zakharov-Shabat Eigenvalue, INFT, Symbol Mapping, Scaling, Pre-Compensation, Gb Insertion, Coherent Detection, Post-Compensation, ML Symbol Detection, Gaussian Noise, Delta-Correlated Gaussian Noise, B-Coefficient, Clipped Linear Mapping, Covariance, Pseudocovariance, RRC, ML Sequence Detection, WDM, MLD, Quantum Noise, Spatial Division Multiplexing (SDM), NLSE, Nonlinear Schrodinger Equation (NLSE)
Introduction
The Nonlinear Fourier Transform (NFT) is widely used in many scientific and industrial applications. It is a good mathematical technique to obtain the solution of nonlinear partial differential equations (NPDEs). One type of nonlinear partial differential equation is Nonlinear Schrodinger equation (NSE). The way NFT technique uses to solve NPDEs is by decomposing complex signals into nonlinear spectrum components. The nonlinear spectrum components are described as discrete eigenvalues and reflection coefficients. We use NFT techniques in fiber optics (FOs) to analyze non-dispersive solitons and nonlinear radiation. It is done directly by addressing nonlinearity with no previous methods to compensate. The capacity in fiber optics (FOs) is a very critical issue since the demands are increasing all the time. The fiber optics nonlinear effects influence the FOs spectral efficiency by limiting it. Nonlinearity compensation is the way to compensate for it and get high fiber optics capacity. Fiber optics suffer from nonlinearity due to the Kerr effect. In this paper, we first discuss the Zakharov - Shabat eigenvalue problem and then characterize the Nonlinear Frequency Division Multiplexing (NFDM) communication scheme. It is a standard scaled modulation scheme b for NFDM. The channel model includes the input and output model and the fiber optic channel law. The symbol wise Maximum Likelihood (ML) detection method is discussed for best performance. The best performances are achieved by using symbol-wise detection. Performances are improved by controlling the error rates (SER/ BER) and signal constellation architecture. The Digital Signal Processing (DSP) in the transmitter and receiver communication system internal functional blocks are described.
System Setup
Zakharov-Shabat Eigenvalue Problem
The forward NFT is computed via the so-called Zakharov-Shabat eigenvalue problem [1-3] for a thorough exposition to the subject. Briefly, the direct NFT is computed from specific solutions of the systems of equations [1]:


NFDM Communication Scheme
Here we shall use a standard scaled b-modulation scheme for NFDM previously used in multiple studies [4–7]. The basics of the scheme are shown in Figure 1. At the transmitter, the bit sequences are mapped into a symbol sequence, each having m-bits which are mapped into the initial loading spectrum consisting of Nsc


is done before transmitting the data through the fiber optic channel. The DSP receiver function is to retrieve information from the received time domain signal. Data retrieving is important, since it is distorted by noise and fiber optic nonlinearity effects. We have communication system opposite functionality blocks at the DSP receiver. The functionality blocks are coherent detection, NFT conversion, data post compensation, data scaling, and ML symbol detection.
Results
The Input Output Model and the Channel Law
The channel model for the b-modulated NFDM transmission was developed in. It reads:

where N
is the circular symmetric delta-correlated Gaussian noise and the last term in the r.h.s describes the processing noise [8]. The latter is important at high values of the input power, and since no analytical model exists for it, we shall not be able to compensate for it.
ML Symbol-Wise Detection
To enable symbol-wise detection, we need to linearize the input-output relation w.r.t. ASE noise. The result is [6]:

Note that both covariance and pseudocovariance matrices above depend on the whole sequence via the absolute value of the input loading spectrum uin
.
The Choice of the Scaling Function
Let us now discuss the choice of the scaling function u → bf(x) - Eq.(3). So far, the only function used in the literature was exponential squeeze mapping for which:

In other words, the relation is linear up to the maximum allowed amplitude of the loading spectrum, xmax = |u
|maxand is cut to this value for larger amplitudes. The reason why we use 1 − € as opposed to the maximum theoretical value of 1 for the clipping threshold is that most numerical NFT routines fail when the b-coefficient approaches this barrier (since this case corresponds to the infinite r coefficient and the infinite burst power) [9]. Therefore, we use a small offset €<< 1 where one can still rely on the numerical accuracy of the NFT routines. The first benefit of using this scaling is that since below the threshold the model (6), (7) becomes exact (in the same regime as (5) is valid) if one assumes:

where the average is taken over an independent and Identically Distributed (i.i.d) constellation symbols with zero mean and variance |cn|2 =
and over the nonlinear spectral bandwidth. This gives a typical scale to be used for the clipping amplitude xmax.
Simplified Covariance and Pseudocovariance Matrices
For symbol-wise detection one must only consider the diagonal elements of the covariance and pseudo-covariance matrices (7) when n = n′. The resulting expressions can be simplified when the nonlinear subcarriers are either non-overlapping (as in finite support carriers of ) or weakly overlapping like e.g. RRC. In this case, one can follow the approach of Ref. and notice that only the current subcarrier contributes to the spectral integrals (7) so that one can√ approximate |
when evaluating the variance and pseudovariance of the sent symbol cn. Thus, the channel becomes not only conditionally Gaussian but ISI-free with the variance and pseudovariance depending only on the sent symbol and the squeezing function:


This simplified model does capture the noise squeezing effect in the channel model for b coefficient higher energy constellation points have reduced noise variance.
ML Sequence Detection
The symbol-wise detection considered previously did not take into account correlations between different symbols and hence is suboptimal. For weakly overlapping symbols, one can approximate the covariance and pseudocovariance matrices as approximately diagonal, which signifies weak correlation between the noise for different symbols. Since the noise statistics conditioned on the input is Gaussian in our model, this signifies conditional independence of the symbols, so that sequence detection should amounts to the symbol-wise detection as considered in the previous subsection
Discussion
In optical communication the maximum achievable capacity is inspecting for getting the maximum data rate a fiber optic channel can transmit. We can get data rate which exceed 100 Tbps/fiber if we use wavelength division multiplexing (WDM) and coherent data technologies. If we use C and L bands then capacity maximization is achievable. It utilizes spatial division multiplexing (SDM) with multi-core fiber optics. The laser transmitter is an element of binary optical communication system and includes a likelihood ratio decision box. The optical communications and achieving optical performance are related the handshake between channel capacity and maximum likelihood detector (MLD). Quantum noise and atmospheric interferences limit the optical channel performances.
We want to transmit data through fiber optic channel with maximum data rate and minimal errors. The fiber optic channel capacity is limited by fiber nonlinearity (Kerr nonlinearity effect) and noises. The fiber nonlinearity increases when the data signal strength increases and create phenomenon of "capacity ceiling". There are few capacity enhancing technologies in use. The maximum likelihood detector (MLD) minimized the probability of errors by choosing the "most likely" transmitted signal which yield the actual received signal. The MLD detector performs a statistical calculation which is based on channel noise model and choose a value that satisfied m = arg max p(c/m), when c is the received signal and m is the signal that is sent. The nonlinear Fourier transform (NFT) is optical fiber advance signal processing technique that take care fiber nonlinearity as a source rather than a channel limitation. The channel signals are transformed into nonlinear spectral data and decouples the nonlinear Schrodinger equation (NLSE). By that we get managed Kerr nonlinearity which increase the spectral efficiency. NFDM method uses NFT to encode information data onto nonlinear spectral components. They are evolve linearity in the transform domain. The "eigenvalue communication" is NFT based communication and help to get the physical reality optical fibers [10-12].
References
- Shabat, A., & Zakharov, V. (1972). Exact theory of two-dimensional self-focusing and one-dimensional self-modulation of waves in nonlinear media. Sov. Phys. JETP, 34(1), 62.
- M. Yousefi and F. Kschischang, “Information transmission using the nonlinear Fourier transform, Part I–III,” IEEE Trans. Inf. Theory 60, 4312 (2014).
- Turitsyn, S. K., Prilepsky, J. E., Le, S. T., Wahls, S., Frumin,L. L., Kamalian, M., & Derevyanko, S. A. (2017). Nonlinear Fourier transform for optical data processing and transmission: advances and perspectives. Optica, 4(3), 307-322.
- Yangzhang, X., Aref, V., Le, S. T., Buelow, H., Lavery, D., & Bayvel, P. (2019). Dual-Polarization Non-Linear Frequency-Division Multiplexed Transmission With-Modulation. Journal of Lightwave Technology, 37(6), 1570-1578.
- X. Yangzhang, V. Aref, S. T. Le, H. Buelow, D. Lavery, andP. Bayvel, “Experimental Demonstration of Dualpolarisation NFDM Transmission with b-Modulation,” IEEE Photonics Technology Letters 31(11), 885–888 (2019).
- Derevyanko, S., Balogun, M., Aluf, O., Shepelsky, D., & Prilepsky, J. E. (2021). Channel model and the achievable information rates of the optical nonlinear frequency division-multiplexed systems employing continuous b-modulation. Optics Express, 29(5), 6384-6406.
- Balogun, M., & Derevyanko, S. (2021). Hermite-Gaussian nonlinear spectral carriers for optical communicatiosystems employing the nonlinear Fourier transform. IEEE Communications Letters, 26(1), 109-112.
- Pankratova, M., Vasylchenkova, A., Derevyanko, S. A., Chichkov, N. B., & Prilepsky, J. E. (2020). Signal-noise interaction in optical-fiber communication systems employing nonlinear frequency-division multiplexing. Physical Review Applied, 13(5), 054021.
- Wahls, S., Chimmalgi, S., & Prins, P. J. (2018). FNFT: A software library for computing nonlinear Fourier transforms. Journal of Open Source Software, 3(23), 597.
- Balogun, M., & Derevyanko, S. (2022). Enhancing the spectral efficiency of nonlinear frequency division multiplexing systems via Hermite-Gaussian subcarriers. Journal of Lightwave Technology, 40(18), 6071-6077.
- Wahls, S. (2017, February). Second order statistics of the scattering vector defining the dt nonlinear fourier transform. In SCC 2017; 11th International ITG Conference on Systems, Communications and Coding (pp. 1-6). VDE.
- Gui, T., Zhou, G., Lu, C., Lau, A. P. T., & Wahls, S. (2018).Nonlinear frequency division multiplexing with b-modulation: shifting the energy barrier. Optics express, 26(21), 27978-27990.

