## Abstract

We propose a time domain structure of channel estimation for coherent optical communication systems, which employs training sequence based equalizer and is transparent to arbitrary quadrature amplitude modulation (QAM) formats. Enabled with this methodology, 1.02Tb/s polarization division multiplexed 32QAM Nyquist pulse shaping signal with a net spectral efficiency of 7.46b/s/Hz is transmitted over standard single-mode fiber link with Erbium-doped fiber amplifier only amplification. After 1190km transmission, the average bit-error rate is lower than the 20% hard-decision forward error correction threshold of 1.5 × 10^{−2}. The transmission distance can be extended to 1428km by employing intra-subchannel nonlinear compensation with the digital back-propagation method.

© 2014 Optical Society of America

## 1. Introduction

Nyquist pulse shaping signal with high spectral efficiency modulation has been shown to be an efficient way towards large capacity optical transmission [1–3]. The polarization tracking in blind equalization for multi-level quadrature amplitude modulation (M-QAM) signal usually requires the constant modulus algorithm (CMA) for pre-convergence, and the cascaded multi-modulus algorithm (CMMA) and/or the decision-directed least mean square (DD-LMS) for stable-state operation [1, 2]. However, the above schemes are format-oriented and require the knowledge of the used modulation formats. For elastic optical networking, it is highly desired that adaptive spectral efficiency and modulation formats are employed [3]. Therefore, format-transparent digital signal processing (DSP) is required to reduce the complexity of hardware implementation. Recently, a so-called format-transparent scheme is proposed by employing quadrature phase shift keying (QPSK) symbols as the training and pilot symbols to realize the initialization stage of CMA and pilot aided polarization tracking in the receiver-side DSP [4].

The scheme of frequency domain equalization (FDE) is based on training sequences and offers a uniform frame to estimate transfer function of channel response for different M-QAM formats. FDE can be applied in the single carrier (SC) systems, which is the so called SCFDE technique [5–7]. Note that in the SCFDE scheme, the data is grouped into blocks and in each block the pilots are inserted to assist phase estimation. Moreover, cyclic extension is inserted into each block to combat with linear impairments such as chromatic dispersion (CD) and polarization mode dispersion (PMD). The insertion of cyclic prefix (CP) and cyclic suffix (CS) requires more redundancy that decreases the spectral efficiency. In [8], we develop a format-transparent digital coherent reception technique by performing training sequence based channel estimation in frequency domain and finite impulse response (FIR) filter based channel equalization in time domain. In this scheme, the cyclic extension is eliminated in the payload part of data symbols, which improves the spectral efficiency [8].

In this paper, we propose a training sequence based channel estimation and equalization scheme in time domain. It adopts similar frame structure like the SCFDE except for no cyclic extension in both training sequences and data payload. Different from [8], the new method obtains the channel response totally in time domain. Since it is based on the training sequences, the approach is basically format-transparent and suitable for parallel implementation and independent of carrier phase recovery. With this technique, we experimentally demonstrate the generation and long distance transmission of Terabit polarization division multiplexed (PDM)-32QAM signal with Nyquist pulse shaping.

Digital back-propagation (DBP) is a straightforward format-independent nonlinearity compensation technique [9, 10]. We also show the benefit of intra-subchannel DBP for PDM- 32QAM superchannel and statistically analyze signal distortions with and without DBP.

The paper is outlined as follows. In Section 2, we present the frame structure of the Nyquist signal and the principle of the training sequence based time domain channel estimation. Section 3 describes the experiment for the generation and transmission of 1.02Tb/s PDM-32QAM superchannel with 22subcarriers. The experimental results are reported after transmitting over standard single-mode fiber (SSMF) with Erbium-doped fiber amplifier (EDFA) only amplification. The benefit of nonlinear compensation with DBP applied in each subchannel is discussed. Finally, we summarize this paper in Section 4.

## 2. Time domain channel estimation

Figure 1 shows the frame structure of PDM Nyquist pulse shaping signals. The preamble performs synchronization and channel estimation. The synchronization sequences consist of two 63-symbol M-sequences and a 130-symbol zero sequence. The training sequences (TS) for channel estimation consist of two 127-symol M-sequences, each of which is followed by a 129-symbol zero sequence. Y polarization is delayed with X polarization by 128 symbols. Thus TS of the dual polarizations X and Y are ${P}^{x}(t)=[\begin{array}{cc}p(t)& 0\end{array}]$ and ${P}^{y}(t)=[\begin{array}{cc}0& p(t)\end{array}]$ with $p(t)$is a known M sequence. In each frame, 38400 data symbols are included after the preamble. We insert one pilot in every 63 data symbols to compensate for phase noise. The total length of the preamble is 768 symbols. Note that in contrast to frequency domain estimation and equalization [6, 7], cyclic extension is not required for both training sequences and data payload in our scheme of time domain equalization.

Figure 2 illustrates the DSP diagrams of the transmitter and the receiver in Nyquist pulse shaping systems. The information data is firstly mapped into M-QAM format and then packed into data frames at the transmitter. After inserting pilots into the data, the preamble including both synchronization and training sequences is inserted into the front of each frame. After 2 samples per symbol up-sampling, both in-phase and quadrature components of the signals are digitally shaped with root-raised-cosine (RRC) filters. After digital to analog convertors (DAC), electrical low-pass filters (ELPF) are used as anti-aliasing filters to remove out-of-band radiation. At the receiver, an FIR filter roughly compensates for the accumulated chromatic dispersion. Then the carrier frequency recovery is conducted by a phase increment estimation algorithm [11]. A matched receiving RRC filter is adopted to satisfy the first Nyquist criterion. After synchronization, the signals are re-sampled to 2 samples per symbol and the training sequences are picked up for linear channel estimation and equalization (LCEE). With LCEE, time domain FIR filtering are extracted from TS and convoluted with the data. The phase is corrected with pilots, and then estimated with the blind phase search (BPS) algorithm [1].

Figure 3 shows the schematic of the proposed linear channel estimation scheme. $h(t=jqT)$ with $j=-K\mathrm{...}K$ is a linear equalizer with $2K+1$ taps. The fractional sampling space is $qT$with $T$ the symbol duration and $q=1/2$. Thus LCEE are performed with 2-point down-sampling. The received training sequences in a polarization-diversity receiver are ${P}_{r}^{x}(t)$ and ${P}_{r}^{y}(t)$. The estimates from the linear equalizer output are ${P}_{e}^{x}(t)$ and ${P}_{e}^{y}(t)$.

The estimation error $\epsilon $ is defined as the difference between TS and the filter output,which is expressed as

The cost function contains an exponentially decaying multiplicative weighting factor ${\lambda}^{j-i}$ and is expressed as [12]

The equalizer coefficient $h(t)$ is updated by the recursive least square (RLS) algorithm according to Eq. (4) [12]. Once we have sounded the channel and its impulse response is known, we can perform polarization demultiplexing and subsequent channel equalization. Compared with the classic CMA based algorithms in blind equalization [1,2], the LCEE based on TS is basically format-transparent, suitable for parallel implementation and independent of carrier phase recovery. The multiplier number of LCEE with RLS in each update is calculated as $16{L}^{2}+20L+2$ where $L=2K+1$is the equalizer tap number. Assuming a tap length of ${L}_{1}$, the CMA requires $8{L}_{1}+8$ multipliers in each update. The CMMA with a tap length of ${L}_{2}$ requires $8{L}_{2}+14$ multipliers in each update. The DD-LMS with a tap length of ${L}_{3}$ requires $8{L}_{3}+4$ multipliers in each update.

Table 1 summarizes the computational complexity of LCEE and CMA + CMMA + DD-LMS. Note that the algorithms based on CMA are performed with all the data symbols, while LCEE is only performed with the training sequences. In this paper, there are ${N}_{training}$ = 512 training symbols and ${N}_{data}$ = 38400 data symbols in each frame. For transmission scenarios discussed in the next section, we choose $L=21$, ${L}_{1}=23$, ${L}_{2}=9$ and ${L}_{3}=15$to achieve similar performance for different equalization schemes. The computation requires about 100 multipliers per symbol for LCEE. In contrast, there are 402 multipliers per symbol for CMA + CMMA + DD-LMS. Therefore, compared with the conventional algorithms based on CMA, LCEE is usually computationally effective.

## 3. Experiment

To validate the proposed scheme, we build two sets of transmitters and carry out transmission experiments of 1.02Tb/s Nyquist PDM-32QAM signal. As shown in Fig. 4, the superchannel consists of 22 subchannels tightly spaced at 6.25GHz by interleaving the odd and the even subchannels with each other. The arbitrary waveform generators operating at 11.6GSample/s with 2-point DAC up-sampling generate baseband signals of 5.8Gbaud. The RRC filters with a roll-off factor of 0.01 are chosen for Nyquist pulse shaping. Analog ELPF with 3dB bandwidth of 4.4GHz are used as anti-aliasing filter. The multistage polarization-maintaining (PM) optical couplers (OC) are used to aggregate all the subchannels. PDM is emulated with a polarization beam splitter/combiner (PBS/PBC) and a tunable optical delay line. Consider hard-decision (HD) forward error correction (FEC) with a redundancy of 20%, the net bit rate is 1.02Tb/s (5.8Gbaud × 10bits/Symbol × 22 × 63/64/1.2/(1 + 768/38400)). The spectral efficiency is 7.46b/s/Hz (5.8Gbaud/6.25GHz × 10bits/Symbol × 63/64/1.2/(1 + 768/38400)). The preamble is 1.96% and the pilots are 1.56% in each frame, which are sufficient to track the time varying channel in our experiment. For practical application, we state that the redundancy ratio should be optimized towards the specific channel and would be similar to other training sequence based schemes [5–7].

The fiber loop has a length of about 238km, which consists of three spans of SSMF with EDFA only amplification. No inline CD compensation is used. In the loop, we apply an optical filter to suppress noise accumulation. In the coherent receiver, the signal and the LO laser are sent into the optical hybrid. Then four balanced detector (BD) are used to convert the beating signals into electrical domain before being sampled by analog to digital convertors (ADC) of the oscilloscope. A loop controller drives two optical switches and triggers the acquisition of the oscilloscope with a sampling rate of 80Gsample/s. For offline processing, the signals are first re-sampled to 10 samples per symbol. An FIR filter with 768 taps is used for CD compensation. After carrier frequency recovery and Nyquist matching filtering, the signals are synchronized and then re-sampled to 2 samples per symbol for time domain equalization. DBP, if switched on, will replace CD compensation. With the oversampling of 10 samples per symbol, we can achieve almost perfect matched filtering and synchronization, and moreover, avoid the aliasing of the newly generated distortions during DBP operation [9]. A PM-EDFA is used before the PDM module to control optical power launched into the fiber loop. Both the transmitter and the optical local oscillator (LO) lasers are tunable external cavity lasers (ECL). The selected subchannel for bit-error rate (BER) measurement uses an ECL with linewidth ~1kHz. While other surrounded subchannels and the LO are sourced by the ECLs with linewidth ~100kHz. For BER measurement, the errors are counted over ten PDM frames with a total of 3.84 × 10^{6} bits.

Figure 5 shows the back-to-back (BTB) BER as a function of optical signal to noise ratio (OSNR) with 0.1nm resolution for both single and super channels. The recovered PDM-32QAM constellations at an OSNR of 22.5dB for single channel and an OSNR of 38.6dB for superchannel are inserted, respectively. For comparison, the solid line gives the simulation results assuming ideal PDM-32QAM signals without laser phase noise distortion. Note that for superchannel OSNR measurement, we consider the total power of 22 subcarriers. The required OSNRs for single and super channels at a BER of 1.0 × 10^{−3} are about 17.7 and 31.9 dB, respectively. Compared to the simulation result, the implementation penalty is 1.4dB for single channel. The superchannel has a further penalty of 0.8dB for subcarrier aggregation.

Figure 6 shows the optimization of total launch power at the target transmission distance of 2380km for single channel and 1190km for superchannel scenarios. The central subchannel (Subch11) is selected to study superchannel operation. We also plot the result of employing DBP in each subchannel as a reference of intra-subchannel nonlinear compensation. The total launch power is optimized towards DBP. For each measurement, the virtual dispersion and nonlinear coefficients in the DBP processing are optimized to achieve the best performance. We also find that one DBP step for each fiber span is sufficient and further increase of DBP steps offers insignificant performance improvement. For single channel operation, at the optimum power of −3dBm the measured BER is 2.43 × 10^{−2} with LCEE, and 5.8 × 10^{−3} with DBP. The corresponding Q improvement over LCEE is 2.14dB for DBP. For superchannel operation, cross-phase modulation (XPM) between different subcarriers would dominate nonlinear distortions. Thus DBP that only consider intra-subchannel nonlinearities will bring moderate performance promotion. For instance, at the optimum power of 6dBm, the BER is 1.27 × 10^{−2} for LCEE and 0.98 × 10^{−2} for DBP. DBP offers a Q improvement of 0.375dB at the optimum launch power. To partially suppress XPM distortions, radio frequency (RF) pilot tone may be employed, which, however, requires a widen guard band and results in a cost of spectral efficiency [10].

Figure 7 shows BER dependence on transmission distance for both single and super channels operating at their respective optimum power. In Fig. 7, we also illustrate the BER threshold of 1.5 × 10^{−2} for 20% HD-FEC, which is equivalent to 6.73dB Q-factor [13]. For single channel transmission, DBP will improve BER performance and enable a maximum reach of 2618km. For superchannel system, the transmission reach is degraded to 1428km mainly due to XPM distortions. Figure 8 shows the optical spectrum and the OSNR degradation of superchannel transmission. The 1.02Tb/s Nyquist PDM-32QAM superchannel occupies 1.1nm bandwidth. The resolution is set as 0.01nm to see spectrum details of the subcarriers. After 1428km transmission, the measured OSNR of superchannel is 30.2dB on average at 6dBm total launch power.

Figure 9 and Fig. 10 respectively show the measured BER results of all 22 subchannels after 1190km and 1428km SSMF transmission with the optimal launch power of 6dBm. In Fig. 9, for the entire superchannel, the average BER is 1.36 × 10^{−2} after LCEE and 0.956 × 10^{−2} after DBP. For future implementation, the subcarriers may co-exist on one silicon chip. FEC would be encoded on a per-superchannel (instead of a per-subchannel) basis [1]. We thus compare the average BER of the whole superchannel with the HD-FEC limit. With LCEE, the average Q margin is about 0.15dB over a 20% HD-FEC threshold. DBP will increase Q margin by 0.51dB. In Fig. 10, the average BER is 2.0 × 10^{−2} if only LCEE is applied. After DBP, the average BER is 1.43 × 10^{−2} which is slightly below the HD-FEC threshold.

In Fig. 11, Figs. 11(a) and 11(b) show typical constellations after 1428km transmission for the central subchannel (Subch11) with LCEE and with DBP based nonlinear compensation, respectively. We further provide a statistical analysis of signal transmission distortions for constellation symbols with different optical filed amplitudes. Figure 12 shows the concentric rings for 32QAM constellation. The 32 constellation points are distinguished with fiver rings according to their Euler distances from the origin. The corresponding probability density functions (PDF) of signal distortions for different rings of 32QAM are shown in Fig. 13. We compare both real and imaginary components with Gaussian distribution fitting.

In Fig. 13, Figs. 13(a)–13(e) of the left column show PDFs with linear channel equalization, while Figs. 13(f)–13(j) of the right column show PDFs with DBP based nonlinear compensation. With LCEE only, the PDFs of Figs. 13(a)–13(c) closely follow Gaussian distribution, while the PDFs of Figs. 13(d) and 13(e) departure from Gaussian distribution moderately. This suggests that even for fiber link with no inline dispersion compensation, the signal distortions of high order modulation formats cannot always be considered as white noise, especially for the outer rings with larger amplitudes. As the net coding gain for the soft decision FEC is typically derived using the additive white Gaussian noise (AWGN) assumption [14], we employ the HD-FEC in our experiments due to the non-Gaussian divergence of the larger rings in Fig. 13.

## 4. Conclusion

In summary, a training sequence based time domain scheme is developed to perform channel estimation in coherent optical communication systems. With this methodology, Terabit Nyquist PDM-32QAM superchannel can be transmitted over 1190km SSMF with the average BER lower than the 20% HD-FEC threshold. With DBP based nonlinear compensation in each subchannel, the transmission distance can be extended to 1428km SSMF. The proposed method provides a time domain approach for format-independent channel estimation and equalization in single carrier systems with multi or hybrid modulation formats.

## Acknowledgment

This work is supported by National Natural Science Foundation of China (no. 61475004), National Basic Research Program of China (no. 2010CB328201), and National Hi-tech Research and Development Program of China (no. 2012AA011302). We also acknowledge the support from Program for New Century Excellent Talents in University. The authors acknowledge the valuable discussions with Chongjin Xie and Junho Cho from Bell-Labs, Alcatel-Lucent.

## References and links

**1. **X. Zhou, L. E. Nelson, P. Magill, R. Isaac, B. Zhu, D. W. Peckham, P. Borel, and K. Carlson, “PDM-Nyquist-32QAM for 450-Gb/s per-channel WDM transmission on the 50 GHz ITU-T grid,” J. Lightwave Technol. **30**(4), 553–559 (2012). [CrossRef]

**2. **X. Zhou, J. Yu, M.-F. Huang, Y. Shao, T. Wang, L. Nelson, P. Magill, M. Birk, P. I. Borel, D. W. Peckham, R. Lingle Jr, and B. Zhu, “64-Tb/s, 8 b/s/Hz, PDM-36QAM transmission over 320 km using both pre- and post-transmission digital signal processing,” J. Lightwave Technol. **29**(4), 571–577 (2011). [CrossRef]

**3. **X. Zhou and L. Nelson, “400G WDM transmission on the 50GHz grid for future optical networks,” J. Lightwave Technol. **30**(24), 3779–3792 (2012). [CrossRef]

**4. **Q. Zhuge, M. Morsy-Osman, M. Chagnon, X. Xu, M. Qiu, and D. V. Plant, “Terabit bandwidth-adaptive transmission using low-complexity format-transparent digital signal processing,” Opt. Express **22**(3), 2278–2288 (2014), http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-22-3-2278. [CrossRef] [PubMed]

**5. **R. Kudo, T. Kobayashi, K. Ishihara, Y. Takatori, A. Sano, and Y. Miyamoto, “Coherent optical single carrier transmission using overlap frequency domain equalization for long-haul optical systems,” J. Lightwave Technol. **27**(16), 3721–3728 (2009). [CrossRef]

**6. **J. Li, C. Zhao, S. Zhang, F. Zhang, and Z. Chen, “Experimental demonstration of 120-Gb/s PDM CO-SCFDE transmission over 317km SSMF,” IEEE Photon. Technol. Lett. **22**(24), 1814–1816 (2010). [CrossRef]

**7. **X. Liu, T. H. Wood, R. W. Tkach, and S. Chandrasekhar, “Demonstration of record sensitivities in optically preamplified receivers by combining PDM-QPSK and M-Ary pulse-position modulation,” J. Lightwave Technol. **30**(4), 406–413 (2012). [CrossRef]

**8. **Z. Zheng, R. Ding, F. Zhang, and Z. Chen, “1.76Tb/s Nyquist PDM 16QAM signal transmission over 714km SSMF with the modified SCFDE technique,” Opt. Express **21**(15), 17505–17511 (2013), http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-21-15-17505. [CrossRef] [PubMed]

**9. **E. F. Mateo and G. Li, “Compensation of interchannel nonlinearities using enhanced coupled equations for digital backward propagation,” Appl. Opt. **48**(25), F6–F10 (2009). [CrossRef] [PubMed]

**10. **T. Kobayashi, A. Sano, A. Matsuura, Y. Miyamoto, and K. Ishihara, “Nonlinear tolerant spectrally-efficient transmission using PDM 64-QAM single carrier FDM with digital pilot-tone,” J. Lightwave Technol. **30**(24), 3805–3815 (2012). [CrossRef]

**11. **A. Leven, N. Kaneda, U. Koc, and Y.-K. Chen, “Frequency estimation in intradyne reception,” IEEE Photon. Technol. Lett. **19**(6), 366–368 (2007). [CrossRef]

**12. **J. G. Proakis, *Digital Communication* (McGraw-Hill, 2001), Edition IV, Chap.11.

**13. **R. Ryf, S. Randel, N. K. Fontaine, M. Montoliu, E. Burrows, S. Corteselli, S. Chandrasekhar, A. H. Gnauck, C. Xie, R.-J. Essiambre, P. J. Winzer, R. Delbue, P. Pupalaikis, A. Sureka, Y. Su, L. Gruner-Nielsen, R. V. Jensen, and R. Lingle, Jr., “32-bit/s/Hz spectral efficiency WDM transmission over 177-km few-mode fiber,” Optical Fiber Communication Conference, Anaheim, CA, March 2013, Postdeadline paper PDP5A.1. [CrossRef]

**14. **J. Cho, C. Xie, and P. J. Winzer, “Analysis of soft-decision FEC on non-AWGN channels,” Opt. Express **20**(7), 7915–7928 (2012), http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-20-7-7915. [CrossRef] [PubMed]