SMF [5]. By exploiting orthogonal spatial and polarization modes of FMFs to transmit independent parallel data streams

 Abstract—Mode division multiplexing (MDM) has the potential of increasing the capacity of optical fiber transmission system. However, primary impaired factor mode-dependent loss (MDL) poses fundamental performance limitations. Recently MIMO space time coding technique has provided a potential solution to mitigate the MDL and hence started receiving considerable interests. In this paper, orthogonal space time block coding (OSTBC) based MDM transmission system was investigated to test its efficiency in mitigating MDL. Considering the importance and necessity of channel estimation for coherent receivers and the paucity of literature on this subject, MMSE channel estimation process was introduced in the MDM system to test the robustness and efficiency of the OSTBC-MDM system in mitigating the MDL. Furthermore, an alternative linear decoding (LD) method was achieved by transferring the orthogonal characteristics of codes in OSTBC to the channel elements in MDM system. The LD method has the same performance as the optimal ML method, excepting that it only exhibits linear computational complexity, in terms of the number of modes and the size of constellations. Simulation results were provided to validate the efficiency and robustness of the proposed OSTBC-MDM system in mitigating the MDL in the presence of MMSE channel estimation.