The need for large data capacity has been increasing tremendously to support future 5G wireless networks, Internet of Things, massive cloud infrastructure deployment and machine-to-machine communications. The growth of always-on broadband internet access network has put incredible pressure on the global communication network industry. Moreover, the maximum transmission capacity of a single fibre is approaching its limit of 100 Tb/s. In the next 15 years, Pb/s or even Eb/s optical communication will be needed to cope with the growth of the always-on internet network. To address this issue, a standard shift from single-core/mode to multicore/mode fibre is essential. In recent years, space division multiplexing (SDM) has attracted significant attention as a feasible solution to overcome the fundamental capacity limit of the optical system. Two types of SDM systems are being investigated in optical communication, namely, multicore fibre and mode division multiplexing (MDM). In multicore fibre, core multiplexing is performed by using a single strand of glass fibre that contains a number of independent single (-or multi) mode cores capable of optical communication signals. While, in the MDM system, each optical mode is regarded as an independent data channel that transmits information signal.

MDM is a breakthrough technology that satisfies the future capacity demand of optical network in combination with wavelength division multiplexing. The increasing number of modes indicates increased data capacity in the MDM system. Inventing different classes of mode-dependent devices has been intensively researched for MDM communication network. This thesis presents studies on several waveguide-based devices for MDM transmission system, namely high-order mode pass filter, switchable multifunction mode processing device, mode multiplexer and demultiplexer (mux/demux) and selective mode filter. These devices will find application in future MDM system to extend further the fibre transmission capacity and optical mode engineering.

Firstly, a unique concept to implement a high-order mode pass filter using mode converters is reported. Mode filter is an essential component in the MDM system and plays an essential role in filtering the undesired mode after demultiplexing different modes. Moreover, mode-pass and mode-rejection filters are needed to construct mode-controlling devices. Despite the importance of these devices, published research on high-order mode pass filter in photonic integrated circuit (PIC) platform is limited thus far. Our proposed design method implements high-order mode pass filter of any order, uses different mode converters available and applies to various PIC material platforms. We fabricate a broadband fundamental mode filter device using a Mach–Zehnder interferometer (MZI) and Y-junctions to demonstrate our idea. The performance of the fabricated device is demonstrated experimentally in the wavelength range of 1.530–1.565 µm (C-band). This filter exhibits a simulated extinction ratio of 37 dB with an excess loss of 0.52 dB for the first-order mode transmission.

Secondly, a switchable multifunction three-mode device for the MDM optical network is proposed. Switchable and flexible MDM system has received considerable attention recently for deploying advance and reconfigurable optical networks. The device can simultaneously function as low- and high-order mode block filters and a mode exchanger by selectively switching different working states. We fabricate the device using standard microfabrication technology. Moreover, the experimental performance of the device agrees reasonably well with the simulated performance. Particularly, the device offers a measured mode extinction ratio of 22.20 dB and 19.46 dB as a low-order mode block filter at 1.550 µm wavelength. The proposed device has great potential for use in a fully flexible and advanced MDM optical system.

Thirdly, a PIC-based three-mode mux/demux device that can multiplex and demultiplex signals that are carried by the fundamental and first- and second-order modes is proposed. This device consists of an unbalanced three-arm MZI sandwiched by three Y-junctions. The first Y-junction is used as a power splitter, whereas the second and third Y-junctions are connected back to back to form a three-mode interferometer. The performance of the device is demonstrated via simulation and experiment. For various multiplexing and demultiplexing conditions, the maximum simulated excess loss of the device is 0.37 dB with a minimum simulated extinction ratio of 25.71 dB at the 1.550 µm wavelength. The three-mode mux/demux is fabricated using standard polymer waveguide microfabrication process and characterised via a tuneable laser source in C-band. This device is compact and offers broadband operation, high extinction ratio and a simple structure.

Finally, a novel design architecture to realise a selective mode filter on the basis of the asymmetric directional coupler structure is proposed. In our proposed structure, any arbitrary high-order mode is allowed to pass, whereas other unwanted modes are blocked. The design architecture is scalable. Furthermore, multiple optical modes can be blocked by only adjusting the structural parameters. As a proof of concept, we experimentally demonstrated a three-mode device. In this device, the fundamental and first-order modes are blocked, and only the second-order mode is allowed to pass. In addition, the scalability of the proposed structure is demonstrated by another design of four-mode filter. In this structure, the third-order mode is allowed to pass, and other remaining modes are filtered out. We fabricate the devices using polymer material and characterise in the C-Band. Both the devices have negligible polarisation dependence. The experimental performance agrees reasonably well with the simulation. The proposed architecture offers scalability and high-design flexibility and has excellent potential to be used in advanced MDM network.