Over the past decades, lithium niobate (LiNbO₃, LN) has been developed as a historical material for integrated optics and photonics, owing to its unique linear and nonlinear properties, including high refractive index (n_e = 2.1 at 1.55 μm wavelength), large electro-optical coefficient (r₃₃ = 33 pm/V), large second-order nonlinear susceptibility (χ⁽²⁾ = 30 pm/V) and wide optical transparency window (0.4-5 μm). Fully exploiting the low-loss window (~1.2 μm to 1.7 μm) of silica-based optical fiber, LN devices with variable functionalities have been developed in fiber-optic communications systems. Passive LN waveguides can perform wavefront manipulation and wavelength selection through different structures such as lenses, linear polarizers (LPs), and Bragg gratings. In high-speed digital communication applications, LN electro-optical modulators provide a very high intrinsic modulation bandwidth and minimized dispersion effects. In nonlinear optics, periodically poled lithium niobate (PPLN) could compensate the relative phase mismatch between interacting optical waves and present efficient wavelength converters. Waveguide amplifiers have also been realized by doping rare-earth (RE) elements into the bulk LN crystal. These devices demonstrate stable performance in different environments, long lifetimes, flexible designs, and the possibility of integration in multi-functional chips.

Recent years have witnessed the rapid development of LN-on-insulator (LNOI) technology, with a sub-micron thickness thin-film LN layer bonded on top of silica (SiO₂) substrate, leading to a high-index-contrast of ~0.7, without affecting any intrinsic properties of the LN material. Benefitting from advanced lithography and nano-machining technologies, the much stronger light confinement in LNOI could dramatically shrink the device footprints to micron-scale from those in several-centimeter-long bulk LN devices. On the other hand, the stronger light confinement could further increase the field strength and power density in the optical waveguides, leading to much more effective usage of photonic properties (e.g., nonlinear coefficient) in LNOI based devices. Since the first achievement of high-Q microring resonators, a good number of ultra-compact, high-performance devices have been developed in the commercial LNOI platform, including high-speed, low-cost electro-optical modulators, wavelength converters, frequency-comb sources, photon-pair generators, on-chip spectrometers, and so on.

Although the applications of LNOI devices are expanding rapidly, some important functionalities for a photonic integrated circuit (PIC), such as polarization manipulation and optical amplification, are still absent in the LNOI platform. On one hand, LN is a kind of hard-to-etch material, and the etched waveguide sidewall slope is usually less than 75°, making refractive index engineering hard to achieve in the LNOI platform using the common nano-fabrication technologies. On the other hand, high-temperature annealing cannot be performed on an LNOI chip due to thermal expansion and pyroelectric issues. Therefore, the RE-doping process which is commonly used to achieve optical amplification in bulk LN cannot be applied directly in LN thin films.

In this thesis, waveguide theories and fabrication techniques in the LNOI platform have been investigated and optimized according to available equipment in the lab. Based on the developed fabrication techniques, passive devices including microring resonators and polarization rotator-splitters (PRSs), and active devices including erbium (Er) -doped waveguide amplifiers have been experimentally demonstrated in the LNOI platform.

The waveguide theory of LNOI microring resonators, and several parameters that would affect device performance, reflected in the quality factor (Q), have been investigated, including metallic-mask-induced sidewall roughness, chemical-mechanical polishing (CMP) processes and waveguide configurations such as waveguide geometries and bus-ring gap in a microring resonator. The parameters have been carefully optimized and finally an average propagation loss of 1 dB/cm in general devices, corresponding to a ~500,000 intrinsic Q (Qi), and >1 million maximum Qi in microring resonators has been demonstrated.

The mode evolution and polarization manipulation function in the LNOI PRS have been investigated and discussed. The mode evolution in the PRS device is achieved by two steps: an adiabatic polarization rotator and an adiabatic mode splitter, in which the input fundamental transverse-electric (TE₀) mode and transverse-magnetic (TM₀) mode are separated and then projected into the TE₀ modes of two output ports of the splitter. The PRS is designed with a half-etched rib waveguide configuration, so that only a single step etching process is required in device fabrication, which is highly compatible to the common fabrication techniques in the LNOI platform. The polarization rotating and splitting performance has been investigated first by imaging the output facets from the rotator and the splitter and then quantitatively characterized through the conversion efficiencies and polarization extinction ratios. Polarization cross talks of approximately -10 dB for both TM₀-input and TE0-input have been achieved in the PRS over a broad wavelength range from 1500 nm to 1630 nm. Furthermore, the PRS also show a potential operation range as wide as 500 nm from 1300 nm to 1800 nm according to simulation prediction.

The optical gains, losses, operation range of wavelength and signal power in the Er-doped LNOI (Er:LNOI) waveguide amplifier have been investigated and discussed. The waveguide amplifier was fabricated in 0.5 mol.% Er-doped LNOI wafer, in which the Er-doping into the thin-film LN was completed during the crystal growth. A maximum on-chip optical net gain per unit length of 10.4 dB/cm has been achieved in the Er:LNOI waveguide amplifier at 1531.6 nm signal wavelength, with a relatively low 980-nm pump power of 20.9 mW. The Er:LNOI waveguide amplifier provides stable optical gain over the entire 1520-1570 nm wavelength range, attributed to the Er³⁺ emission. A net gain of 1.8 dB has also been achieved in the same waveguide amplifier at signal wavelength of 1550 nm. The operation range of signal power in the Er:LNOI waveguide amplifier has been further characterized by applying fixed pump power and increasing signal power. The waveguide amplifier provides stable optical net gain in a signal power range from <10 nW to ~100 μW, indicating a dynamic operation range of 50 dB supported by the Er:LNOI waveguide amplifier.

To summarize, fabrication technology for low-loss rib-like LNOI waveguide devices has been developed. Based on the fabrication technology, high-Q microring resonators, broadband PRSs and efficient Er-doped LNOI waveguide amplifiers have been experimentally demonstrated in the LNOI platform. The work described in this thesis could provide solutions for the long-awaited missing functionalities, including polarization manipulation and optical signal amplification, thus expanding the applications and functional degrees of LNOI photonic devices.