Abstract
Nowadays, microscopy data analysis is highly valued in the field of biomedical research. The video data of biological samples collected during the process of microscopic imaging are obtained through the short-term continuous shooting of the same sample. Given the limited light/electron available to capture images, real-world microscopy data accordingly tend to be affected by the low signal-to-noise ratio (SNR). In general, microscopy data understanding is intended to learn representative noise-invariant features from noisy videos or images to recover high-quality data. In this connection, this thesis aims to clarify the following three key research questions concerning deep neural networks (DNN): a) how to improve the quality of microscopy videos by utilizing spatial and temporal information; b) how to estimate the per-pixel motion in microscopy videos; and c) how to achieve unsupervised microscopy image denoising in the absence of clean signals. To achieve these objectives, three empirical studies are further conducted with target data ranging from cryogenic electron microscopy (cryo-EM), transmission electron microscopy (TEM), and fluorescence microscopy (FM).More precisely, we focused on microscopy video enhancement tasks, especially those associated with low-SNR cryo-EM movies whose target is to recover a high-quality frame from the noisy frame and its neighboring frames. The existing video enhancement methods have the following two main problems. Firstly, supervised methods require clean ground truth to train the model, making the microscopy training data difficult or impossible to collect. Meanwhile, unsupervised methods that are inferior to supervised ones cannot be applied to low-SNR data. Secondly, they rely more on models based on convolutional neural networks (CNN) or recurrent neural networks (RNN) to extract spatial or temporal features to perform an enhancement. This feature makes them inadequate to deal with extremely low-SNR data. To solve the first problem, this thesis proposed a novel synthetic data pipeline to generate clean cryo-EM movie frames and multiple ground truth attributes. On this basis, the learned noise patterns were applied to clean frames through noise modeling for real data to obtain synthetic noisy cryo-EM data. Additionally, to alleviate the second problem, the proposed model extracted representative features from both spatial and temporal domains to characterize the low-SNR cryo-EM movies. Experimental results from synthetic and real cryo-EM datasets demonstrated the effectiveness and robustness of the proposed method relative to existing models.
Limited by the nature of end-to-end deep learning (DL) models, high-frequency details of the frames are also removed during the enhancement, resulting in the enhanced frames not being available for some downstream tasks, such as the reconstruction of high-resolution 3D models from 2D cryo-EM data. To overcome the foregoing limitation, we further explored the learning of the per-pixel motion of cryo-EM movies through DL-based approaches. The estimated motions can be used for frame alignment and lossless micrograph generation. Specifically, we proposed a multi-frame stacking module and a denoising module to extract noise-invariant features from the low-SNR movies to compute the correlation volume, thereby iteratively refining the optical flow from the computed volume by RNNs. Extensive experiments on synthetic and real cryo-EM datasets indicate that the proposed method is superior to the state-of-the-art alternatives.
In closing, we explored DL-based algorithms through a more challenging problem, which can be summarized as unsupervised microscopy image denoising without corresponding clean signals. Existing unsupervised and self-supervised denoising methods in the field of computer vision typically assume that the noise in data follows a Gaussian distribution and is signal- or pixel-independent. The related research, however, reveals that the signal-dependent Poisson shot noise dominates the microscopy data, which can be attributed to the limited light/electron available to capture images. Meanwhile, the noise is spatially correlated due to the data acquisition process. To tackle this problem, we proposed the use of a shatter to break the dependency of noise before denoising, training the model on the pairwise noisy images obtained from the video data with a well-designed unsupervised loss. Simply put, our model achieved optimal performance across a wide range of microscopy data types, encompassing cryo-EM, TEM, and FM.
As outlined above, we investigated the effective DL-based approaches related to the visual understanding of microscopy data in the field of computational biology. By innovatively developing task-specific extraction and fusion structures, the learned noise-invariant features contribute to a more comprehensive understanding of low-SNR microscopy data analysis. Moreover, the restored high-quality microscopy videos and images can advance various downstream tasks in biomedical research, such as classification, segmentation, and 3D reconstruction in the future. Particularly, high-quality cryo-EM data can be utilized to obtain structure determination of biomacromolecules, thereby facilitating structure-based drug design and fragment-based drug discovery.
| Date of Award | 29 Aug 2023 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | Wing Ho Howard LEUNG (Supervisor) & Qing Li (External Co-Supervisor) |