Deep Learning-Enhanced Multimodal Data Transformation: From Nanoscale Image Resolution to Electronic Skin Voice Reconstruction

Abstract

This thesis explores the application of deep learning models to enhance multimodal data quality, particularly in the micro and nanoscale domains. In this research, “multimodal data” refers to the independent processing of different types of data, specifically, microscopic images and throat vibration signals, using generative deep learning algorithms to improve the quality of each modality. By leveraging deep learning’s ability to learn complex patterns and representations, the research aims to address quality improvement across these distinct data types. The study specifically targets two applications: enhancing visual image resolution and reconstructing voice data. Through these focused approaches, the research demonstrates how deep learning can transform low-quality data into high-quality outputs, effectively tackling unique challenges in data enhancement across these different modalities.

In the field of visual image enhancement, deep learning algorithms are employed to convert low-resolution optical images into high-resolution scanning electron microscope (SEM) images. This transformation enhances image resolution, thereby improving overall image quality. This study details the design and implementation of an optical super-resolution imaging system, scanning superlens microscopy (SSUM), featuring microsphere lenses combined with atomic force microscopy (AFM) probes. Achieving a remarkable 80nm resolution, surpassing optical diffraction limits, this system required additional enhancement for imaging comparable to SEM. Then, a cycle-consistent generative adversarial network (CycleGAN) model is trained on paired SSUM and SEM images to learn the mapping between domains. The model is then used to transform previously unseen SSUM test images. Quantitative analysis shows the reconstructed images achieve a mean peak signal-to-noise ratio 0.74 dB higher than the input SSUM images. Qualitative assessment further demonstrates the model’s ability to generate results with high structural detail. This technique overcomes key SEM constraints while preserving nanoscale resolution, promising wide applicability for challenges such as chip-level defect detection and biological sample analysis where coating or vacuum requirements pose obstacles.

In the field of voice processing, this dissertation introduces an AI-driven sensing technique for sound signal acquisition and reconstruction using a wearable acoustic resonance cavity throat sensor and deep learning methods, effectively converting vocal fold vibration signals into clear, high-fidelity human speech. This sensor exhibits high sensitivity across a broad dynamic pressure range, accurately capturing electrical signals generated by vocal fold and muscle movements. This research integrates a reference-based, supervised generative deep learning multi-task Conformer for both voice recognition and reconstruction from throat signals. Unlike traditional sensor-based voice recognition tasks that primarily involve classification, our model generates complete voice signals, thereby utilizing throat signals more comprehensively. In our multi-task Conformer model, voice recognition serves as an auxiliary task, while voice reconstruction is the primary task. This configuration not only improves the quality of voice reconstruction from throat signals but also optimizes overall performance, thereby enhancing human-machine interface interactions. This research also demonstrated that voice reconstruction and generation for arbitrary English words can be accomplished by training on a limited dataset along with fundamental English phoneme data. The results indicate that this deep learning-based throat vibration sensor achieved an accuracy rate of 86.6% in voice-to-text classification. Furthermore, the voice quality of normal voice signals, reconstructed from signals detected by the throat sensor, was subjectively rated with a mean opinion score (MOS) of 3.91. This breakthrough from traditional throat sensing techniques to AI-enabled perception of throat information significantly enhances signal acquisition and reconstruction, enabling efficient communication in complex and noisy environments, interactive gaming, multilingual translation devices, and sophisticated human-machine interfaces.

This research emphasizes the significance of transforming low-quality data into high-quality data. While obtaining low-quality data is relatively easy across various fields, acquiring high-quality data often presents challenges and comes with higher costs. To address this issue, this study employs a reference-based, supervised generative deep learning algorithm. This algorithm demonstrates outstanding performance in the field of optical super-resolution, producing higher-resolution images that provide more precise tools for observing the microscopic structure of materials with greater clarity and detail. Moreover, this reference-based generative deep learning approach is pivotal in tasks involving voice signal enhancement. It allows throat sensors to operate more effectively in complex and noisy environments, supporting secure communications, interactive gaming, multilingual translation, and sophisticated human-machine interfaces. Looking ahead, this research is poised to offer significant therapeutic and rehabilitative support to populations such as stroke survivors, individuals with amyotrophic lateral sclerosis (ALS), and those recovering from throat or vocal cord surgeries.

Date of Award	4 Sept 2024
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Wen Jung LI (Supervisor)