Abstract
In the era of digital abundance, efficient management and compression of vast data, such as multimedia data and scientific data, have become imperative. Large-scale data compression techniques have emerged as indispensable technology across various domains, facilitating the storage, transmission, and analyses of immense volumes of information. Effectively processing this data to make it easier to store and distribute has become a significant challenge. This thesis explores two areas of large-scale data compression: video coding technologies based on the latest video standard and genome compression. It is structured into three main parts: 1) an all-zero block detection tailored for the latest video coding standard H.266/VVC; 2) a lightweight model for partition skipping decisions; 3) a complexity-configurable learning-based genome codec. The first two parts focus on encoding optimization methods for videos, while the third part addresses the efficient compression of genome data.In the first topic, a low-complexity method for early detection of all-zero blocks (AZB) is proposed to reduce the encoding complexity from transform and quantization phase. A two-stage detection method is introduced to detect potential zero-quantized coding blocks. The first stage devises a theoretical upper bound for the sum of absolute differences in residual signals to detect genuine all-zero blocks in the spatial domain. The second stage detects all-zero blocks in the frequency domain using a coefficient-level threshold. Additionally, an RD estimation method identifies pseudo all-zero blocks before trellis-coded quantization. By early identifying AZBs in both spatial and frequency domains, our method significantly saves transform and quantization time with negligible loss in encoding efficiency.
In the second part, the thesis further explores the method for low-complexity encoding through mode pruning techniques. Based on the AZB detection result, a partition decision model leveraging the AZB feature is introduced to reduce encoding complexity by pruning unnecessary partition temptation. Additionally, we extend the mode decision method for hybrid screen content images. By fully considering the characteristics of hybrid screen content images, we propose a method that combines frame-level text detection and coding tree unit-level screen content classification for adaptive decision-making, significantly reducing the complexity of the encoding process for such content.
In the third part, we explore high efficient compression method for genome data. We propose a configurable, learning-based genome compression codec that achieves lossless coding while supporting parallel coding mode, multi-stride coding mode, and bidirectional coding mode. This codec strikes an excellent balance between coding complexity and compression ratio. Specifically, we introduce the concept of group-of-bases (GoB) in genome compression, where each GoB can be encoded and decoded independently. To address the cold start dilemma and achieve high compression ratios, we employ a hybrid model, which consists of a Markov model and a learning-based model. Powered by the learning-based model, the codec effectively balances complexity and compression ratio while maintaining lossless coding capabilities.
In conclusion, this thesis improves the compression efficiency for large-scale data. 1) For conventional videos, the encoding complexity is reduced with all-zero block detection method and partition skip method. 2) encoding complexity is minimized by integrating semantic information into mode pruning decisions. 3) Genome data is effectively compressed using a configurable, learning-based contextual model. Extensive experimental results validate the effectiveness of these methods.
| Date of Award | 16 Aug 2024 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | Shiqi WANG (Supervisor) & Tak Wu Sam KWONG (External Co-Supervisor) |