Abstract
As the volume and complexity of data continue to grow at the edge, distributed learning is poised to play a pivotal role in enabling large-scale learning applications across various domains, including healthcare, the Industrial Internet of Things (IIoT), and finance. In a distributed learning setting, each client holds a subset of the data, and the models are trained by exchanging information among these clients. This approach offers several advantages, including improved model performance, scalability, privacy protection, and efficient utilization of distributed resources. By distributing the training process, the computational burden can be shared among multiple clients, allowing faster and more efficient model training. In addition, by keeping data decentralized and training models locally on individual clients, sensitive information can be safeguarded.Despite its benefits, distributed learning faces two critical challenges, particularly in wireless networks. First, as clients integrate more powerful computing resources (such as GPUs), they can train complex models with larger parameter sizes. However, the increasing size of model parameters poses challenges to scarce wireless communication resources (e.g., bandwidth and spectrum) for exchanging model parameters. Second, while exchanging model parameters (instead of raw data) provides some level of privacy protection, existing research has revealed that sharing model parameters or gradients can lead to privacy leakage. Adversaries may infer sensitive information about individual data samples by analyzing the model’s output or parameters. These challenges underscore the need for efficient communication strategies and robust privacy-preserving techniques in distributed learning deployments.
To address communication efficiency, this thesis firstly introduces a novel framework for dynamic gradient descent that collaboratively leverages both quantization and sparsification techniques. Specifically, we propose Adaptively-Compressed Stochastic Gradient Descent, which dynamically adjusts the number of quantization bits and sparsification size based on gradient norms, communication budgets, and remaining iterations. By combining these techniques, we aim to improve communication efficiency while maintaining model accuracy.
The heavy-tailed nature of gradient distributions in distributed deep learning poses challenges for existing compression methods. To address this, we propose a unique compression scheme that combines gradient truncation with quantization. This tailored approach, designed specifically for heavy-tailed gradients, is implemented within a communication-limited distributed SGD framework. We analyze the convergence error bound under both uniform and non-uniform quantization scenarios.
In order to safeguard privacy, we leverage the benefits of quantization, thereby enhancing the protection of sensitive information. Our innovative technique, known as Layered Randomized Quantization (LRQ), effectively compresses the updates of the raw model by utilizing a layer multishift coupler. By carefully adjusting the parameters of LRQ, we are able to shape the quantization error to conform to an anticipated Gaussian distribution, thereby guaranteeing differential privacy at the client level. Furthermore, we offer a comprehensive theoretical analysis that quantifies the trade-offs of LRQ in terms of communication, privacy, and convergence performance. Moreover, we bolster the convergence performance by incorporating dynamic private budget allocation and quantization bit allocation through an optimization formula.
In summary, this thesis addresses the challenges of communication efficiency and privacy protection in distributed learning through innovative compression strategies and optimization formulations. Our proposed approaches significantly improve model accuracy and provide valuable insights into the trade-offs inherent in distributed learning systems.
| Date of Award | 30 Aug 2024 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | Linqi SONG (Supervisor) |