Towards Resource-Efficient Deep Learning on Edge and Embedded Systems

Abstract

The rapid proliferation of deep learning technologies has fundamentally transformed various aspects of modern life. Traditionally, deep learning models have been predominantly deployed on cloud servers due to their substantial computational requirements. However, the reliance on cloud infrastructure introduces challenges such as unstable network connections and potential privacy breaches, which hinder the broader adoption of deep learning in diverse applications. Consequently, there is a pressing need to shift deep learning models from the cloud to the edge, encompassing devices like single-board computers, mobile phones, and microcontroller units (MCUs). Edge and embedded devices, however, are typically constrained by limited hardware resources, including restricted memory size, computational capability, and power supply. These limitations often result in degraded model accuracy and delayed responses, thereby diminishing the efficacy of deep learning applications. This dissertation explores strategies to optimize the utilization of hardware resources on edge and embedded devices to support efficient deep learning applications through software-hardware co-design. Specifically, we first introduce a library tailored for the efficient deployment and execution of deep learning models on MCUs. This library addresses critical challenges in model compression and deployment by incorporating pruning unit selection, preexecution pruning optimizations, runtime acceleration, and post-execution low-cost storage. These innovations bridge the gap for the effective deployment and execution of pruned models on resource-constrained devices. Next, to further augment the deep learning capabilities of edge devices, we develop a preemptible multi-DNN inference framework designed to enhance the temporal efficiency of mobile edge GPUs in multi-task environments. A significant challenge in this context is the lack of preemptible scheduling support in GPUs, which is crucial for real-time multi-DNN inference systems. Our solution involves slicing DNN tasks into chunks, the smallest executable units within our runtime framework. Additionally, we integrate early-exit technologies to improve the framework’s efficiency, ensuring that preempted tasks can resume execution without missing their deadlines. Finally, we propose a dynamic voltage and frequency scaling (DVFS) design aimed at optimizing energy efficiency for small language models (SLMs) on mobile edge devices during inference. The primary challenge is understanding the dynamic relationship between frequency, power, and latency, which is influenced by the autoregressive nature of SLMs. To address this, we develop a formulation that predicts power and latency for SLM workloads, using matrix multiplication scale and device hardware data as key intermediaries. Based on these predictions, we design a governor that periodically adjusts processor frequencies, thereby minimizing energy consumption while ensuring timely token generation. In summary, this dissertation presents a comprehensive investigation into resource efficient deep learning on edge and embedded devices. By proposing innovative solutions and enabling insightful application designs, this work contributes to the advancement of deep learning technologies in resource-constrained environments.

Date of Award	15 Aug 2025
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Zhenjiang LI (Supervisor)