Systematic Pruning and Acceleration for TinyML on Extremely Weak IoT Devices

This paper introduces DTMM (Deploying TinyML Models on MCUs), a library for efficiently deploying and running machine learning (ML) models on low-power embedded devices such as microcontroller units (MCUs). The motivation for designing DTMM comes from the emerging field of tiny machine learning (TinyML), which explores extending the reach of machine learning to a large number of embedded devices such as MCUs to achieve ubiquitous intelligence. Since MCUs are low-end devices, it is often necessary to compress ML models by pruning enough weights before deploying the model. Although model pruning has been extensively studied on many computing platforms, two key issues with pruning methods are exacerbated on MCUs due to extremely limited memory resources and computing power: models need to be deeply compressed without compromising accuracy too much, and should also perform efficiently after pruning. So far, due to the lack of dedicated solutions, pruned models are only effective in terms of execution efficiency or accuracy on MCUs, but not both. In this paper, we find that pruned models have great potential for efficient deployment and execution on MCUs, but requires a systematic library to manage aspects of pruned models before, during, and after execution. Therefore, we propose DTMM, a specialized library with pruning unit selection, pre-execution pruning optimizations, runtime acceleration, and post-execution low-cost storage, to fill this gap. DTMM is integrated into TensorFlow Lite Micro via a custom sparse convolution operator that extends its standard inference pipeline. We develop a prototype system. Extensive experiments on various models show promising gains compared to state-of-the-art methods. © 2026 IEEE.