Overcoming Memory Constraint for Improved Target Classification Performance on Embedded Deep Learning Systems

Pattern recognition applications such as face recognition, detection of broken eggs, and classification of agricultural products are all using image classification in deep neural networks to improve the quality of services. However, traditional cloud inference models suffer from several problems such as network delay fluctuations and privacy leakage. In this regard, most real-Time applications currently need to be deployed on edge computing devices. Constrained by the computing power and memory limitations of edge devices, the use of an efficient memory manager for model reasoning is the key to improving the quality of service. This study firstly explored the incremental loading strategy of model weights for the model reasoning. Next, the memory space at runtime is optimized through data layout reorganization from the spatial dimension. In particular, our proposed schemes are orthogonal and transparent to the model. Experimental results demonstrate that the proposed approach reduced the memory consumption by 43.74% on average without additional reasoning time overhead.