On-Device Deep Multi-Task Inference via Multi-Task Zipping

Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks locally on-device. Yet the complexity of deep models needs to be trimmed down to fit in mobile storage and memory. Previous studies squeeze the redundancy within a single model. In this work, we reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. MTZ supports typical network layers and applies to inference tasks with different input domains. Evaluations show that MTZ can fully merge the hidden layers of two VGG-16 network. Moreover, MTZ can effectively merge nine residual networks for diverse inference tasks and models for different input domains. With the joint model merged by MTZ, the latency to switch between these tasks on memory-constrained devices is reduced by 8.71. © 2021 IEEE