Moses: Exploiting Cross-device Transferable Features for On-device Tensor Program Optimization

Achieving efficient execution of machine learning models on mobile/edge devices has attracted significant attention recently. A key challenge is to generate high-performance tensor programs for each operator inside a DNN model efficiently. To this end, deep learning compilers have adopted auto-Tuning approaches such as Ansor. However, it is challenging to optimize tensor codes for mobile/edge devices by auto-Tuning due to limited time budgets and on-device resources. A key component of DNN compilers is the cost model that can predict the performance of each configuration on specific devices. However, current design of cost models cannot provide transferable features among different hardware accelerators efficiently and effectively. In this paper, we propose Moses, a simple yet efficient design based on the lottery ticket hypothesis, which fully takes advantage of the hardware-Agnostic features transferable to the target device via domain adaptation to optimize the time-consuming auto-Tuning process of DNN compiling on a new hardware platform. Compared with state-of-The-Art approaches, Moses achieves up to 1.53X efficiency gain in the search stage and 1.41X inference speedup on challenging DNN benchmarks. © 2023 ACM.