Improving Lightweight AdderNet via Distillation From ℓ₂to ℓ₁-norm

To achieve efficient inference with a hardware-friendly design, Adder Neural Networks (ANNs) are proposed to replace expensive multiplication operations in Convolutional Neural Networks (CNNs) with cheap additions through utilizing ℓ1 -norm for similarity measurement instead of cosine distance. However, we observe that there exists an increasing gap between CNNs and ANNs with reducing parameters, which cannot be eliminated by existing algorithms. In this paper, we present a simple yet effective Norm-Guided Distillation (NGD) method for ℓ1 -norm ANNs to learn superior performance from ℓ2 -norm ANNs. Although CNNs achieve similar accuracy with ℓ2 -norm ANNs, the clustering performance based on ℓ2 -distance can be easily learned by ℓ1 -norm ANNs compared with cross correlation in CNNs. The features in ℓ2 -norm ANNs are encouraged to achieve intra-class centralization and inter-class decentralization to amplify this advantage. Furthermore, the roughly estimated gradients in vanilla ANNs are modified to a progressive approximation from ℓ2 -norm to ℓ1 -norm so that a more accurate optimization can be achieved. Extensive evaluations on several benchmarks demonstrate the effectiveness of NGD on lightweight networks. For example, our method improves ANN by 10.43% with 0.25× GhostNet on CIFAR-100 and 3.1% with 1.0× GhostNet on ImageNet.

© 2023 IEEE