Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

3 Scopus Citations
View graph of relations

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Subtitle of host publicationCVPR 2021
PublisherInstitute of Electrical and Electronics Engineers, Inc.
Pages2392-2401
ISBN (electronic)9781665445092
ISBN (print)9781665445108
Publication statusPublished - 2021

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919
ISSN (electronic)2575-7075

Conference

Title2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)
LocationVirtual
Period19 - 25 June 2021

Abstract

Nested networks or slimmable networks are neural networks whose architectures can be adjusted instantly during testing time, e.g., based on computational constraints. Recent studies have focused on a “nested dropout” layer, which is able to order the nodes of a layer by importance during training, thus generating a nested set of subnetworks that are optimal for different configurations of resources. However, the dropout rate is fixed as a hyperparameter over different layers during the whole training process. Therefore, when nodes are removed, the performance decays in a human-specified trajectory rather than in a trajectory learned from data. Another drawback is the generated sub-networks are deterministic networks without well-calibrated uncertainty. To address these two problems, we develop a Bayesian approach to nested neural networks. We propose a variational ordering unit that draws samples for nested dropout at a low cost, from a proposed Downhill distribution, which provides useful gradients to the parameters of nested dropout. Based on this approach, we design a Bayesian nested neural network that learns the order knowledge of the node distributions. In experiments, we show that the proposed approach outperforms the nested network in terms of accuracy, calibration, and out-of-domain detection in classification tasks. It also outperforms the related approach on uncertainty-critical tasks in computer vision.

Research Area(s)

  • Neural Network Compression, Uncertainty Calibration

Bibliographic Note

Information for this record is supplemented by the author(s) concerned.

Citation Format(s)

Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression. / Cui, Yufei; Liu, Ziquan; Li, Qiao et al.
Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition: CVPR 2021. Institute of Electrical and Electronics Engineers, Inc., 2021. p. 2392-2401 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review