Incorporating Side Information by Adaptive Convolution

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

58 Scopus Citations
View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publication31st Conference on Neural Information Processing Systems (NIPS 2017)
EditorsI. Guyon, S. Bengio, U.V. Luxburg
Pages3868-3878
Publication statusPublished - Dec 2017

Publication series

NameAdvances in Neural Information Processing Systems
ISSN (Print)1049-5258

Conference

Title31st Annual Conference on Neural Information Processing Systems (NIPS 2017)
LocationLong Beach Convention Center
PlaceUnited States
CityLong Beach
Period4 - 9 December 2017

Abstract

Computer vision tasks often have side information available that is helpful to solve the task. For example, for crowd counting, the camera perspective (e.g., camera angle and height) gives a clue about the appearance and scale of people in the scene. While side information has been shown to be useful for counting systems using traditional hand-crafted features, it has not been fully utilized in counting systems based on deep learning. In order to incorporate the available side information, we propose an adaptive convolutional neural network (ACNN), where the convolution filter weights adapt to the current scene context via the side information. In particular, we model the filter weights as a low-dimensional manifold within the high-dimensional space of filter weights. The filter weights are generated using a learned "filter manifold" sub-network, whose input is the side information. With the help of side information and adaptive weights, the ACNN can disentangle the variations related to the side information, and extract discriminative features related to the current context (e.g. camera perspective, noise level, blur kernel parameters). We demonstrate the effectiveness of ACNN incorporating side information on 3 tasks: crowd counting, corrupted digit recognition, and image deblurring. Our experiments show that ACNN improves the performance compared to a plain CNN with a similar number of parameters. Since existing crowd counting datasets do not contain ground-truth side information, we collect a new dataset with the ground-truth camera angle and height as the side information.

Citation Format(s)

Incorporating Side Information by Adaptive Convolution. / Kang, Di; Dhar, Debarun; Chan, Antoni B.
31st Conference on Neural Information Processing Systems (NIPS 2017). ed. / I. Guyon; S. Bengio; U.V. Luxburg. 2017. p. 3868-3878 (Advances in Neural Information Processing Systems).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review