Incorporating Side Information by Adaptive Convolution
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Pages (from-to) | 2897–2918 |
Journal / Publication | International Journal of Computer Vision |
Volume | 128 |
Issue number | 12 |
Online published | 2 Jul 2020 |
Publication status | Published - Dec 2020 |
Link(s)
Abstract
Computer vision tasks often have side information available that is helpful to solve the task. For example, for crowd counting, the camera perspective (e.g., camera angle and height) gives a clue about the appearance and scale of people in the scene. While side information has been shown to be useful for counting systems using traditional hand-crafted features, it has not been fully utilized in deep learning based counting systems. In order to incorporate the available side information, we propose an adaptive convolutional neural network (ACNN), where the convolution filter weights adapt to the current scene context via the side information. In particular, we model the filter weights as a low-dimensional manifold within the high-dimensional space of filter weights. The filter weights are generated using a learned “filter manifold” sub-network, whose input is the side information. With the help of side information and adaptive weights, the ACNN can disentangle the variations related to the side information, and extract discriminative features related to the current context (e.g. camera perspective, noise level, blur kernel parameters). We demonstrate the effectiveness of ACNN incorporating side information on 3 tasks: crowd counting, corrupted digit recognition, and image deblurring. Our experiments show that ACNN improves the performance compared to a plain CNN with a similar number of parameters and achieves similar or better than state-of-the-art performance on crowd counting task. Since existing crowd counting datasets do not contain ground-truth side information, we collect a new dataset with the ground-truth camera angle and height as the side information. We also perform ablation experiments, mainly for crowd counting, to study the helpfulness of the side information, and the effect of the placement of the adaptive convolutional layers in order to get insight about ACNNs.
Research Area(s)
- Convolutional neural network (CNN), Deep learning, Crowd counting
Citation Format(s)
Incorporating Side Information by Adaptive Convolution. / Kang, Di; Dhar, Debarun; Chan, Antoni B.
In: International Journal of Computer Vision, Vol. 128, No. 12, 12.2020, p. 2897–2918.
In: International Journal of Computer Vision, Vol. 128, No. 12, 12.2020, p. 2897–2918.
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review