Gated Hierarchical Attention for Image Captioning

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

10 Scopus Citations
View graph of relations

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationComputer Vision – ACCV 2018
Subtitle of host publication14th Asian Conference on Computer Vision: Revised Selected Papers
EditorsC. V. Jawahar, Hongdong Li, Greg Mori, Konrad Schindler
Pages21-37
VolumePart IV
ISBN (electronic)978-3-030-20870-7
Publication statusPublished - Dec 2018

Publication series

NameLecture Notes in Computer Science
Volume11364
ISSN (Print)0302-9743
ISSN (electronic)1611-3349

Conference

Title14th Asian Conference on Computer Vision (ACCV 2018)
Location
PlaceAustralia
CityPerth
Period2 - 6 December 2018

Abstract

Attention modules connecting encoder and decoders have been widely applied in the field of object recognition, image captioning, visual question answering and neural machine translation, and significantly improves the performance. In this paper, we propose a bottom-up gated hierarchical attention (GHA) mechanism for image captioning. Our proposed model employs a CNN as the decoder which is able to learn different concepts at different layers, and apparently, different concepts correspond to different areas of an image. Therefore, we develop the GHA in which low-level concepts are merged into high-level concepts and simultaneously low-level attended features pass to the top to make predictions. Our GHA significantly improves the performance of the model that only applies one level attention, e.g., the CIDEr score increases from 0.923 to 0.999, which is comparable to the state-of-the-art models that employ attributes boosting and reinforcement learning (RL). We also conduct extensive experiments to analyze the CNN decoder and our proposed GHA, and we find that deeper decoders cannot obtain better performance, and when the convolutional decoder becomes deeper the model is likely to collapse during training.

Research Area(s)

  • Hierarchical attention, Image captioning, Convolutional decoder

Citation Format(s)

Gated Hierarchical Attention for Image Captioning. / Wang, Qingzhong; Chan, Antoni B.
Computer Vision – ACCV 2018: 14th Asian Conference on Computer Vision: Revised Selected Papers. ed. / C. V. Jawahar; Hongdong Li; Greg Mori; Konrad Schindler. Vol. Part IV 2018. p. 21-37 (Lecture Notes in Computer Science; Vol. 11364).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review