Controllable Scene Generation from Natural Language

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Pages (from-to)122-131
Journal / PublicationProcedia Computer Science
Volume209
Online published7 Nov 2022
Publication statusPublished - 2022

Conference

Title2022 International Symposium on Biomimetic Intelligence and Robotics, ISBIR 2022
PlaceChina
CityYunnan
Period26 - 29 July 2022

Link(s)

Abstract

We propose a novel framework to generate recognizable scenes conditioned on natural language (NL) descriptions. The proposed modular approach decomposes the scene synthesis process into several manageable steps, in which it first infers a spatial layout of the desired scene from input descriptions by a spatial layout generator and generates the scene with a scene generator. Specifically, the proposed approach allows interactive tuning of the synthesized scene via NL, which helps to generate more complex and meaningful scenes, and to correct training errors or bias. We demonstrate the capability of the proposed approach on the challenging MS-COCO dataset and show that our approach can improve the quality of generated scenes, interpretability of the drawn scenes and semantic alignment to the input language descriptions.

Research Area(s)

  • discrete event system, natural language, scene generation

Citation Format(s)

Controllable Scene Generation from Natural Language. / Cheng, Yu; Sun, Zhiyong; Shi, Yan et al.
In: Procedia Computer Science, Vol. 209, 2022, p. 122-131.

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

Download Statistics

No data available