Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

View graph of relations


  • Zhentao Tan
  • Qi Chu
  • Menglei Chai
  • Dongdong Chen
  • Qiankun Liu
  • Bin Liu
  • Gang Hua
  • Nenghai Yu

Related Research Unit(s)


Original languageEnglish
Number of pages18
Journal / PublicationIEEE Transactions on Pattern Analysis and Machine Intelligence
Online published27 Sep 2022
Publication statusOnline published - 27 Sep 2022


Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level or even instance-level multimodal results, still remains a challenge. In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at both semantics and instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. To further extend the user interaction function of the proposed method, we also introduce sketches into the network. In addition, specially designed generator modules, Progressive Growing Module and Multi-Scale Refinement Module, can be used as a general module to improve the performance of complex scene generation. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Codes are available at

Research Area(s)

  • Controllability, Generators, Image synthesis, Instance-level Diversity, Layout, Modulation, Probability distribution, Probability Distribution Modeling, Semantic Image Synthesis, Semantics