Learning Locality and Isotropy in Dialogue Modeling

Research output: Conference Papers (RGC: 31A, 31B, 32, 33)32_Refereed conference paper (without host publication)peer-review

View graph of relations

Author(s)

  • Mingjie Zhan
  • Gangming Zhao
  • Shaoqing Lu
  • Ding Liang

Detail(s)

Original languageEnglish
Publication statusPublished - 2023

Conference

Title11th International Conference on Learning Representations (ICLR 2023)
LocationHybrid
PlaceRwanda
CityKigali
Period1 - 5 May 2023

Abstract

Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks. More in-depth analyses further confirm the effectiveness of our proposed approach. We release the code at https://github.com/hahahawu/SimDRC.

Citation Format(s)

Learning Locality and Isotropy in Dialogue Modeling. / Wu, Han; Tan, Haochen; Zhan, Mingjie et al.
2023. Paper presented at 11th International Conference on Learning Representations (ICLR 2023), Kigali, Rwanda.

Research output: Conference Papers (RGC: 31A, 31B, 32, 33)32_Refereed conference paper (without host publication)peer-review