Domain-Adaptive Pretraining Methods for Dialogue Understanding

Han Wu; Kun Xu; Linfeng Song; Lifeng Jin; Haisong Zhang; Linqi Song

doi:10.18653/v1/2021.acl-short.84

Domain-Adaptive Pretraining Methods for Dialogue Understanding

Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, Linqi Song

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

16 Citations (Scopus)

Abstract

Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

Original language	English
Title of host publication	Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing
Subtitle of host publication	Short Papers
Publisher	Association for Computational Linguistics
Pages	665-669
Number of pages	5
Volume	2
ISBN (Print)	978-1-954085-53-4
DOIs	https://doi.org/10.18653/v1/2021.acl-short.84
Publication status	Published - Aug 2021
Event	59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) - Virtual Duration: 1 Aug 2021 → 6 Aug 2021 https://2021.aclweb.org/

Publication series

Name	ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference
Volume	2

Conference

Conference	59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021)
Period	1/08/21 → 6/08/21
Internet address	https://2021.aclweb.org/

Access Output

10.18653/v1/2021.acl-short.84

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Wu, H., Xu, K., Song, L., Jin, L., Zhang, H., & Song, L. (2021). Domain-Adaptive Pretraining Methods for Dialogue Understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Short Papers (Vol. 2, pp. 665-669). (ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference; Vol. 2). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-short.84

Wu, Han ; Xu, Kun ; Song, Linfeng et al. / Domain-Adaptive Pretraining Methods for Dialogue Understanding. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Short Papers. Vol. 2 Association for Computational Linguistics, 2021. pp. 665-669 (ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference).

@inproceedings{7efa3e2b7427400ea85464e5cbda7614,

title = "Domain-Adaptive Pretraining Methods for Dialogue Understanding",

abstract = "Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.",

author = "Han Wu and Kun Xu and Linfeng Song and Lifeng Jin and Haisong Zhang and Linqi Song",

year = "2021",

month = aug,

doi = "10.18653/v1/2021.acl-short.84",

language = "English",

isbn = "978-1-954085-53-4",

volume = "2",

series = "ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference",

publisher = "Association for Computational Linguistics",

pages = "665--669",

booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing",

address = "United States",

note = "59th Annual Meeting of the Association for Computational Linguistics<br/>and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) ; Conference date: 01-08-2021 Through 06-08-2021",

url = "https://2021.aclweb.org/",

}

Wu, H, Xu, K, Song, L, Jin, L, Zhang, H & Song, L 2021, Domain-Adaptive Pretraining Methods for Dialogue Understanding. in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Short Papers. vol. 2, ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference, vol. 2, Association for Computational Linguistics, pp. 665-669, 59th Annual Meeting of the Association for Computational Linguistics
and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), 1/08/21. https://doi.org/10.18653/v1/2021.acl-short.84

Domain-Adaptive Pretraining Methods for Dialogue Understanding. / Wu, Han; Xu, Kun; Song, Linfeng et al.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Short Papers. Vol. 2 Association for Computational Linguistics, 2021. p. 665-669 (ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference; Vol. 2).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Domain-Adaptive Pretraining Methods for Dialogue Understanding

AU - Wu, Han

AU - Xu, Kun

AU - Song, Linfeng

AU - Jin, Lifeng

AU - Zhang, Haisong

AU - Song, Linqi

PY - 2021/8

Y1 - 2021/8

N2 - Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

AB - Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

UR - http://www.scopus.com/inward/record.url?scp=85122221651&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85122221651&origin=recordpage

U2 - 10.18653/v1/2021.acl-short.84

DO - 10.18653/v1/2021.acl-short.84

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-1-954085-53-4

VL - 2

T3 - ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference

SP - 665

EP - 669

BT - Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

PB - Association for Computational Linguistics

T2 - 59th Annual Meeting of the Association for Computational Linguistics<br/>and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021)

Y2 - 1 August 2021 through 6 August 2021

ER -

Wu H, Xu K, Song L, Jin L, Zhang H, Song L. Domain-Adaptive Pretraining Methods for Dialogue Understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Short Papers. Vol. 2. Association for Computational Linguistics. 2021. p. 665-669. (ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference). doi: 10.18653/v1/2021.acl-short.84

Domain-Adaptive Pretraining Methods for Dialogue Understanding

Abstract

Publication series

Conference

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this