Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods

Ning Miao; Yuxuan Song; Hao Zhou; Lei Li

doi:10.18653/v1/2020.acl-main.314

Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods

Ning Miao, Yuxuan Song, Hao Zhou, Lei Li

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

4 Citations (Scopus)

Abstract

It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to underestimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at https://github.com/NingMiao/MC-tailor. © 2020 Association for Computational Linguistics.

Original language	English
Title of host publication	ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
Editors	Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Place of Publication	Stroudsburg, PA
Publisher	Association for Computational Linguistics
Pages	3436-3441
ISBN (Print)	9781952148255
DOIs	https://doi.org/10.18653/v1/2020.acl-main.314
Publication status	Published - Jul 2020
Externally published	Yes
Event	58th Annual Meeting of the Association for Computational Linguistics (ACL 2020) - Virtual, United States Duration: 5 Jul 2020 → 10 Jul 2020 https://acl2020.org/

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)
Abbreviated title	ACL2020
Country/Territory	United States
Period	5/07/20 → 10/07/20
Internet address	https://acl2020.org/

Access Output

10.18653/v1/2020.acl-main.314

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Miao, N., Song, Y., Zhou, H., & Li, L. (2020). Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (pp. 3436-3441). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.314

Miao, Ning ; Song, Yuxuan ; Zhou, Hao et al. / Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods. ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. editor / Dan Jurafsky ; Joyce Chai ; Natalie Schluter ; Joel Tetreault. Stroudsburg, PA : Association for Computational Linguistics, 2020. pp. 3436-3441 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{a02c616305c1418f845bf08e843c53de,

title = "Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods",

abstract = "It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to underestimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at https://github.com/NingMiao/MC-tailor. {\textcopyright} 2020 Association for Computational Linguistics.",

author = "Ning Miao and Yuxuan Song and Hao Zhou and Lei Li",

year = "2020",

month = jul,

doi = "10.18653/v1/2020.acl-main.314",

language = "English",

isbn = "9781952148255",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

pages = "3436--3441",

editor = "Dan Jurafsky and Joyce Chai and Natalie Schluter and Joel Tetreault",

booktitle = "ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference",

address = "United States",

note = "58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), ACL2020 ; Conference date: 05-07-2020 Through 10-07-2020",

url = "https://acl2020.org/",

}

Miao, N, Song, Y, Zhou, H & Li, L 2020, Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods. in D Jurafsky, J Chai, N Schluter & J Tetreault (eds), ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Stroudsburg, PA, pp. 3436-3441, 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), United States, 5/07/20. https://doi.org/10.18653/v1/2020.acl-main.314

Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods. / Miao, Ning; Song, Yuxuan; Zhou, Hao et al.
ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. ed. / Dan Jurafsky; Joyce Chai; Natalie Schluter; Joel Tetreault. Stroudsburg, PA: Association for Computational Linguistics, 2020. p. 3436-3441 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods

AU - Miao, Ning

AU - Song, Yuxuan

AU - Zhou, Hao

AU - Li, Lei

PY - 2020/7

Y1 - 2020/7

N2 - It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to underestimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at https://github.com/NingMiao/MC-tailor. © 2020 Association for Computational Linguistics.

AB - It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to underestimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at https://github.com/NingMiao/MC-tailor. © 2020 Association for Computational Linguistics.

UR - http://www.scopus.com/inward/record.url?scp=85098680414&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85098680414&origin=recordpage

U2 - 10.18653/v1/2020.acl-main.314

DO - 10.18653/v1/2020.acl-main.314

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9781952148255

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 3436

EP - 3441

BT - ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

A2 - Jurafsky, Dan

A2 - Chai, Joyce

A2 - Schluter, Natalie

A2 - Tetreault, Joel

PB - Association for Computational Linguistics

CY - Stroudsburg, PA

T2 - 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)

Y2 - 5 July 2020 through 10 July 2020

ER -

Miao N, Song Y, Zhou H, Li L. Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods. In Jurafsky D, Chai J, Schluter N, Tetreault J, editors, ACL 2020 - The 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Stroudsburg, PA: Association for Computational Linguistics. 2020. p. 3436-3441. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2020.acl-main.314

Do you have the right scissors? Tailoring pre-trained language models via Monte-Carlo methods

Abstract

Publication series

Conference

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this