OCNLI: Original Chinese Natural Language Inference

Hai Hu; Kyle Richardson; Liang Xu; Lu Li; Sandra Kübler; Lawrence S. Moss

doi:10.18653/v1/2020.findings-emnlp.314

OCNLI: Original Chinese Natural Language Inference

Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, Lawrence S. Moss

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

66 Citations (Scopus)

4 Downloads (CityUHK Scholars)

Abstract

Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ∼56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (∼12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese natural language understanding. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language. © 2020 Association for Computational Linguistics

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics Findings of ACL
Subtitle of host publication	EMNLP 2020
Editors	Trevor Cohn, Yulan He, Yang Liu
Publisher	Association for Computational Linguistics
Pages	3512-3526
Number of pages	15
ISBN (Electronic)	978-1-952148-90-3
DOIs	https://doi.org/10.18653/v1/2020.findings-emnlp.314
Publication status	Published - Nov 2020
Externally published	Yes
Event	2020 Conference on Empirical Methods in Natural Language Processing - Virtual, Online Duration: 16 Nov 2020 → 20 Nov 2020 https://aclanthology.org/2020.emnlp-main.0/ https://aclanthology.org/volumes/2020.findings-emnlp/

Publication series

Name	Findings of the Association for Computational Linguistics Findings of ACL: EMNLP

Conference

Conference	2020 Conference on Empirical Methods in Natural Language Processing
Abbreviated title	EMNLP 2020
City	Virtual, Online
Period	16/11/20 → 20/11/20
Internet address	https://aclanthology.org/2020.emnlp-main.0/ https://aclanthology.org/volumes/2020.findings-emnlp/

Funding

This work was supported by the CLUE benchmark and the Grant-in-Aid of Doctoral Research from Indiana University Graduate School. Special thanks to the beaker team at AI2 for providing technical support for the beaker experiment platform. Computations on beaker.org were supported in part by credits from Google Cloud.

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2020.findings-emnlp.314

309079774.pdfFinal Published version, 664 KBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Hu, H., Richardson, K., Xu, L., Li, L., Kübler, S., & Moss, L. S. (2020). OCNLI: Original Chinese Natural Language Inference. In T. Cohn, Y. He, & Y. Liu (Eds.), Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020 (pp. 3512-3526). (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.314

Hu, Hai ; Richardson, Kyle ; Xu, Liang et al. / OCNLI : Original Chinese Natural Language Inference. Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. editor / Trevor Cohn ; Yulan He ; Yang Liu. Association for Computational Linguistics, 2020. pp. 3512-3526 (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP).

@inproceedings{f389ae8e441648eca6a7cd0b4d289068,

title = "OCNLI: Original Chinese Natural Language Inference",

abstract = "Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world{\textquoteright}s languages. In this paper, we present the first large-scale NLI dataset (consisting of ∼56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (∼12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese natural language understanding. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language. {\textcopyright} 2020 Association for Computational Linguistics",

author = "Hai Hu and Kyle Richardson and Liang Xu and Lu Li and Sandra K{\"u}bler and Moss, {Lawrence S.}",

year = "2020",

month = nov,

doi = "10.18653/v1/2020.findings-emnlp.314",

language = "English",

series = "Findings of the Association for Computational Linguistics Findings of ACL: EMNLP",

publisher = "Association for Computational Linguistics",

pages = "3512--3526",

editor = "Trevor Cohn and Yulan He and Yang Liu",

booktitle = "Findings of the Association for Computational Linguistics Findings of ACL",

address = "United States",

note = "2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020 ; Conference date: 16-11-2020 Through 20-11-2020",

url = "https://aclanthology.org/2020.emnlp-main.0/, https://aclanthology.org/volumes/2020.findings-emnlp/",

}

Hu, H, Richardson, K, Xu, L, Li, L, Kübler, S & Moss, LS 2020, OCNLI: Original Chinese Natural Language Inference. in T Cohn, Y He & Y Liu (eds), Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Findings of the Association for Computational Linguistics Findings of ACL: EMNLP, Association for Computational Linguistics, pp. 3512-3526, 2020 Conference on Empirical Methods in Natural Language Processing, Virtual, Online, 16/11/20. https://doi.org/10.18653/v1/2020.findings-emnlp.314

OCNLI: Original Chinese Natural Language Inference. / Hu, Hai; Richardson, Kyle; Xu, Liang et al.
Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. ed. / Trevor Cohn; Yulan He; Yang Liu. Association for Computational Linguistics, 2020. p. 3512-3526 (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - OCNLI

T2 - 2020 Conference on Empirical Methods in Natural Language Processing

AU - Hu, Hai

AU - Richardson, Kyle

AU - Xu, Liang

AU - Li, Lu

AU - Kübler, Sandra

AU - Moss, Lawrence S.

PY - 2020/11

Y1 - 2020/11

N2 - Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ∼56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (∼12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese natural language understanding. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language. © 2020 Association for Computational Linguistics

AB - Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ∼56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (∼12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese natural language understanding. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language. © 2020 Association for Computational Linguistics

UR - http://www.scopus.com/inward/record.url?scp=85101031367&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85101031367&origin=recordpage

U2 - 10.18653/v1/2020.findings-emnlp.314

DO - 10.18653/v1/2020.findings-emnlp.314

M3 - RGC 32 - Refereed conference paper (with host publication)

AN - SCOPUS:85101031367

T3 - Findings of the Association for Computational Linguistics Findings of ACL: EMNLP

SP - 3512

EP - 3526

BT - Findings of the Association for Computational Linguistics Findings of ACL

A2 - Cohn, Trevor

A2 - He, Yulan

A2 - Liu, Yang

PB - Association for Computational Linguistics

Y2 - 16 November 2020 through 20 November 2020

ER -

Hu H, Richardson K, Xu L, Li L, Kübler S, Moss LS. OCNLI: Original Chinese Natural Language Inference. In Cohn T, He Y, Liu Y, editors, Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Association for Computational Linguistics. 2020. p. 3512-3526. (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP). doi: 10.18653/v1/2020.findings-emnlp.314

OCNLI: Original Chinese Natural Language Inference

Abstract

Publication series

Conference

Funding

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this