MELA: Multilingual Evaluation of Linguistic Acceptability

Ziyin Zhang; Yikang Liu; Weifang Huang; Junyu Mao; Rui Wang; Hai Hu

doi:10.18653/v1/2024.acl-long.146

MELA: Multilingual Evaluation of Linguistic Acceptability

Ziyin Zhang (Co-first Author), Yikang Liu (Co-first Author), Weifang Huang, Junyu Mao, Rui Wang^*, Hai Hu^*

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

6 Citations (Scopus)

47 Downloads (CityUHK Scholars)

Abstract

In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability—MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. https://github.com/sjtu-compling/MELA. © 2024 Association for Computational Linguistics.

Original language	English
Title of host publication	Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics
Editors	Lun-Wei Ku, Andre Martins, Vivek Srikumar
Publisher	Association for Computational Linguistics
Pages	2658-2674
Volume	1: Long Papers
ISBN (Electronic)	9798891760943
DOIs	https://doi.org/10.18653/v1/2024.acl-long.146
Publication status	Published - Aug 2024
Externally published	Yes
Event	62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Bangkok, Thailand Duration: 11 Aug 2024 → 16 Aug 2024 https://2024.aclweb.org/

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
Place	Thailand
City	Bangkok
Period	11/08/24 → 16/08/24
Internet address	https://2024.aclweb.org/

Funding

We thank Bingjie Mao, Jinchi Jiang, Wen Shi, Jialin Guo, Chenhui Liu and Licen Liu for their help in data collection. We also appreciate the suggestions and comments from the anonymous reviewers. This study is supported by Shanghai Pujiang Program awarded to Hai Hu (22PJC063). Ziyin Zhang and Rui Wang are partially supported by the National Natural Science Foundation of China (62176153) and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102, as the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University)

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2024.acl-long.146

309042828.pdfFinal Published version, 720 KBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Zhang, Z., Liu, Y., Huang, W., Mao, J., Wang, R., & Hu, H. (2024). MELA: Multilingual Evaluation of Linguistic Acceptability. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers, pp. 2658-2674). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.146

Zhang, Ziyin ; Liu, Yikang ; Huang, Weifang et al. / MELA : Multilingual Evaluation of Linguistic Acceptability. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. editor / Lun-Wei Ku ; Andre Martins ; Vivek Srikumar. Vol. 1: Long Papers Association for Computational Linguistics, 2024. pp. 2658-2674 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{b3c7dc140f3645a9b2cee980eb157204,

title = "MELA: Multilingual Evaluation of Linguistic Acceptability",

abstract = "In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability—MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. https://github.com/sjtu-compling/MELA. {\textcopyright} 2024 Association for Computational Linguistics.",

author = "Ziyin Zhang and Yikang Liu and Weifang Huang and Junyu Mao and Rui Wang and Hai Hu",

year = "2024",

month = aug,

doi = "10.18653/v1/2024.acl-long.146",

language = "English",

volume = "1: Long Papers",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

pages = "2658--2674",

editor = "Lun-Wei Ku and Andre Martins and Vivek Srikumar",

booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics",

address = "United States",

note = "62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 ; Conference date: 11-08-2024 Through 16-08-2024",

url = "https://2024.aclweb.org/",

}

Zhang, Z, Liu, Y, Huang, W, Mao, J, Wang, R & Hu, H 2024, MELA: Multilingual Evaluation of Linguistic Acceptability. in L-W Ku, A Martins & V Srikumar (eds), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. vol. 1: Long Papers, Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, pp. 2658-2674, 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand, 11/08/24. https://doi.org/10.18653/v1/2024.acl-long.146

MELA: Multilingual Evaluation of Linguistic Acceptability. / Zhang, Ziyin (Co-first Author); Liu, Yikang (Co-first Author); Huang, Weifang et al.
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. ed. / Lun-Wei Ku; Andre Martins; Vivek Srikumar. Vol. 1: Long Papers Association for Computational Linguistics, 2024. p. 2658-2674 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - MELA

T2 - 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024

AU - Zhang, Ziyin

AU - Liu, Yikang

AU - Huang, Weifang

AU - Mao, Junyu

AU - Wang, Rui

AU - Hu, Hai

PY - 2024/8

Y1 - 2024/8

N2 - In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability—MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. https://github.com/sjtu-compling/MELA. © 2024 Association for Computational Linguistics.

AB - In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability—MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. https://github.com/sjtu-compling/MELA. © 2024 Association for Computational Linguistics.

UR - http://www.scopus.com/inward/record.url?scp=85204436622&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85204436622&origin=recordpage

U2 - 10.18653/v1/2024.acl-long.146

DO - 10.18653/v1/2024.acl-long.146

M3 - RGC 32 - Refereed conference paper (with host publication)

AN - SCOPUS:85204436622

VL - 1: Long Papers

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 2658

EP - 2674

BT - Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

A2 - Ku, Lun-Wei

A2 - Martins, Andre

A2 - Srikumar, Vivek

PB - Association for Computational Linguistics

Y2 - 11 August 2024 through 16 August 2024

ER -

Zhang Z, Liu Y, Huang W, Mao J, Wang R, Hu H. MELA: Multilingual Evaluation of Linguistic Acceptability. In Ku LW, Martins A, Srikumar V, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Vol. 1: Long Papers. Association for Computational Linguistics. 2024. p. 2658-2674. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2024.acl-long.146

MELA: Multilingual Evaluation of Linguistic Acceptability

Abstract

Publication series

Conference

Funding

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this