Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Yikang Liu; Wanyang Zhang; Yiming Wang; Jialong Tang; Pei Zhang; Baosong Yang; Fei Huang; Rui Wang; Hai Hu

doi:10.18653/v1/2025.emnlp-main.633

Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Yikang Liu
, Wanyang Zhang
, Yiming Wang
, Jialong Tang
, Pei Zhang
, Baosong Yang
, Fei Huang
, Rui Wang^*
, Hai Hu^*

^*Corresponding author for this work

Department of Linguistics and Translation

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Citation (Scopus)

2 Downloads (CityUHK Scholars)

Abstract

Translationese refers to linguistic properties that usually occur in translated texts. Previous works study translationese by framing it as a binary classification between original texts and translated texts. In this paper, we argue that translationese should be graded instead of binary and propose the first measure for translationese-the translationese-index (T-index), computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index can generalize to unseen genres, authors, and language pairs. Moreover, T-index computed using two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can effectively capture translationese, as demonstrated by alignment with human pointwise ratings and pairwise judgments. Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE. © 2025 Association for Computational Linguistics.

Original language	English
Title of host publication	Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Editors	Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
Publisher	ACL Anthology
Pages	12521-12538
ISBN (Print)	9798891763326
DOIs	https://doi.org/10.18653/v1/2025.emnlp-main.633
Publication status	Published - Nov 2025
Event	30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) - Suzhou, China Duration: 4 Nov 2025 → 9 Nov 2025 https://aclanthology.org/volumes/2025.emnlp-main/

Publication series

Name	EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Conference

Conference	30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)
Abbreviated title	30th EMNLP
Place	China
City	Suzhou
Period	4/11/25 → 9/11/25
Internet address	https://aclanthology.org/volumes/2025.emnlp-main/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. The Research Unit(s) information for this record is based on the then academic department affiliation of the author(s).

Funding

This work is funded by Shanghai Pujiang Program (22PJC063) awarded to Hai Hu and the General Program of National Natural Science Foundation of China (62176153) awarded to Rui Wang.

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2025.emnlp-main.633

494595225.pdfFinal Published version, 892 KBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Liu, Y., Zhang, W., Wang, Y., Tang, J., Zhang, P., Yang, B., Huang, F., Wang, R., & Hu, H. (2025). Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese. In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 12521-12538). (EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference). ACL Anthology. https://doi.org/10.18653/v1/2025.emnlp-main.633

Liu, Yikang ; Zhang, Wanyang ; Wang, Yiming et al. / Translationese-index : Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. editor / Christos Christodoulopoulos ; Tanmoy Chakraborty ; Carolyn Rose ; Violet Peng. ACL Anthology, 2025. pp. 12521-12538 (EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference).

@inproceedings{2e63439d1a8c4d1c9d6fbcc9c47aa758,

title = "Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese",

abstract = "Translationese refers to linguistic properties that usually occur in translated texts. Previous works study translationese by framing it as a binary classification between original texts and translated texts. In this paper, we argue that translationese should be graded instead of binary and propose the first measure for translationese-the translationese-index (T-index), computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index can generalize to unseen genres, authors, and language pairs. Moreover, T-index computed using two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can effectively capture translationese, as demonstrated by alignment with human pointwise ratings and pairwise judgments. Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE. {\textcopyright} 2025 Association for Computational Linguistics.",

author = "Yikang Liu and Wanyang Zhang and Yiming Wang and Jialong Tang and Pei Zhang and Baosong Yang and Fei Huang and Rui Wang and Hai Hu",

note = "Full text of this publication does not contain sufficient affiliation information. The Research Unit(s) information for this record is based on the then academic department affiliation of the author(s).; 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), 30th EMNLP ; Conference date: 04-11-2025 Through 09-11-2025",

year = "2025",

month = nov,

doi = "10.18653/v1/2025.emnlp-main.633",

language = "English",

isbn = "9798891763326",

series = "EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference",

publisher = "ACL Anthology",

pages = "12521--12538",

editor = "Christodoulopoulos, \{Christos \} and Chakraborty, \{Tanmoy \} and Rose, \{Carolyn \} and Peng, \{Violet \}",

booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",

address = "Australia",

url = "https://aclanthology.org/volumes/2025.emnlp-main/",

}

Liu, Y, Zhang, W, Wang, Y, Tang, J, Zhang, P, Yang, B, Huang, F, Wang, R & Hu, H 2025, Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese. in C Christodoulopoulos, T Chakraborty, C Rose & V Peng (eds), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, ACL Anthology, pp. 12521-12538, 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), Suzhou, China, 4/11/25. https://doi.org/10.18653/v1/2025.emnlp-main.633

Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese. / Liu, Yikang; Zhang, Wanyang; Wang, Yiming et al.
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. ed. / Christos Christodoulopoulos; Tanmoy Chakraborty; Carolyn Rose; Violet Peng. ACL Anthology, 2025. p. 12521-12538 (EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Translationese-index

T2 - 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

AU - Liu, Yikang

AU - Zhang, Wanyang

AU - Wang, Yiming

AU - Tang, Jialong

AU - Zhang, Pei

AU - Yang, Baosong

AU - Huang, Fei

AU - Wang, Rui

AU - Hu, Hai

N1 - Full text of this publication does not contain sufficient affiliation information. The Research Unit(s) information for this record is based on the then academic department affiliation of the author(s).

PY - 2025/11

Y1 - 2025/11

N2 - Translationese refers to linguistic properties that usually occur in translated texts. Previous works study translationese by framing it as a binary classification between original texts and translated texts. In this paper, we argue that translationese should be graded instead of binary and propose the first measure for translationese-the translationese-index (T-index), computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index can generalize to unseen genres, authors, and language pairs. Moreover, T-index computed using two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can effectively capture translationese, as demonstrated by alignment with human pointwise ratings and pairwise judgments. Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE. © 2025 Association for Computational Linguistics.

AB - Translationese refers to linguistic properties that usually occur in translated texts. Previous works study translationese by framing it as a binary classification between original texts and translated texts. In this paper, we argue that translationese should be graded instead of binary and propose the first measure for translationese-the translationese-index (T-index), computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index can generalize to unseen genres, authors, and language pairs. Moreover, T-index computed using two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can effectively capture translationese, as demonstrated by alignment with human pointwise ratings and pairwise judgments. Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE. © 2025 Association for Computational Linguistics.

UR - https://www.scopus.com/pages/publications/105040265051

UR - https://www.scopus.com/pages/publications/105040265051#tab=impact

U2 - 10.18653/v1/2025.emnlp-main.633

DO - 10.18653/v1/2025.emnlp-main.633

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798891763326

T3 - EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

SP - 12521

EP - 12538

BT - Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

A2 - Christodoulopoulos, Christos

A2 - Chakraborty, Tanmoy

A2 - Rose, Carolyn

A2 - Peng, Violet

PB - ACL Anthology

Y2 - 4 November 2025 through 9 November 2025

ER -

Liu Y, Zhang W, Wang Y, Tang J, Zhang P, Yang B et al. Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese. In Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. ACL Anthology. 2025. p. 12521-12538. (EMNLP - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference). doi: 10.18653/v1/2025.emnlp-main.633

Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Abstract

Publication series

Conference

Bibliographical note

Funding

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this