Machine translation and evaluation

: online systems

Abstract

The Internet has exerted a powerful influence over the development of machine translation (MT). The enormous quantity of multilingual documents online is a valuable resource for MT to support the development of online MT systems, many of which are free of charge. In order for a user to select the most suitable option, MT evaluation plays a critical role. Most users may not have a good enough understanding of MT, leading to inappropriate evaluation practices and subjective judgment about the usefulness of an MT system. In recent years, a number of quantitative translation evaluation metrics have been developed. By providing language independent translation quality scoring measurements with respect to human translations as gold standards, translations by various MT systems can be compared in a fair and reliable manner. Their translation performance can then be evaluated objectively and comparably, but still in high correlation with human judgment. In this research, a number of representative online MT systems are evaluated from the users’ perspective using quantitative evaluation metrics of their performance in different languages. Legal texts, which represent one of the most difficult text genres to translate, are conventionally conceived to be suitable for MT, and were thus selected as test data for this evaluation. On the basis of exploring the current MT development, the proper use of online MT for legal translation will be discussed and a user-oriented MT evaluation metric will be proposed. This metric is comparable with the other evaluation metrics for MT performance, and can reveal the translation quality in an intuitive and highly readable way. The reliability and usability of these evaluation metrics in the context of legal translation will then be examined. A horizontal comparison of the translation performance of a number of popular online MT systems will be carried out using these metrics on a large scale corpus of legal texts, to show the relative strengths and weaknesses of different systems. The evaluation results from the automatic quantitative scoring using real legal texts provide us with an objective view of the suitability of these systems for legal translation in different language pairs. As a whole, the evaluation shows that there is no particular one of these MT systems outperforming the others for all language pairs. Different systems have different strengths and weaknesses for different language pairs. It is expected that these evaluation results can help a user to select the most suitable online MT system for a particular translation task.

Date of Award	15 Jul 2008
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Chun Yu KIT (Supervisor)