Comparative Evaluation of Neural Machine Translation of fiction literature: A case study

Abstract

The development of machine translation has significantly raised the caliber of translations. However, it is unfortunate that not all language pairings or genres are equally advantaged by this technology. This study investigates the quality of neural machine translation (NMT) output in the novel genre from English into Arabic languages. It examines two Machine Translation (MT) systems; Google translate and Reverso for translating quotes in Charles Dicken’s novel; Hard Times. The study aims to reveal whether human translators can benefit from incorporating MT into their work and which MT system is valuable for the translation of such genre. For that purpose, a comparable corpus of 50 English quotes and their Arabic translations was used to assess the output quality of the two MT’s systems. The corpus was gathered using CLiC (Corpus Linguistics in Context) software for literary analysis and it was evaluated through the BLEU (Bilingual Evaluation Understudy) metric. BLEU compares MT outputs with a professionally published human translation, thereby generating scores for comparison. Based on the precision parameter used by BLEU, the results show that Google Translate slightly outperforms Reverso in producing quality output. The results will help us to evaluate machine translation outputs of the novel genre in comparison to a human translation. It is concluded that the precision of human translators cannot be equaled by the most advanced machine translation technology (NMT) when it comes to the novel genre. However, translators can still benefit from MT systems in their work.