Evaluating LLMs' Arabic-English Translation of Metaphor in Al-Ashmawi’s Poetry

Abstract

The current study aims to evaluate LLMs' effectiveness of metaphor translation in the poetry of Al-Ashmawi, one of the prominent poets in the Kingdom of Saudi Arabia, in comparison to that of human translators. It also aims to identify LLMs' potential, strengths, and limitations when translating complex literary forms and show the significance of humans' expertise in refining LLMs' translation. For data collection, a sample of 20 poetic verses were selected purposefully, after ensuring their validity and suitability for the study objectives, from the poem of "Fortitude in the Time of Fracture". The mixed methods design (quantitative and qualitative) was adopted for data analysis with the help of the comparative analytical approach where the translation quality of both systems were evaluated by four Yemeni professors specialized in literature and translation, using Likert five-point scale, then analyzed descriptively through SPSS and Multidimensional Quality Metrics (MQM) model, focusing on three parameters: linguistic accuracy, poetic fidelity, and emotional impact. The findings showed that human translators consistently outperformed LLMs, particularly in areas that require cultural sensitivity, cognitive and creative interpretation, stylistic coherence, and emotional resonance. In comparison, though LLMs outperformed human translators in preserving lexical and semantic features and produced an accurate readable translation, they simplified and literalized  metaphor translation, resulting in the strip of the poem of its aesthetic beauty and artistic weight. The study called for further research on training LLMs in computational linguistics and highly symbolic poetic language besides using a hybrid approach that combines LLMs' efficiency and human's creativity to achieve contextually relevant, linguistically accurate, and artistically authentic literary translation.