Lingnan University study: AI falls short on context and cultural rhetoric. Research published in Humanities and Social Sciences Communications (Nature Portfolio journal)

The research team from Lingnan University and the Chongqing University of Posts and Telecommunications analysed 16 Chinese-language speeches delivered at the United Nations General Assembly (UNGA) between 2008 and 2023, and compared the official English interpretations by professional UN conference interpreters with AI-generated translations produced by ChatGPT-4o, examining how each handled language in different contexts.

Before generating the AI translations, the researchers designed detailed prompts that included the speaker's official position, institutional background, year of delivery, audience, and broader sociopolitical stance in order to approximate the contextual information available to professional interpreters. However, despite providing the AI model with extensive contextual information, they found major differences between AI-generated translations and human interpretations in both contextual understanding and translation strategies.

One key difference concerns the use of personal pronouns. As Chinese frequently omits subjects, professional interpreters were more likely to introduce pronouns such as “our” and “they” to reflect interpersonal meanings and relationships between speakers and audiences, reinforcing collective identity and shared responsibility. AI-generated translations, by contrast, tended to produce more literal renderings with fewer personal pronouns.

For example, a Chinese sentence referring to vaccines as a powerful weapon against the pandemic was rendered by a professional interpreter as:

“Vaccination is our powerful weapon against COVID-19.”

whereas ChatGPT-4o translated it as:

“Vaccines are a powerful weapon against the pandemic.”

The researchers found that the interpreter’s addition of “our” strengthened the sense of collective identity, while the AI translation adopted a more neutral tone. The study also identified distinct differences in how obligation and responsibility were expressed. Professional interpreters were more likely to adjust modal verbs according to context, using expressions such as “should” and “need to” to convey persuasive rather than mandatory obligation. AI-generated translations, however, relied more heavily on “must” and passive constructions, making responsibility less explicit.

For example, the professional interpretation reads:

“We need to enhance coordinated global COVID-19 response and minimise the risk of cross-border virus transmission.”

whereas the AI translation states:

“International joint prevention and control must be strengthened, and the cross-border spread of the virus must be minimised.”

The researchers found that the AI version obscures the agent responsible for action by using passive constructions.

The study also examined culturally embedded metaphors. More than half (52.63 per cent) of the AI translations reduced culturally specific metaphors to their literal meanings, weakening their rhetorical force. By contrast, professional interpreters adopted more flexible strategies, preserving, adapting, and explaining metaphorical expressions according to context. In about one-third of the cases (31.6 per cent), interpreters retained the metaphor and also conveyed its intended meaning.

One example involved the traditional Chinese metaphor of people travelling “in the same boat”. The professional interpreter translated it as:

“We are called upon by our times to unite as one and work together for mutual benefit and win-win progress like passengers in the same boat.”

While ChatGPT-4o rendered it as “Working together and achieving mutual benefits and win-win outcomes are the objective demands of our time.”

According to the researchers, the AI translation conveyed the general meaning, but omitted the metaphorical imagery and its rhetorical impact.

The team noted that ChatGPT-4o generally produces fluent and grammatically accurate translations capable of completing translation tasks effectively. However, drawing on socio-cognitive theory, the study argues that professional interpreters consider not only the source text itself but also factors such as the speaker's identity, communicative setting, audience, cultural background, stance, and rhetorical purpose when deciding how to translate. This suggests that current large language models have yet to replicate fully the human capacity to interpret context and cultural meaning.

Prof Wang Binhua, Professor of the Department of Translation and Head of the Centre for English and Additional Languages at Lingnan University, member of the SIG on Artificial Intelligence in Translation and Interpreting of the European Language Council (ELC), said “Large language models still face the challenge of the ‘black box’, meaning that the mechanisms through which they produce particular translations remain difficult to explain. Unlike professional interpreters, who work within established professional ethical standards and are accountable, AI systems generate translations by identifying patterns in large volumes of language data and do not possess an intrinsic ethical framework. In translation tasks that require careful attention to cultural meaning and contextual understanding, human interpreters remain indispensable in making informed judgements about interpersonal relationships, rhetorical choices, and cultural expression.”

He added that AI is better positioned to augment rather than replace professional translators and interpreters. When integrated with human expertise, AI has the potential to improve efficiency while leaving context-sensitive and culturally informed decision-making in human hands.