The hardest pair in the 2025 evaluation was not the one anyone expected
English into Italian was named jointly the most challenging language pair in the WMT25 evaluation, level with Egyptian Arabic. A well-resourced European language written in the Latin alphabet. That finding is worth understanding, because it dismantles how most people think about machine translation difficulty.
The finding
The Conference on Machine Translation's 2025 evaluation was deliberately built to be harder than its predecessors — the organisers titled their paper Time to Stop Evaluating on Easy Test Sets. It covered 30 language pairs and 60 systems, with professional annotators marking real errors.
Their summary named two pairs as the most challenging in the whole evaluation: English into Egyptian Arabic, and English into Italian.
One of those is unsurprising. The other is a well-resourced European language, written in the Latin alphabet, with enormous quantities of parallel text behind it, spoken in a major economy that has been translating commercially for as long as commercial translation has existed.
Why this matters more than the Italian market alone
The comfortable assumption in this industry is that difficulty tracks distance: European languages are easy, non-European languages are hard, and low-resource languages are hardest. The Italian result says that model is wrong.
Difficulty also depends on how demanding the readers are, and on what kind of content is being translated. Italian buyers, editors and readers have exacting expectations of register, rhythm and idiom. Output that would pass as acceptable in a language with a more tolerant commercial culture is rejected in Italian. The engine has not got worse; the bar is higher.
The practical consequence for you: if your quality assumptions are organised by language family, they are wrong somewhere. The only way to know where is to test your own content in your own pairs — which costs one batch of files and an afternoon.
What actually goes wrong in Italian
Register collapse. Italian distinguishes formal and informal address and maintains sharper boundaries between commercial, technical, legal and marketing registers than English does. Engines translate meaning accurately and land in the wrong register, producing text that a native reader immediately identifies as foreign without being able to point at an error.
Rhythm and sentence structure. Italian tolerates and expects different sentence architecture from English. Literal structural transfer produces text that is grammatical, comprehensible and visibly translated — which in a market where presentation carries commercial weight is a real cost.
Agreement chains. Gender and number agreement propagate across long sentences. Engines break the chain in exactly the long, qualified sentences that legal and technical documents are made of.
False friends at scale. English and Italian share vast Latinate vocabulary, and the overlap is where confident errors live. Eventualmente does not mean eventually. Attualmente does not mean actually. Engines get the common cases right and the specialised, technical cases wrong.
Terminology in technical and legal content. Italian technical and legal vocabulary is prescriptive, and the correct term is frequently not the most natural-sounding one.
What we do
- We set the register explicitly at the start, by audience and content type, rather than accepting the engine's default.
- We score register and idiom as a named error category, separately from accuracy, because that is where Italian output actually fails.
- We use reviewers matched to the domain — Italian legal, technical and medical vocabulary each behave differently and a good generalist does not cover all three.
- We check agreement and structure across long sentences as a discrete pass.
Where Italian work is worth most
Manufacturing and industrial machinery documentation, where Italy has a deep industrial base and where documentation is a compliance artefact. Legal and regulatory content. Luxury, design and consumer brands, where the entire commercial proposition is presentation and translated-sounding text is a direct cost.