Where the engines are weakest and the benchmarks are least trustworthy

African languages are where the gap between what translation systems claim and what they deliver is widest — and where the consequences of an unchecked translation most often land on someone with no alternative source of information.

--:--:-- [PST]
13.95 → 2.29
one model's benchmark score against its score on real-world text, in a low-resource language
2025 academic audit of a 200-language benchmark
S01

Languages we cover

Swahili, Amharic, Somali, Tigrinya, Hausa, Yoruba, Igbo, isiZulu, isiXhosa, Afrikaans, Oromo and others on request. Coverage varies by language and we will tell you plainly where we can review to a standard worth paying for and where we cannot.

S02

The evidence problem

Most engine vendors advertise coverage of 200 or more languages. That claim rests largely on one standard multilingual benchmark.

A 2025 academic audit re-examined that benchmark and found that its own reference translations — the supposedly correct answers — fell below its claimed quality standard. The audit also compared benchmark scores against performance on real-world text. In one language, a model scored 13.95 on the benchmark and 2.29 on real-world content.

Two things follow. First, the published quality gap between well-resourced and low-resourced languages is narrower than the real gap. Second, a supplier's language count tells you almost nothing about whether the output in any particular one of those languages is usable.

If you are relying on machine translation into African languages and have not had the output independently checked, you do not currently know what you are publishing.

S03

What actually goes wrong

Training data is scarce and uneven. These are low-resource languages in the technical sense — the constraint is digitised parallel text, not speakers. Swahili is comparatively better served; Tigrinya, Oromo and many others are not, and the difference between them is much larger than any vendor's documentation suggests.

Noun class systems. isiZulu, isiXhosa, Swahili and the wider Bantu group organise nouns into classes that drive agreement across the whole sentence. Engines break agreement chains in exactly the long sentences that policy, medical and legal documents are made of.

Agglutination. A single word can carry what English expresses in a clause. Errors happen inside words, where they are harder to spot and harder to correct cleanly.

Tone marking. Yoruba and other tonal languages distinguish meaning by tone, and tone marks are frequently absent from source data and inconsistently produced in output. Text that looks fine can be ambiguous or wrong to a reader.

Non-Latin scripts. Amharic and Tigrinya use Ge'ez script, with its own rendering, sorting and input behaviour that Latin-tested systems handle badly.

Terminology gaps in technical and medical content. Standardised technical vocabulary does not exist for every domain in every one of these languages. The correct treatment — borrow, coin, gloss or explain — is a judgement about the audience that an engine cannot make.

Variety and standardisation. Several of these languages have significant regional variation and, in some cases, contested standards. "Translate it into Swahili" is not one deliverable.

S04

What we do

  • We tell you what we can and cannot staff. This is the language group where overpromising is most common in this industry and most harmful. Where we do not have a reviewer we can stand behind, we will say so.
  • We decide variety and terminology policy explicitly, per audience, and record it.
  • We check agreement, tone marking and script rendering as named categories, because these are where the errors concentrate.
  • We assess comprehensibility separately from accuracy for public health and citizen-facing content, where being technically correct and practically unreadable is the most common failure.
  • We document reviewer qualifications, which in this language group is both harder and more important than in any other.
S05

Where this work is worth most

Public health, humanitarian and development content, where the reader has no alternative source and the consequence of misunderstanding is direct. Government and citizen-facing information. Energy, mining and infrastructure documentation, where safety procedures are read by multilingual workforces. Telecommunications and financial inclusion content aimed at mass-market consumers.

S06

Talk to us about African languages

Tell us the language, the audience and what the content is for. You will get a straight answer about what we can review properly — including when the answer is that you need a different supplier.

Talk to us about African languages