October 28, 2026 · Blogs
New research finds some languages need up to 17 times more tokens than others, making AI slower and dearer for non-English users. What it means for UK businesses with multilingual customers.
Most AI services charge by the token, the small chunks of text a model reads and writes. New research shows that the same request can use far more tokens in some languages than in others, so non-English users can pay several times more for the same work. Rest of World reported the findings on 26 October 2026, and they matter to any UK business serving customers in more than one language.
What the research found
A study published in September by the Coalition for Language Equity in AI Research calls the effect a hidden "tokenizer tax". A tokenizer is the tool that breaks text, images or audio into tokens before a model processes them.
- Among the 20 most popular languages on the internet, the most expensive used between 1.5 and 5 times more tokens than the cheapest.
- Across 200 languages, the gap in one model grew to 17 times.
- Languages written in scripts the tokenizers rarely saw, such as Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki, were consistently the worst affected.
- Script matters as much as how widely a language is spoken. Asturian, with few speakers, does relatively well because it uses the Latin alphabet and is related to well-represented languages such as Italian.
"A startup in Bengaluru may pay a surcharge on every prompt and completion while its competitor in Austin does not," the study says.
It is not only about cost
More tokens also mean slower responses and less room in the model's context window, the amount of text it can consider at once. Pratyush Choudhury of Activate AI gave the example of an AI tutor: if a conversation's history fills the window faster in one language, the tutor forgets earlier mistakes sooner and the lesson gets worse.
Dheemanth Reddy of Maya Research expects falling token prices to soften the cost gap over time, but said fixing it properly will need labs to design future tokenizers to handle these languages and scripts better. Choudhury has called on every major lab to publish a language-by-language scorecard of what comparable tasks cost.
What it means for UK businesses
London businesses often serve customers in many languages, and AI features such as chatbots, translation, document summaries and voice assistants are where the tokenizer tax shows up.
- Budget by language. Estimate AI running costs using real samples in each language your customers use, not English alone.
- Test quality, not just cost. Check accuracy and response times in every supported language before launch, especially in long conversations.
- Compare models. Tokenizers differ between providers, so the cheapest model in English may not be the cheapest in Arabic, Bengali or Polish.
- Design for context limits. Summarise long histories and keep prompts lean so non-English users get the same quality of service.
Webmasters LDN builds multilingual websites, apps and AI assistants for UK businesses. If you are planning an AI feature for customers who do not all speak English, we can help you test cost and quality across languages before you commit.
This article summarises news reports published up to 28 October 2026.