How to measure AI support quality language by language

Measure AI support quality one language at a time, on real tickets. Build a test set of about 50 tickets per language, score every AI reply with one four-part rubric, have a native speaker review it, and track resolution, escalations, repeat contacts and the rubric pass rate per language every month. A vendor's language count tells you what the AI can attempt. Only this test tells you what it gets right.

Why a language count says nothing about quality

Vendors quote how many languages their AI handles. CoSupport AI Agent, for example, detects the customer's language and answers in it for the whole conversation, across 120+ languages. That number tells you where the AI will answer. How well it answers has to be checked in each language separately.

A 2025 benchmark called MMLU-ProX put 36 language models through the same 11,829 questions in 29 languages and found performance declines markedly in low-resource languages, with gaps of up to 24.3%. CSA Research's write-up of a TAUS Vancouver 2026 panel puts the drop-off at roughly the top 20 to 30 languages, where training data often doesn't exist.

Hallucinations follow a different pattern. A 2025 study found that, once adjusted for answer length, the hallucination rate seems unrelated to how much online text a language has. And a 2026 study of multilingual benchmarks found that they mostly measure math and factual recall rather than language skill. Neither a leaderboard nor the size of a language tells you how the AI does on your tickets in it.

Customers care which language the answer comes in. CSA Research's Can't Read, Won't Buy survey of 8,709 consumers in 29 countries found that 75% are more likely to buy from a brand again if customer care is in their language. Even among those most confident in reading English, 60% favored care in their own language.

English isn't a safe fallback either, even in Europe. The European Commission's Eurobarometer on languages found that 47% of people in the EU can speak English, which leaves more than half who can't.

Step 1. Pick the markets and build a test set per language

Start with the three to five languages that bring the most tickets. For each one, pull tickets that customers actually wrote in that language and that your team has already resolved. Translated English tickets miss how people really write, with slang and local formats.

Run each ticket through the AI in a setup where nothing reaches the customer, and keep the reply your team actually sent next to it. The reviewer needs that reply as the reference for facts and policy.

Fifty tickets per language is our recommendation: enough to spot a weak language and the ticket types where it fails, without weeks of review work. For a language with fewer than 50 tickets a month, take three months of history. For a language you're about to launch, you have no resolved tickets yet, so ask a native speaker to write 50 from your top ticket types, the way customers in that market would.

Step 2. Score every reply with one rubric

Use the same rubric in every language, so the scores compare. Four criteria, scored 0, 1 or 2:

A reply passes at 7 or 8 points out of 8. That means at most one criterion scored 1 and none scored 0, so a wrong policy, price or promise fails the reply on its own. We'd go live in a language when at least 43 of the 50 replies (85%) pass. Set the bar higher for languages where a wrong answer is expensive, such as payments or health. Our guide to measuring AI response accuracy covers how to count accuracy across the whole queue.

The rubric is a simplified cousin of MQM, the framework translation teams use to score quality. The MQM Council describes how each error gets a type and a severity, and how the counts turn into a score. Its guidance also warns that reviewers who aren't trained may arrive at very different results, which is why calibration in Step 3 matters.

Step 3. Choose who reviews each language

The rubric needs a reader who knows the language and the market. Here are the options, from the most thorough to the fastest:

Back-translation is a fast filter with a known weak spot. WHO's translation guidelines for its WHODAS questionnaire call it the method in common use for a long time, and add that its success was uneven, depending on the care taken. A classic study of round-trip translation found that a bad translation can still come back as quite reasonable English, so a clean round trip proves little about the reply the customer reads.

The AI judge needs the most care. MM-Eval, a benchmark of AI evaluators, found them unfair and inconsistent when scoring lower-resourced languages, which are the languages where you'd most like the help.

Whoever reviews, calibrate first. Have two reviewers score the same ten replies, compare, and agree on the reading of each criterion before they score the rest.

Where multilingual AI support breaks

Six failure types are worth a deliberate test in every language:

Step 4. One knowledge base or five

Start from what differs between your markets. Where the product and the policies are the same everywhere, one well-kept knowledge base can serve every language, as long as each language passes the test. Where the content differs by market, a translated answer stays wrong.

Localize the pages the test flags and leave the rest in one language. Expect the flagged pages to be returns, delivery, billing and legal. Our guide on how AI search works across languages in a knowledge base covers the indexing side.

Step 5. Run the numbers every month

After launch, the 50-ticket test becomes a monthly sample, and four numbers per language tell you where to look. "Resolved by AI" counts conversations closed without a person, "escalated" counts handovers, and the rest are still open or were dropped by the customer. A repeat contact is the same customer back on the same issue within seven days. Here's an illustrative month for a team with three markets:

Read across the row. Polish has the lowest resolution, the highest escalations and the most repeat contacts, and it sits below the 85% bar. In this example, the failed replies cluster in returns, where the AI is translating an English policy that doesn't apply in Poland. Fix the source, re-run the 50 tickets, and compare next month.

Don't read too much into small gaps. Fifty replies is enough to spot a weak language and too few to rank two good ones: the six points between English and German can be chance. Act when a language falls below the bar, or when its failures sit in the same ticket type two months running. When every live language passes two months in a row, add the next market and start again from Step 1.

CoSupport AI Agent handles the conversation in the customer's language, and CoSupport AI Assistant translates in both directions inside the helpdesk, so one support rep can handle a ticket in a language they don't speak. The full set of multilingual features is on our multilingual customer service page.

What to do when a language fails the test

A failed language doesn't have to be switched off. Read the failed replies by ticket type first and check whether they cluster, then pick the smallest fix that clears them:

1. Fix the source. If the failures sit in one topic, such as returns in Poland, the AI is probably answering from a policy that doesn't apply there. Add the local version to the knowledge base and re-run the tickets that failed.
2. Narrow the scope. Let the AI keep the ticket types it passes in that language, such as order status and password resets, and send the rest to a person until the next test.
3. Hand over with a translation. When a ticket goes to someone who doesn't speak the language, give them the translated thread and a draft reply they can check. Our guide to the AI-to-human handoff covers what the person should receive.
4. Pause it. Pause a language when the failures are spread across every ticket type, or when it sees too few tickets to test fairly. Re-test after the next knowledge base update.

Note which option you chose for each language and why, so next month's sample can confirm it.

Where native speakers stay in the loop

In the languages that pass the test, the AI takes the routine tickets. People keep two jobs.

They review the monthly sample, since the rubric and the calibration in Step 3 need a native reader. And they take the tickets where a wrong answer costs more than a slow one: rare languages with too little volume to test, and upset customers writing in their own language because the problem matters to them.