How to evaluate an AI copilot for support agents before you run a pilot

Evaluate an AI copilot on 100 of your own tickets. Let it draft the replies and have two agents score every draft blind: send as is, edit, or discard. Then check what it read, where it lives, what happens to your data, what one reply costs, and whether agents still open it in week three.

What you're evaluating: drafts a person sends

In an Alibaba field experiment reported in a 2026 preprint, 17.9% of agents who were given an AI assistant never used it once. The tool worked. The agents didn't want it. That's the risk a demo can't show you, and it's why the evaluation below runs on your queue, with your agents.

An AI copilot for support agents, also called agent assist, works next to a person. It reads the incoming ticket, drafts a reply, suggests the steps to resolve the case, summarizes a long thread, and translates when the customer writes in a language the agent doesn't speak. The customer never talks to the copilot.

An autonomous AI agent answers the customer on its own and hands the hard cases to a person. If you're still choosing between the two, CoSupport AI Assistant's page covers that choice. This article starts after the choice: you're looking at a copilot, and you want to know it's good before a team depends on it.

A vendor demo runs on the vendor's tickets: clean, one question per message, a polite customer, a knowledge base written for the demo. Your tickets have typos, two questions in one line, an angry opening paragraph, and an answer that lives in last month's resolved tickets rather than in the help center. Seven checks tell you how a copilot handles that queue. The pass marks are ours; set your own if your queue argues for different ones.

Check 1. Draft accuracy on 100 of your own tickets

Build the sample first. Take 100 tickets that are already resolved, so the right answer is known. Spread them across your ticket types in the same proportion as your real volume: if a third of your queue is billing, a third of the sample is billing. Make 10 of the 100 deliberately ugly: mixed questions, a missing order number, a customer who's already been bounced twice.

Connect the copilot to the same sources your agents use, run it on the 100 tickets, and score the drafts blind. Two agents, each on their own, mark every draft: send as is, small edit (a name, a sentence, the tone), large edit (the draft got the policy or the facts wrong), or discard. Where the two disagree, a third person decides, and the decision gets written down as a rule.

Count every column. A study of more than 5,000 support agents published in the Quarterly Journal of Economics in 2025 found that agents followed the AI's suggestions 38% of the time on average, and the middle half of agents followed between 23% and 50% of them. Most of the value and most of the risk sit in the edited drafts.

Put senior agents on the scoring panel and in the pilot. The same study found that less skilled and less experienced agents resolved 30% more issues per hour with the assistant, while the most experienced and highest-skilled agents saw small gains in speed and small declines in quality. That study covered a chat assistant at one software company, so treat the numbers as a pattern to test for, and score your seniors' drafts on quality.

Pass mark: at least 80% of drafts go out as is or with a small edit, per ticket type, and no large edit hides a policy or money error that an agent would miss on a busy day.

Check 2. What it reads: knowledge coverage

There are four sources a copilot can read, and the drafts show which ones it did.

Test it directly. Pull 20 more tickets whose correct answer isn't in the help center: it sits in a past ticket, an internal policy, or the order system. A copilot that reads only public docs still produces a draft for each of them, and the draft sounds sure of itself.

Two more questions. Does the draft show its sources, so the agent can see which policy section and which past tickets it leaned on? A draft with no sources costs the agent the lookup the copilot was supposed to remove. And how often does the copilot re-read the sources? A policy you changed on Monday belongs in Tuesday's drafts.

CoSupport AI Assistant, for example, learns from the help center, historical tickets, internal documentation, company policies, and business systems, and backs each suggested resolution with the sources it used, such as the policy section and the past tickets it drew on. That's the standard to hold any copilot to.

Pass mark: the 20 "not in the help center" tickets get correct drafts, and every draft names its sources.

Check 3. Where it lives: inside the helpdesk, no code, time to first draft

Agents don't switch tabs. The draft has to appear inside the ticket view of Zendesk, Freshdesk, Zoho Desk, or whichever helpdesk your team already uses, with the send button where it always was.

"No code" has a precise meaning here: an admin on your team connects the helpdesk, the knowledge base, and the business systems without a developer, and changes the tone rules or the escalation rules later without one either. If every rule change waits on a vendor ticket, you're buying a service contract.

Measure the time to the first useful draft. Connecting CoSupport AI Assistant takes about 10 minutes, and full deployment takes about 3 days. Whatever the vendor you're testing promises, write the real number down during the pilot.

Pass mark: drafts appear inside the ticket view, an admin changes rules without a developer, and the first useful draft shows up within days.

Check 4. Data handling: five questions to ask

Get these five answers in writing before the pilot:

1. Where are tickets stored and processed, and in which region?
2. Does the vendor train shared models on your tickets? The answer should be no. CoSupport's wording is "your tickets never train anyone else's model"; ask any vendor for the equivalent sentence.
3. Who at the vendor can read your tickets, and is each client's data isolated from every other client's?
4. Which certifications and legal bases apply: ISO 27001, GDPR, CCPA, and a data processing agreement you can sign?
5. What's deleted when the pilot ends, and how fast?

Ask for the certificate itself and the data processing agreement; the page that mentions them is a claim. Then send the vendor three of your real tickets with personal data in them and ask what happens to each field on the way to the model and back.

Pass mark: all five answers in writing, a signed data processing agreement before real tickets flow, and per-client isolation confirmed.

Check 5. Cost per reply versus cost per seat

Most copilots are billed one of two ways: a flat fee per agent seat per month, or a fee per AI response. The difference shows up in Check 6.

A seat fee charges for every agent, including the ones who never open the copilot. In the Alibaba experiment, agents with access used the AI suggestion in 21.5% of chats. Under seat pricing, the 17.9% of agents who never used it cost the same as the ones who used it on every ticket.

A per-response fee charges when a draft is produced. Ask the follow-up question: does a discarded draft cost the same as a sent one? Then do the arithmetic yourself. Monthly cost divided by the number of drafts your agents used (sent as is plus edited) gives the cost per useful draft. Put it next to the minutes the draft saved times your loaded hourly cost.

Pass mark: cost per useful draft, measured on your pilot, is below the agent time it saves.

Check 6. Agent adoption after week two

Track three numbers per agent from day one: the share of tickets where the agent used the draft, the size of the edits, and time-to-send compared with the four weeks before the pilot. Then look at the spread. In the Alibaba experiment, 17.9% of agents sat at one end, never using the assistant, and 6.2% at the other, using it in every chat. The gap between those groups holds your reasons: drafts that arrive too late, a tone that doesn't match the brand, missing context the agent had to look up anyway, or plain distrust after one bad draft.

Expect the biggest gains from newer agents. The Quarterly Journal of Economics study found that agents with two months of tenure and an AI assistant performed as well as agents with more than six months of tenure and no assistant. If your team is mostly seniors, expect small gains: that study found small speed gains and small quality declines for its most experienced agents, so score their drafts on quality.

Salesforce's State of Service report from 2025, based on 6,500 service professionals, found that service representatives spend less than half their time, 46%, with customers, partly because of administrative tasks and internal responsibilities. Time-to-send before and after the pilot is the cleanest way to see whether the copilot moved that number for your team.

Pass mark, ours and deliberately high: by week three, at least two thirds of agents use the draft on half their tickets or more, and time-to-send is down for the ticket types the copilot covers.

Check 7. The exit: when drafts go out unchanged

When a ticket type's drafts go out unchanged four times out of five for a month, the agent adds nothing to those tickets except a click. That type is ready for an autonomous AI agent that resolves it without a person, while the copilot keeps drafting the rest.

So the last check is whether the vendor can make that move. Ask whether the same system runs a copilot on some ticket types and resolves others on its own, with one knowledge base and one queue, or whether "autonomous" means a second product with a second setup.

Rank ticket types by cost to serve, and check that the autonomous agent hands hard cases back to a person with the context attached.

Pass mark: a written path from "copilot drafts it" to "AI resolves it" for at least one ticket type, inside the same system.

Run the numbers

Fill the scorecard in from the 100-ticket test and the pilot. A copilot passes all seven, or you know exactly where it failed.

Here's an example run of Check 1, with the 100 tickets split by type in proportion to volume:

76 of 100 drafts went out as written or with a small fix, under the 80% bar overall. Order status (90%) and how-to (88%) clear it. Billing (72%), account access (70%) and everything else (20%) miss it, and five billing drafts needed large edits because they quoted a refund window the policy doesn't allow. That's a knowledge coverage problem: the copilot hadn't read the refund policy. Fix the source, re-run the 25 billing tickets, and score again. Order status is the closest to Check 7: 63% went out unchanged, against the 80% that would make it a candidate for autonomous resolution.

Where a person still owns the answer

A copilot writes the first draft. A person owns the answer.

The exceptions stay with people: the refund that policy doesn't allow but the situation deserves, the customer who's already angry when they open the chat, the ticket where the right answer depends on something no system records.

Judging the draft stays with people too. In the Alibaba experiment, top agents adopted 19.7% of suggestions whose diagnosis matched the final issue and 15.5% of those that didn't. The authors read that gap as too small to show that reviewers caught the AI's errors. Reading a draft critically is a skill, and the pilot is where your team learns it.