A month after a customer service AI goes live, someone presents the dashboard: thousands of bot conversations, a high containment rate, first response time down to seconds. The numbers are usually true and easy to produce. They do not answer the question the budget owner cares about: did customers get the right answer, faster, at a lower cost than before? These are the metrics I use to answer that question, with how to compute each one and how each one can mislead.
Which customer service AI metrics actually matter?
Useful metrics measure an outcome for the customer or a cost for the company. Vanity metrics measure activity: conversations the bot opened, tickets the classifier touched. An assistant that answers everything and gets half of it wrong has excellent volume.
| Metric | How to compute it | The trap |
|---|---|---|
| Accuracy vs human agents | Correct decisions / total, on the same sample you use to score people | An overall average hides categories the system always gets wrong |
| Time to resolution | Median time from ticket open to final close | Rushed closes that reopen later |
| Real resolution | Cases closed with no repeat contact on the same issue within a few days | Counting every conversation that ended without an agent as resolved |
| Missed calls or messages | Contacts nobody answered, human or AI | Measuring only during business hours |
| Handoff rate | Cases passed to a person / total | Driving it to zero by hiding the route to a human |
| Cost per resolved contact | (staff + platform + model calls) / resolved contacts | Dividing by handled contacts instead of resolved ones |
Every row has a trap because every metric can improve on paper while service stays the same. I read them in pairs for that reason: accuracy next to handoff rate, time to resolution next to reopen rate.
Why do you need a baseline before the AI goes live?
Without a starting value, no post-launch number means anything. At a major Italian bank, an LLM agent routes more than 1 million internal help desk tickets a year into 400+ categories and picks the right one 81% of the time. On its own, 81% says nothing. Next to 75%, the accuracy of manual routing on the same kind of tickets, it becomes a result. The full numbers are in the LLM ticket routing case study.
My rule with clients is to measure the system against metrics the company already tracks. If no baseline exists, measuring one is the first step of the project, before any code. A history export and a few weeks of data on the current process are enough.
How do you measure the accuracy of AI ticket routing or classification?
Take a sample of real requests, settle the correct answer for each one, and compare it with the system's answer. The hard part is the correct answer. The agent's original assignment is not always right, so every case where the system and the agent disagree needs a second look from someone who knows the categories well.
Then break the average down by category. With 400 categories, 81% overall can coexist with a few categories that are almost always wrong, usually the ones that overlap or that nobody ever defined in writing. Finally, weight errors by what they cost. A formal complaint routed to the wrong queue matters more than ten misfiled information requests. How the routing itself works is covered in the guide to AI ticket triage.
Why does time to resolution matter more than first response time?
First response time is the easiest metric to improve and the least useful. An auto-reply saying "we got your request" takes it to a few seconds and changes nothing for the customer.
Time to resolution measures what the customer actually waits for. At the bank, the largest gain from the routing agent showed up there: average resolution time fell from 5 or 6 days to 2 or 3. Most of the delay came from tickets assigned to the wrong team, which bounced between groups before reaching someone who could close them. A few points of accuracy at the front removed days at the back.
Containment or resolution: how should you judge a support chatbot?
Containment rate, the share of conversations a bot handles without an agent, is the most quoted chatbot number and the most misleading. A customer who gives up and closes the chat counts as contained. In a Gartner survey of 5,728 customers, only 14% of customer service issues were fully resolved in self-service, and only 36% of the issues customers themselves called very simple (Gartner, August 2024).
To measure real resolution I count repeat contacts. If the same customer writes or calls again about the same issue within a few days, the first conversation was not resolved, whatever the bot logged. This requires linking chat, email and phone to one customer record, which is also why many vendors do not report it.
What should you measure for an AI voice agent?
On the phone, two things matter more than in ticketing: missed calls and latency. A call nobody answers is often a booking that goes elsewhere. A reply that comes too late makes callers talk over the agent, and the conversation falls apart.
IGEA, the voice agent for clinic reception that I co-founded and lead as CTO, missed 0 calls in its first month at Centro Medico Luni Care, a clinic with 40+ doctors, with a 96.7% success score. It handled about €3,500 of bookings and recovered around 10 hours a week of staff time. It replies in under 1 second from the end of the caller's speech. The engineering details are in the real-time voice agent case study.
| Metric | What it measures | IGEA, first month |
|---|---|---|
| Missed calls | Calls with no answer, including when the clinic is closed | 0 |
| Success score | Overall outcome of calls handled by the AI | 96.7% |
| Latency | End of caller's speech to start of reply | Under 1 second |
| Staff time recovered | Phone hours the front desk no longer covers | About 10 hours a week |
| Value handled | Bookings completed by the AI | About €3,500 |
Write down what counts as success before launch, call type by call type: booking recorded, cancellation done, correct handoff to a person. A success score defined after the fact, while looking at the data, tends to produce the result everyone hoped for.
How do you calculate cost per contact and the real saving?
Cost per resolved contact puts everything the service costs in the numerator: staff, platform, model calls, system upkeep. The denominator is resolved contacts. Put handled contacts there instead and a chatbot that closes many conversations without solving them looks cheap.
I measure the saving in hours returned and errors avoided. The roughly 10 hours a week at Luni Care are phone hours the front desk no longer has to cover. Track the review rate next to cost: how many cases still go through a person. When it drops very fast, the usual cause is someone who stopped checking.
What are the common mistakes when measuring customer service AI?
The first is launching without a baseline and finding out later that there is nothing to compare against. The second is reading only the overall accuracy figure, without a per-category breakdown and without weighting expensive errors. The third is rewarding containment: a bot tuned never to hand off to an agent produces customers who call back. The fourth is letting the vendor choose and compute the metrics, with the definitions that suit them. The last is measuring once. New products, seasonality and team changes move the numbers, so review them every month.
Not sure your customer service AI is working?
Tell me what you measure today and which numbers your vendor showed you. I will tell you which ones are missing and how to get them from your own data.
Let's talkIf you are weighing a custom agent for tickets, email or phone, the AI agent development page explains how I set up evaluation, checks and handoff to people.



