Giacomo Mantovani

Article · November 5, 2026 · By Giacomo Mantovani

Customer service AI metrics: what to measure before and after launch

Containment rate and bot conversations say little. The metrics that show whether AI in customer service works, how to compute them, and where they mislead.

  • customer service
  • metrics
  • KPIs
  • help desk
  • voice AI
Bot conversations don't measure answers. Accuracy 75% → 81%, Resolution 5–6 → 2–3 days, 0 missed calls.

A month after a customer service AI goes live, someone presents the dashboard: thousands of bot conversations, a high containment rate, first response time down to seconds. The numbers are usually true and easy to produce. They do not answer the question the budget owner cares about: did customers get the right answer, faster, at a lower cost than before? These are the metrics I use to answer that question, with how to compute each one and how each one can mislead.

Which customer service AI metrics actually matter?

Useful metrics measure an outcome for the customer or a cost for the company. Vanity metrics measure activity: conversations the bot opened, tickets the classifier touched. An assistant that answers everything and gets half of it wrong has excellent volume.

A scorecard for customer service AI
MetricHow to compute itThe trap
Accuracy vs human agentsCorrect decisions / total, on the same sample you use to score peopleAn overall average hides categories the system always gets wrong
Time to resolutionMedian time from ticket open to final closeRushed closes that reopen later
Real resolutionCases closed with no repeat contact on the same issue within a few daysCounting every conversation that ended without an agent as resolved
Missed calls or messagesContacts nobody answered, human or AIMeasuring only during business hours
Handoff rateCases passed to a person / totalDriving it to zero by hiding the route to a human
Cost per resolved contact(staff + platform + model calls) / resolved contactsDividing by handled contacts instead of resolved ones

Every row has a trap because every metric can improve on paper while service stays the same. I read them in pairs for that reason: accuracy next to handoff rate, time to resolution next to reopen rate.

Why do you need a baseline before the AI goes live?

Without a starting value, no post-launch number means anything. At a major Italian bank, an LLM agent routes more than 1 million internal help desk tickets a year into 400+ categories and picks the right one 81% of the time. On its own, 81% says nothing. Next to 75%, the accuracy of manual routing on the same kind of tickets, it becomes a result. The full numbers are in the LLM ticket routing case study.

My rule with clients is to measure the system against metrics the company already tracks. If no baseline exists, measuring one is the first step of the project, before any code. A history export and a few weeks of data on the current process are enough.

How do you measure the accuracy of AI ticket routing or classification?

Take a sample of real requests, settle the correct answer for each one, and compare it with the system's answer. The hard part is the correct answer. The agent's original assignment is not always right, so every case where the system and the agent disagree needs a second look from someone who knows the categories well.

Then break the average down by category. With 400 categories, 81% overall can coexist with a few categories that are almost always wrong, usually the ones that overlap or that nobody ever defined in writing. Finally, weight errors by what they cost. A formal complaint routed to the wrong queue matters more than ten misfiled information requests. How the routing itself works is covered in the guide to AI ticket triage.

Why does time to resolution matter more than first response time?

First response time is the easiest metric to improve and the least useful. An auto-reply saying "we got your request" takes it to a few seconds and changes nothing for the customer.

Time to resolution measures what the customer actually waits for. At the bank, the largest gain from the routing agent showed up there: average resolution time fell from 5 or 6 days to 2 or 3. Most of the delay came from tickets assigned to the wrong team, which bounced between groups before reaching someone who could close them. A few points of accuracy at the front removed days at the back.

Containment or resolution: how should you judge a support chatbot?

Containment rate, the share of conversations a bot handles without an agent, is the most quoted chatbot number and the most misleading. A customer who gives up and closes the chat counts as contained. In a Gartner survey of 5,728 customers, only 14% of customer service issues were fully resolved in self-service, and only 36% of the issues customers themselves called very simple (Gartner, August 2024).

To measure real resolution I count repeat contacts. If the same customer writes or calls again about the same issue within a few days, the first conversation was not resolved, whatever the bot logged. This requires linking chat, email and phone to one customer record, which is also why many vendors do not report it.

What should you measure for an AI voice agent?

On the phone, two things matter more than in ticketing: missed calls and latency. A call nobody answers is often a booking that goes elsewhere. A reply that comes too late makes callers talk over the agent, and the conversation falls apart.

IGEA, the voice agent for clinic reception that I co-founded and lead as CTO, missed 0 calls in its first month at Centro Medico Luni Care, a clinic with 40+ doctors, with a 96.7% success score. It handled about €3,500 of bookings and recovered around 10 hours a week of staff time. It replies in under 1 second from the end of the caller's speech. The engineering details are in the real-time voice agent case study.

Voice agent metrics
MetricWhat it measuresIGEA, first month
Missed callsCalls with no answer, including when the clinic is closed0
Success scoreOverall outcome of calls handled by the AI96.7%
LatencyEnd of caller's speech to start of replyUnder 1 second
Staff time recoveredPhone hours the front desk no longer coversAbout 10 hours a week
Value handledBookings completed by the AIAbout €3,500

Write down what counts as success before launch, call type by call type: booking recorded, cancellation done, correct handoff to a person. A success score defined after the fact, while looking at the data, tends to produce the result everyone hoped for.

How do you calculate cost per contact and the real saving?

Cost per resolved contact puts everything the service costs in the numerator: staff, platform, model calls, system upkeep. The denominator is resolved contacts. Put handled contacts there instead and a chatbot that closes many conversations without solving them looks cheap.

I measure the saving in hours returned and errors avoided. The roughly 10 hours a week at Luni Care are phone hours the front desk no longer has to cover. Track the review rate next to cost: how many cases still go through a person. When it drops very fast, the usual cause is someone who stopped checking.

What are the common mistakes when measuring customer service AI?

The first is launching without a baseline and finding out later that there is nothing to compare against. The second is reading only the overall accuracy figure, without a per-category breakdown and without weighting expensive errors. The third is rewarding containment: a bot tuned never to hand off to an agent produces customers who call back. The fourth is letting the vendor choose and compute the metrics, with the definitions that suit them. The last is measuring once. New products, seasonality and team changes move the numbers, so review them every month.

Next step

Not sure your customer service AI is working?

Tell me what you measure today and which numbers your vendor showed you. I will tell you which ones are missing and how to get them from your own data.

Let's talk

If you are weighing a custom agent for tickets, email or phone, the AI agent development page explains how I set up evaluation, checks and handoff to people.

Frequently asked questions

How often should you review the metrics after launch?

Weekly for the first few weeks, then monthly. Re-score a fresh hand-labelled sample whenever products, categories or teams change, because that is when accuracy drops without anyone noticing.

How many tickets do you need to measure accuracy reliably?

A few hundred real tickets are enough for a first measurement, as long as you keep the hard ones in. With many categories, make sure the sample has enough examples of the most frequent categories and of those where an error is expensive.

Is CSAT enough to evaluate customer service AI?

No. Only a small share of customers fill in the survey, often the happiest or the angriest, and the score arrives after the problem has happened. Use it as a signal next to real resolution and time to resolution.

Who should own customer service AI metrics?

The head of customer service, not the vendor and not IT. The vendor can compute them, but definitions, baselines and alert thresholds belong to whoever answers to customers for the results.

Author

Giacomo Mantovani

AI Engineer & Consultant for Companies and Agencies

LinkedInGitHub

Next step

Tell me the workflow. We scope it on a free call, then a working version is live within 10 days.

Book a scoping call

Related articles