A woman seen from behind walks round inside a glass revolving door at dusk, while a mint green ring of light traces the loop on its floor.

AI agent analysis

How to evaluate conversational AI performance from chats

How to evaluate conversational AI performance from real chats: four weekly questions, a worked example, and what counting failed chats cannot show.

Parimal Savaj

6 min read

Share

Your dashboard says your AI agent closed most chats on its own. Then a customer writes to say it never answered her question. Here is how to evaluate conversational AI performance: read real chats and ask four things of each one. Was the question answered? Was the answer right? Did it hand over when it should? If it failed, why?

A closed chat is not a solved chat, and the customer who gave up does not show on your dashboard.

Conversational AI is another name for an AI agent: the software that chats with your customers for you.

Key takeaways

  • To evaluate conversational AI performance, read real chats and ask four questions of each one.
  • A chat the AI agent closed without a person is not always a solved chat. The customer may have given up.
  • Sort the failed chats by reason. The biggest pile is this week's fix.
  • Counting failed chats ranks the reasons by how often they happen, not by how much harm they do. A person still decides.

How to evaluate conversational AI performance when the numbers look fine

Many dashboards lead with one number: chats the AI agent closed without a person. It goes up, and everyone relaxes.

But a chat closes for two reasons. The customer got what they needed, or the customer gave up. The number counts both the same way.

New York City's consumer protection department has named the second kind. In October 2026 it proposed a rule against customer service "doom loops". It would make it "a deceptive and unconscionable trade practice" to offer customer service and then "repeatedly and unreasonably fail to address consumers' inquiries". It is a proposal, not yet a rule.

A loop still ends. On your dashboard, it looks like a win.

Two more sources make a related point. In October 2026 the UK Financial Conduct Authority (FCA) said that firms using AI-supported tools "remain responsible for consumer outcomes". Its advice: "Monitor live outcomes so they can spot and deal with issues quickly." That paper is about UK financial firms, so read it as a principle, not a rule for your business. The U.S. National Institute of Standards and Technology (NIST) says tests are "predominantly done in controlled testing environments that cannot account for real-world dynamics". That report covers monitoring AI systems in general, not customer service chats.

The cost comes later. The customer who gave up on Monday phones on Tuesday to ask again, angrier. Or they do not come back.

How do you measure AI agent performance in real conversations?

Ask four questions of every chat you read.

  1. Was the question answered? Not "did the chat end". Did the customer get what they came for?
  2. Was the answer right? It should match your policy, your prices and your product.
  3. Did it hand over to a person when it should? For example, when the customer asked twice, or asked for a person.
  4. If it failed, why? Write the reason in a few words.

The first three give you a yes or a no. The fourth gives you a list of reasons, and that is the part you can act on.

Here is what a list of reasons can look like. The examples are invented for this post, not customer data.

Reason the chat failed The customer wrote The AI agent replied
Repeats the same answer "You already told me that. It did not work." "Please restart the app and try again."
Answers a different question "Do you deliver on Sundays?" "Our stores are open seven days a week."
A made-up answer (your warranty says otherwise) "Is the screen covered if it cracks?" "Yes, cracked screens are always covered."
No handing over to a person "Can I talk to someone, please?" "I can help with that. What is your question?"

How do you evaluate a week of chats by hand?

  1. Take last week's chats. Take all of them if you can. If not, take every chat from one full day, for example all of Tuesday, so nobody picks the easy ones.
  2. Read both sides and answer the first three questions. Take this chat. The customer wrote, "Do you deliver on Sundays?" The AI agent replied, "Our stores are open seven days a week." Answered: no. It answers a different question. Note the answers in a sheet, one row per chat. Our post on reviewing a single conversation covers this step in detail.
  3. Write the reason for each failed chat. For that chat the reason is "answers a different question". Reuse a reason you already have before you add a new one.
  4. Count the chats under each reason. If "repeats the same answer" has the most chats, it is this week's fix.

Here is one chat, worked through. It is an example, not customer data.

Customer: "Can I change the delivery address on my order? It ships tomorrow."

AI agent: "You can update your address in Account settings."

Customer: "That changes my profile, not this order."

AI agent: "You can update your address in Account settings. Is there anything else I can help with?"

The customer leaves. The dashboard counts a chat closed without a person. Your sheet says something else.

  • Answered: no.
  • Right: the reply is true, but it answers a different question.
  • Handed over: no, though the customer said the answer did not help.
  • Reason: repeats the same answer.

One chat like this is a bad chat. A pile of them under one reason is a finding: the AI agent has no answer for changing an order that is already placed.

What does a good week look like?

There is no number here, on purpose. A good week reads like this.

  • Most customers got what they came for.
  • No answer about money, policy or safety was wrong.
  • Every customer who asked for a person got one.
  • The biggest pile is smaller than last week's, or it is a new reason because you fixed the old one.

In a bad week, the same reason tops the list again and nobody owns it.

The FCA paper names a common weakness: firms that left "long gaps between reviews of live outcomes". It sets no timescale. A weekly read is a place to start.

How does Metricsense do this every week?

Each step starts with the problem, then shows what Metricsense does about it.

  1. Reading every chat takes more time than anyone has. You upload the chat logs from your AI agent. Metricsense reads each conversation, both sides: the customer's messages and the AI agent's replies.
  2. Your four questions live in a spreadsheet. You write each one as a check in plain words, such as "Did the customer ask for a person and not get one?" There is no tagging and no model training.
  3. Finding the exact message again takes another hour. Every finding links to the exact message behind it, so you can read it when someone asks to see it.
  4. A flagged chat is not a verdict. Someone who knows your policy reads the messages and decides what it is. That person also decides the fix. Sometimes the AI agent is wrong. Sometimes it repeated your help page faithfully, and the help page is wrong.

Here is what a flagged chat looks like. Illustrative data, not actual customer data.

Check that failed The customer wrote The AI agent replied
Right answer: matches our rescheduling policy "Can I move my appointment to Friday?" "Yes, you can change it at any time."

What you learn from this: the reply promises something the rescheduling policy does not. A person opens the policy, confirms it, and fixes what the AI agent was told.

What can counting failed chats not tell you?

A count ranks reasons by how often they happen. It does not rank them by how much harm they do. One made-up answer about a price or a policy can matter more than a large pile of clumsy greetings.

So read the small piles too, starting with anything about money. We do not have a clean rule for weighing a rare, serious failure against a common, small one, and that call stays with you.

Want this run on your own chats? A pilot reads around 50 of your conversations, with a readout inside a week. Book a free pilot on your calls or chats.

Share this article

Back to the blog