Your support bot reports a deflection rate: the share of chats that ended without reaching a person. Your report counts each of those chats as an agent contact you no longer pay for. The number cannot tell a customer whose problem was solved from one who closed the window and called you the next morning. Both count as savings, and only one of them is.

What customers and regulators see
Cost is a large part of why these bots exist. A 2023 Issue Spotlight from the CFPB, the US consumer finance regulator, states that chatbots "were introduced in large part to reduce the costs of human customer service agents." It describes "doom loops" of unhelpful answers "without an offramp to a human customer service representative," and names harms to customers that include wasted time, feeling stuck and frustrated, and inaccurate information.
Customers remember a bad bot experience. A survey of 3,566 customers, published in 2026 by a research firm, found only 27% would try a chatbot again after a negative experience, and 87% say access to a person is essential when a company uses generative AI in customer service. Its advice: bots should be connectors to human support, "not containment traps."
What your chats actually show
Your company runs 12,000 bot chats a month. The dashboard shows 68% deflected: 8,160 chats that never reached an agent, all counted as savings. At $7 per agent-handled contact, the report credits the bot each month with about $57k of avoided agent work.
Then the bot's logs are joined to your tickets and call records by customer. In 2,450 of those 8,160 chats, the customer reached an agent about the same issue within seven days, by email, phone or a later chat. The problem was not solved, and the agent starts the conversation again. The bot resolved at most 5,710 chats, under 48% of the total. If each of those replaced one $7 contact, the handling cost avoided each month is about $40k, before the bot's own running cost. The other $17k of that claim, about $206k over a year, was counted as savings while your agents did the work, on the customer's second attempt.
Even that is generous. A customer who gave up and bought elsewhere never contacts you again, so a recontact count misses them. Those losses show up later, in cancellations, fewer repeat orders and renewals you do not win, and no dashboard links them to the bot.
When a measure becomes a target
This is Goodhart's law, named after the economist Charles Goodhart: once a number becomes the target people are rewarded on, it stops measuring what it was chosen to measure. Deflection was chosen to estimate how many issues the bot solves. Once the bot's team is judged on it, the fastest way to raise it is to make reaching a person harder: hide the button, end the chat after one answer, or loop the customer back to the main menu.
Each of those moves can raise deflection without solving anything. Nobody has to intend it; the dashboard rewards it, and changes that move it get kept. Your cost report and your customers then disagree, and your management meeting sees only the cost report.
Define resolution in the data
Start with a definition your systems can compute every week. A chat counts as resolved when the same customer does not contact you about the same issue within seven days, on any channel: the bot, email or phone. Treat that figure as a ceiling, since a customer who gave up silently still passes the test. Every chat that ends without a person is contained, which your dashboard calls deflected; report contained and resolved side by side.
Measuring it and acting on it takes three pieces of work:
- A shared customer key. Match bot sessions, tickets and calls by account, email or phone.
- One issue label. A classifier tags chats, tickets and calls from one issue list.
- A visible path to a person. The hand-off passes the chat, so nobody explains twice.
Feed the failures back
Every unresolved chat is evidence of where the bot fails. Have a person read fifty of them each week, grouped by issue, plus a sample of chats that passed the seven-day test but ended without a confirmed answer. Then change what the reading shows: an answer that is wrong, a policy the bot cannot apply, or an issue it should hand over at once. In consumer finance, the CFPB notes that a bot may fail to recognize a customer invoking a federal right, and that a wrong answer may breach legal obligations; send those issues to a person by rule.
Customers can also tell you directly. On the help pages your bot links to, GreatFeedback.ai adds a small widget that lets people flag a problem on the page itself, with the context they choose to add, so a customer can say where the answer failed. It never records sessions.
When deflection is good enough
When your bot handles narrow, factual questions, such as opening hours, order status or a password reset, contained and resolved may be close, but never assume it. Measure the gap with the seven-day join. If resolved stays within five points of contained for a quarter, and the weekly sample agrees, keep the dashboard and repeat the check each quarter.
Three steps to take first
- Export a month closed a week ago. With the customer key, issue and end time.
- Join it to tickets and calls. Count repeats on one issue within seven days.
- Report both numbers. Contained and resolved, side by side, every month.
Bring your bot's data to a call
A fifteen-minute call works out how your bot's true resolution rate can be measured from the systems you already run, and what the gap between contained and resolved is likely costing you. If your bot already resolves what it contains, you hear that plainly.
Where the join, the issue labels and the hand-off need building, Engineering builds them with your team, so your people own the measure and keep it honest. For a regular outside read of whether your customer-facing AI earns its cost, Advisory gives you one.
References
- Consumer Financial Protection Bureau. Chatbots in consumer finance. Issue Spotlight, published 2023-06-06.
- Gartner. Gartner Survey Finds Only 27% of Customers Would Try a Chatbot Again After a Negative Experience. Gartner Newsroom, published 2026-09-02.


