You launched a chatbot. A dashboard shows thousands of conversations. Is it working?
Not necessarily. Many chatbot dashboards show big numbers that feel good but say little about results. This guide lists twelve KPIs that actually matter, how to measure each, and the vanity metrics that can mislead you.
It applies to support, sales, and booking chatbots. For the bigger picture on building one, see our guide to AI customer support chatbots.
Before you measure: decide the goal
KPIs only make sense against a goal. Pick the main one:
- Support: resolve questions faster and reduce the load on your team
- Sales: capture and qualify leads, book meetings
- Operations: complete tasks such as bookings and order lookups
Then choose KPIs that show progress towards it, and record your baseline before the chatbot so you can see the change.
The 12 KPIs that matter
Quality and outcomes
1. Resolution rate The share of conversations where the customer's issue is solved without a human. Count it only when the customer's problem is actually resolved, not merely when the chat ends.
2. Escalation (handover) rate How often the bot passes to a human. Too low can mean customers are trapped; too high means the bot is not helping. Look at the reasons for escalation as well as the number.
3. Customer satisfaction (CSAT) A quick rating at the end of the chat. Always read it alongside resolution rate. A bot that "resolves" many chats while customers are unhappy is a warning sign.
4. Answer accuracy Sample conversations each week and score whether answers were correct and complete. This is how you catch wrong answers early. See our article on stopping hallucinations.
5. Fallback / unanswered rate How often the bot says "I don't know" or fails to understand. The list of unanswered questions is a to-do list for your content.
Speed and effort
6. First response time Bots should answer in seconds. Track it, especially in busy periods.
7. Time to resolution How long it takes from the first message to a solved problem, for both bot-resolved and human-resolved chats.
8. Repeat contact rate How often the same customer returns about the same issue within a few days. A high rate suggests the answer did not really solve the problem.
Business impact
9. Ticket deflection / support volume Change in the number of tickets reaching your team for topics the bot covers. Compare with your baseline.
10. Cost per conversation or per resolution Include AI usage, platform fees, and the cost of human review. Compare with the cost of a human-handled conversation. Our guides to LLM API costs and ROI of AI automation help with the maths.
11. Lead conversion (for sales bots) Share of conversations that become qualified leads or booked meetings, and then customers. Track the full path, not only the first step.
12. Goal completion rate For task-based bots, such as booking an appointment or checking an order, the share of started tasks that finish successfully.
Vanity metrics to treat with caution
| Metric | Why it misleads |
|---|---|
| Total conversations | More chats can mean more confusion, not more value |
| Total messages | Long conversations might mean the bot is struggling |
| Time spent in chat | Longer is not better for support |
| "Containment" on its own | Can hide customers giving up or being blocked from a human |
| Number of questions answered | Says nothing about whether the answers were right |
| Uptime alone | A bot that is up but wrong is not working |
Vanity metrics are not useless. They show activity. But they are not outcomes.
How to build a simple dashboard
Keep it small. Start with five numbers:
- Resolution rate
- Escalation rate and top reasons
- CSAT
- Answer accuracy (from sampling)
- Cost per resolution
Add the rest as you mature. Review weekly in the first months, then monthly.
How to measure without heavy tools
- Use your chatbot platform's analytics for volume, escalation, and fallback.
- Add a one-click rating at the end of each chat.
- Sample 20–50 conversations a week and score them in a spreadsheet.
- Tag escalation reasons so you can see patterns.
- Pull ticket volume by topic from your helpdesk.
Set targets carefully
Avoid copying numbers from vendor reports. Your questions, customers, and data are different. Use your own baseline, set a realistic target for the first 90 days, and improve step by step.
Use the numbers to improve
- Top unanswered questions become new content.
- Frequent escalation reasons show where to improve flows or add handover rules.
- Low-CSAT conversations show tone and accuracy problems.
- High repeat contact shows answers that need clearer wording.
Our chatbot testing checklist helps you turn each finding into a test so the problem does not return.
We can set up measurement for you
We build chatbots with analytics, review workflows, and improvement cycles included. See our chatbot development service, or book a free consultation.
