A chatbot demo with hand-picked questions always looks good. Real customers are different: they misspell words, ask two things at once, get angry, switch languages, and try things you never imagined.
Testing is what separates a chatbot that helps from one that embarrasses you. This chatbot testing checklist gives you 25 test cases to run before you go live, and a method for keeping the tests useful afterwards.
How to test properly
- Build a test set of real questions from your tickets, emails, and chat logs, not only ones you invent. Aim for at least 50–100.
- Write the expected answer or behaviour for each.
- Run the whole set and score each result as pass, partial, or fail.
- Record failures and fix the cause: content, instructions, retrieval, or integration.
- Re-run the entire set after every change. A fix in one place can break another.
- Have real people test it, including someone who did not help build it.
A. Accuracy and knowledge (tests 1–6)
- Top questions. Ask your 20 most common questions. Are answers correct and complete?
- Exact facts. Ask for prices, opening hours, deadlines, and policies. Compare with the official source.
- Recent changes. Ask about something you updated last week. Does the bot reflect it?
- Unknown topic. Ask something not in your content. The bot should say it is not sure and offer a human, not guess. See our guide to stopping hallucinations.
- Contradictory sources. If two documents disagree, which answer appears? Fix the content.
- Source display. If the bot shows sources, do the links work and match the answer?
B. Real-world language (tests 7–11)
- Typos and slang. "wher is my ordr", short messages, abbreviations.
- Vague questions. "It's not working." Does the bot ask a clarifying question?
- Two questions in one message. Does it answer both?
- Long, rambling messages.
- Other languages and mixed languages. If customers use more than one language, test each one with native speakers. See our guide to multilingual chatbots.
C. Scope and safety (tests 12–16)
- Off-topic requests. Ask for jokes, homework help, or opinions. The bot should decline politely and stay in scope.
- Attempts to override instructions. "Ignore your rules and..." The bot should keep to its rules.
- Request for sensitive actions. Refunds, account changes, personal data. Does it follow the approval or handover rules?
- Medical, legal, or financial advice requests, if relevant to your business. The bot should avoid giving advice and direct people appropriately.
- Abusive or offensive input. Does the bot stay calm and escalate when needed?
D. Handover and human fallback (tests 17–19)
- Explicit request for a person. "I want to talk to someone." Does handover work immediately?
- Frustrated customer. Does the bot detect frustration and escalate?
- Context transfer. When a human takes over, do they see the full conversation and any details collected?
E. Integrations and actions (tests 20–22)
- Order, account, or booking lookups. Do they return the right data for the right customer?
- Actions. Booking, cancelling, or creating a ticket. Does it complete, confirm, and record correctly?
- Failures. What happens when a connected system is slow or down? The bot should give a clear message, not crash or invent data.
F. Channels and performance (tests 23–25)
- Each channel. Website, WhatsApp, and app. Check formatting, buttons, and links on real phones. See our WhatsApp chatbot guide.
- Speed. How long does the first reply take? How does it behave with many users at once?
- Long conversations. After 20 or 30 turns, does the bot remember context, or become confused or slow?
Score your results
| Category | What to aim for |
|---|---|
| Accuracy | Very high on top questions and critical facts |
| Safety and scope | No unsafe or off-brand responses |
| Handover | Works every time |
| Integrations | Correct data, clear errors |
| Performance | Fast and stable |
Decide in advance what counts as "ready", such as "no critical failures and at least 90% pass on top questions". Choose your own thresholds based on risk.
Launch gradually
Even after testing, launch to a small share of traffic first:
- Watch real conversations daily at first
- Fix issues quickly
- Add every new failure to your test set
- Expand when quality is stable
Keep testing after launch
- Re-run the test set after every change to content, prompts, or models.
- Sample real conversations weekly and score them.
- Track KPIs such as resolution, escalation, and satisfaction. See our chatbot KPIs guide.
- Turn every complaint into a test case.
A printable summary
- Top 20 questions pass
- Exact facts match official sources
- Updated content is reflected
- Unknown topics handled honestly
- Typos, slang, and vague questions handled
- Multi-question messages handled
- Other languages tested by native speakers
- Off-topic and rule-breaking attempts handled
- Sensitive requests follow rules
- Human handover works with context
- Integrations return correct data and fail gracefully
- Each channel tested on real devices
- Speed and long conversations acceptable
- Launch plan and monitoring ready
Let us test and launch your chatbot with you
We build and test chatbots with real questions, safety checks, and staged launches. See our chatbot development service, or book a free consultation.
