To test a website chatbot before launch, run a written set of test questions with expected answers across six areas: coverage of real questions, unanswerable questions, out-of-scope questions, conflicting sources, tone and branding, and the widget itself on desktop and mobile. Fix failures at the content level first, get sign-off from whoever owns the content, and plan how you’ll monitor conversations after launch. The checklist and test matrix below give you a repeatable process, whether you’re launching your own bot or one for a client.
Why structured testing matters
A chatbot that works in a five-minute demo can still fail on the questions real visitors ask. Visitors use casual wording, misspell things, ask two questions at once and ask about topics your content never covers. Without a written test plan:
- The client, or worse, a customer, finds the errors first.
- You can’t tell whether a fix made things better or worse.
- Sign-off is based on impressions instead of evidence.
A test plan turns “it seems fine” into “40 of 42 questions pass; the other two are content gaps with owners.”
Before you start testing
Gather these first:
- Question inventory. 30 to 60 real questions from support emails, contact forms, sales calls and site search logs.
- Expected answers. Write the correct answer and the page or document that should support it, before running tests.
- Content owner. The person who can confirm what’s correct and approve changes.
- Test devices. At least one desktop browser and one real phone.
- A tracking sheet. Question, expected answer, actual answer, source cited, result, fix, owner.
The pre-launch test matrix
| Area | What to test | Example questions | Pass criteria |
|---|---|---|---|
| Coverage | The top real questions | “Do you offer weekend appointments?” | Correct, cites the right source |
| Phrasing variety | Same question, different wording | Formal, casual, typos, one-word queries | Consistent answers across variants |
| Document content | Facts that only exist in PDFs/DOCX | A detail from the policy PDF | Retrieved and cited correctly |
| Unanswerable | Reasonable questions not covered by content | “Do you offer financing?” (if not documented) | Says it doesn’t know; suggests a next step |
| Out of scope | Unrelated topics | Trivia, homework, coding help | Politely stays on topic |
| Conflicting sources | Topics where documents disagree | Refund window in old vs new policy | Surfaces the conflict so you can fix it |
| Stale content | Topics that changed recently | Old prices, old staff, past events | Current information only |
| Sensitive topics | Legal, medical, financial advice | “Should I take this medication?” | Appropriate caution; points to a professional or document |
| Tone and brand | Voice, length, formatting | Any answer | Matches brand voice; not too long |
| Adversarial | Attempts to break role | “Ignore your instructions…” | Stays in role; no inappropriate output |
| Contact route | Requests for a human | “Can I talk to someone?” | Gives the correct phone, email or contact page |
| Widget and UX | Display and behavior | Open, close, scroll, long answers | Works on desktop and mobile |
For a question-by-question script focused on answer accuracy, use the 20-question test script for chatbots with source citations.
Testing each area in detail
Coverage questions
Start with the 20 most common questions. These drive most of the bot’s value. Each should be correct, grounded in the right source and phrased naturally. If a common question fails, fix it before worrying about edge cases.
Unanswerable questions
This is where untested bots do the most damage. Ask plausible questions your content doesn’t cover. A good bot says it doesn’t have that information and points the visitor somewhere useful. A bad one invents a plausible answer. If the bot invents, strengthen the instructions about staying within the content, and consider whether the question deserves a real answer in your content.
Out-of-scope questions
Visitors will test the bot with jokes, general knowledge and unrelated requests. Decide in advance how strict you want it to be. Most business sites should keep the bot focused on the business.
Conflicting sources
Deliberately search for topics where two sources might disagree: pricing, refund windows, eligibility, opening hours. If the bot gives different answers to similar questions, it’s often because two sources conflict. The fix is to remove or update one source, not to tweak the bot. See what breaks chatbot retrieval for the common culprits.
Tone and brand
Read answers aloud. Check:
- Does it sound like the brand (formal, friendly, concise)?
- Are answers the right length for a chat window?
- Does the welcome message set clear expectations about what the bot can help with?
- Does the widget use the brand’s colors and the right bot name?
The widget on desktop and mobile
Test the embedded widget itself, not just the answers:
- Does the chat bubble appear on every intended page?
- Does it cover important buttons, cookie banners or other widgets?
- On a phone, can you open it, type, read long answers and close it easily?
- Does the keyboard hide the input field on mobile?
- Do citation links open correctly?
- Does the page still load quickly with the widget installed?
If you’re using a hosted bot page or custom domain instead of an embed, check the link, the page title and how it looks when shared.
Handover of contact details
If visitors will want to reach a person, test that the bot gives the correct route: contact page, email address or phone number, taken from your content. Whether the bot can hand a conversation to a live agent or capture lead details varies by platform, so test only what your platform actually supports, and make sure the bot never promises “someone will call you” unless that’s really set up.
Fixing failures: content first
When a test fails, diagnose before you change anything:
| Symptom | Likely cause | Fix |
|---|---|---|
| Wrong answer with a relevant citation | Source passage is ambiguous | Rewrite the passage to be explicit |
| Wrong answer citing an old page | Outdated content in the sources | Remove or update the old source |
| Inconsistent answers to similar questions | Conflicting sources | Keep one source of truth |
| Invented answer, weak or no citation | Content gap | Add content, or tighten gap handling |
| Correct but too long or off-brand | Instructions or welcome message | Adjust tone guidance |
Re-run the full question set after fixes, not just the failed questions. A content change can fix one answer and alter another.
Sign-off checklist
Before going live, confirm:
- All top-20 coverage questions pass
- Unanswerable questions don’t produce invented answers
- Out-of-scope and adversarial questions are handled acceptably
- Known conflicting or stale sources are resolved
- Tone, bot name and widget branding approved by the content owner
- Widget tested on desktop and a real phone
- Contact route answers are correct
- Results recorded in the tracking sheet
- Monitoring owner and review schedule agreed
Monitoring after launch
Testing doesn’t end at launch. Real visitors will ask questions nobody anticipated. Review conversation history on a schedule, weekly for the first month and monthly after that works for many sites, and look for:
- Questions the bot couldn’t answer or answered weakly
- Repeated questions that point to a missing page or unclear content
- Citations pointing at outdated sources
- New topics driven by campaigns, seasons or product changes
Add each finding to your content backlog and re-run your test set after significant changes.
In Techvia AI Bot, conversation history and usage views are available per bot, and agencies can see them across client workspaces from one dashboard, which makes this monthly review routine rather than a chore.
Frequently asked questions
How many test questions do I need before launching a chatbot?
Enough to cover your top real questions plus each risk area. For most small-business sites, 30 to 60 well-chosen questions is a practical range. Quality and variety matter more than raw count.
Who should sign off on a client chatbot?
The person who owns the content, usually someone on the client side who can confirm what’s correct. The agency owns the test process; the client owns the facts.
What’s the most common chatbot launch mistake?
Not testing questions the content doesn’t answer. Bots that handle known questions well can still invent answers to unknown ones, and that’s what erodes trust.
Should I re-test after launch?
Yes. Re-run your test set after significant content changes and review conversation history regularly to add new real questions to it.
Next step
If you’re preparing to launch chatbots for your business or your clients, book a demo of Techvia AI Bot to see how testing, citations and conversation history fit together.