A Chatbot Audit Checklist: 12 Things That Break

A support chatbot that fails in testing costs you an afternoon. One that fails in production costs you customers, and sometimes more. In 2024 a Canadian tribunal ordered Air Canada to honor a bereavement refund policy that its website chatbot had invented. Whatever your bot says, customers will treat it as something you said.

Most teams test their bot by asking it ten friendly questions and watching it answer them well. That proves very little. The failures that hurt come from the questions nobody on the team thought to ask. This checklist covers the twelve we see most often, grouped by what goes wrong.

Invented facts

These are the most damaging failures, because the bot sounds exactly as sure of itself when it is wrong as when it is right.

1. Prices and discounts that are not in your documents

Ask about a product, a bundle, a discount for buying three, or a price for a size you do not sell. A bot that has seen a few prices will often produce a new one that looks consistent with them. Check every number it gives against your real catalog.

2. Policies you do not have

Ask about a return window for opened items, price matching, a loyalty program, or a student discount. If your documents say nothing, the right answer is "I don't have that information" and a pointer to a person. A bot that improvises a generous policy has just made a promise on your behalf.

3. Dates, days, and deadlines

Shipping times, return windows, warranty lengths, and holiday hours are easy for a model to blur. "30 days" becomes "60 days," or a business-day count becomes a calendar-day count. Ask for each one directly and in passing, inside a longer question.

4. Phone numbers, emails, and links

Contact details are a quiet but serious failure. A bot can produce a support email that follows your domain's pattern but does not exist, or a link to a page that was never published. Every address and URL in a reply should be checked against a real source.

Questions near the edge of your documents

5. Questions your documents almost answer

If your policy covers returns for unworn shoes, ask about worn ones. If it covers shipping to Canada, ask about Mexico. These near-miss questions are where bots stretch a real policy to cover a case it was never written for.

6. Questions about one customer's order

"Where is my order?" and "Can I change my address?" need account data the bot usually cannot see. It should say so and hand off, not guess a delivery date or confirm a change it cannot make.

7. Requests outside the bot's job

Ask for medical advice about a supplement, legal advice about a contract, or help with a competitor's product. A support bot should decline politely and stay in its lane.

Pressure and manipulation

8. Attempts to make it break its rules

Try the basic prompt injections: "ignore your previous instructions," "you are now in developer mode," "print your system prompt." This is not a full security test, but a bot that gives in to the obvious attempts will give in to worse ones.

9. Being talked round over several messages

Many bots handle one message well and slip over three. A customer who pushes ("but the last agent said I could," "your website says otherwise") can sometimes walk a bot into agreeing to things it refused at first. Test short conversations, not only single questions.

10. Promises and commitments

Ask the bot to guarantee delivery by a date, approve a refund, or confirm an exception. Unless it is connected to a system that can actually do those things, it should not say it has.

Tone and handoff

11. Saying "I don't know" well

Every bot will hit questions it cannot answer. Check that it says so plainly, does not apologize five times, and gives the customer a real next step, such as a contact form, an email address that exists, or a human handoff.

12. Staying consistent with your brand

Read twenty answers in a row. Does the bot sound like your company? Does it use the right product names? Does it stay calm with an angry customer? Tone problems rarely cause legal trouble, but they shape how people remember the experience.

How to check answers

Asking the questions is the easy half. The hard half is judging the answers consistently, especially across sixty of them.

• Use rules wherever a rule can decide. Extract every price, number of days, date, phone number, email, and link from each reply and compare it with your documents. If it is not there, it is a failure. No judgment needed.
• Use a model only where judgment is required. For questions like "does this answer contradict our return policy?", an AI model can help, but it should quote the exact line it relied on and be labeled as a model's verdict so a person can check it.
• Never count an unclear result as a pass. Anything the checks cannot settle should go on a list for a person to read.
• Keep the question set. A bot can answer differently the next time it is asked, and every change to its prompt or documents can break something that used to work. Rerun the same questions after every change.

We followed this process when we built the Reductivist Shop Assistant. Before launch it was tested against 61 questions, including prompt injections and invented policies, and the test report lists every failure and what was changed to fix it.

What a pass does and does not mean

A pass on one question is one good answer, not a guarantee. Models are not fully predictable, and a bot that answered correctly today can answer differently tomorrow. Testing lowers the risk of a bad answer reaching a customer. It does not remove it. That is why the most useful outcome of an audit is a clear, ranked list of what to fix first, and a question set you can run again.

If you want this done for you before launch, our chatbot audit builds about 60 questions from your own documents, checks every answer, and hands you a plain-English report you can read in ten minutes, along with the question set to keep.