A guest asks whether you allow dogs. Your chatbot, which has never been told anything about pets, replies that pets are welcome for a small fee. Nobody wrote that. The model produced it because producing an answer is what it was built to do.

This is the single most common way business chatbots fail, and it is worth understanding why — because the fix is architectural, not a matter of better prompting.

General models answer everything

A general-purpose language model is trained to continue text plausibly. Ask it about your refund policy and it will generate something that reads like a refund policy, drawn from the thousands it has seen. It has no mechanism for noticing that it does not know yours.

In casual use that is harmless. In customer service it is a liability, because the customer has no way to tell an invented answer from a real one. They act on it, and you find out when they arrive with a dog.

Retrieval changes the question

The fix is to stop asking the model what it knows and start asking it to read. Before answering, the system searches your own documents for the passages that actually relate to the question, and the model is then instructed to answer using only those passages.

If your documents do not mention pets, there is no passage to retrieve, so there is nothing to answer from — and the honest reply becomes the only available one.

That constraint is what makes it safe to leave running unsupervised overnight.

Three failure modes that survive retrieval

Retrieval is necessary but not sufficient. Three problems remain, and all three are fixable:

1. The documents are wrong

An out-of-date price list produces confidently wrong answers with perfect grounding. The system is working exactly as designed and still misleading customers. Keeping documents current is the real maintenance work, and it never goes away.

2. The follow-up loses its subject

Somebody asks which of your properties are near central London, gets a good answer, and types “anywhere else?”. On its own that phrase means nothing — searched in isolation it matches whatever happens to be closest, which might be the Wi-Fi instructions. The system has to carry the previous turns into the search, not just into the reply.

3. The wrong scope answers

A business with eight properties has eight sets of parking instructions. A guest at one address asks about parking and gets another address’s answer — grounded, accurate, and completely useless. The conversation has to be pinned to one location, and the pin has to expire so yesterday’s guest does not haunt today’s.

What good looks like

A well-built assistant should be able to do all of the following:

• Refuse. “I don’t have that information” is a correct answer and should be a common one at first.
• Hand over. A refusal should route the conversation to a person, not end it.
• Stand down. Once a human replies, the bot should go quiet rather than talking over a colleague.
• Show you its gaps. Every question it could not answer is a line in your content backlog.

The uncomfortable part

A chatbot that refuses questions looks worse in a demo than one that answers everything. It is also the only kind you can safely leave talking to customers while you sleep. Judge the two on what happens when they are asked something nobody anticipated — which is most of what real customers ask.