Why AI replies go wrong, and six controls that keep a Digital Mind grounded

An AI reply goes wrong when the model guesses instead of admitting doubt, leans on facts that have aged, sounds sure without cause, drifts from an instruction, answers a different question, or handles a language poorly. Six controls in a Digital Mind lower each risk. None removes it, so a person stays involved in consequential decisions.

Why do language models make things up?

Hallucination is the everyday word for what NIST calls confabulation: “confidently stated but erroneous or false content” (opens in a new tab)^[3]. NIST's generative AI profile (July 2024, voluntary US government guidance) calls confabulations a natural result of how these models are built, because they predict likely text from patterns in their training data. It singles out open-ended prompts for long-form responses and domains that require contextual or domain expertise, which is where an expert business wants an AI assistant to work.

A 2025 paper by researchers at OpenAI and Georgia Tech, “Why Language Models Hallucinate” (opens in a new tab)^[1], puts part of the blame on grading. Most benchmarks award a point for a right answer and nothing for admitting uncertainty, so a guess always scores at least as well as an honest “not sure”. The authors write that models are “optimized to be good test-takers.” Their own example: an open-source model was asked for a researcher's birthday, with an instruction to answer only if it knew, and gave three different wrong dates in three attempts.

Facts with no pattern behind them are the hardest. For such facts, birthdays among them, the paper's analysis sets a floor under a pretrained model's error rate, roughly equal to the share of those facts that appeared only once in training. The paper does not test business details, so what follows is this article's inference: a shop's opening hours, prices and refund terms look like facts of this kind, because nothing in language lets a model work them out. They have to be supplied.

Guessing is not compulsory, though. The authors propose grading with explicit confidence targets, so that admitting uncertainty below the target is no longer penalised, and they argue that weighing the accuracy of what a model says above how comprehensive it is matters for cutting hallucinations. The cost is a model that says less. That trade between coverage and certainty is the one a handoff to a person is meant to ease.

Does answering from your own documents fix it?

It helps, and it does not finish the job. One measured case comes from law, where vendors had said their tools avoid or eliminate hallucinations by drawing on trusted sources. In a preregistered Stanford and Yale study (opens in a new tab)^[2] (preprint, May 2024), researchers ran 202 legal research queries between March and May 2024. A response counted as a hallucination if it contained incorrect information or falsely claimed that a source supported a statement. On that definition, Lexis+ AI and Ask Practical Law AI hallucinated on 17% of the queries, and Westlaw's AI-Assisted Research on 33%. GPT-4, answering without a knowledge base, hallucinated on 43%.

Grounding lowered errors against the ungrounded model but did not remove them. Ask Practical Law AI shows the trade-off: its hallucination share matched Lexis+ AI's, yet it gave incomplete answers, meaning refusals or correct answers that cite no supporting authority, on more than 60% of queries.

Two cautions apply. The study tested those legal products and says nothing about any other tool, Plandora included. And its authors designed the queries to be hard, so the figures are not an unbiased estimate of real-world error rates.

Citations do not settle the question either. The study counts a real source that does not support the claim as a hallucination, and its authors conclude that users “must continue to verify that key propositions are accurately supported by citations.”

How does an AI reply go wrong?

1. Made-up facts. A plausible price, policy or credential that appears in nothing you supplied. This is the guessing described above.
2. Stale knowledge. An answer that was true once. In the Stanford study, one Lexis+ AI answer treated a rule as good law while citing the later case that overruled it, and the authors note that legal documents often contain outdated information. Your own material ages the same way: last season's price list, an offer that ended, a retired refund policy.
3. Over-confident tone. Wrong answers arrive in the same voice as right ones. The OpenAI paper observes that bluffs are often overconfident and specific, “September 30” rather than “Sometime in autumn”. NIST notes that the risk arises when users believe false content, often because of the confident nature of the response.
4. Ignoring boundaries. NIST's definition of confabulation includes “generated outputs that diverge from the prompts or other input”. A Boundary is part of that input, and a system that generates text statistically can slip past an instruction it was given.
5. Misread intent. The model answers the question it expected, or finds material for a different one. In the Stanford study, poor retrieval, meaning a failure to find the most relevant source, contributed to between 20% and 47% of hallucinated responses, depending on the tool (the study's causes overlap). Asked to define the “moral wrong doctrine”, Lexis+ AI relied on a source defining moral turpitude, a similar-sounding but unrelated term. In another example the authors show, a tool explained why a justice dissented in a case where that justice had joined the majority, instead of correcting the premise.
6. Language and translation errors. NIST notes that “an LLM may perform less well for non-English languages or certain dialects”, and that people may wrongly trust a system to perform the same for every group. A Mind that serves visitors in several languages needs testing in each one.

What six controls keep a Digital Mind grounded?

1. Approved knowledge only

A Digital Mind speaks from the material you give it: documents and FAQ answers in Drive, and posts from connected Social media accounts. Anything in Trash is ignored. Search web is a separate switch, best left off unless current public information helps the conversation. Start with the five questions you are asked most, write each answer once, and test them in the chat preview, where replies may show source citations that visitors can open. A citation is a pointer to check, not proof, for the reason the legal study gives. How a Digital Mind learns from your documents covers the detail.

2. Validity dates

Set dates… on a document or file limits its use to the window between two dates, which suits seasonal material, offers and policies with an end. Move superseded versions to Trash rather than leaving two prices in Drive. Dates only set the window in which an item is used. Keeping prices, hours and terms accurate stays with you, so schedule a regular look at them.

3. Boundaries in the Playbook

The Playbook has four sections: Goal, Role & style, Boundaries and Flow. Boundaries holds your limits and says when a question needs a person rather than an AI answer. The documentation's example is “Send refund requests to @Sam”. Tag the person from the picker, because typing a name as plain text creates no tag. Keep facts such as prices in Knowledge, not in Boundaries. Short replies in Role & style may help too: in the Stanford study, of the three legal tools, the one with the longest answers (about 350 words on average excluding refusals, against 175 to 219 for the others) also hallucinated most, and the authors note that longer answers hold more claims that can be false and take longer to check. Three tools cannot show that length caused the difference. How to write a Playbook walks through each section.

4. Handoff and Take over

When a conversation needs a person, the Mind can request one. Your team is notified, and the Mind keeps replying until someone takes over, so a handoff is not instant. Open the conversation in Inbox and choose Take over, or simply send a reply, which pauses the Mind and assigns the conversation to you. Select Release to AI when the Mind should answer again. The Unreplied and AI handling filters show where to look first. Inbox sits under Audience, which can be available through guided rollout or self-serve depending on your plan. Human handoff for AI conversations covers the design.

5. Improve suggestions

Improve drafts edits to your instructions. In the Playbook and in Specialists it can draw suggestions from real chats in the last seven days, such as unanswered questions, weak answers and handoffs. In a Specialist or an Automation you can also describe a change yourself, for example “also ask for their budget”. Edits appear in place, with the old text struck through and the new text highlighted, and you accept or reject each one. Improve is AI drafting too, so read every change instead of relying on Accept all, and test again afterwards: accepted changes save like any other edit.

6. Reviewing conversations in the Inbox

Read real conversations, not only dashboards. In Inbox, the AI handling filter lists what the Mind is handling, Contact details shows collected answers and notes, and Add comment saves an internal note the contact never sees. In Insights, the Conversations report lists top questions and answer coverage, and for a No answer or Partial answer, Add an answer opens the FAQ with the question filled in. Plandora's documentation says that Answered is a system assessment, not proof the visitor was happy, so read the conversation before judging quality. Answer analysis can depend on your plan.

Which control covers which failure?

Can any control remove the errors?

No. Each control lowers the chance or the cost of an error, and none brings either to zero. This article makes no accuracy claim about Plandora, and the studies cited here measured other tools. A Digital Mind is not the human expert and should not be presented as one.

AI replies can be wrong, so keep a person involved in financial, legal, medical and other consequential decisions. NIST gives a reason beyond the error rate. It describes automation bias, excessive deference to automated systems, as something that can worsen the confabulation risk, and it says confabulated content may be especially important to monitor in applications involving consequential decision making. In practice, put those topics in Boundaries with a named person tagged, and test them. If you work in a regulated field such as financial advisory, confirm with a qualified professional what your regulator expects of AI in client conversations. This article is general information, not legal, medical or financial advice.

Where should you start?

1. Write answers to your five most common questions in the FAQ, and set dates on anything seasonal.
2. In Boundaries, list the topics a person must handle and tag that person.
3. In the chat preview, test a normal request, a request that should be handed off, and a question your material does not answer.
4. Once the Mind is live, read the first week of conversations in Inbox before you trust any summary.
5. Use Improve and Insights to fix what you find, one change at a time, and test after each change.