Why AI Support Gives Wrong Answers—and How to Diagnose the Failure
Trace a failed customer promise through its policy, retrieved evidence, applicability, timing and action status. A worked return-support example shows what to fix, who should own it and how to verify the repair.
The Short Answer
Start with the specific customer promise that failed. Check whether an approved rule existed, whether the relevant evidence reached the assistant, whether it applied to this case and time, and whether any promised action actually completed. Repair the first confirmed break and test nearby ordinary cases too.
Key takeaways
- Audit individual claims: an unsupported promise, a contradicted statement and a false completion claim require different evidence.
- Preserve the sources, relevant configuration and transaction status from the time of the incident.
- Use controlled comparisons to distinguish retrieval failures from interpretation and execution failures.
- Give customer recovery and permanent repair clear owners, and verify useful ordinary behavior remains available.

An AI assistant tells a customer that an opened product can be returned. The warehouse later refuses it. The customer now has two problems: the original purchase and a promise your business did not keep. Calling the incident a hallucination describes how the answer felt. It does not tell you which part of the operation needs to change.
My starting point is to treat a wrong support answer as a broken decision, not merely a bad sentence. The assistant needed to identify the request, find an approved rule, establish whether that rule applied, and stay within what it could verify or do. A failure at any of those steps can produce the same confident-looking reply. Replacing the model before examining those steps is rather like replacing the delivery van because a parcel had the wrong address.
This guide gives a practical way to investigate the failure. We will work through a fictional return-policy incident, separate five kinds of evidence, and decide what to fix first. The examples are deliberately small enough to reproduce in a spreadsheet. They are teaching scenarios, not DTCUP customer results or claims about a particular assistant’s accuracy.
The useful outcome is a diagnosis somebody can act on: which claim was wrong, which evidence should have supported it, where the decision departed from that evidence, and what test would show that the correction worked. You can use the method with a hosted chatbot, a custom retrieval system, or an assistant connected to a CRM. The tools differ; the obligation to keep a customer promise does not.
1. Define what was wrong before deciding why it happened
Consider a fictional brand called Fieldwork that sells a rechargeable desk lamp and replacement batteries. Its ordinary return policy allows an unused lamp to be returned within thirty days. An opened replacement battery has a separate handling rule: support must assess its condition before offering a return route. These are invented operating rules for our example, not legal guidance or a description of any real retailer’s obligations.
A customer writes, “I bought a replacement battery three weeks ago, opened the package, and it doesn’t hold a charge. Can I send it back?” The assistant replies, “Yes. All products can be returned within thirty days, even if opened. I have arranged your free return label.” The reply contains several distinct claims. The time window may be relevant, the eligibility statement may be wrong, the shipping promise may be unsupported, and the claimed action may never have happened.
Do not give that whole paragraph one undifferentiated failure label. Break it into statements that could be checked independently. In this case: the product belongs to the ordinary return route; opening the package does not change eligibility; return shipping is free; and a label has been created. A single answer can have a correct sentence, an unsupported sentence, a contradicted sentence, and a false completion claim. Your investigation should preserve those differences.
Also identify the customer’s actual decision. They need a safe, legitimate next step for a possibly faulty battery. They did not ask for a general summary of a returns page. An answer that accurately quotes the ordinary policy but sends this customer down the wrong route is still an operational failure. Factual accuracy at the sentence level is necessary, but it is not the complete definition of useful support.
Write the expected outcome before opening the model settings. For Fieldwork, a satisfactory reply would recognize the battery category, explain that a condition check is needed, avoid promising eligibility or a free label, and collect only the information needed for the approved next step. If the assistant cannot perform that check, it should route the case to a person with the relevant authority. The expected answer need not use one exact script.
This distinction matters when teams assess “accuracy.” If the benchmark simply asks whether the phrase “thirty days” appears, the failed reply might pass. If it demands a fixed sentence word for word, several useful replies might fail. Instead, define required facts, forbidden promises, necessary questions, and allowed actions. That creates a test of the decision rather than a test of whether the assistant memorized your preferred wording.

There is a further distinction between an unsupported answer and a false answer. You may not yet know whether a particular customer qualifies for a free label. The immediate problem is that the assistant claimed certainty without the necessary evidence. Record that as unsupported until an authorized source settles the fact. Otherwise, the investigation quietly replaces the assistant’s guess with the reviewer’s guess.
Finally, decide what the incident is not. A correct policy that disappoints a customer is not automatically a model error. A carrier changing a delivery estimate after a properly qualified answer is not necessarily a failure either. But an assistant presenting an estimate as a guarantee is a failure of wording and scope. Clear boundaries let your team improve reliability without pretending it can eliminate all uncertainty from customer service.
2. Capture the evidence that existed at the time
A screenshot of the reply is a useful beginning. It is not enough to explain the incident. By the time someone investigates, a policy page may have changed, the search index may have refreshed, and a tool connection may have recovered. Looking only at the current system can make yesterday’s failure impossible to reproduce. You need a small incident record that preserves the relevant state without collecting an unnecessary copy of the customer’s life.
For the Fieldwork case, save the customer’s relevant messages, the assistant’s reply, the time and market, the product identifier if available, and the conversation or request identifier. Keep the approved policy version that applied at that time. Record the actual snippets delivered to the model, including their source identifiers and dates. A link to an entire knowledge base is not the same thing as evidence of what the assistant saw.
If the assistant can use tools, preserve the tool name, the non-sensitive inputs needed to understand the action, the returned status, and a reference to the authoritative transaction record. A label-creation request being sent is not proof that a label exists. A successful network response is also not sufficient if its body says the business operation was rejected. The evidence should distinguish an attempted action from an accepted, completed action.
Do not ask staff to copy raw logs containing passwords, payment details, or unrelated private messages into an unrestricted document. Use access-controlled incident records and redact fields that are not needed. Preserve stable record references when the underlying system can answer the question more safely. A good diagnostic file lets an authorized reviewer reconstruct the decision; it should not become a new source of customer-data exposure.
You will also want the relevant configuration version: model identifier, instructions version, retrieval settings, and available tool permissions. These identifiers explain what was running, but they are not a substitute for observing its behavior. Two requests with the same configuration may still produce different outputs. When a provider cannot expose every internal detail, record what is available and label the missing evidence rather than inventing a precise explanation.
Suppose your hosted chatbot gives you only the transcript and cited pages. You can still perform a useful investigation. Compare each claim with the applicable policy, reproduce the question with controlled variations, and ask the vendor for the retrieved context or action status. Just do not report “the model ignored the correct snippet” unless you know that snippet reached the model. The distinction between a confirmed cause and a plausible cause should survive the incident report.
A compact record can use five columns: observation, expected evidence, evidence available, confidence in diagnosis, and next check. For example, “Label claimed created” belongs in observation; “label identifier and successful transaction status” belongs in expected evidence. If no tool call exists, that supports a false completion diagnosis. If logs are unavailable, the status is unresolved, and the next check is to inspect the label system through an authorized route.
Keep the original failed response alongside any corrected reply. Overwriting the transcript with a better answer makes the file look tidy but removes the material needed to test the fix. The same principle applies to policy versions. You are preserving a small, relevant evidentiary trail, not creating a permanent archive of all customer conversations. Set retention and access according to the business’s actual needs and obligations.
3. Locate the first broken boundary
I use five diagnostic boundaries: authority, retrieval, applicability, freshness, and execution. These are working categories, not a claim that every AI system has exactly five components. They are useful because each category points to a different owner and a different correction. A case may cross several boundaries, but identifying the earliest confirmed break prevents the team from polishing an answer that never had a reliable foundation.
Authority: was there an approved answer to find?
Start with the source material. Does the business have a current, approved rule for the question? Fieldwork’s marketing page might say “easy thirty-day returns,” while an internal handling note contains the battery exception. If nobody has reconciled those statements or decided which one may be communicated publicly, the assistant has inherited an unresolved business decision. More elaborate prompting cannot authorize a policy that the business itself has not settled.
Look for ownership, approval status, intended audience, and scope. A support agent’s generous exception in an old ticket is not automatically a general policy. A draft launch document is not a promise to customers. A supplier’s statement about a component is not necessarily the retailer’s warranty commitment. The source may be perfectly genuine and still lack the authority required for the answer being given.
The corrective action here usually belongs to the policy or knowledge owner. Publish the approved rule, identify its audience and applicability, and remove or clearly supersede contradictory material. Do not merely add “follow the latest policy” to the instructions while leaving several undated documents in circulation. The assistant cannot reliably infer which of two apparently official sources your team privately considers authoritative.
Retrieval: did the necessary evidence reach the answer?
Now assume the battery rule exists, is approved, and can legitimately be used. Did the system actually retrieve it? Search may favor the broad returns page because it contains more familiar words. The product identifier may be missing from the query. A document may have failed to index. Permissions may correctly exclude an internal note, leaving no public version. Each possibility requires a different correction.
Compare the source that should have been found with the exact context delivered to the model. Do not stop at “the document is in the knowledge base.” A library can own a book without placing the right page on your desk. Test the customer’s wording, a precise product query, and a version that includes the missing category. This shows whether the problem is absence, matching, filtering, or loss of context during document splitting.
Anthropic’s discussion of contextual retrieval describes why isolated document fragments can lose useful meaning and explores adding context to retrieval. The practical lesson here is limited: a relevant fragment must retain enough surrounding information to be interpreted correctly. It does not follow that one retrieval technique will make every support answer reliable.
Applicability: did the assistant use the evidence within its limits?
Suppose the delivered context says, “Unused lamps may be returned within thirty days. Replacement batteries require a condition review.” If the assistant still says “all products,” it has expanded a scoped rule into a universal promise. The retrieval step worked; the interpretation did not. Adding more copies of the same document will probably make the context longer without repairing that inference.
Check product, market, purchase channel, effective date, customer status, and any required condition. These are examples of applicability fields, not a mandatory form for every question. A customer asking how to clean a lampshade should not face an account-verification obstacle course. But a request to disclose an order address or issue a refund has different evidence and permission requirements. Apply the fields that change the decision.
Freshness: was the evidence still valid for this decision?
A policy can be authoritative, retrievable, and correctly interpreted yet no longer apply. The search index may hold an older version, the assistant may reuse a cached answer, or the customer’s order may fall under a previous policy. “Use the newest document” is not always sufficient: the newest rule may apply to purchases made after a particular date, while an older rule still governs earlier orders.
Separate the time the answer was generated, the time the fact was observed, and the time the rule became effective. For an order status, observation time matters. For return eligibility, purchase date and policy effective date may matter. For a promotional price, the offer window matters. If the evidence cannot establish the relevant time, qualify the answer or retrieve the missing fact instead of quietly treating all dates as interchangeable.
Execution: did the promised action actually happen?
Finally, inspect the gap between language and action. Fieldwork’s assistant may have no label tool at all. It may have attempted a call that failed. It may have received a request identifier for a pending workflow and described that as a completed label. Each is a different execution defect, even though the customer sees the same sentence: “I have arranged your return.”
A completion statement should be tied to evidence from the authoritative system and the level of certainty it actually provides. “Your request has been sent for review” and “Your return label is ready” describe different states. Treat that distinction as a product requirement. Polite confidence cannot move a request from one state to another.

4. Use controlled checks instead of repeatedly asking the same question
Typing the failed question into the chatbot until it produces a good reply is not a repair. It shows that a good reply is possible under some conditions. You need comparisons that isolate a cause. Change one meaningful part of the setup, keep the rest as stable as practical, and record the result. This is especially important when generated answers vary between attempts.
For Fieldwork, begin with a replay of the original question and preserved configuration in a safe test environment. Capture the retrieved material and any attempted actions. Then run an evidence-supplied test: provide the approved battery rule directly in the context, without changing its wording or adding the desired answer. If the reply becomes consistently appropriate, the retrieval path deserves attention. If it still invents eligibility, the problem extends beyond retrieval.
Next, run an applicability test. Keep the policy unchanged but compare “unused desk lamp” with “opened replacement battery.” The expected route should differ. If both receive the ordinary return answer, inspect whether the assistant recognizes the product category and exception. If both are sent to a person, the system may have become excessively cautious. A fix that refuses every question can reduce wrong answers while also eliminating useful assistance.
A missing-information test is equally revealing. Remove the purchase date or product category where it changes the decision. Does the assistant ask the necessary question, state a conditional answer, or assume a value? Record which missing field caused the uncertainty. “Please provide more details” is less useful than “Was this the complete lamp or a replacement battery?” because the second question is tied to an actual decision boundary.
Then test the tool boundary. Give the assistant a simulated pending status, a rejected request, and a confirmed label result. It should describe each state differently. Use a test double or a controlled non-production account so the experiment does not create real returns. If your vendor cannot safely simulate transactions, test the wording and evidence checks separately before attempting a narrowly scoped live validation.
Include a freshness comparison with explicit policy dates. Imagine the lamp rule changes on the first of a month, while an order was placed the previous week. The expected outcome depends on the fictional business’s approved transition rule, which you must supply. Do not let the reviewer invent that transition while grading. The point is to test whether the assistant respects an established rule, not whether it guesses the reviewer’s unstated preference.
Run more than one trial for cases where the output can vary, but choose the number according to the consequence and your available effort. There is no magic count that proves an assistant safe. A small series can expose obvious inconsistency; a larger, representative test set is needed to estimate performance more credibly. Preserve the denominator and the exact cases tested so “passed testing” means something concrete.
Anthropic’s guide to evaluating agents separates tasks, repeated trials, grading, and the final state of an action. That separation is useful here: the transcript tells you what the assistant said, while the transaction record tells you what happened. Neither should be treated as a complete substitute for the other.

The strongest finding is not “the new prompt worked once.” It is something like: “The original retrieval returned the general policy but not the approved battery rule. Supplying that rule corrected the eligibility claim in the diagnostic trials, while the assistant still falsely claimed a completed label. We therefore have a retrieval defect and a separate completion-state defect.” That statement supports two concrete fixes and avoids pretending one change solved the whole incident.
Check the diagnosis with an ordinary case too
A diagnostic intervention should have a negative control: a nearby case that should not change. If you add the battery exception, replay an ordinary unused-lamp return. If you repair the label status mapping, replay a general policy question that does not need a tool. These controls help detect a correction that works by broadly suppressing the assistant rather than improving the intended decision.
Also separate a result that supports your explanation from proof of the assistant’s internal reasoning. Supplying a missing policy and seeing a better answer is evidence that the context matters. It does not reveal every internal cause of the original generation. You do not need that level of certainty to make a sensible repair. You do need to avoid presenting an inferred mechanism as an observed fact.
Consider a second fictional failure: a customer asks whether a lamp fits a narrow shelf, and the assistant invents a dimension. Providing the product specification corrects the answer. Your diagnosis should still check whether the specification was absent, retrieved under the wrong product variant, or dropped when the context became too long. Those possibilities suggest different maintenance work. The same final sentence can conceal different paths to failure.
Keep one test case outside the set used to tune the correction. For example, use a different wording of the battery question or another product with a separate condition rule. If the repair succeeds only on the exact incident sentence, it may have taught the system a narrow response without establishing the underlying distinction. This held-out check is modest evidence of generalization, not a promise that all future cases are covered.
Finally, ask another qualified person to review the expected outcome when the rule is disputed. Two reviewers disagreeing about the correct return route is a signal to resolve the business rule before judging the assistant. Otherwise, your evaluation score will measure who happened to grade the answer that day.
5. Repair the right layer, and keep the useful behavior
Once the diagnosis is specific, the intervention can be smaller. If the policy is absent, write and approve it. If the correct policy is excluded by a publishing permission, resolve that permission deliberately or create an approved public answer. Do not bypass access controls merely to improve answer coverage. Some information should remain unavailable to a public assistant, even when it would make answering easier.
The OWASP RAG Security guidance treats access control and untrusted retrieved material as security concerns throughout a retrieval system. For support operations, the practical boundary is straightforward: finding a document does not make it safe to disclose, and reading instructions inside a document does not authorize an action. Keep permission checks outside the assistant’s persuasive interpretation of the text.
If the problem is search matching, improve the evidence path rather than hiding the symptom in the final response. Product aliases may need to resolve to a canonical identifier. A policy fragment may need its heading and exception attached. An indexing failure may need monitoring. Validate the change against nearby products too, because a rule added for replacement batteries should not accidentally override the ordinary lamp policy.
If the assistant has the right evidence but expands its meaning, make the decision requirement explicit. For example: distinguish general rules from exceptions; do not infer return eligibility from a time window alone; ask for a product category when it changes the route; and do not promise a benefit absent from the source. These instructions are useful behavior constraints, not a substitute for an approved source or an authorization check.
For completion errors, fix the state mapping. A pending request should produce pending language. A failed request should produce an honest failure message and an available next step. A confirmed label should expose the actual approved delivery route for that label. The assistant should not construct a plausible-looking label link or invent a confirmation number. A reassuring sentence is not a transaction receipt.
Be careful with broad negative instructions. After a painful incident, a team may write, “Never discuss returns.” That can be appropriate as temporary containment, but it should be labeled as such. Otherwise, a short-term safety measure becomes a permanent product limitation without anyone deciding whether that tradeoff is worthwhile. Record the trigger for restoring the narrower capability and the evidence required to do so.
I would also resist changing the model, retrieval settings, policy wording, and escalation process in the same release unless an urgent incident makes that necessary. Bundled changes make causal diagnosis harder: if the results improve, you cannot tell which change helped; if a new error appears, rollback becomes more disruptive. A sequence of small, testable repairs often teaches the team more than one large “AI optimization” project.
There are times when the underlying model really is the limiting factor. It may struggle to follow a supported distinction even with clean evidence and clear instructions. Compare a candidate model on the same held-out cases, with the same permitted tools and sources. Judge the distribution of failures, response time, and total operating cost. A model that improves difficult policy reasoning but introduces false tool-completion claims is not an uncomplicated upgrade.
6. Measure the cost of the error without hiding it in averages
Not every wrong answer deserves the same response. A mistaken link to a size guide is different from an unauthorized disclosure or a fabricated refund. Rank incidents by customer consequence, exposure, reversibility, and the time needed to discover them. These dimensions help prioritize work, but do not turn them into a decorative score whose precision exceeds the evidence. A clear severity description is often more useful than a mysterious number.
For Fieldwork, the false return promise can cause wasted shipping, a delayed resolution, and a dispute with a customer who acted on the company’s words. The false label claim adds another failure: the customer may wait for something that does not exist. Even if a later agent resolves the issue, that recovery consumes work and trust. Count the recovery instead of letting the final happy ending erase the original mistake.
Imagine a fictional weekly review of one hundred return-related conversations. Eight contain an unsupported eligibility promise, three contain a false completion claim, and two contain both. That means nine distinct conversations contain at least one of those two errors: eight plus three minus two. Reporting eleven affected customers would double-count the overlap. Reporting only eight “policy problems” would hide the execution defect.
The denominator also matters. Those nine cases are nine percent of the one hundred reviewed return conversations, not nine percent of all support traffic unless the sample represents all support traffic. If the review deliberately selected complaints, it estimates the composition of complaints, not the overall error rate. Keep a representative sample for monitoring and a separate difficult-case collection for stress testing. They answer different questions.
Now compare a proposed fix on the same evaluation set. Suppose unsupported eligibility promises drop, but the assistant sends many more ordinary lamp questions to a person. That may still be a worthwhile temporary tradeoff. To judge it, track whether the handoff was accepted, whether the customer had to repeat the story, and whether resolution took longer. A lower wrong-answer count is not automatically a better complete service.
A useful dashboard separates at least three views: harmful or unsupported claims, successful completion of the intended task, and recovery workload. You can add latency and cost, but avoid collapsing everything into one average too early. An assistant that answers quickly and cheaply while creating rare expensive disputes may look excellent in the wrong aggregate. Review the small number of high-consequence cases individually.

Use a follow-up window that fits the task. A customer can discover a missing return label within hours, but a mistaken warranty promise may surface much later. A conversation ending without another message is not proof of resolution. Where appropriate and permitted, connect the conversation to the subsequent support outcome using a stable reference. Do not assume silence means success simply because silence is convenient to count.
When reporting results, say what changed and what remains uncertain. “The five known battery-policy cases now route correctly in repeated tests; ordinary lamp answers remain available; broader production monitoring continues” is much more useful than “hallucinations solved.” It tells the business what has been demonstrated and what still needs observation. Reliability improves through this accumulation of bounded evidence, not through a single reassuring percentage.
7. Give the customer a recovery route and the team an owner
Diagnosis should not leave the affected customer waiting while the team debates taxonomy. Once a harmful promise is discovered, route the individual case to somebody who can determine the appropriate remedy. Explain the current status clearly, acknowledge the conflicting information, and avoid issuing a second unverified promise to soften the first. The person handling recovery needs the transcript and the relevant evidence, not just the label “AI issue.”
For the business, assign an owner to the defective boundary. The knowledge owner can approve a policy, but may not be able to repair a failed search index. The integration owner can map a label status, but cannot decide a disputed return exception. A support lead may coordinate the incident while separate owners handle the source and system changes. Clear ownership prevents every problem from disappearing into an undifferentiated engineering queue.
DTCUP’s existing guide to building a support escalation process covers the operational handoff in more detail. Use that process for the actual customer case. The diagnostic method in this article has a narrower purpose: identify why the answer failed and what must be verified before the same capability is trusted again.
A useful repair ticket contains the failed claim, the expected behavior, the evidence that confirms the cause, the affected scope, the proposed correction, and the test that will close the ticket. Include any temporary restriction and who can remove it. Avoid tickets such as “improve chatbot accuracy.” They invite endless tweaking because nobody has defined the condition under which the work is finished.
Suppose the Fieldwork investigation confirms two failures. The knowledge-search owner must ensure the approved battery rule is retrieved for the relevant product questions. The integration owner must prevent a pending or absent label action from becoming a completion statement. The support lead approves the customer-facing language and tests a small set of ordinary and exceptional requests. Each owner has a bounded deliverable, and none can declare the whole case fixed alone.
After release, watch for the original failure and for plausible side effects. A new exception filter could suppress useful lamp information. A stricter completion check could produce confusing messages when a tool succeeds slowly. A refreshed source could lose an important historical transition rule. Monitoring should follow the changed boundary, not merely repeat a generic checklist that has no connection to the repair.
If the same issue recurs, check whether the fix reached every answering surface. A website assistant, an internal agent helper, and an email drafting tool may share some knowledge but use different indexes, caches, or permissions. Do not assume that correcting one published source immediately updates every consumer. Record which surfaces were verified and which are still pending, especially during a staged release.
8. A worksheet for your next wrong-answer incident
You can implement the method with one row per claim and a short summary per conversation. Begin with the customer’s requested outcome. Then copy only the relevant sentence from the assistant’s answer and mark it as supported, contradicted, unsupported, or unresolved. Attach the applicable evidence reference and its version. This keeps the review focused on what the customer could reasonably understand from the reply.
In the next columns, identify the first confirmed broken boundary and any later safeguard that also failed. For the battery case, the broad policy may have been retrieved instead of the exception, but a missing-evidence rule should still have prevented an unconditional promise. Recording both avoids a false choice between “retrieval problem” and “prompt problem.” Systems need good primary evidence and sensible behavior when evidence is incomplete.
Add a diagnostic test and its observation. Write “supplying approved battery rule removed the eligibility claim in these trials” rather than “RAG fixed.” The first statement is inspectable. The second hides the conditions and overstates the conclusion. Keep configuration versions, tested cases, and the number of trials nearby so another person can repeat the check without guessing what the original reviewer meant.

The final columns are operational: customer recovery owner, permanent repair owner, release condition, and follow-up status. These fields keep the investigation connected to service delivery. A beautifully classified incident that never reaches an owner has not improved the customer experience. Conversely, a customer recovery that does not repair the underlying cause leaves the next person exposed to the same mistake.
For teams starting from scratch, choose a handful of recent incidents with different consequences rather than collecting hundreds of loosely labeled examples. Work them through carefully. You will discover where evidence is missing, which policies are ambiguous, and which systems cannot distinguish attempted actions from completed ones. Those findings tell you what instrumentation and knowledge work are worth doing next.
If you are also deciding whether to configure existing software or build a custom assistant, read the support assistant build, buy, or configure guide alongside this method. The incident evidence makes that decision more concrete. You can ask a supplier to demonstrate the exact retrieval, permission, and action-state behavior your operation needs instead of comparing broad promises about “human-like” service.
My recommendation is to judge progress by the quality of the team’s explanations. At the beginning, the report may say, “The bot made something up.” After a disciplined investigation, it should say which promise lacked authority, which evidence was missing or misapplied, who owns the correction, and how the result will be checked. That is a manageable operating problem. It is also the foundation for an assistant customers can use without having to wonder whether its confidence is doing the work of evidence.
Sources
Still have a question?Ask the DTCUP Community ↗

