DTC technology × growth systems — built to compound.
DTCUP / FIELD NOTES / BUILD

Your RAG Chatbot Found a Document. Did It Find the Answer?

Trace a support answer from source extraction through search, ranking and final context. A controlled product-compatibility example shows how relevant documents can still leave the answer unsupported.

Published Oct 2, 202624 min read
Jack Hu
Jack Hu

7 years of experience building and operating brand websites, specializing in website development, paid advertising and SEO. Built a brand website from launch to US$10 million in annual sales.

The Short Answer

Check whether the retrieved material supports the exact claim, not merely the topic. Trace required passages through eligibility, initial retrieval, ranking and final context. Preserve identifiers and conditions, then evaluate answer correctness and appropriate abstention separately.

Key takeaways

  • Topic relevance, product applicability and claim support are different requirements.
  • Inspect extracted text before changing the search model; tables and headings can lose essential relationships.
  • Trace the required passage through every selection stage to locate the first observed break.
  • Test valid, invalid, ambiguous and unavailable-answer cases without treating a retrieval score as a truth probability.
Editorial concept of a magnifying lens on a forest-green rail inspecting one of three translucent evidence plates with an orange alignment pin.

A customer asks whether filter FL-210 fits purifier P1. The assistant finds a page about replacement filters, quotes a sentence from it, and confidently says yes. The page is real. The citation works. The answer is still wrong: the quoted sentence describes purifier P2, while the P1 compatibility row was never included in the material given to the model.

This is the awkward middle of retrieval-augmented generation, usually shortened to RAG. The system has retrieved something relevant to the topic without retrieving enough evidence to answer the actual question. A successful search request and a well-written answer can hide that distinction. If the team only checks whether a document was returned, the failure can look like an inexplicable model hallucination.

I would diagnose this kind of problem by following the evidence through the retrieval process. Did the right source exist? Was it eligible for this customer and task? Did the search find the passage? Did the selected context preserve the conditions? Did the final answer stay within what that context supported? Each question points to a different repair.

We will use a small fictional purifier knowledge collection throughout this guide. Product names, compatibility rules, counts and test results are invented for explanation; they are not real specifications or DTCUP customer results. The goal is to help a website or support owner work with an engineer on a useful investigation, without needing to become a search researcher first.

1. Separate finding a document from supporting an answer

Think of retrieval quality in three layers. Topic relevance means a result concerns the same general subject. Case applicability means it concerns the correct product, version, market or situation. Answer support means the available evidence establishes the particular statement the assistant is about to make. A passage can pass the first layer and fail the other two.

For example, a general filter-care article is topically relevant to FL-210. A P1 replacement guide may mention FL-210 explicitly. Neither necessarily answers whether it fits P2. The compatibility table might answer that question, but only if the row labels and column headings survive extraction. A search result containing the word “compatible” is not enough when the missing heading determines compatible with what.

Do not collapse these layers into one score called accuracy. A retrieval score usually helps rank candidates under a particular method. It is not automatically a probability that every claim in a generated response is correct. A high-ranked result can be the best of several inadequate candidates. The system needs a way to recognize insufficient evidence rather than treating first place as a certificate of truth.

It also helps to distinguish complete from partial support. A passage may establish that a filter fits but say nothing about whether it is in stock, included under warranty or available in the customer’s country. If the customer asks all four questions, the assistant may need different sources or a live lookup. One relevant document should not become a license to answer every nearby question.

This is why I start with the proposed answer’s claims. Write down the statements that would need evidence before looking at the search results. For the initial example, the central claim is a compatibility relationship between two exact identifiers. If the answer also promises delivery by Friday, that is a separate claim with a different evidence requirement. Keeping the claims visible prevents the investigation from drifting into a generic search-tuning exercise.

The broader AI support diagnosis guide covers knowledge, timing and action failures. Here we are concentrating on the path between approved knowledge and the context the answering model receives. Fixing that path can improve answer quality, but it cannot turn an incorrect product specification into a correct one.

Three retrieval checks distinguish a page about filters, a passage about the correct purifier model, and the approved rule that FL-210 is not compatible with P1.
Topic relevance, case applicability and claim support are separate checks. Passing one does not imply the others.

2. Build a small collection whose answers you can inspect

Before changing a production search system, create a controlled collection that reproduces the failure. It should be small enough for a person to read completely and rich enough to contain plausible distractions. A single perfect document will not reveal whether the system confuses versions or ranks a general article above a precise table.

Our fictional collection has six documents. D1 is an approved compatibility table: FL-110 fits P1, FL-210 fits P2, and neither cross-model combination is approved. D2 is a current filter-care guide for both purifiers. D3 is an older launch note describing a prototype filter with similar wording. D4 explains how to identify the purifier’s model label. D5 is a public shipping-process page. D6 is an internal service note that a public assistant is not permitted to retrieve.

The example in the introduction contains the wrong pairing: FL-210 does not fit P1 under the approved fictional rule. A passage saying it fits P2 cannot establish compatibility with P1. Also test the valid pairing, FL-210 with P2, which should receive a supported positive answer. Keeping both positive and negative cases prevents a system from appearing safe merely because it rejects every compatibility request.

Give every document a stable identifier, revision, intended audience and source owner. Record the exact extracted text that enters the index, not only the attractive document as a person sees it. The visible table can be correct while the extraction output is scrambled. If a PDF parser emits all product names first and all compatibility values afterward, the model may receive a list of true words with the relationships removed.

Preserve a separate answer key for the evaluation questions. For a compatibility question, list the required relationship and the source passage that establishes it. For an identification question, list the approved distinguishing detail. For a question about current inventory, mark that the static collection cannot answer it. The answer key should express the expected decision, not require one exact sentence.

Include at least one unavailable-answer case and one permission boundary. The public system should not use D6 even if it appears to contain a convenient solution. It should not infer live stock from D5’s shipping description. These cases reveal whether the retrieval process respects eligibility and whether the answering stage can stop when the eligible evidence is insufficient.

A small collection is a diagnostic instrument, not a production benchmark. It lets the team reproduce a specific failure and inspect the stages. Once the repair works there, test a broader set of real authorized cases and nearby counterexamples. Do not claim that six clean documents prove the system will behave well across thousands of mixed-format records.

3. Inspect the extracted relationship before changing the search model

The simplest retrieval improvement is sometimes fixing the source representation. A table cell saying “Yes” has almost no meaning without its row and column. A paragraph beginning “This version” may depend on a heading several screens earlier. A warning can be separated from the procedure it restricts. These are representation problems, and a stronger model may only make better guesses around them.

For D1, a useful extracted unit reads: “Purifier P2 accepts replacement filter FL-210. FL-110 is not approved for P2. Source: compatibility table, approved revision C.” The wording in this teaching example preserves the relationship explicitly. In a real system, the transformation must remain faithful to the approved table, and a reviewer should verify it. Do not generate new compatibility statements from a table whose layout the parser has not understood.

Compare the rendered document with the indexed text side by side. Check identifiers, units, negative words, exceptions and headings. Pay particular attention to “not,” “only,” “before,” and “except.” Losing a decorative border may be harmless; losing a negative can reverse the rule. If the source includes a footnote that changes eligibility, the extracted record must retain the footnote’s relationship to the relevant claim.

Splitting a document into smaller passages is often called chunking. The useful unit is not necessarily a fixed number of characters. It is a passage that can be interpreted correctly with the context supplied alongside it. A short compatibility row may need the table title and version. A procedure may need several ordered steps and a warning. A very long passage may bury the decisive sentence among irrelevant material.

I would test chunk boundaries with actual support questions. Ask a reviewer to read only the passage that would reach the assistant. Can they identify the product, the condition and the answer without opening another document? If not, add the missing context or retrieve a linked parent section. Do not assume the model will reconstruct relationships that a careful human cannot establish from the same input.

Anthropic’s contextual retrieval research examines adding document-specific context to passages before indexing. The useful principle here is that isolated text can lose meaning. Its reported benchmark improvements are results from its experiments, not a forecast for your support system. Test any contextualization method against your own questions, and verify that generated context does not introduce facts absent from the source.

Keep source text and added context distinguishable in the implementation. If a summary says a passage applies to all products when the original mentions only P2, the summary has created a new error upstream of the answer. An audit should be able to identify which words came from the approved source and which were added to aid retrieval. Helpful context should clarify authority, not impersonate it.

An isolated yes cell lacks meaning, while a complete extracted compatibility statement preserves purifier P2, filter FL-210, the excluded alternative and revision C.
Preserve the relationship the customer needs. More accurate search cannot reliably repair a relationship that extraction removed.

4. Give exact identifiers and ordinary language different jobs

Customers do not always use the words in your documentation. They may ask for “the cartridge inside the air cleaner” instead of “replacement filter.” Searching by meaning can help connect those phrases. But when a customer supplies FL-210, the difference between FL-210 and FL-110 is not a stylistic variation. It is part of the question’s identity.

A useful retrieval design can combine these needs. Exact identifier fields help locate the correct product or error code. Text and semantic retrieval help with descriptions, symptoms and alternative wording. The combination is often called hybrid search. The details vary by system; the operating requirement is that broad semantic similarity must not silently override a material identifier mismatch.

Microsoft’s Azure AI Search documentation describes combining full-text and vector search results, with rank fusion used in its implementation. It also notes the value of keyword search for identifiers and specialized terms. That is a useful implementation reference, not a claim that every hybrid configuration will improve your results or that a rank-fusion score establishes factual support.

Normalize identifiers carefully. Removing harmless spaces or standardizing letter case may improve matching when the product system defines those variants as equivalent. Removing every hyphen or letter can be dangerous if the business uses them to distinguish revisions. Keep a documented normalization rule and test it on known distinct items. Search convenience should not rewrite the product catalog’s identity rules.

Typos need a separate decision. If a customer types FL-21O with a letter O instead of zero, the system may suggest FL-210 as a possible match. It should not always treat the suggestion as confirmed, especially when another valid identifier could fit the input. A clarifying question or a visible candidate selection can be more reliable than silently correcting the customer’s product into the wrong one.

When no identifier is available, retrieve the approved identification guidance first. The customer’s description may narrow the possibilities without settling the answer. In our example, “the larger purifier” is not enough unless the approved documentation establishes a unique mapping. Ask for a label or another distinguishing fact that the user can reasonably provide. Do not fill the gap from a product photograph that the system cannot inspect reliably.

The practical test set should contain both language and identity variations. Use an exact code, an approved alias, a common typo, a description without a code, and two similar valid codes. If a change improves the broad descriptive questions but starts confusing identifiers, the average result is hiding an important regression. Review the types separately before choosing the configuration.

5. Decide which evidence is eligible before arguing about rank

A search engine can rank a document very highly even when the assistant should not use it. The prototype launch note D3 may contain the exact phrase in the question. The internal note D6 may be unusually detailed. Neither fact makes those documents suitable for a public customer answer. Relevance and permission solve different problems.

Define the eligible set for the task. It may depend on workspace, audience, publication state, product version and effective period. Some conditions should be enforced as hard restrictions; others may require a question before the system can choose. The implementation must enforce access outside the language model. An instruction saying “do not reveal internal notes” is not a substitute for preventing unauthorized notes from entering the context.

Be careful with unknown conditions. If the customer has not identified the purifier, filtering to the latest model is an assumption, not an evidence-based decision. The system can retrieve identification guidance or present a bounded comparison of the possible models. It should not quietly choose whichever product sells most. A popular wrong answer is still wrong.

Effective dates require similar care. The current service policy might differ from the policy that applied to a historical purchase. Some answers depend on the present service route; others depend on the terms relevant to the original transaction. The retrieval configuration needs the business’s rule for that distinction. A blanket “newest document wins” rule can be as misleading as retrieving an obsolete page.

Log why a candidate was excluded when doing controlled QA, while respecting the user’s permission boundary. If the expected compatibility document is missing because it was incorrectly marked internal, changing the embedding model will not solve the problem. If it is correctly restricted, the answer key may be wrong to expect a public answer from it. The investigation needs to distinguish bad metadata from a legitimate restriction.

The maintained knowledge structure is covered in the AI support knowledge-base guide. Retrieval depends on that upstream work. Search cannot reliably infer approval, ownership and audience from the tone of a document. Those conditions need to be represented deliberately, then carried through the index and checked where the system selects evidence.

6. Inspect what survives ranking and the context budget

Many systems first retrieve a broad candidate set and then select a smaller set for the answering model. A reranker may reorder candidates using a different relevance method. The final prompt may trim passages to fit a context limit. A source can therefore be found early and still disappear before generation. Looking only at the initial search results misses this failure.

Record the passage identifiers at each stage: eligible corpus, initial candidates, reranked candidates, and final context. For the failed compatibility question, ask exactly where D1’s required row disappears. If it is absent from the initial candidates, investigate indexing, query interpretation or retrieval. If it appears there but falls below the final cutoff, investigate ranking. If it survives but loses its heading during prompt construction, investigate context assembly.

Increasing the number of returned passages can help when necessary evidence falls just below the cutoff. It can also add obsolete, redundant or distracting text. The right number depends on the question and the quality of the candidates. I would test several bounded configurations instead of treating a large context window as free insurance. More text creates more opportunities for conflicting conditions to be combined.

Deduplicate at the level that matters. Three copies of the same P1 guide should not crowd out the single P2 compatibility table. Near-duplicate pages with different effective dates should not be merged blindly either. Preserve the authoritative revision while retaining the distinctions needed for the task. A candidate count that looks diverse can actually be several fragments of the same unhelpful document.

Multi-part questions may require several evidence units. “Does it fit, and how do I replace it?” needs compatibility plus an approved procedure. Selecting only the highest-ranked passage may answer the first half and encourage the model to improvise the second. Build a claim-to-evidence checklist for the expected answer and inspect whether the final context covers every material part. Partial coverage should produce a partial answer or a useful follow-up.

Watch how the system handles contradictions. Giving the model both an obsolete prototype note and a current specification without clear authority can create an avoidable interpretation problem. The solution may be to exclude the obsolete note from the public path, not to add another paragraph asking the model to be careful. When legitimate sources conflict, the system should expose the unresolved conflict rather than blend them into a new rule.

Evidence trace follows a required compatibility row through eligible sources, initial candidates, reranking and final context, identifying the stage where a row can be lost.
Trace the required passage through every selection stage. “Search found it” does not mean the answering model received it.

7. Measure evidence completeness with a clear denominator

A retrieval evaluation needs an answer key that identifies the required evidence for each question. Some questions need one passage. Others need several complementary passages. If two sources are valid alternatives, either may be sufficient. Document these conditions before scoring results, or the metric will reward whatever happens to be easiest to count.

For a small operational review, I like a simple task-level measure: the proportion of answerable questions for which the final context contains all required evidence. Call it complete-evidence coverage and define it explicitly. It is not a standard claim about every retrieval system, and it is not the same as answer correctness. Its purpose is to reveal whether the model had a fair opportunity to answer the specific task.

Suppose our synthetic evaluation contains 20 questions. Sixteen are answerable from the eligible collection, and four deliberately lack sufficient eligible evidence. In the baseline configuration, 12 of the 16 answerable questions receive complete evidence. Complete-evidence coverage is therefore 12 divided by 16, or 75%. Do not use 20 as the denominator, because the four unanswerable cases cannot correctly receive the missing evidence.

Now inspect the final answers separately. Suppose ten of the sixteen answerable questions receive a correct, useful answer. Two questions had complete evidence but still received an incorrect interpretation. The other four lacked required evidence. This tells the team that retrieval improvements alone cannot explain all six answer failures. Some work belongs in the answer construction or decision logic.

The four unanswerable questions need a different measure. Suppose three receive an appropriate clarification or handoff, while one receives an invented stock claim. Report that result directly: three appropriate abstentions out of four, with one unsupported answer. Do not hide it inside the 75% evidence score. A system can improve retrieval coverage while becoming more willing to make unsupported promises.

After a repair, imagine that fifteen of the sixteen answerable questions receive complete evidence, and fourteen receive correct answers. The respective rates are about 94% and 88%. All four unanswerable questions receive an appropriate bounded response. These invented numbers illustrate how to report different outcomes; they are not benchmarks or promised results. With such a small set, one case changes the rate substantially, so inspect the actual failures rather than celebrating the percentage alone.

For larger evaluations, engineers may use recall at a chosen cutoff, precision, ranking metrics and graded relevance judgments. Those can be valuable when the labels and definitions match the task. Ask what counts as a relevant passage, how multiple required passages are handled, and whether scoring occurs before or after the final context is assembled. A familiar metric name does not remove the need to understand its denominator.

For example, retrieving one of two required passages gives passage recall of one half, but it may still leave the customer decision entirely unsupported. Retrieving either of two equivalent authoritative passages may be sufficient. The evaluation record must distinguish complementary evidence from interchangeable evidence; otherwise the same arithmetic can describe two very different levels of readiness to answer.

Synthetic twenty-question evaluation separates sixteen answerable cases from four insufficient-evidence cases, with baseline complete-evidence coverage of twelve of sixteen and correct answers of ten of sixteen.
Measure retrieved evidence, final answers and appropriate abstention separately. The numbers describe different parts of the system.

8. Run controlled comparisons before changing everything

A failed answer often triggers several simultaneous changes: a new model, a longer prompt, more retrieved passages, different chunk sizes and a revised knowledge article. If the result improves, the team cannot tell which change helped. If another question breaks, the team cannot tell which change caused it. Controlled comparisons make progress easier to explain and reverse.

Begin with a saved baseline. Preserve the approved source revision, extracted passages, retrieval configuration, answer instructions and evaluation questions. You do not need to expose secrets or customer records in the review report. You do need enough version information to reproduce the relevant behavior. A screenshot of the final reply alone is not a reproducible baseline.

First test an oracle context: give the answering stage the reviewed evidence it should have received, without changing the question. If the answer remains wrong, retrieval is not the only problem. The model may misinterpret a negative, combine incompatible conditions or make an unsupported extra promise. If the answer becomes correct, that supports investigating retrieval and context assembly, although repeated tests are still needed because outputs may vary.

Next, repair one observed retrieval break. If exact product identifiers are being confused, test the identifier handling. If a table row lost its heading, fix the representation. If duplicate pages crowd out the relevant source, test deduplication. Keep the other components stable where practical. This is more informative than adopting a fashionable search technique before confirming what failed.

Run the repaired configuration across the whole small evaluation set, not just the original question. Include ordinary answerable requests and the insufficient-evidence cases. A configuration that fixes P2 by filtering everything to P2 may damage P1. A configuration that suppresses every uncertain answer may increase handoffs on straightforward questions. The repair should improve the intended decision without silently moving the cost somewhere else.

Repeat cases when generation variability matters, and keep the trials separate from the number of distinct questions. Twenty runs of one question are not twenty different customer situations. They can reveal instability, while a broader case set reveals coverage. Report both dimensions when comparing systems. This avoids treating a single fortunate answer as evidence that a recurring failure is gone.

Finally, keep a short change note: observed break, proposed repair, configuration difference, cases improved, cases worsened and release decision. If the evidence is mixed, state that. A useful engineering discussion can end with “the table extraction is fixed, but the stock question still needs a live-data route.” That is clearer than saying the new RAG pipeline is smarter.

9. Make missing evidence useful to the customer

A system that cannot support the requested answer should still help the customer move forward. The right response depends on why evidence is missing. If the product version is unknown, ask for the distinguishing detail. If the approved source is temporarily unavailable, explain the limitation and offer the appropriate support route. If the question requires live stock, use an authorized live lookup or say that current availability cannot be confirmed.

Avoid generic uncertainty language that leaves the customer doing all the work. “I am not sure” is honest but incomplete. “I can confirm compatibility once we identify the model label on the rear panel” is more useful when that is the approved identification method. The follow-up should request the minimum information that can change the decision, not a long list of unrelated account details.

Do not disclose restricted evidence while explaining the limitation. If D6 is internal, the assistant should not say, “Our private service note says this is a recurring defect.” The public response must stay within its permitted knowledge and process. The fact that an internal source might answer the question does not authorize its use in a public explanation.

Similarly, do not convert a search failure into a product claim. “I could not find a compatibility record” does not necessarily mean the parts are incompatible. In our controlled collection, an explicit approved negative supports a negative answer. In a production collection, absence may mean incomplete data, an indexing problem or an unsupported combination. The wording should reflect which of those the business can establish.

Give the handoff enough context to avoid making the customer start again. Record the question, known identifiers, the missing fact and the evidence limitation in the authorized support workflow. Do not include unnecessary personal detail or a large dump of irrelevant search results. A concise statement of what remains unresolved is often more useful to the next agent than the entire generated conversation.

10. Judge cost and latency alongside the failure pattern

A more elaborate retrieval pipeline can improve some cases while adding delay, cost and maintenance work. The business decision is not simply whether it scores better on a benchmark. Ask whether the improvement addresses failures that matter, whether the added steps are reliable, and whether customers still receive a timely useful response.

Measure the stages separately during testing. Search latency, reranking latency and generation latency can behave differently. A slow response might come from an external source lookup rather than the model. If a timeout causes the system to answer without the required evidence, the apparent performance improvement has created an accuracy problem. Define what should happen when an essential stage fails or exceeds its allowed time.

Use the simplest configuration that meets the observed task requirements. Exact product lookup may be enough for some identifier questions. Semantic retrieval may help with vague descriptions. A multi-source policy question may justify additional retrieval and review. Different tasks can have different routes, provided the routing rules are explicit and tested rather than hidden inside an unpredictable prompt.

Be cautious with caching. A cached answer may depend on product revision, audience, market and time. Reusing it across a different context can bypass the careful selection logic you just built. Cache stable approved facts or appropriately scoped results only when the implementation preserves those conditions and can invalidate them after changes. Faster repetition of an obsolete answer is not a successful optimization.

Discuss operating cost in terms of useful outcomes. If an extra retrieval step prevents an expensive wrong replacement recommendation, its value differs from an extra step that only makes a greeting more polished. Conversely, a complex pipeline that barely changes answer quality may not deserve its maintenance burden. Connect the technical comparison to the customer decision and the team’s ability to keep the system working.

11. Read one complete trace before accepting the repair

Here is how the fictional failure could look in a review. The question asks about FL-210 and P1. The query parser retains FL-210 but drops P1 while rewriting the request as “replacement filter compatibility.” Initial retrieval returns D1, the general care guide and the prototype launch note. A later selection step chooses the paragraph about FL-210 and P2 because it repeats the filter identifier most clearly. The model then answers the original question with that paragraph as its evidence.

There are two breaks to investigate. The rewritten query lost a material constraint, and the final selection accepted a passage about a different purifier. The answer stage also failed to notice the mismatch. The repair should preserve both identifiers through query construction, ensure the compatibility relationship reaches the final context, and require the answer to refer to the actual pair being asked about. Simply adding “be accurate” to the prompt would not explain or repair those specific breaks.

Now rerun the original question with the approved D1 row supplied directly. The expected answer is a clear negative for this pairing, followed by the supported alternative if the source establishes it. If the assistant says that FL-210 fits P2 but never answers whether it fits P1, it has provided a true statement without completing the customer’s task. Mark that as incomplete rather than allowing factual fragments to stand in for useful resolution.

Run the positive pairing next: FL-210 with P2. The expected answer changes, while the source and identity rules stay the same. Then ask about an unspecified purifier. The system should identify what is missing and request the relevant model detail. Finally, ask about a product identifier absent from the collection. It should not map that identifier to the nearest familiar one without confirmation. These four questions reveal whether the repair understands the relationship or merely memorizes the original negative answer.

Inspect the final context for each run. It should preserve the exact product pair, the approved revision and the relevant constraint. Confirm that the internal note remains excluded and the prototype note has not become an alternative authority. Then review the generated reply for extra claims. An answer that correctly rejects the pairing but promises a free replacement has solved the compatibility problem while creating a policy problem. The acceptance check should follow the whole customer-facing promise.

If the trace is clean in the controlled collection but fails in production, compare the actual indexed revisions and routing configuration. Perhaps production still holds an earlier extraction, the product filter is not applied on one channel, or an answer cache bypasses the repaired route. The investigation should follow the observed difference. Do not assume that the code deployed successfully means every relevant index, cache and channel received the new behavior.

This end-to-end trace is worth saving as a compact acceptance record. Include the four questions, expected decisions, source revision, final passage identifiers and observed outcomes. It can become a regression check when the team changes document parsing, retrieval providers or answer models. Its value is not that it predicts every future failure. It protects a specific business distinction that the team has already learned matters.

12. Give an engineer an evidence brief they can act on

A useful bug report begins with the customer question and the specific unsupported or incorrect claim. Include the expected decision and the approved source that supports it. Then attach the relevant trace: source revision, extracted passage, candidate positions and final context. This gives the engineer a concrete path to inspect instead of an abstract request to improve hallucinations.

State the boundary conditions too. Was the user public or authenticated? Was the product identified? Which market or effective period applied? Was the required evidence eligible? If the answer needs a live record, say so. These details help prevent a repair that works by removing a necessary permission or assuming a convenient product version.

Ask for a comparison that shows the ordinary cases as well as the failure. The brief might request the baseline and proposed configuration on sixteen answerable questions and four insufficient-evidence questions, with the failed claims listed. The exact numbers are not important outside our example. The important part is that the acceptance decision includes both useful answering and safe behavior when evidence is absent.

Keep the release decision understandable to the business owner. “Hybrid reranking v3 improved relevance” may be technically accurate but operationally incomplete. “The approved compatibility row now reaches the model for fifteen of sixteen answerable cases; one table extraction remains unresolved; no restricted notes appeared in the public tests” gives the owner something concrete to assess. Only make those statements when the recorded checks support them.

My preferred sequence is to repair source representation, eligibility and identity handling before buying complexity to compensate for them. Then tune ranking and context selection against explicit questions, and test the final answer separately. A retrieval system earns trust when the team can explain why the evidence applies—not merely when it can display a link beneath a confident reply.

Retrieval investigation brief lists the failed claim, approved evidence, missing pipeline stage, controlled repair and acceptance checks for answerable and insufficient-evidence cases.
A concrete evidence brief connects the customer failure to a repair that can be tested and reviewed.

Sources

  1. www.anthropic.com — contextual-retrieval
  2. learn.microsoft.com — hybrid-search-overview

Still have a question?Ask the DTCUP Community ↗

Your workspace

Selected services

DTCUP / YOUR NEXT PERSPECTIVE

Stay curious.
Stay connected.

Ideas for the work behind growth—websites, brands and the apps that connect them.