Confirm that a supported answer exists
Locate the current source passage a knowledgeable reviewer would use. If the documentation does not contain the answer, the chatbot’s inability to provide it is not necessarily a product failure. Improve the source through the normal editorial process or preserve an honest limit. Do not add an unsupported answer only to satisfy the test or encourage the model to rely on general knowledge outside the agreed scope.
Check the actual imported material
Verify that the relevant article and its operative section are available in the collection. Pay attention to tables, attachments and content behind interactions. A visible article title does not prove every important detail was extracted. If the product cannot expose the imported text, record that uncertainty and use a small targeted question to investigate, rather than claiming the source is present from the URL alone.
Test meaning-preserving wording
Ask the same reader task in a clear alternative phrase without supplying the expected answer. If only a particular keyword retrieves the source, note the terminology gap. Improve headings or synonyms where appropriate, but do not teach the assistant a special case that hides a broader problem. Keep both phrasings in the test record so the apparent repair can be checked again after other changes.
Avoid the more-data reflex
Importing the whole website can introduce outdated or contradictory material without fixing the retrieval issue. Add a source only when it supplies a distinct missing answer and has an owner. If a targeted repair still leaves important questions unresolved, consider a direct article link or a different interface. A smaller well-maintained collection is often easier to evaluate than an unbounded collection with unclear authority.
Sources used for this page
These records support the facts and comparisons above. Merchant-controlled records are labelled so you can separate product claims from independent evidence.
- SiteGPT getting started — Merchant documentation · sitegpt.ai · Merchant-controlled · checked 2026-09-25
- LangChain: evaluation datasets and reference answers — Platform documentation · docs.langchain.com · Merchant-controlled · checked 2026-09-25