Follow the information, not the chat bubble
A reader asks a question, the system looks for relevant material, and a model produces a response. In a retrieval-based setup, this is not the same as permanently teaching a model every fact in your product. The source collection and the retrieval step still matter. A polished sentence can be based on incomplete, outdated or irrelevant material. Start your evaluation by asking which exact documents were available for that answer.
Separate three kinds of failure
A missing page is a source-coverage problem. A relevant page that is present but not found is a retrieval problem. A response that misstates a found passage is an answer problem. The remedies differ. Rewriting instructions will not restore an excluded document, and importing more documents will not necessarily fix an unsupported conclusion. Keep these categories separate in your review notes before changing several controls together.
Define a narrow first job
For a fictional export feature, a useful first job might be answering how to download a CSV from the current product version. It should not include looking up a customer’s private balance or promising a refund. Write the allowed job, public source URL and expected handoff before choosing software. This gives you something observable to test, instead of asking whether the chatbot generally sounds intelligent or friendly.
Keep a non-chat route open
A reader should still be able to browse the source article or use ordinary documentation search. Chat is an additional route, not a reason to remove readable instructions. Prefer static guidance when the answer is short, stable and easy to find. This publication offers documentation-led comparisons and original evaluation methods; it does not certify any vendor’s accuracy or replace your review of the actual deployed configuration.
Sources used for this page
These records support the facts and comparisons above. Merchant-controlled records are labelled so you can separate product claims from independent evidence.
- SiteGPT getting started — Merchant documentation · sitegpt.ai · Merchant-controlled · checked 2026-09-25
- LangChain: evaluation datasets and reference answers — Platform documentation · docs.langchain.com · Merchant-controlled · checked 2026-09-25
- LangChain: retrieval-augmented generation — Platform documentation · docs.langchain.com · Merchant-controlled · checked 2026-09-25