Practical guide

Evaluate clarification when a reader’s question is ambiguous

Last materially reviewed 2026-09-25

Quick answerAsk a short clarifying question when a missing detail changes the answer; do not force a clarification when the source already supports a safe response.
What to know

Find the detail that changes the answer

A question such as how do I export may omit the object, format or product version. List which missing details materially change the documented steps. A good response can ask for that detail or state a clearly labelled assumption. It should not guess silently when the choice could produce the wrong instruction. Keep the expected clarification narrow enough that a reader can answer it easily.

What to know

Avoid an interrogation script

Not every brief question needs a follow-up. If the public documentation has one supported export workflow and the question is otherwise clear, answer directly with its limits. Excessive clarification adds friction and can make a simple task slower than ordinary search. Evaluate whether each requested detail is necessary for correctness rather than rewarding a chatbot simply for asking more questions.

What to know

Test the conversation, not only the first turn

Use a fictional exchange in which the reader supplies the missing version after a clarification. Check that the final answer uses that version rather than reverting to the default. Keep the conversation and its expected source together. Do not count a sensible first question as a completed success when the subsequent answer ignores the reply or blends instructions from different releases.

What to know

Record assumptions explicitly

When a response proceeds under an assumption, ask whether the assumption is visible and easy to correct. A reader should not need to infer which plan or version the assistant chose. Use these findings to improve source labels and examples as well as chatbot instructions. Ambiguity is sometimes a documentation design problem, and a clearer article may help both chat and non-chat readers.

  • Record the missing detail, expected clarification and final answer after clarification; the first response alone cannot establish that the task succeeded.
Continue when useful

Next: Keep product versions from contaminating chatbot answers

Choose one default version and make exceptions explicit; a correct answer for the wrong release is still a failed answer.

Open Keep product versions from contaminating chatbot answers →

Sources used for this page

These records support the facts and comparisons above. Merchant-controlled records are labelled so you can separate product claims from independent evidence.

  1. LangChain: evaluation datasets and reference answers — Platform documentation · docs.langchain.com · Merchant-controlled · checked 2026-09-25