Build a small, decision-focused chatbot question set

Start from tasks rather than features

Collect a small initial set covering the pilot’s supported reader jobs. For each question, write the expected source passage, essential answer points and unacceptable statements. The count is less important than coverage of materially different situations. Include ordinary questions, missing prerequisites, ambiguous wording and requests outside the documented scope. Do not claim a small sample proves universal accuracy or covers every future reader.

Make the expected answer inspectable

A reference answer does not have to prescribe exact prose. For a fictional export question, it might require the correct permission, supported format and current navigation path. It should forbid an invented automatic backup. This lets reviewers accept different clear explanations while rejecting material errors. Keep the supporting source URL and version so a later editor can understand why those requirements were chosen.

Separate development from evaluation

Use some questions to improve the setup and reserve others for a later check. If every visible failure becomes a rule in the instructions, the bot may look better on the same sample without generalizing. Keep the reserved set out of routine tuning notes. LangChain’s evaluation guidance supports datasets with expected outputs and document references; our worksheet adapts that idea into a manual, vendor-neutral review.

Record what the set cannot establish

A question set is an instrument for finding problems, not a certificate. It will miss unfamiliar language, changed source pages and unusual combinations. Track coverage gaps explicitly and add cases only when they represent a new failure class. Preserve the old cases so improvements do not erase regressions. Avoid adding many cosmetic paraphrases merely to make the test total look impressive.

• Use columns for question, reader task, source version, required facts, forbidden claims and the reviewer’s reason for the final classification.