Keep a useful change log for chatbot source and answer changes

Record the smallest meaningful change

A useful entry identifies the source or setting, the previous state, the new state, the owner and the reason. Avoid a vague note such as improved accuracy. If a response was wrong, reference the sanitized test case and the expected source passage. This makes the repair understandable to someone who was not present during the investigation and prevents the same debate from restarting later.

Separate cause from coincidence

If you change the source collection, model and instructions at once, it becomes harder to explain why an answer improved. Prefer a scoped change when practical and record concurrent changes when they are unavoidable. Do not claim causation from a better score alone. The log should retain uncertainty about the cause rather than turning a convenient story into a product-performance claim.

Attach the verification result

Name the cases rerun and preserve their outputs or approved references. Record failures as well as passes. For a fictional export correction, the log might show the updated permission paragraph, the refreshed source and two questions rechecked against it. It should not claim that every topic was retested if only those two cases were observed. Precision makes future regression work smaller and more useful.

Keep the record portable and minimal

Store the essential source list, expected answers and decisions in a format the team can export and read independently of a single vendor. Avoid retaining unnecessary customer data or secrets in the log. A portable record supports a later migration or rollback without recreating the entire evaluation from memory. Review the log’s ownership and retention alongside the assistant’s operating responsibilities.

• Use fields for changed object, reason, prior version, new version, related cases, verification result and owner; leave uncertain causes explicitly unresolved.