Resolve disagreement between chatbot answer reviewers

Identify what the reviewers disagree about

One reviewer may judge factual correctness while another judges readability. Ask each to identify the exact passage and the criterion used. Separate a missing fact from an awkward explanation or a disputed source version. A single numeric rating hides these differences. Keep a short rationale with each judgment so a disagreement becomes a useful correction to the review process rather than a popularity contest.

Return to the authoritative material

Open the expected source and inspect the relevant qualification. If the source itself is unclear, record an unresolved documentation issue rather than forcing agreement. A senior reviewer’s confidence is not a substitute for evidence. Where the question requires product knowledge absent from the public source, update the documentation through its owner or narrow the assistant’s claim to what can actually be supported.

Calibrate on a few examples

Before a larger review, jointly assess a small set containing a correct answer, an incomplete answer and an unsupported answer. Discuss why each classification applies. Use fictional or approved sanitized examples and preserve the agreed criteria. This does not create a scientifically validated accuracy measure; it makes the team’s working standard more explicit and reduces accidental changes in interpretation between review sessions.

Preserve unresolved cases

Do not discard difficult examples because they make the result look untidy. Hold them separately with the missing evidence and the person responsible for resolving it. If disagreement affects a release-blocking fact, do not publish the affected capability until the basis is clear. Consistent review is valuable, but unanimous agreement on a mistaken assumption still needs a source check.

• Write both initial rationales before discussion, then preserve the agreed source interpretation or the exact evidence still needed to resolve it.