Documentation chatbot or better site search?

Identify the failed step

Watch where the reader gets stuck using permitted, privacy-respecting feedback. Is the article impossible to find, difficult to understand, or split across several pages? A chatbot may help combine relevant material, but a missing navigation label can be a much smaller repair. Do not interpret every failed search as demand for an AI assistant. The underlying question may need a better heading, alias or direct link.

Run a fair non-chat baseline

Give a reviewer the question and the current documentation search. Record whether they find the correct article and complete the task without coaching. Then compare an authorized chatbot evaluation on the same question. Use separate reviewers or vary the order to reduce learning effects. A second attempt is often easier simply because the person now knows the answer, not because the second interface is better.

Preserve the cost of checking

A chat response can feel faster while requiring a longer fact check. Include time spent opening citations, correcting a wrong version and recovering from an unsupported instruction. Use a small written observation log rather than a claimed universal speed improvement. If the chatbot adds a verification burden to simple navigation questions, consider linking the relevant article directly instead of generating an answer.

Keep a mixed interface possible

Search, navigation and chat can coexist. Let readers choose without hiding the original articles or forcing conversational steps. For a compact stable manual, improved headings and search synonyms may be sufficient. For a cross-article task, a grounded summary may be useful if citations remain inspectable. This is an editorial decision framework, not a measured claim that one interface universally improves engagement or support costs.

• Record the article found, task completed, verification effort and unresolved mistake for both interfaces using the same reader scenario.