SiteGPT vs DocsBot for a maintained documentation corpus

Freeze one corpus for both candidates

Use the same permitted public articles, the same version boundary and the same date. Record which documents each tool actually imports, not simply which URLs you supplied. If one trial contains a newer article, you are testing different knowledge rather than different systems. Keep source import failures visible and exclude unsupported connectors from your assumed feature list until you have checked the current plan.

Different breadth can be useful or unnecessary

DocsBot’s current product offering includes a free entry point and broader sources and actions. Those capabilities deserve attention when the job extends beyond a public website collection. SiteGPT’s content-to-embedded-chat workflow may be sufficient for a narrower project. Extra capabilities are not automatically better: they can add configuration, access and review obligations that a read-only documentation assistant does not need.

Compare decisions a reader can verify

Ask each system how to perform the same supported operation, what happens at a documented limit and how it responds when the answer is absent. Check citation relevance and version accuracy, then record failures by category. Do not average a dangerous invented instruction into a pleasing overall score. A useful comparison preserves the actual answer, expected passage and reviewer rationale so a second reviewer can disagree meaningfully.

Choose the simpler acceptable operation

Select on observed task performance, maintenance effort and the current billing basis together. Different credit or message units prevent a straight price-per-number comparison. We have not conducted a paid side-by-side benchmark and do not declare a measured accuracy winner. If both fail the same question because your source is incomplete, improve that source first; switching vendors is not a substitute for publishing the missing answer.

• Keep a two-column result sheet: imported source coverage, failed cases, review effort and actual plan meter for each candidate.