Build an AI recheck scorecard before you claim an improvement

A better-looking number can answer a different question

Before a team changes a page, ask what it intends to improve: being named, receiving original-source credit, being described accurately, or helping a visitor complete a useful task. Those outcomes can move independently. A more prominent mention with an incorrect price is not simply a better result.

Write one primary outcome and a few guardrails. For a product comparison page, the primary outcome might be an accurate explanation of a buyer-relevant constraint. Guardrails could include a working evidence link and the absence of invented features. This makes a recheck a decision aid rather than a search for a flattering screenshot.

A real research scorecard has several columns

The BBC–EBU report released in October 2025 evaluated 2,709 core-question responses collected May 24–June 10, 2025. Its headline rates were 45% with any significant issue, 31% with significant sourcing issues, 20% with significant accuracy issues, and 14% with significant context issues. The categories overlap; they are not slices of a whole.

The headline denominator is the evaluated core-response set, not the wider 3,113 core-and-custom questions asked. The project covered 22 media organizations, 18 countries and 14 languages. These are dated news-task findings, not a score for your product or for today’s models.

Do not turn two research headlines into your before-and-after

The earlier BBC report, published in February 2025, found significant issues in 51% of 362 reviewed responses to news questions. Those responses were collected December 5–6, 2024. Placing that headline beside the international study’s 45% does not isolate a page edit, a stable audience, or your business’s outcome.

The later report includes its own BBC-only comparison. That analysis should not be replaced with an improvised comparison of differently pooled headlines. For your own content, preserve the actual question set, product, collection dates, rating rules and evidence before claiming that an edit improved the result.

Specify a pass before you look at the new answer

For each question, write an answer requirement a second reviewer could apply. For example: “States whether guest access is available, gives the documented restriction, and links to a source that supports both.” This is a fictional review criterion, not a claim about a particular product.

Keep three verdicts: passes the criterion, fails the criterion, or cannot be evaluated. Record why a verdict is unresolved. A provider refusal, an incomplete collection and a genuinely unsupported answer are different events. Define which belong in the primary denominator before collection, then show exclusions and failures separately so a cleaner-looking rate cannot hide missing evidence.

Keep a matched recheck sheet

One row should hold a stable question ID, exact wording, buyer intent, language, geography if controlled, interface or API, search mode, model label when available, date and time, saved answer, linked sources and reviewer verdict. Add the page change and its publication timestamp. Retain the earlier response instead of overwriting it.

Where practical, review saved answers without showing the reviewer which version is newer. Use the same rules on both sets. If the product changes its model, your question list changes, or the page cannot yet be retrieved, record that event. A new collection may still be useful, but its comparison needs a narrower claim.

Use the calculator as a warning about precision

The calculator uses a Wilson 95% interval for one binary rate, following the NIST statistical handbook. Its initial 10 passes out of 20 trials is a mathematical illustration, not BBC–EBU data, a customer result or a DoneHat benchmark. A 50% observed rate from 20 hypothetical trials has an interval of about 29.9%–70.1%; with 200 hypothetical trials at the same observed rate, it is about 43.1%–56.9%.

This calculation assumes independent binary observations under a stable underlying probability. Repeating one AI question many times can produce correlated answers and does not create that assumption. The calculator does not adjust for clustering, multiple tests, changing models or selective reporting. It does not test whether two rates differ, and interval overlap is not a universal significance rule.

State what changed, then what you still cannot tell

A useful decision note might say: “The new sample contains more answers that meet our prewritten criterion, but the samples are small and the retrieval system changed between checks. We will keep the clearer page because it corrects a documented gap, and collect another planned sample before claiming an answer-visibility lift.” This is an illustrative format, not a measured result.

A controlled comparison can support a stronger causal claim when its design and assumptions are sound. An ordinary before-and-after observation usually cannot isolate your edit from other changes. Do not compensate for that uncertainty by hiding the dates, adding unrelated questions to the denominator, or reporting only the most favorable engine.

Give the recheck an operating purpose

Choose the smallest monitoring job that would change a decision: detect an outdated price, verify an important constraint, or check whether a corrected source is still being misrepresented. A saved report can support that work without an endless schedule of provider calls.

For repeat monitoring, ask whether a paid workflow preserves comparable evidence and helps someone act on a material change. That is a clearer upgrade reason than a promise that every score will rise. Read /methodology before interpreting results, /demo for a worked product flow, or /pricing when a recurring evidence job is worth funding.

Sources

• News Integrity in AI Assistants — October 2025, methods and core-response results — BBC and European Broadcasting Union (accessed 2026-10-17)
• Representation of BBC News content in AI Assistants — February 2025 — BBC Responsible AI Team; retained on News Integrity research site (accessed 2026-10-17)
• Confidence intervals for a proportion — Wilson method — NIST/SEMATECH e-Handbook of Statistical Methods (accessed 2026-10-17)