Hallucination hunt: an exercise on a doctored output

1Four families of error and how to check them

A language model predicts the most plausible text. It does not consult a base of facts, and when the prompt demands a precision the source does not contain, it fills it in. Microsoft says so at the top of its advice: always review and verify. Asking the model to correct itself on its own is not enough: research shows that, without external feedback, self-correction often degrades the answer.

Four families of errors keep coming back. The false fact: a date, a name or a condition altered from the source. The invented reference: a study, a report or an article that does not exist, or that does not say that. The distorted figure: 68% of respondents becomes 86% of employees. The circular argument: the conclusion repeats the premise, "this measure improves productivity since employees are more productive".

Each family has its verification. The fact is confronted with the attached document, passage by passage. The reference is searched for: an opened link, or it does not exist. The figure is recalculated, checking the base (respondents or employees). The argument is tested: does the conclusion bring information absent from the premise?

The signal that triggers verification is always the same: precision without origin. A percentage, a year, the name of a body, an "according to a study" without a link. A confident tone is no sign of reliability.

Tap to enlarge

2A note summary where nothing comes from elsewhere

An HR manager asks for a summary of an attached internal memo, remote_work_memo.docx, for the executive committee.

Weak prompt:

Summarise this memo in five bullets for management.

Summarised answer: five fluent bullets, including a reference to an external report absent from the memo, a rounded percentage and a shifted date, with nothing to tell the memo apart from the addition.

Strong prompt:

Summarise remote_work_memo.docx in five bullets for the executive committee. Rely exclusively on this file: add no reference, figure or date that does not appear in it. For each bullet, quote in brackets the passage of the memo used. If a piece of information is missing, write "not found". End with a list "Points to be checked by a human".

Summarised answer: five bullets, each followed by its quotation, no external reference, the exact percentage with its base, and a final list pointing to the effective date and the evaluation deadline.

What changes: the source is restricted, so nothing can come from elsewhere. Every bullet can be traced in ten seconds. The gap has an honest output, "not found", instead of an invention.

3Find the four planted errors in a summary

Here is the source memo, fictional, then a doctored output that contains four planted errors. Circle them before reading the correction, and note for each the verification you would use.

The memo: "Internal memo of 12 September 2026. From 1 March 2027, remote working goes from one to two days a week for eligible employees, on a voluntary basis. The internal survey of June 2026 (412 respondents out of 650 employees) shows that 68% of respondents want two days. Tuesday remains a mandatory on-site day. An evaluation will be presented to the works council in September 2027."

The doctored output:

1. Remote working goes to two days a week from 1 January 2027 for eligible employees, on a voluntary basis.
2. The internal survey of June 2026 shows that 86% of employees want two days.
3. According to the National Remote Work Observatory (2025 report), this arrangement cuts absenteeism by 12%.
4. This measure will improve productivity, since more productive employees work better remotely.
5. Tuesday remains a mandatory on-site day; an evaluation will be presented to the works council in September 2027.

Correction. Bullet 1, false fact: the memo says 1 March 2027, not 1 January; verification by confrontation with the document. Bullet 2, distorted figure: 68% of respondents (412 out of 650) has become 86% of employees; verification by recalculation and checking the base. Bullet 3, invented reference: no observatory or report in the memo, no link; verification by search, and removal until an opened link confirms it. Bullet 4, circular argument: the conclusion repeats the premise and the memo says nothing about productivity; verification by logical test. Bullet 5 is accurate: do not flag it, a false positive costs time too.

Then rewrite the starting prompt "Summarise this memo in five bullets" with the restriction to the source, the quotation per bullet, the "not found" instruction and the list of points to be checked. The composer below guides you on the Sources and Verification fields.

Self-assessment grid: (a) the four errors are spotted, with no false positive; (b) each error has its verification method; (c) the corrected prompt closes the door to inventions.

Open the prompt composer