Lesson 2 · Beginner · 15 min
How ChatGPT Responds: Token Prediction and Probabilities
Understand ChatGPT token prediction engine: statistical probabilities, attention mechanisms, and the root causes of plausible hallucinations.
- Goal
- You will analyze the probabilistic nature of ChatGPT to prevent plausible hallucinations using strict source framing constraints.
- Skills
- Check
Your first attempt, unaided
Ask ChatGPT to complete a rare proverb while strictly forbidding it from guessing if it lacks high confidence.
ChatGPT operates as a probabilistic autoregressive engine that computes the statistically most probable following token at each step. While this architecture guarantees fluent syntax, it has no intrinsic comprehension of verified reality. When deprived of reference material or strict instructions, it bridges gaps with plausible fabrications known as hallucinations. Always supply source data and explicitly mandate negative fallback statements.
1The statistical mechanics of next-token prediction
The statistical mechanics of next-token prediction clarifies why ChatGPT demonstrates remarkable eloquence while occasionally fabricating factual statements. The model parses input into statistical fragments called tokens and evaluates probability distributions across tens of thousands of candidate tokens.
At every step, the attention layers analyze the active prompt context and select the next token that best satisfies learned text patterns. If verified knowledge is missing from its context window, the model blends surface associations into synthetic assertions. This behavior constitutes the technical root of factual hallucinations.
Core mechanism Role in ChatGPT Operational risk Recommended mitigation Tokenization Breaks raw text into sub-word tokens Token budget variations across languages Monitor context token limits Probability scoring Calculates likely next words Grammatical perfection without truth Demand literal source quotes Plausible hallucination Fills factual gaps with plausible prose Convincing but false assertions Enforce strict negative fallback rules Attention mechanism Weights tokens across input prompts Early and trailing prompts dominate focus Position critical constraints at boundaries 
Diagram of token predictionDiagram generated by AI and reviewed 2Preventing the fabrication of imaginary legal citations
Preventing the fabrication of imaginary legal citations ensures organizational decisions never rely on synthetic case law produced by a probability engine.
A compliance officer requests recent appellate case law regarding overtime calculation rules.
Weak prompt.
Give me the three latest court decisions from 2024 penalizing companies for overtime calculation errors with official docket numbers.ChatGPT produces three authentic-looking case citations complete with docket numbers and dates, none of which exist in official legal gazettes.
Strong prompt.
Context: Regulatory compliance research. Task: Based exclusively on the statutory excerpts pasted below [paste official statute], list overtime calculation rules. Constraint: If official precedent is missing from this excerpt, reply: Precedent not found in source text. Never invent docket numbers.What changes. The explicit negative boundary instructs the model to acknowledge document limits rather than interpolating synthetic citations.
3Trigger a hallucination and neutralize it with constraints
Trigger a hallucination and neutralize it with constraints to observe directly how strict framing shifts model behavior from speculative storytelling to factual rigor.
Ask ChatGPT a leading question about a fabricated regulation (for example an imaginary 2023 international treaty on remote work ergonomics).
Inspect its unconstrained answer, then issue this corrective prompt in a new conversation:
"You are an objective auditor. Verify the existence of the regulation described above. If primary official legal sources do not confirm this agreement, reply solely with: Document unverified in official corpus."
Self-evaluation rubric: (a) the unconstrained turn exposes the model's bias toward pleasant agreement; (b) the amended instruction enforces rigorous verification; (c) the final response cleanly rejects speculative fabrication.
4Believing that grammatical fluency implies factual accuracy
Believing that grammatical fluency implies factual accuracy leads professionals to circulate unverified synthetic data simply because it sounds authoritative.
A consultant copies market share figures generated by ChatGPT into an executive deck, captivated by the balanced sentence structure and precise percentages. During the client presentation, the executive team realizes the cited statistics were statistically generated fiction rather than audited market data.
Correction: isolate every factual metric and trace it back to an identifiable, primary source document.
Rule to remember: impeccable prose frequently serves as the disguise for statistical language hallucinations.
5Quiz
Three questions, instant feedback. Each option comes with an explanation.
6Proof of mastery
Build a prompt querying an obscure fact, run it unconstrained to observe fabrication, then apply a negative constraint to elicit an honest refusal.
Criteria
What you wrote at the start of the lesson
Going further
Review glossary entries for token, hallucination, and context window. Next, master practical prompt framing in Writing an effective ChatGPT prompt: the ROCC method. For advanced verification techniques, examine Hunting ChatGPT hallucinations.
Frequently asked questions
What is a token in ChatGPT?
A token is a sub-word text chunk averaging about 4 characters in English. Language models ingest and produce text sequentially token by token.
Why does ChatGPT state false claims with high confidence?
The model optimizes for the most statistically plausible sequence of words, lacking an innate database of verifiable facts.
How can you force ChatGPT to admit when it lacks information?
Include an explicit negative constraint: 'If the answer is not present in the provided source text, reply solely with: Information not found'.