Evaluating a prompt on a set of ten cases

1Ten cases, three runs, a score instead of an opinion

Anthropic opens its prompt engineering guide with a piece of advice few people follow: first define a success criterion and a way to test it, then only work on the prompt. Without measurement, every change is a bet, and you keep the one you liked on the last try. With ten cases and a score, the discussion changes in nature: you compare figures.

Building the set takes about an hour. Choose ten representative fictional or anonymised inputs: six ordinary cases, three hard cases (double topic, missing data, ambiguity) and one trap case whose expected answer is "not found".

Always write the expected answer before any run. Without this, you will evaluate Copilot based on its confidence rather than its factual accuracy. The score simply reflects the count of validated cases out of ten.

Why three runs? Because the same request produces different answers from one time to the next. Wang and his co-authors built an entire method, self-consistency, on this variability: several runs followed by a vote clearly improve the result. For you, three runs give a range (8, 9, 8) instead of a misleading figure.

The golden rule is isolation: a single change between two series. If you change the doubt rule and add an example simultaneously, any measured gain remains inexplicable. Microsoft recommends iterating and regenerating; the test set objectively adjudicates which revision to preserve.

Have outputs scored by an independent peer in a blind review. Huang and co-authors demonstrated that models struggle to correct reasoning defects autonomously. The same cognitive bias affects prompt authors, who naturally favour their latest drafts.

Tap to enlarge

2A message sorting prompt from 5 to 9 out of 10

A support team wants to classify the messages of a shared mailbox into five categories.

Turn 1 (Evaluating the baseline prompt across the ten-case benchmark).

Classify these messages by category. Messages: [ten fictional messages]

Over three runs, the scores are 5/10, 6/10 and 5/10. Copilot invents different categories on each run, puts one message in two categories and assigns an arbitrary category to the trap case.

Turn 2 (Injecting an unambiguous precedence rule and measuring gains).

Classify each message below into exactly one category: Billing, Delivery, Product complaint, Information request, Other. If a message covers two topics, pick the one requiring operational action. In case of dual intent, complaints always take precedence over simple inquiries. If unsure, classify as Other and explain in five words. Output table: Number, Category, One-sentence rationale. Messages: [ten fictional messages]

Across the three evaluation runs, scores rise to 9/10, 10/10, and 9/10. Explicit arbitration handled mixed inquiries cleanly without degrading the negative test case.

What changes. Closed categories eliminate hallucinations. The doubt fallback gives the model an honest exit instead of wild guesses. Measuring before and after an isolated change validates prompt modifications scientifically.

3Score a sorting prompt before and after one rule

Here is a ready-to-use test set of ten customer messages received by the "NovaTech SAS" support team, along with the expected output for each:

The "Load exercise" button in the composer pre-fills the weak starting prompt:

"Summarise this customer email in one sentence and assess its urgency (Urgent / Normal)."

1. First series (3 runs): Run this prompt without modification on the ten cases, three times in fresh conversations. Count the number of outputs matching expectations.

2. Isolated modification: Add a single explicit rule in the Verification field of the composer: "Rule: classify as Urgent only if delivery is blocked, equipment is unusable, or deadline is under 48 hours. Otherwise, classify as Normal."

3. Second series (3 runs): Replay three runs over the ten cases and compare your numbered scores.

Self-assessment grid: (a) all ten expected answers are written before the first run; (b) six numbered scores are recorded, three before and three after; (c) a single rule changed, and the gain is measured.

Open the prompt composer