Lesson 22 · Expert · 20 min
Evaluating a prompt on a set of ten cases
Measure a prompt instead of feeling it: ten cases with an expected answer, three runs, one rule changed at a time, and a compared score.
- Goal
- You will be able to build a set of ten cases with an expected answer, measure a prompt over three runs and attribute every gain to a single, isolated change.
- Skills
- Check
Your first attempt, unaided
Take a prompt you are happy with and run it three times in a row in Copilot Chat, in a fresh conversation, on three different cases. Count how many times the result suits you with no rework.
A test set is a table of ten realistic cases, each with the expected answer written before anything is run. You run your prompt three times over the ten cases, count the right answers, change a single rule, and start again. The score replaces the impression. It is what lets you tell a colleague "this prompt passes nine cases out of ten" rather than "it works well".
1Ten cases, three runs, a score instead of an opinion
Anthropic opens its prompt engineering guide with a piece of advice few people follow: first define a success criterion and a way to test it, then only work on the prompt. Without measurement, every change is a bet, and you keep the one you liked on the last try. With ten cases and a score, the discussion changes in nature: you compare figures.
Building the set takes an hour. Choose ten fictional or anonymised inputs that look like the real ones: six ordinary cases, three difficult cases (two topics in one message, a missing piece of data, an ambiguous wording) and one trap case whose right answer is "not found". Write the expected answer before any run, otherwise you will judge Copilot's answer by its confidence rather than its accuracy. The score is simple: number of cases passed out of ten.
Why three runs? Because the same request produces different answers from one time to the next. Wang and his co-authors built an entire method, self-consistency, on this variability: several runs followed by a vote clearly improve the result. For you, three runs give a range (8, 9, 8) instead of a misleading figure.
The golden rule is isolation: a single change between two series. If you change the doubt rule and add an example at the same time, the gain is real but inexplicable. Microsoft recommends iterating and regenerating; the test set says which of the attempts to keep. Last point: have the scoring done by a peer who does not know which version produced which answer. Huang and his co-authors showed that models correct themselves poorly without external feedback; the same finding applies to the prompt's author, too pleased with the latest version.
Diagram "Evaluate a prompt on a test set"Diagram generated by AI and reviewed 2A message sorting prompt from 5 to 9 out of 10
A support team wants to classify the messages of a shared mailbox into five categories.
Weak prompt.
Classify these messages by category. Messages: [ten fictional messages]Over three runs, the scores are 5/10, 6/10 and 5/10. Copilot invents different categories on each run, puts one message in two categories and assigns a category to the trap case, which has none.
Strong prompt.
Classify each of the messages below into a single category among: Billing, Delivery, Product complaint, Information request, Other. If a message belongs to two categories, choose the one that requires an action and flag it with an asterisk. If you hesitate, classify as Other and explain in five words. Output: table Number, Category, Justification in one sentence. Messages: [ten fictional messages]Scores: 8/10, 9/10, 8/10. The two cases missed are those mixing a delivery question and a complaint. A single change, "in case of a double topic, the complaint takes precedence over the question", raises the scores to 9, 10, 9.
What changes. Closed categories remove invention. The doubt rule gives the model an honest output instead of a guess. The score, measured before and after an isolated change, proves that the rule added is the right one, and the table becomes the test card the team keeps.
3Score a sorting prompt before and after one rule
Starting prompt, to be measured then improved:
"Summarise this client email in one sentence and say whether it is urgent."
Write ten fictional emails, each with the expected summary and the urgency verdict, including one email with no element of urgency and one email mixing two requests. Run the prompt three times, record the scores. Add a single rule, for instance the definition of "urgent" ("deadline under 48 hours or an order blocked"), run three more times, compare. The composer below helps you phrase the verification rule.
Self-assessment grid: (a) the ten expected answers are written before the first run; (b) six scores are recorded, three before and three after; (c) a single rule changed, and you can say whether it helped.
4The test set that never makes the prompt fail
A support manager builds her ten-case set from the last ten emails received, on a quiet Tuesday morning: ten short, polite information requests on a single topic. The classification prompt scores 10, 10 and 10 over three runs. She publishes it in the library marked "reliable". On the first day of real use, an email mixing a complaint and a delivery question is classified twice, and an email with no identifiable subject receives an invented category. What should have been seen: a test set measures only what it contains; ten easy cases yield a perfect score that says nothing about the real inbox. Correction: build the set as the lesson describes, six ordinary cases, three hard cases (two topics in one message, a missing piece of data, an ambiguous wording) and one trap case whose expected answer is "Other" or "not found". Add every error met in real use to the set, so that 10 out of 10 becomes hard to get again. Rule to remember: a test set that never makes the prompt fail tests nothing.
5Quiz
Three questions, instant feedback. Each option comes with an explanation.
6Proof of mastery
Paste the table of ten cases with the expected answer, the prompt tested, the scores of the three runs before and after the change, and the rule changed with an excerpt of the answer before and after.
Criteria
What you wrote at the start of the lesson
Going further
The test set measures the version kept after a meta-prompt and serves as a check before sharing an agent; it also reveals the hallucinations the eye lets through.
Frequently asked questions
Are ten cases enough?
For a team prompt, yes: ten cases are enough to reveal the frequent defects, provided you include two or three difficult cases and one trap case whose expected answer is "not found". The set then grows with every error met in real use.
Can I ask Copilot to score its own answers?
With caution. Research shows that models correct themselves poorly without external feedback. Copilot can pre-fill the comparison between its answer and the expected one; a human, preferably a peer who does not know which version produced what, validates the score.
Where should the test set be kept?
In a clean Excel file, one row per case, next to the prompt's library card. Copilot Notebooks, open to Copilot Chat users since June 2026, also lets you gather the set, the prompt and the results in the same place.