Exercise: Spot the errors a novice would miss
Time required: 20 minutes
This exercise is designed to make the relationship between expertise and evaluation viscerally obvious — not as a theory, but as something you experience directly.
Step 1—Part A: Evaluate in Your Area of Expertise
Choose a topic you know deeply — your professional field, a subject you studied extensively, a skill you've practised for years.
Ask an AI the following (adapting to your domain):
"Explain the key principles of [your area of expertise] to someone who needs to apply them in practice. Include common mistakes people make and how to avoid them. Be specific and detailed."
What topic did you choose for your area of expertise?
Now evaluate the response using all five quality checks:
- Accuracy — Read every claim carefully. Are the facts correct? Are the technical terms used properly? Are there subtle errors that a non-expert wouldn't notice?
- Relevance — Does this address practical application, as requested? Or has the AI drifted into theoretical overview?
- Completeness — What's missing? What would you add if a colleague showed you this? What crucial nuance has been flattened?
- Bias — Does it favour one school of thought, one methodology, one approach? Does it present debatable opinions as consensus?
- Usability — Could someone genuinely learn from and apply this? Or would following this advice lead them astray in important ways?
Record your findings below. Be specific — note exact phrases that are wrong, misleading, or incomplete.
| Quality Check | Rating (1–5) | Specific Issues Found |
|---|---|---|
| Accuracy | ||
| Relevance | ||
| Completeness | ||
| Bias | ||
| Usability |
Describe your most significant findings in detail. What errors, omissions, or misleading statements did you spot that a non-expert would likely miss?