5Evaluating AI output in under a minuteDetermine

When to stop iterating and start over

5 min read847 words

The Meta-Question

The first two lessons focused on evaluating what the AI produced. This lesson asks a different question: is the AI communicating with you in the most effective way possible?

This is what we might call meta-evaluation — stepping back from the content to assess the interaction itself. It's the difference between asking "Is this report good?" and asking "Is the way we're working on this report actually productive?"

Most people never think to evaluate the interaction. They accept whatever communication style the AI defaults to and work around its limitations. But the interaction style is something you can shape, and shaping it well can dramatically improve the quality of everything that follows.

Common Interaction Mismatches

Too verbose when you need concise. You asked a simple question and got six paragraphs. The answer was in the first sentence; the rest was padding. This isn't just annoying — it's a practical problem. When AI buries the useful content in unnecessary elaboration, you have to do extra work to extract what matters, and you risk missing important details hidden in the noise.

Too surface-level when you need depth. The opposite problem. You asked about a complex topic and got a response that reads like a Wikipedia summary — technically correct but lacking the nuance, detail, or analytical depth you need to actually make a decision. AI often defaults to breadth over depth unless you specifically direct it otherwise.

Too agreeable when you need pushback. You shared your plan and the AI said it was excellent. You shared a different plan and the AI said that was excellent too. AI has a strong tendency towards agreement and validation — what researchers call sycophancy. This feels pleasant but is deeply unhelpful when you need honest assessment, devil's advocacy, or genuine critique.

Wrong register for your audience. The AI is writing at a postgraduate level when you need something accessible, or it's oversimplifying when you need technical precision. It's using corporate jargon when you need plain language, or casual phrasing when you need formality.

Wrong format for your needs. You're getting essays when you need tables. You're getting bullet points when you need flowing prose. You're getting abstract principles when you need concrete examples.

Adjusting the Interaction

The good news is that all of these mismatches are fixable, usually with a single direct instruction. Here are examples:

  • "Be more concise. Give me the key point in one sentence, then supporting detail only if I ask."
  • "Go deeper on this. I don't need an overview — I need detailed analysis with specific evidence and nuanced consideration of edge cases."
  • "Challenge my assumptions. I don't want validation — I want you to find the weaknesses in my thinking and tell me about them directly."
  • "Write this for an audience of [specific description]. They know [X] but don't know [Y]. Use [formal/casual/technical/plain] language."
  • "Present this as a table with columns for [X], [Y], and [Z]" or "Give me this as a numbered list I can work through sequentially."

Notice that these instructions are specific and direct. "Be better" doesn't work. "Be more concise, and lead with your main recommendation before providing supporting reasoning" does work. The more precisely you can name what you want to change, the more effectively the AI will adjust.

When to Change Your Approach Entirely

Sometimes the right evaluation isn't "How do I fix this interaction?" — it's "Should I be having this interaction at all?"

Consider changing tools when the AI consistently struggles with your task type. If you're trying to do precise calculations, a spreadsheet might serve you better. If you need current information, a search engine is more reliable. AI is powerful but it's not the right tool for every job.

Consider changing your approach when you've been iterating for several rounds without meaningful improvement. If you've refined your prompt five times and the output still isn't right, the problem might not be your prompting — it might be that you need to break the task into smaller pieces, provide different context, or approach the problem from a different angle entirely.

Consider starting fresh when the conversation has accumulated too much context that's pulling the AI in the wrong direction. Sometimes a clean slate with a better-crafted initial prompt will outperform ten rounds of correction on a poorly started conversation. Don't fall victim to the sunk cost fallacy — the time you've already spent on a conversation is not a reason to keep going if starting over would be faster.

Key Takeaways

  • 1Evaluate not just what the AI produced, but whether the interaction itself is working effectively for you.
  • 2Common mismatches include responses that are too verbose, too shallow, too agreeable, or in the wrong format for your needs.
  • 3You can fix most interaction problems with a single specific, direct instruction telling the AI exactly how to adjust.
  • 4Sometimes the best move is to change tools, break the task differently, or start a fresh conversation rather than continuing to iterate.