When the answer is right but the reasoning is broken

Beyond the Answer

Imagine you're helping your child with their maths homework. They show you their answer: 42. You check the back of the textbook — the answer is indeed 42. Great, right?

Not necessarily. If they got there by guessing, or by making two errors that happened to cancel each other out, the correct answer is hiding a broken process. Next time, with a different problem, those same errors will produce a wrong answer.

The same principle applies to AI. A correct-looking output can emerge from flawed reasoning. And if you only check the answer without examining the reasoning, you'll miss the flaw — until it matters.

Process evaluation means looking at how the AI arrived at its response, not just what it produced. This is especially important for complex tasks: analysis, strategy, problem-solving, and anything where the reasoning matters as much as the conclusion.

Common AI Reasoning Failures

AI doesn't reason the way humans do — it generates text that resembles reasoning, based on patterns in its training data. This means it can produce impressively structured arguments that contain fundamental logical problems. Here are the most common ones to watch for.

Logical leaps

The AI presents a chain of reasoning where one step doesn't actually follow from the previous one. "Your company's revenue grew last quarter. Your main competitor also grew. Therefore, the market is expanding." That sounds reasonable, but two companies growing doesn't prove market expansion — they might be taking share from smaller players.

These leaps are easy to miss because AI presents them fluently, without the hesitation or hedging a human might use when they're unsure of a connection.

Circular reasoning

The AI uses its conclusion as evidence for itself. "This strategy will succeed because it's the right approach for your situation, and the reason it's the right approach is that it will lead to success." It sounds like an argument. It's actually a circle.

AI is particularly prone to this when asked "why" questions. It can restate the claim in different words multiple times, creating the appearance of supporting reasoning without actually providing any.

Cherry-picking evidence

When asked to support a position, AI will often select only the evidence that fits and ignore everything that doesn't. This isn't deliberate deception — it's a consequence of how language models generate text. Once they start building an argument in one direction, the next sentence is statistically more likely to continue in that direction than to introduce a contradiction.

If you ask, "What are the benefits of remote work?" you'll get an enthusiastic list. Ask, "What are the problems with remote work?" and you'll get an equally enthusiastic list in the opposite direction. Neither response will voluntarily mention the other side.

Conflating correlation with causation

"Countries with higher chocolate consumption have more Nobel laureates. Therefore, eating chocolate improves cognitive function." AI can present correlational data as if it demonstrates causation, particularly when the correlation aligns with an intuitively appealing narrative.

Watch for phrases like "this leads to," "this results in," or "because of this" when the evidence only shows that two things occur together, not that one causes the other.

Signs the AI Has Gone Off Track

Sometimes you don't need to dissect the logic — there are surface-level signals that something has gone wrong with the AI's reasoning process.

Inserting ideas you already rejected. You told the AI you don't want to use social media marketing. Three exchanges later, it's recommending an Instagram campaign. This happens because AI doesn't truly "understand" your constraints the way a human collaborator would — it's generating likely text, and sometimes the most statistically likely recommendation is the one you've already ruled out.

Fixating on one interpretation. You describe a nuanced situation with multiple possible explanations. The AI picks one and builds its entire response around it, ignoring the others. When you push back, it acknowledges the other interpretations briefly but then returns to its original fixation. This "anchoring" behaviour is common and frustrating.

Contradicting itself. In longer responses, AI can make claims in paragraph three that directly contradict claims in paragraph one. This happens because each sentence is generated based primarily on the nearby context, not on a holistic review of everything it's already said. The AI doesn't go back and check for consistency.

Excessive confidence on uncertain topics. If the AI is making definitive claims about something genuinely uncertain — future predictions, contested scientific questions, complex human behaviour — that confidence is itself a red flag. Appropriate reasoning about uncertain topics includes acknowledgement of uncertainty.

How to Ask AI to Show Its Working

You don't have to guess at the AI's reasoning process. You can ask it to make that process visible. Here are prompts that help:

• "Walk me through your reasoning step by step." This forces the AI to make each logical connection explicit, making it easier for you to spot where a step doesn't follow.
• "What assumptions did you make in reaching this conclusion?" AI always makes assumptions — about your context, your goals, the meaning of ambiguous terms. Making them explicit lets you check whether they're valid.
• "What evidence supports this, and what evidence contradicts it?" This counters the cherry-picking tendency by explicitly requesting both sides.
• "How confident should I be in this answer, and why?" This invites the AI to flag its own uncertainties, which it often won't do unless asked.
• "What would need to be true for this recommendation to be wrong?" This is a powerful question that forces the AI to consider failure conditions, which it typically glosses over.

These aren't magic phrases — they're tools for making the AI's process more transparent so your evaluation can be more thorough.