The Prediction Machine

The Prediction Machine — A Mental Model for Generative AI

Forget Everything You Know About Software

Most software you've used in your career works on a straightforward principle: someone writes rules, and the computer follows them. If you type a formula into a spreadsheet, it calculates the answer. If you click "sort A–Z," it alphabetises your list. The programmer anticipated what you'd want to do and wrote specific instructions for each scenario.

Generative AI is fundamentally different. Nobody wrote rules telling it how to answer your questions. Instead, it learned patterns — billions upon billions of patterns — and uses those patterns to predict what text should come next.

Here's the mental model that will serve you well throughout this course:

Think of an AI assistant as an incredibly well-read colleague. Imagine someone who has read virtually every book, article, website, forum post, and document published in English (and dozens of other languages) up to a certain date. They've absorbed writing styles, factual information, logical arguments, creative techniques, coding conventions, and conversational patterns from all of that reading. When you ask them a question, they draw on everything they've absorbed to compose a thoughtful, relevant response.

But here's the crucial part of the analogy: this colleague has never actually experienced anything. They've never visited Paris, even though they can describe it beautifully. They've never felt frustrated by a difficult project, even though they can empathise convincingly. They've never run a business, even though they can discuss strategy with apparent expertise. Everything they know, they know from text.

That distinction — vast knowledge from reading, zero knowledge from experience — will help you understand almost every strength and limitation we'll cover in this module.

What Are Large Language Models?

The specific type of AI we're discussing is called a Large Language Model, or LLM. Let's break that term apart:

• Language — it works with human language (and code, which is really just another form of language). It takes text in and produces text out.
• Model — it's a mathematical model, essentially a massive collection of numerical relationships that represent patterns in language. When you ask it something, it's running your input through these mathematical relationships to generate output.
• Large — and this is where things get interesting. Modern LLMs contain hundreds of billions of parameters (think of these as adjustable dials that shape the model's behaviour). GPT-4, Claude, and Gemini are all built at this scale. "Large" is something of an understatement — these are the most complex mathematical models ever constructed.

The Three Ingredients Behind the Breakthrough

People have been trying to build AI that understands language for decades. So why did it suddenly start working so impressively in recent years? Three things came together at roughly the same time:

1. An Architectural Breakthrough: The Transformer

In 2017, researchers at Google published a paper with the unassuming title "Attention Is All You Need." It introduced a new way of processing language called the transformer architecture. Without getting into the mathematics, the key innovation was a mechanism called "attention" — it allows the model to look at all the words in a passage simultaneously and understand how they relate to each other, rather than processing them one by one in sequence. This was dramatically more effective than previous approaches, and virtually every modern AI assistant is built on this foundation.

2. Massive Training Data

Transformers are powerful, but they need something to learn from. The models behind today's AI assistants were trained on enormous datasets — we're talking about significant portions of the publicly available internet, digitised books, academic papers, code repositories, and much more. The sheer volume of text gives the model enough examples to learn remarkably subtle patterns about how language, logic, and knowledge work.

3. Enormous Computing Power

Training these models requires staggering amounts of computing power. We're talking about thousands of specialised processors (called GPUs) running for weeks or months at a time, at a cost of tens or even hundreds of millions of pounds. This kind of computing power simply wasn't available or affordable until relatively recently. The investment from companies like OpenAI, Google, Anthropic, and Meta made it possible to train models at a scale that produces genuinely useful results.

Pre-Training vs Fine-Tuning: Two Stages of Learning

Building an AI assistant happens in two main stages, and understanding them helps explain a lot about how these tools behave.

Pre-training is the first stage. This is where the model reads vast quantities of text and learns patterns. Crucially, the training objective is deceptively simple: given a sequence of text, predict what comes next. That's it. By doing this prediction task across billions of examples, the model develops a remarkably rich understanding of language, facts, reasoning patterns, and writing styles. But after pre-training alone, the model is essentially a very sophisticated autocomplete engine — it can continue any text in a plausible way, but it doesn't know how to be helpful.

Fine-tuning is the second stage. Here, the model learns to behave like an assistant. Human trainers provide examples of good, helpful, honest responses. The model also learns safety guidelines — how to decline harmful requests, when to express uncertainty, and how to be forthcoming about its limitations. This stage is what transforms a raw prediction engine into the helpful conversational tool you interact with.

Think of it this way: pre-training gives the AI its knowledge and language ability, while fine-tuning gives it its personality, helpfulness, and guardrails.