How Models Are Trained

The prediction machine described in the previous lesson doesn't arrive ready-made. Before a model can predict language well, it has to be trained — an intensive, expensive, multi-phase process that takes months and costs tens of millions of dollars. Understanding it at a high level changes how you think about what you're actually interacting with.

The four phases of training

Building a frontier language model isn't a single event. It's a pipeline, each phase building on the last:

Pre-training

Weeks to months

The model reads a massive corpus of text and learns to predict the next token — billions of times per second, across trillions of examples. No human guidance yet. Pure pattern learning at scale.

Supervised Fine-Tuning

Days to weeks

Human contractors write examples of ideal conversations — good questions answered well, harmful requests declined gracefully. The model is trained on these examples to produce outputs that look like helpful responses, not just plausible text.

Reinforcement Learning from Human Feedback

Weeks

Human raters compare pairs of model responses and pick the better one. A separate model learns to predict those ratings, then the main model is optimized to score higher. This is what makes the model feel conversational and aligned with human preferences rather than just statistically correct.

Safety Evaluation & Red-Teaming

Ongoing

Teams of researchers try to break the model — elicit harmful content, expose reasoning failures, find edge cases. Fixes are applied. This phase never fully ends; it continues post-deployment as new attack patterns are discovered.

The result is a model that has internalized statistical patterns across virtually every domain of human knowledge encoded in text — and has been shaped, through feedback, to apply those patterns in ways humans find genuinely useful.

The compute involved

Training a frontier model is one of the most computationally intensive operations in the history of the industry. It runs on clusters of thousands of specialized AI chips (GPUs or TPUs) operating continuously for months.

To put that in concrete terms: training GPT-4 is estimated to have required roughly 25,000 NVIDIA A100 GPUs running for approximately 90 days. These chips draw hundreds of watts each. The electricity bill alone — before you count the hardware, the engineers, the data, or the buildings — runs into the millions.

$100M+estimated cost to train a frontier model

GPT-4, Claude 3 Opus, and Gemini Ultra are all estimated in this range. Next-generation models may cost several times more.

This is why the frontier AI market is effectively a two-company race at the top tier. Only organizations with access to hundreds of millions in capital — and the relationships to acquire thousands of chips at once, which are themselves in short supply — can play at this level. Open-source models like Meta's Llama can be trained for a fraction of this cost, but they start from a capability disadvantage that smaller training budgets make difficult to close.

What "parameters" actually means

You'll often see models described by their parameter count: GPT-3 had 175 billion parameters; estimates for GPT-4 range into the trillions. A parameter is simply one of the numerical weights adjusted during training — a single dial in an enormous bank of dials, calibrated across trillions of training examples.

More parameters means more capacity to encode patterns. It also means more memory, more compute, and more cost. This is why smaller models (7B, 13B parameters) run on a laptop while the largest ones require data-center-scale infrastructure just to serve a response.

Why RLHF matters

The most significant step in the pipeline — and the one most responsible for the modern AI assistant experience — is Reinforcement Learning from Human Feedback (RLHF).

Before RLHF, language models were technically impressive but practically difficult. A raw pre-trained model would complete your text in statistically plausible ways, but it didn't understand that you wanted help, not just continuation. It would answer harmful questions, produce biased outputs, and generally do whatever the training text patterns suggested, regardless of usefulness.

RLHF is the step that turns a text-completion engine into an assistant. Human raters compare model responses and pick the better one; the model learns to produce outputs that score higher. It's how the same underlying technology becomes helpful, how it learns to say "I don't know" rather than make things up, and how it develops the conversational instincts that make it feel usable.

It's also the step that introduces the most controversy. The human raters doing this work are often contract workers, frequently in lower-income countries, paid modest wages to evaluate enormous volumes of content — including disturbing, violent, or exploitative material. The working conditions of these contractors have become a significant ethical concern in the industry.

The training cutoff

Training is not continuous. At some point the data collection stops, the training run concludes, and the model's knowledge is frozen. This is the training cutoff date — the point after which the model has no direct knowledge of world events.

This is why models can seem confidently wrong about recent developments: they aren't being evasive. From their perspective, those events haven't happened yet. The next lesson explains where the training data itself comes from — and why that question is more complicated than it sounds.