Inference: The Model in Use
Training builds the model. Inference is everything that happens after.
Every time you send a message to an AI assistant, you're running inference: your input passes through the trained model, and the model produces output. The weights — those billions of numerical parameters shaped over months of training — are frozen. Nothing about the model changes. Inference is the model being used, not being built.
Training vs. inference
The two processes are easy to conflate because they both involve the same model architecture. But they're different operations with different resource profiles and different purposes.
| Training | Inference | |
|---|---|---|
| What it does | Adjusts model weights based on data | Runs fixed weights to produce output |
| When it happens | Once (or periodically, for updates) | Every single time a user sends a message |
| Compute required | Enormous — thousands of GPUs for months | Moderate — seconds on server hardware |
| Cost | Tens to hundreds of millions of dollars | Fractions of a cent per query |
| Who does it | The AI lab | The AI lab's servers, on your behalf |
| What changes | The model itself | Nothing — the model stays the same |
Think of it this way: training is writing and printing a book. Inference is someone reading it.
What actually happens during inference
When you send a message, a sequence of operations runs in milliseconds:
- Tokenization — your text is broken into tokens (the chunks covered in the next module)
- Embedding — each token is converted into a high-dimensional numerical vector the model can process
- Forward pass — the token representations travel through the model's transformer layers; each layer applies learned attention patterns, refining the representation
- Output distribution — the final layer produces a probability distribution across the model's entire vocabulary: how likely is each possible next token?
- Sampling — the model picks a token based on that distribution (influenced by temperature, as covered earlier)
- Repeat — the new token is appended to the context, and the whole process runs again for the next token
One response might involve hundreds or thousands of these forward passes chained together. Each pass is fast — modern inference hardware handles this in milliseconds — but the compute adds up at scale.
The economics of scale
Training a frontier model costs tens to hundreds of millions of dollars — a one-time (or infrequent) expense that produces the model. Inference is the ongoing cost: every query from every user, running continuously.
At consumer scale, this is significant. ChatGPT reportedly handles over 10 million queries per day. Each query runs thousands of forward passes. The inference bill for a widely-used model can rival the training cost over time — which is a large part of why AI products charge subscription fees and usage-based pricing, and why more efficient models (fewer parameters, faster inference) are commercially valuable even when they're slightly less capable.
This is also why on-device AI has become a strategic priority for Apple, Google, and Qualcomm. Running inference locally on a phone or laptop means no server costs, no latency, and no privacy concerns about sending data to a cloud. The tradeoff is capability — only smaller models fit on consumer hardware.
The one lever you have
Here's what all of this means for you as a user: at inference time, the model is fixed. The only variable is your input.
The weights won't change because you phrased something differently. The model's knowledge doesn't update between messages. The training data cutoff doesn't move. What changes — the only thing you can actually control — is the prompt you write.
This is why prompting is a real skill and not just a matter of being polite to a chatbot. The model is a sophisticated, frozen function. You are the only dynamic part of the equation. Understanding how to shape your input to get better output from a fixed model is exactly what the next module covers.