What Are Tokens?

When you type a message to an AI, it doesn't read your words the way you do. Before any thinking happens, your text gets sliced up into small chunks called tokens.

A token is roughly a word — but not always. It might be a whole word, part of a word, a punctuation mark, or even a space. The way text gets divided depends on the model's tokenizer, a kind of dictionary that maps text chunks to numbers.

A few examples

• The word "hello" → 1 token
• The word "unbelievable" → 3 tokens: un + believ + able
• The word "ChatGPT" → 2 tokens: Chat + GPT
• A single space can be its own token

Common English words are often a single token. Rare words, technical terms, and words in other languages tend to use more tokens — because they appear less often in training data, so the tokenizer doesn't have a compact representation for them.

Why does this matter?

Three reasons:

1. Cost. AI providers charge per token — both the tokens you send in (input) and the tokens the AI sends back (output). Understanding tokens helps you understand the bill.

2. Context windows. Every AI model has a maximum number of tokens it can handle at once — its context window. Long documents, long conversations, or complex instructions all eat into that limit.

3. How the model "thinks." The model processes tokens, not words or sentences. This is partly why it can make strange errors — it's pattern-matching on token sequences, not comprehending meaning the way humans do.

Try it yourself

Head to the Tokenizer Explorer lab to see exactly how your own text gets broken into tokens — and what it costs across different models.