A token in AI is one item from the vocabulary an LLM uses to represent and generate text. Depending on the model's tokenizer, a token can correspond to a word, part of a word, punctuation, whitespace, or even part of an encoded character.
Tokens are not a universal unit of language. They are a model-specific way to turn text into a sequence of numbered entries that a model can process.
A token is a codebook entry
Think of a tokenizer as a codebook shared by the software that prepares text and the LLM that processes it. The codebook contains permitted text or byte sequences and assigns each one an integer ID.
Suppose an illustrative tokenizer splits:
The cat sat.
into:
The | cat | sat | .
Those visible pieces might map to IDs such as:
[814, 3290, 7112, 13]
The numbers here are invented to show the mechanism. Real IDs and boundaries depend on the tokenizer. A leading space may be grouped with the following word, so cat and cat can be different entries.
In everyday explanations, token can mean either the visible piece or its ID. The distinction matters: the model receives the IDs, not little strings of readable text. Each ID selects an initial learned numeric representation that the model can work with.
Why models use pieces smaller than words
A whole-word vocabulary would need an entry for every name, spelling, inflection, technical term, and newly coined word the model might encounter. Any missing word would be unknown.
A character-only vocabulary avoids unknown words, but it turns ordinary text into much longer sequences. That increases the number of steps the model has to process.
Subword tokenization is a compromise. Frequent sequences can get their own entries, while rare words can be assembled from smaller entries. A common word might be one token. A rare name might become several fragments. Punctuation and spaces can be tokens or parts of tokens too.
The fragments are chosen for useful coverage and compact sequences, not because every fragment has an independent dictionary meaning. A piece such as ment, a leading-space word, or a run of bytes is a valid token if it is in the tokenizer's vocabulary.
How tokenization works
The exact procedure varies, but the path through a text model has the same general shape:
text
↓
model-specific tokenizer
↓
token pieces
↓
integer token IDs
↓
language model
↓
next token ID
↓
tokenizer decoder
↓
generated textFirst, the tokenizer applies its fixed rules to the input. Depending on its design, it may normalize the text, make an initial split, and then segment it with an algorithm such as BPE, Unigram, or WordPiece.
Next, it looks up each resulting piece in its vocabulary and emits the corresponding ID. It can also add special IDs for purposes such as marking boundaries or request structure. These special tokens do not necessarily appear as visible text.
The model processes the input IDs. When generating text, it assigns probabilities to possible next-token IDs and selects one according to its decoding settings. That new ID joins the sequence, and the process repeats. Finally, the decoder turns the generated IDs back into bytes or text.
Tokenization is normally reversible for a complete valid sequence: decoding the IDs reproduces the text. An individual token does not always decode into a readable character on its own, especially with byte-level tokenizers. Neighboring tokens may need to be combined first.
One word can have different token counts
There is no exact word-to-token conversion.
In an OpenAI tokenizer example, antidisestablishmentarianism is split by one encoding into five tokens:
ant | idis | establishment | arian | ism
Another encoding splits it into six:
ant | idis | establish | ment | arian | ism
Neither split changes the spelling. Neither proves that one model understands the word better. The encodings simply have different vocabularies and segmentation rules.
Counts also vary with language, spacing, punctuation, code, emoji, and unusual character sequences. Even a small edit can change the boundaries around it. Rough averages can help with an early estimate, but the only reliable count comes from the tokenizer or counting endpoint for the model and complete request you will use.
Why tokens matter
They determine how much fits. A model's context window is measured in tokens, not words. The input, prior messages, generated output, and sometimes other request content compete for that capacity according to the provider's rules.
They measure usage. Many AI APIs report and price input tokens and output tokens separately. The rates and categories vary, so a token count alone is not a price.
They affect generation time. Text generation proceeds token by token. More output tokens generally require more generation steps, though total latency also depends on the model, hardware, load, and input processing.
They shape text efficiency. Two strings with similar human-readable length can consume different numbers of tokens. This matters when an application handles multiple languages, source code, structured data, or unusual identifiers.
They connect the tokenizer to the model. The model learns with a particular mapping from IDs to vocabulary entries. Swapping in an unrelated tokenizer is not a cosmetic change: the same ID could point to a different piece, so the model would receive the wrong learned representation.
Common misconceptions
One token is one word
Sometimes a short, common word is one token. A token can also be a word fragment, punctuation mark, space-bearing fragment, character, or byte sequence. Treat any tokens-per-word figure as an estimate for a particular kind of text.
Token boundaries show what the model thinks a concept means
Token boundaries come from the tokenizer's vocabulary and segmentation rules. They are not a map of the model's concepts. A word split into four tokens is not necessarily four ideas, and a phrase stored as one token is not necessarily understood as one indivisible idea.
All models count the same text the same way
Different vocabularies produce different boundaries and counts. Even models from one provider can use different encodings. Count with the specific model interface rather than reusing a count from another model.
Only the text you typed counts
An API request can include system instructions, conversation history, tool definitions, file or image representations, and formatting added by the service. Some of this may contribute tokens even though it is not visible in the final chat interface. Use the provider's usage report or full-request counting tool when an exact count matters.
More tokens mean more meaning
Token count measures the length of the model's representation, not the amount or quality of information. Repetitive text can use many tokens. A compact formula can use few tokens while carrying substantial meaning.
AI tokens are crypto tokens
They share a name, not a mechanism. A text token is a model vocabulary unit. A crypto token is a digital asset or record on a blockchain.
How tokens fit into the wider system
The tokenizer's vocabulary is usually established before the language model is trained. Training text is converted into IDs with that tokenizer, and the model learns patterns over those IDs. At inference time, the same mapping turns a new prompt into input IDs and turns generated IDs back into text.
This makes the tokenizer part of the model, even when it ships as a separate file or library. The model, vocabulary, tokenizer rules, and special-token definitions must agree.
The word token can also be used more broadly in multimodal systems. A provider may convert image patches, audio intervals, or video frames into units that it reports as tokens. Those units serve a similar role in model processing and accounting, but they are not pieces produced by a text tokenizer.
Where to go next
Read What Is a Context Window? to see how tokens set a model's working limit. For API usage categories, continue to Input vs. Output vs. Reasoning Tokens.