Tokenization
Tokenization is the step that turns text into tokens, the small units a language model actually reads: the input is split according to a fixed vocabulary and each token is mapped to a number, so the model processes IDs instead of words.
No jargon, no math
In plain words, what is it?
Before a model reads a single word, the text has to become numbers. Tokenization does it in two moves. First, split: the text is chopped into tokens using a fixed vocabulary of subword pieces, typically around 100,000 entries. Common words become one token; rare or compound words are broken into smaller pieces ("artificial" can be two tokens). Second, map: every token is swapped for its numeric ID, so the model receives a plain sequence of numbers. On the way out the same process runs in reverse: the model predicts the next token ID one step at a time, and the detokenizer rebuilds the text. Three practical consequences follow. Cost is metered in tokens, so a longer prompt costs more and every limit is a token count. A token is not a word: Italian text uses more tokens per sentence than English, and numbers, code and typos shatter into many small pieces. And the vocabulary is fixed at training time, so a brand-new term arrives already broken into fragments.
- 01
Split. The text is chopped into tokens using a fixed vocabulary of subword pieces.
- 02
Map. Every token is swapped for its numeric ID from the vocabulary.
- 03
Read. The model receives only a sequence of numbers.
- 04
Predict. It outputs the next token ID, one step at a time.
- 05
Reassemble. IDs map back to tokens and the detokenizer rebuilds the text.
Terms that live nearby.
Token
A token is the chunk of text a language model actually reads and writes: on average about three quarters of an English word, converted to a number from a fixed vocabulary before the model can process it.
ReadContext Window
The context window is the model's working memory, measured in tokens: everything in a single prompt (instructions, documents, the conversation so far) must fit inside it, and anything beyond the limit is simply invisible to the model.
ReadLarge Language Model
A large language model (LLM) is an AI system trained on massive collections of text to predict the next token, which gives it a general ability to write, summarise, translate and reason without being programmed for each task.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.