Transformer
The transformer is the neural network architecture, published by Google researchers in 2017, behind every modern language model: it processes all tokens of a text at once and uses attention to let each token weigh the relevance of every other token.
No jargon, no math
In plain words, what is it?
A transformer reads the whole sentence in one go, not word by word. Your text is split into tokens, and each token becomes a vector, a list of numbers that encodes its meaning. From each vector the model derives three smaller vectors: query, key and value. Attention computes a score between every query and every key: a high score means 'look at me'. Those scores become weights, and each token absorbs a weighted blend of all the other tokens' values, so information flows across the whole sentence in a single pass. That block, attention plus a small feed-forward network, is stacked dozens of times: layer by layer the representation sharpens, from spelling to grammar to 'this penalty refers to that clause'. Because there is no loop over positions, everything runs in parallel, which is why training on trillions of words became practical. Two boundaries: cost grows with the square of the sequence length, capping how much text fits in one pass, and the model learns statistical patterns, not rules, so rare cases can still surprise it.
- 01
Tokenise. Text is split into tokens, each given an ID.
- 02
Vectorise. Each token becomes a numeric vector carrying its meaning.
- 03
Score attention. Every token measures how relevant every other token is to it (query times key).
- 04
Blend. Each token's value is mixed in proportion to those scores.
- 05
Stack. Dozens of layers refine the mix; position information keeps word order.
- 06
Predict. The final vector feeds a probability for the next token.
Terms that live nearby.
Attention Mechanism
Attention is the mechanism inside a transformer that lets each token weigh how much every other token matters before being processed, so the model can track what 'it' refers to or which date belongs to which clause.
ReadNeural Network
A neural network is a function built from layers of simple units. Each unit multiplies its inputs by weights, adds them, passes the sum through a small nonlinearity, and forwards the result. Training adjusts the weights until the whole stack produces useful outputs.
ReadLarge Language Model
A large language model (LLM) is an AI system trained on massive collections of text to predict the next token, which gives it a general ability to write, summarise, translate and reason without being programmed for each task.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.