Attention Mechanism
Attention is the mechanism inside a transformer that lets each token weigh how much every other token matters before being processed, so the model can track what 'it' refers to or which date belongs to which clause.
No jargon, no math
In plain words, what is it?
Attention gives each token three vectors. The query says what this token is looking for, the key says what it can offer, and the value is the actual content on offer. The model multiplies queries against keys to get a score for every pair: when 'penalty' meets the liquidated-damages clause, the score spikes. A softmax turns the scores into weights that sum to one, and each token rebuilds itself as a weighted sum of all the values. Think of reading a contract with a highlighter: every word decides for itself which other words to shade in, and the shading drives what the word means in context. Multi-head attention runs this several times in parallel with different views: one head catches grammar, another catches cross-references. The failure mode is distance: in a very long document the few tokens that matter can be outshouted by the many that merely surround them, which is why models miss the one buried number.
- 01
Query. Each token asks, in vector form, what it is looking for.
- 02
Key. Every token broadcasts what information it can offer.
- 03
Score. Query times key produces a relevance score for every pair of tokens.
- 04
Weigh. A softmax turns the scores into weights that sum to one.
- 05
Blend. Each token becomes a weighted mix of all the values, so context is baked in.
Terms that live nearby.
Transformer
The transformer is the neural network architecture, published by Google researchers in 2017, behind every modern language model: it processes all tokens of a text at once and uses attention to let each token weigh the relevance of every other token.
ReadLarge Language Model
A large language model (LLM) is an AI system trained on massive collections of text to predict the next token, which gives it a general ability to write, summarise, translate and reason without being programmed for each task.
ReadEmbedding
An embedding is a list of numbers that captures the meaning of a piece of text, image or audio, so that content with similar meaning gets similar numbers and the computer can measure closeness with plain arithmetic.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.