Inference
Inference is using a trained model: you send input in, the model applies what training encoded in its parameters, and you get an answer out, one token at a time.
No jargon, no math
In plain words, what is it?
After training, the parameters are frozen and the work changes name: inference. Your prompt is tokenized into numbers. The model runs them through its layers and out comes a probability for every possible next token. It picks one (how boldly is set by temperature), appends it, and runs again, now with a longer input. That loop is why answers appear word by word and why long answers cost more: every token is a fresh pass through billions of parameters, usually on a GPU. The KV cache remembers past calculations so tokens after the first are cheaper. It breaks at the context window: too many tokens and the model literally cannot see the beginning of your document.
- 01
Tokenize. Your prompt becomes a list of numbered tokens.
- 02
First pass. The model computes probabilities for the next token.
- 03
Pick. One token is sampled, the boldness set by temperature.
- 04
Append and repeat. The new token joins the input; the loop runs until a stop signal.
- 05
Stream. Tokens return as they are generated, so the answer appears word by word.
Terms that live nearby.
Training
Training is the process that turns an untrained model into a working AI: you feed it thousands or millions of examples, compare its guesses with the correct answers, and nudge its internal settings until the guesses get good.
ReadToken
A token is the chunk of text a language model actually reads and writes: on average about three quarters of an English word, converted to a number from a fixed vocabulary before the model can process it.
ReadTemperature
Temperature is a setting on a language model that controls how random its word choices are: low values make it pick the most likely next word every time, high values let less likely, more varied options win.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.