Latency
Latency is the delay between a request and its answer, measured in milliseconds. In AI systems it adds up across queueing, prompt processing, token-by-token generation and network hops, and it decides whether a tool feels instant or unusable.
No jargon, no math
In plain words, what is it?
Every AI answer travels a chain, and latency is the total time of the chain. First, queueing: your request waits behind others for a free slot. Second, prefill: the model reads your whole prompt, fast but not free. Third, generation: the model writes the answer one token at a time, and each token needs a full pass through the model. This is usually the long leg, and it grows with the length of the answer. Fourth, the network round trips between you, the API and back. You trim latency where it actually lives: a shorter prompt cuts prefill, asking for a shorter answer cuts generation, a nearby server or an on-device model cuts the network. The boundary is human perception. A chat that answers in 300 milliseconds feels instant; past about a second users notice the wait; past a few seconds they abandon the task. Voice is stricter: turn-taking breaks above roughly 800 milliseconds.
Terms that live nearby.
Inference
Inference is using a trained model: you send input in, the model applies what training encoded in its parameters, and you get an answer out, one token at a time.
ReadInference Optimization
Inference optimization is the engineering discipline of making a trained model cheaper and faster to run: measure where time and compute go on each request, then apply levers like caching, quantization, batching and prompt trimming to cut both.
ReadEdge AI
Edge AI is running AI models on the device where the data is born: a camera, a sensor, a phone, a factory gateway, instead of shipping everything to a cloud data centre and waiting for the answer.
Read§ 03
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.