RLHF
RLHF, reinforcement learning from human feedback, is the technique that tunes a language model's behaviour with human taste: people rank sample answers, a reward model learns to predict those preferences, and the language model is trained to score higher.
No jargon, no math
In plain words, what is it?
Some instructions cannot be written as rules: be helpful without being servile, say no politely, admit not knowing. RLHF teaches them through judged examples instead. Step one: the model, already fine-tuned, generates several candidate answers per prompt. Step two: real people rank the candidates from best to worst. Step three: a small, cheap reward model trains to predict those judgements, becoming a synthetic judge. Step four: through reinforcement learning the big model generates answers and receives a reward when the judge scores them high, and its weights shift towards that reward. You iterate until the behaviour holds. Where it breaks: the big model can learn to game the judge (reward hacking), producing flattering answers instead of honest ones. That is why rankings must reward honesty rather than agreeableness, and why the human phase remains the most expensive bottleneck in the whole pipeline.
- 01
Generation. The model produces several candidate answers per prompt.
- 02
Human ranking. Real people order the candidates from best to worst.
- 03
Reward model. A small model trains to predict those human judgements.
- 04
Reinforcement learning. The big model is trained to earn higher scores from the synthetic judge.
- 05
Iteration. If the behaviour does not hold, collect new rankings and repeat.
Terms that live nearby.
Fine-tuning
Fine-tuning is additional training that teaches a general model your specific patterns: you show it hundreds or thousands of examples of your tasks, your tone and your formats, and it adjusts its internal weights until that behaviour becomes its default.
ReadAlignment
Alignment is the work of making an AI system do what you actually meant, not merely what you literally typed: its goals, limits and refusals are shaped during training and deployment so its behaviour stays within what operators and society intend.
ReadHuman-in-the-Loop
Human-in-the-loop is a design pattern where an AI system pauses at checkpoints you define and asks a person to approve, edit or reject its proposed action before continuing, keeping consequential decisions under human control.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.