Reinforcement learning from human feedback (RLHF) trains a model to prefer outputs that human raters rank higher, usually after supervised fine-tuning. It is a major reason chat models sound helpful and refuse some requests — and a source of debates about whose preferences get encoded.
RLHF turns preference data into a training signal: humans (or proxies) rank answers, a reward model learns those rankings, then the policy model is optimized against it.
It is why assistants apologize, hedge, and follow chat norms that raw pretrained models lack.
Latent policy and labor stories increasingly ask who the raters are, what rubrics they use, and how preference data becomes a competitive asset.
No. Labs also use RLAIF, direct preference optimization (DPO), constitutional methods, and rule-based filters alongside or instead of classic RLHF.
Rater bias, reward hacking, and over-refusal can make models sycophantic or less capable on legitimate tasks.