Reinforcement Learning from Human Feedback (RLHF) Explained

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the training methodology that transformed large language models from impressive text predictors into genuinely useful AI assistants. It is the key ingredient behind systems like ChatGPT, Claude, and Gemini — enabling them to follow instructions, refuse harmful requests, and produce responses that humans actually prefer.

In this article, we break down the three-stage RLHF pipeline, examine the math behind it, and walk through practical code examples.

Why Supervised Fine-Tuning Alone Isn't Enough

Pre-trained language models learn to predict the next token. Supervised fine-tuning (SFT) teaches them to follow instructions by training on curated prompt-response pairs. But SFT has a fundamental limitation: it optimizes for imitation, not quality. A model trained with SFT might produce grammatically correct but unhelpful, verbose, or subtly harmful outputs — because the training signal doesn't capture nuanced human preferences.

RLHF addresses this by letting humans rank model outputs and then training the model to maximize those preference signals.

The Three Stages of RLHF

Stage 1: Supervised Fine-Tuning (SFT)

Start with a pre-trained base model and fine-tune it on high-quality demonstration data. This gives the model a solid foundation for following instructions.

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

training_args = TrainingArguments(
    output_dir="./sft-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-5,
    bf16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sft_dataset,  # curated instruction-response pairs
)
trainer.train()

Stage 2: Reward Model Training

Next, we train a reward model (RM) that learns to predict human preferences. Human annotators are shown pairs of model responses and asked to pick the better one. The reward model is trained on these comparisons using a Bradley-Terry preference model:

$$\mathcal{L}{RM} = -\log\sigma(r\theta(x, y_w) - r_\theta(x, y_l))$$

Where $y_w$ is the preferred response and $y_l$ is the rejected one.

import torch
import torch.nn as nn

class RewardModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.backbone = base_model
        self.reward_head = nn.Linear(base_model.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids, attention_mask=attention_mask)
        last_hidden = outputs.last_hidden_state[:, -1, :]
        reward = self.reward_head(last_hidden)
        return reward

def preference_loss(reward_chosen, reward_rejected):
    return -torch.log(torch.sigmoid(reward_chosen - reward_rejected)).mean()

Stage 3: Reinforcement Learning via PPO

Finally, the SFT model is optimized using Proximal Policy Optimization (PPO) to maximize the reward model's scores while staying close to the original SFT policy. The objective is:

$$\max_\pi \mathbb{E}{x,y \sim \pi}[r\theta(x,y)] - \beta \cdot \text{KL}[\pi | \pi_{\text{SFT}}]$$

The KL-divergence penalty prevents the model from "reward hacking" — finding degenerate outputs that game the reward model.

from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    batch_size=16,
    learning_rate=1.4e-5,
    kl_penalty="kl",
    init_kl_coef=0.2,
)

ppo_trainer = PPOTrainer(
    model=sft_model,
    config=ppo_config,
    ref_model=ref_model,       # frozen copy of SFT model
    reward_model=reward_model,
    tokenizer=tokenizer,
)

for batch in dataloader:
    query_tensors = batch["input_ids"]
    response_tensors = ppo_trainer.generate(query_tensors)
    rewards = reward_model(query_tensors, response_tensors)
    ppo_trainer.step(query_tensors, response_tensors, rewards)

Recent Advances: DPO and Beyond

Direct Preference Optimization (DPO) simplifies RLHF by eliminating the separate reward model entirely. Instead of training an RM and running PPO, DPO directly optimizes the language model on preference data using a closed-form loss:

$$\mathcal{L}{DPO} = -\log\sigma\left(\beta \log\frac{\pi\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)$$

This approach is simpler to implement, more stable to train, and has become the preferred method at many research labs.

Key Challenges

• Reward hacking: Models can exploit weaknesses in the reward model rather than genuinely improving.
• Annotation quality: RLHF is only as good as the human feedback — disagreements between annotators introduce noise.
• Scalability: Collecting high-quality human preferences is expensive and slow.
• Alignment tax: RLHF can slightly reduce raw capability in exchange for safety and helpfulness.

Conclusion

RLHF represents a paradigm shift in how we train AI systems — moving from purely optimizing prediction accuracy to aligning models with human values and preferences. While newer methods like DPO are simplifying the pipeline, the core insight remains: human feedback is the bridge between capable models and useful ones.