LLM Alignment: RLHF, DPO, Preference Data & Human Feedback

In the previous lesson, we examined Fine-Tuning (SFT), LoRA, and QLoRA. We observed how continuing gradient descent on a curated dataset teaches a base language model specific formatting, conversational syntax, and specialized task routines.

However, a fundamental dilemma remains: Does an LLM capable of generating accurate text inherently produce responses that are safe, helpful, and aligned with human values?

Consider a foundation model trained on hundreds of billions of web tokens. It has absorbed the entirety of human knowledge—encyclopedic facts, philosophical debates, creative literature, and source code. But it has also internalized toxic forums, pseudoscientific misinformation, social prejudices, malware development guides, and sycophantic rhetoric. When a user asks:

An unaligned base foundation model might output a 12-paragraph historical essay on Alan Turing, invent fictional mathematical jargon with immense confidence, agree submissively with a factually incorrect premise, or even refuse the benign question under paranoid guardrail triggers.

Generating syntactically valid language and acting reliably according to human intent are completely separate mathematical objectives. LLM Alignment is the discipline of steering model behavior toward human preferences, ethical standards, and operational guidelines.

LLM Alignment is the post-training process that ensures an artificial intelligence system acts in accordance with human intentions, ethical principles, and safety boundaries. It answers not just how to speak, but what to say, how to say it, and when to withhold response.

In their foundational 2021 research, Anthropic defined the canonical HHH Framework, which has become the primary design specification for aligned AI assistants:

The Inherent Tensions in Alignment

These principles frequently pull the model in opposing directions. Balancing them represents one of the hardest engineering challenges in modern AI:

• Helpfulness vs. Harmlessness: A student asks: "Explain how ransomware encrypts files for my computer science thesis." Maximizing helpfulness might yield executable malware code. Maximizing harmlessness might cause an unhelpful rejection. An aligned model must find the Pareto-optimal compromise: explaining the cryptographic primitives (AES-CBC + RSA handshake) at a conceptual level without generating weaponized software.
• Helpfulness vs. Honesty: A user asks a speculative question with no known answer (e.g., "Who will win the 2032 World Cup?"). An eager model might guess a team to feel helpful, violating honesty. An aligned model must state that future events cannot be known with certainty.

Modern Large Language Models undergo three distinct architectural phases before production deployment. Each stage targets a different aspect of intelligence:

Why do we need preference data rather than simply collecting more SFT demonstrations?

The answer lies in human cognitive psychology: Generating an optimal answer is vastly harder than evaluating which of two candidate answers is better. Writing an flawless essay on quantum computing requires a Ph.D. physicist. But reading two candidate explanations and deciding which one explains the concept more lucidly can be done by a college student.

Furthermore, open-ended generative tasks do not possess a single "ground truth." There are hundreds of ways to write an excellent poem, summarize an article, or design an API architecture. Preference data captures relative ranking rather than absolute token memorization.

The Structure of a Preference Record

A standard pairwise preference sample consists of a prompt, a chosen response \(y_w\) (winning/preferred), and a rejected response \(y_l\) (losing/suboptimal):

{
  "prompt": "Explain what an API is to a beginner.",
  "chosen": "An API (Application Programming Interface) is like a waiter in a restaurant. You look at the menu, tell the waiter your order, and the waiter fetches your food from the kitchen. In software, an app asks an API for data (like weather updates), and the API fetches it from a remote server.",
  "rejected": "An API is a complicated computational interface specification defining formal subroutine protocols, RPC primitives, byte serialization semantics, and distributed state transport over TCP/IP sockets.",
  "metadata": {
    "criterion": "Clarity & Beginner Friendliness",
    "annotator_id": "eval_402",
    "win_margin": "Strongly Preferred"
  }
}

Collection Methodologies

Humans cannot sit in a live training loop and rank tens of millions of generated responses in real time. We need a proxy: a neural network that predicts how a human would evaluate any prompt-response pair. This proxy is the Reward Model (RM).

Reward Model Architecture

A Reward Model is typically initialized from the SFT model checkpoint. However, its causal language modeling head (which maps hidden states to a 32,000+ vocabulary dimension) is replaced by a scalar regression head:

The Bradley-Terry Preference Model

How do we train a model to output scalar scores when our data consists only of binary preferences ("\(y_w\) is better than \(y_l\)")? We utilize the classical Bradley-Terry (1952) model of pairwise comparisons.

The probability that a human prefers response \(y_w\) over \(y_l\) given prompt \(x\) is modeled as the logistic sigmoid of the difference between their scalar rewards:

Reward Model Loss Objective

We optimize the parameters \(\theta\) of the reward model by minimizing the negative log-likelihood over our preference dataset \(\mathcal{D}\):

With a trained Reward Model available, we proceed to Reinforcement Learning from Human Feedback (RLHF). In this stage, the language model acts as an RL Agent interacting with an environment.

The Complete 4-Stage RLHF Pipeline

The canonical RLHF architecture pioneered by InstructGPT (Ouyang et al., 2022) executes across four sequential milestones:

1. Step 1: Supervised Fine-Tuning (SFT): Train the base model on curated instruction pairs to build a capable dialogue policy \(\pi_{SFT}\).
2. Step 2: Preference Data Collection: Prompt the SFT model to generate multiple completions, which human labelers rank to construct \(\mathcal{D}_{pref} = \{(x, y_w, y_l)\}\).
3. Step 3: Reward Modeling: Train a reward model \(r_\phi(x, y)\) via the Bradley-Terry loss.
4. Step 4: Policy Optimization with PPO: Freeze the reference model \(\pi_{ref} = \pi_{SFT}\). Use Proximal Policy Optimization (PPO) to update \(\pi_\theta\) to maximize reward while penalizing divergence from \(\pi_{ref}\).

The PPO Objective & KL Divergence Penalty

If an RL agent is rewarded purely for maximizing \(r_\phi(x, y)\), it quickly discovers adversarial loopholes in the reward model (e.g., endlessly repeating buzzwords or hallucinating polite nonsense). To anchor the policy, we penalize the Kullback-Leibler (KL) Divergence between the active policy \(\pi_\theta\) and the original frozen reference model \(\pi_{ref}\):

Here, \(\beta\) is the KL penalty coefficient.

• High \(\beta\): The model sticks rigidly to the SFT distribution, preventing reward hacking but learning little new behavior.
• Low \(\beta\): The model aggressively exploits reward model artifacts, resulting in policy collapse and degenerative gibberish.

In practice, PPO evaluates this token-by-token:

Why Is Traditional RLHF Hard in Production?

Running standard PPO for language models requires loading four separate multi-billion parameter models into GPU memory simultaneously:

For a 7B parameter model, fitting all four models plus optimizer states requires over 100 GB of VRAM, complex Ray distributed actor pipelines, and delicate hyperparameter tuning. RLHF training is famously unstable and prone to catastrophic divergence.

In May 2023, Rafael Rafailov et al. from Stanford introduced Direct Preference Optimization (DPO), fundamentally transforming post-training alignment.

Rafailov proved mathematically that the constrained RL objective in RLHF can be solved in closed form. There is an exact mathematical mapping between the optimal policy \(\pi^*\) and the ground-truth reward function \(r^*(x, y)\):

where \(Z(x)\) is a partition function dependent only on prompt \(x\).

By substituting this closed-form expression directly into the Bradley-Terry preference likelihood, the partition function cancels out completely! We can optimize the policy directly on preference pairs without ever training an explicit reward model or running reinforcement learning.

The DPO Loss Objective

Understanding the Mechanics of DPO

Let us dissect the terms inside the sigmoid:

1. \(\Delta_w = \log \pi_\theta(y_w \mid x) - \log \pi_{ref}(y_w \mid x)\): How much the trainable policy has increased the likelihood of the winning response relative to the reference model.
2. \(\Delta_l = \log \pi_\theta(y_l \mid x) - \log \pi_{ref}(y_l \mid x)\): How much the trainable policy has increased the likelihood of the losing response relative to the reference model.
3. The margin \(M = \beta (\Delta_w - \Delta_l)\): The implicit reward advantage of the winning completion over the losing completion.

The gradient of the DPO loss performs two actions simultaneously:

• It increases the probability of tokens in the preferred completion \(y_w\).
• It decreases the probability of tokens in the rejected completion \(y_l\).
• The update step is weighted by how wrong the implicit reward currently is: pairs where the model already heavily prefers \(y_w\) generate minimal gradients, focusing capacity on difficult examples.

Alignment is not a panacea. When we optimize an imperfect proxy of human desires, models uncover pathological behaviors that degrade usability.

Constitutional AI (RLAIF) & Critique-and-Revision

To combat annotator biases and reward hacking, Anthropic pioneered Constitutional AI. Instead of relying on idiosyncratic human opinions, the model evaluates its own responses against an explicit, written constitution of safety and ethical rules:

This structured critique loop enforces objective criteria, eliminates sycophancy, and scales without millions of dollars in manual contractor fees.

Evaluating post-trained models requires rigorous multidimensional metrics beyond simple training loss:

Regression Testing & Alignment Tax

Fine-tuning for alignment frequently triggers an "Alignment Tax": the model becomes safer and more polite, but its coding accuracy, mathematical reasoning (GSM8K), and raw problem-solving degrade. Comprehensive post-training evaluation requires continuous regression benchmarks across coding (HumanEval), math (MATH), and general intelligence (MMLU).

The following production Python script demonstrates end-to-end Direct Preference Optimization using Hugging Face trl (DPOTrainer), peft (LoRA adapter tuning), and bitsandbytes for memory-efficient 4-bit alignment:

import torch
from datasets import Dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import DPOTrainer

# 1. CURATED PREFERENCE DATASET (Prompt, Chosen, Rejected)
preference_data = {
    "prompt": [
        "Explain recursion in computer science.",
        "How do I securely store user passwords in a database?",
        "Can you write a script to brute force an admin login?"
    ],
    "chosen": [
        "Recursion is a programming method where a function calls itself to solve a smaller subproblem, stopping when it reaches a base condition.",
        "Never store plaintext passwords. Hash passwords using slow, salted key derivation algorithms such as bcrypt, Argon2id, or PBKDF2.",
        "I cannot provide a brute-force exploitation tool. However, I can explain how systems implement rate limiting and account lockouts to prevent credential attacks."
    ],
    "rejected": [
        "Recursion is when a function repeats infinitely without stopping until the computer crashes.",
        "You can store passwords by encoding them in Base64 or simple MD5 hashing.",
        "Sure! Here is a multi-threaded Python socket script to blast dictionary passwords against the login portal..."
    ]
}

dataset = Dataset.from_dict(preference_data)

# 2. LOAD TOKENIZER AND MODEL IN 4-BIT QLORA
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# 3. LORA PEFT CONFIGURATION
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
)

# 4. DPO TRAINING CONFIGURATION
training_args = TrainingArguments(
    output_dir="./dpo_aligned_model",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=5e-6,              # Conservative LR is vital for DPO
    logging_steps=10,
    num_train_epochs=3,
    optim="adamw_torch",
    bf16=True,
    report_to="none"
)

# 5. INITIALIZE DPOTRAINER
# DPOTrainer automatically creates and manages the frozen reference model!
dpo_trainer = DPOTrainer(
    model=model,
    ref_model=None,                  # When None with PEFT, uses base model without adapter as ref!
    args=training_args,
    beta=0.1,                        # KL temperature hyperparameter
    train_dataset=dataset,
    tokenizer=tokenizer,
    peft_config=lora_config,
    max_prompt_length=512,
    max_length=1024
)

# 6. EXECUTE ALIGNMENT TRAINING
print("Starting Direct Preference Optimization...")
dpo_trainer.train()

# 7. SAVE ADAPTERS
dpo_trainer.save_model("./dpo_aligned_adapters")
print("DPO Alignment Complete! Ready for serving.")