Understanding How Models Learn to Prefer Better Answers Through a DPO Alignment Demo

What Is Alignment?

Alignment means making model behavior better aligned with human goals.

A pretrained model has learned a large amount of language knowledge, but its original objective is mainly next-token prediction. That objective is powerful, but it does not automatically teach the model which answer is more polite, safer, or more helpful.

For example, a user asks:

Write a polite rejection email to a job applicant.

The model could generate:

You did not pass the interview. Do not come again.

It could also generate:

Thank you for your interest in this role. Your background is impressive, but we have decided to move forward with candidates whose experience more closely matches the current opening. We wish you all the best in your job search.

Both responses are grammatically valid, but the second one is clearly more aligned with human preference. Alignment focuses on exactly this difference: a model should not only be able to answer, but should also be more likely to produce the better answer.

Where Alignment Fits in the LLM Generation Pipeline

A typical LLM training and generation pipeline can be simplified as:

Pretraining
-> SFT
-> Preference Alignment, such as RLHF or DPO
-> Inference
-> Post-processing or safety checks
-> Final answer

Each stage has a different role:

Stage	Main purpose
Pretraining	Learn language, knowledge, and general reasoning patterns
SFT	Learn how to follow instructions
Alignment	Learn which answers better match human preferences
Inference	Generate an answer from a prompt
Safety / Policy	Apply additional checks and constraints to the output

From this perspective, Alignment is not a decorative step. It is a key part of moving a model from “can speak” toward “can answer reliably.”

The importance of Alignment is that it further shapes the model’s generation ability toward the kind of behavior humans actually want.

What Is DPO?

DPO stands for Direct Preference Optimization.

DPO data is not the prompt + response format commonly used in SFT. Instead, it uses:

{
  "prompt": "Write a polite rejection email to a job applicant.",
  "chosen": "Thank you for your interest in this role. Your background is impressive, but we have decided to move forward with candidates whose experience more closely matches the current opening. We wish you all the best in your job search.",
  "rejected": "You did not pass the interview. Do not come again."
}

The three fields mean:

Field	Meaning
`prompt`	The user question or instruction
`chosen`	The answer that better matches human preference
`rejected`	The worse, less appropriate, or less preferred answer

The core idea of DPO is not to make the model memorize chosen. Instead, it teaches the model:

For the same prompt, the chosen answer should be preferred over the rejected answer.

Why DPO Matters

In real LLM applications, many questions do not have one single correct answer.

For example:

Will AI replace humans?

An extreme answer might be:

Yes. Everyone will lose their job.

A more balanced answer might be:

AI is a powerful tool that will change many types of work. However, it lacks human emotion, judgment, and creativity, so the more realistic future is collaboration rather than complete replacement.

This is not simply a matter of right or wrong. It involves tone, risk framing, caution, helpfulness, and the boundaries of the answer.

DPO is well suited for this kind of problem. It uses chosen / rejected preference pairs to teach the model which style of response humans are more likely to prefer.

Its value includes:

Making answers more polite
Making responses more balanced
Reducing rude or extreme replies
Reducing unnecessary overconfidence
Improving sensitivity to human preference
Making the model more likely to choose the better answer among alternatives

Running DPO Training

In this demo, DPO training is completed in Colab:

1. Download Qwen/Qwen2.5-0.5B-Instruct from Hugging Face
2. Load the tokenizer
3. Load the policy model
4. Load the reference model
5. Read data/preference_dataset.jsonl
6. Train with TRL DPOTrainer
7. Save the model to models/dpo_aligned_model/

During training, it is normal to see output like:

trainable params: 8,798,208 || all params: 502,830,976 || trainable%: 1.7497

This means LoRA is training only a small portion of the parameters, which is suitable for a small-GPU Colab demo.

If the training result is saved as a LoRA adapter, the output directory will contain files such as:

adapter_config.json
adapter_model.safetensors
tokenizer_config.json
tokenizer.json

Demo Goal

DPO_alignment_demo is a teaching project for learning DPO Alignment.

Project URL:

https://github.com/jackie20260501/DPO_alignment_demo

The goal of this Demo is not to train a production-grade model. Instead, it uses a runnable small project to explain the DPO mechanism: how preference data is organized, how the policy model and reference model work together, how to compare outputs before and after training, and how DPO makes the model more likely to produce answers in the style of chosen.

The environment and tool responsibilities are:

WSL: Manage the project, prepare data, run demo scripts, and organize comparison materials
Colab: Use GPU to run DPO training
Model: Qwen/Qwen2.5-0.5B-Instruct
Trainer: Hugging Face TRL DPOTrainer

Two Model Roles in DPO

DPO training usually involves two model roles:

Policy model
Reference model

The policy model is the model being trained. Its parameters are updated by the DPO loss, so it gradually becomes more likely to prefer the chosen style.

The reference model is usually copied from the same base model before training and remains unchanged during DPO training.

A simple way to understand this is:

The policy model learns.
The reference model provides the reference point.

The reference model matters because it helps prevent the policy model from drifting too far away from its original language ability while trying to fit a small preference dataset.

In this demo, Qwen/Qwen2.5-0.5B-Instruct is used as the base model source:

Qwen/Qwen2.5-0.5B-Instruct
-> policy model: trained
-> reference model: not trained, used as reference

After training, the result is saved to:

models/dpo_aligned_model/

If LoRA is used, this directory usually stores an adapter rather than a full model.

Demo Workflow Breakdown

The whole Demo can be understood in six steps.

Step 1: Prepare Preference Data

The core file is:

data/preference_dataset.jsonl

Each line is a JSON object containing:

prompt
chosen
rejected

A good DPO sample should make the preference clear. For example, chosen should be more polite, specific, and safe, while rejected may be rude, vague, or overly assertive.

Step 2: Check the Data Format

Run:

python scripts/01_prepare_dataset.py

This step checks:

Whether each line is valid JSON
Whether all required fields exist
Whether any field is empty
Whether chosen and rejected are meaningfully different
Whether the data can be loaded by the training script

Data quality is the foundation of DPO. If chosen and rejected are too similar, the model will struggle to learn a stable preference.

Step 3: Generate Baseline Outputs

Before training, run:

python scripts/02_baseline_generate.py

The output is saved to:

outputs/baseline_outputs.jsonl

The purpose of the baseline is to create a comparison point. After DPO training, the same or highly consistent prompts should be used again so the comparison is fair.

Step 4: Run DPO Training in Colab

The training script is:

scripts/03_train_dpo.py

It reads:

configs/dpo_config.yaml
data/preference_dataset.jsonl

Important training parameters include:

Parameter	Meaning
`model_id`	Base model, such as Qwen/Qwen2.5-0.5B-Instruct
`learning_rate`	Learning rate, usually small for DPO
`beta`	Controls how far the policy model can move away from the reference model
`max_steps`	Number of Demo training steps
`max_length`	Maximum length of prompt plus answer
`max_prompt_length`	Maximum prompt length
`use_lora`	Whether to use LoRA to reduce training cost

beta is a key DPO hyperparameter. It controls how strongly the model should stay close to the reference model while learning preferences.

Step 5: Generate DPO Outputs

After the training result is copied back to WSL, generate outputs from the same prompts:

outputs/dpo_outputs.jsonl

When comparing before and after, the prompts should remain the same. Otherwise, differences may come from input changes rather than the effect of DPO.

Step 6: Generate comparison.md

Finally, run:

python scripts/04_compare_outputs.py

The result is:

outputs/comparison.md

This report is useful for showing:

What the model answered before training
What the model answered after DPO
What preference chosen/rejected was meant to teach
Whether the model moved closer to the chosen style
Which changes are only observations and should not be overinterpreted

Relationship Between DPO and RLHF

DPO and RLHF are both Alignment methods. Their shared goal is to make the model not merely generate fluent text, but generate text that better matches human preferences.

Their common foundation is preference:

Which answer do humans prefer?
Which answer is more helpful?
Which answer is safer, more balanced, and more polite?

In other words, DPO and RLHF are not simply teaching the model to memorize standard answers. They train the model to understand which type of answer is more worth choosing.

The difference is the engineering path.

Traditional RLHF usually includes:

1. Collect human preference data
2. Train a Reward Model
3. Use reinforcement learning to optimize the language model

This process is effective, but it is more complex. It requires training a separate reward model and dealing with the stability challenges of reinforcement learning.

DPO is more direct:

Do not train a separate Reward Model.
Use chosen / rejected preference data directly to optimize the language model.

So the common point is that DPO and RLHF both serve Alignment and both perform preference alignment. The difference is that RLHF usually optimizes indirectly through a Reward Model and reinforcement learning, while DPO directly optimizes the model with chosen / rejected preference pairs.

That is why DPO is very suitable for a teaching Demo: it preserves the core idea of preference alignment while making the engineering workflow lighter than full RLHF.

Key Difference Between DPO and SFT

The simplest summary is:

SFT learns answers.
DPO learns preferences.

A more complete comparison:

Item	SFT	DPO
Full name	Supervised Fine-Tuning	Direct Preference Optimization
Data format	`prompt + response`	`prompt + chosen + rejected`
Training goal	Make the model generate the target answer	Make the model prefer chosen over rejected
Focus	Task ability, format, instruction following	Human preference, safety, answer quality
Common position	Usually earlier	Usually after SFT
Intuition	The teacher gives a sample answer	The teacher compares two answers

Why is SFT usually done before DPO?

SFT solves “Can the model answer?”
DPO solves “Does the model answer well?”

If the model cannot follow basic instructions yet, preference optimization alone is limited. Usually, SFT first gives the model basic task ability, then DPO adjusts answer style and preference direction.

How to Interpret Demo Results

This Demo is for teaching, not for rigorous benchmarking.

Small data, few training steps, and a lightweight model can demonstrate:

How DPO data is organized
How DPOTrainer participates in training
How policy and reference models work together
How to compare outputs before and after training
How preference learning can influence response style

But the results should not be overstated as:

The model capability has significantly improved
The model has completed production-grade safety alignment
A small number of samples can represent a full evaluation

A more careful interpretation is:

This demo shows the training mechanism and engineering workflow of DPO alignment.
Post-training outputs can be used as observations of preference movement, but not as a strict evaluation conclusion.

Returning to the Essence of Alignment

LLM generation is not simply “input a prompt, output text.” A usable model is usually shaped by multiple layers:

Pretraining gives the model language and knowledge foundations.
SFT teaches the model to follow instructions.
DPO/RLHF further moves the model closer to human preference.
Inference and safety policies make the final output more controllable.

The value of Alignment is that it moves the model from “can answer” toward “should answer this way.”

DPO provides a relatively direct and practical path: instead of first training a separate Reward Model, it uses preference pairs to tell the model which answer is more desirable.

Summary

DPO_alignment_demo connects the key ideas of DPO through a clear small project:

Use WSL to manage code, data, and demo materials.
Use Colab GPU to run DPO training.
Use Qwen/Qwen2.5-0.5B-Instruct as the base model.
Use prompt/chosen/rejected to build preference data.
Use the policy model to learn preferences.
Use the reference model to control drift.
Use comparison.md to observe before/after changes.

If SFT teaches the model to imitate a good answer, DPO teaches the model to choose the answer that better matches human preference.

Final takeaway:

Alignment determines whether a model behaves more like a reliable assistant; DPO uses intuitive preference pairs to train the model to place better answers higher.