Fine-Tuning AI

Chapter 10. Alignment and Preference Optimization

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Your fine-tuned model from Chapter 9 knows medical terminology. It can rattle off symptoms of various illnesses, and it’s meeting the tone of voice from your Q&A dataset.

But this is just the start of communicating effectively; a model that has memorized the right facts can still deliver them in the wrong tone. Supervised fine-tuning (SFT) teaches the model what to say. Alignment teaches it how to say it.

This chapter introduces the techniques that bridge that gap: Reinforcement Learning from Human Feedback (RLHF) and its modern, more practical successor, Direct Preference Optimization (DPO). By the end, you’ll understand why alignment is a separate stage from fine-tuning, how to build the preference datasets that drive it, and how to run a DPO training loop on your medical Q&A model—all without significant hardware investments! It’s a light introduction to a massive topic, but hopefully will be enough to help you understand how impactful strong ...