Large Language Models: The Hard Parts

Chapter 8. LLM Alignment: A Case Study

A people that values its privileges above its principles soon loses both.

Dwight D. Eisenhower

Warning

Please be advised that in this chapter, we discuss harmful prompts (such as asking an LLM how to make a bomb) and unsafe model behavior for the purpose of evaluating and mitigating safety risks. Examples are abbreviated or abstracted where appropriate.

In this chapter, we use alignment to mean adapting a model’s behavior to a defined policy or set of preferences. Policy alignment is the challenge of ensuring that an LLM behaves in accordance with our values, organizational guidelines, and other norms to reliably generate the right text in the right way for the right reasons. An LLM trained purely on internet data learns to predict text statistically, which means it also learns human biases, harmful content, misinformation, and toxic language patterns alongside genuinely useful knowledge.

More operationally, policy alignment is the set of techniques we use to take that raw capability and shape it into a system that is helpful, honest, and harmless in a predictable, controllable way. Without alignment, we have a powerful but unpredictable system that might generate brilliant financial advice one moment and dangerous misinformation the next. With alignment, we can build LLMBAs whose behavior we can trust, and deploy responsibly.

The distinction between safety (what we covered in the previous chapter) and alignment is subtle but important. ...