Large Language Models: The Hard Parts

Chapter 7. Evaluating LLMs for Safety

If you get on the wrong train, get off at the first station. The longer you stay on, the more expensive the return trip will be.

Japanese proverb

Warning

Please be advised that in this chapter, we discuss harmful prompts (such as asking an LLM how to make a bomb) and unsafe model behavior for the purpose of evaluating and mitigating safety risks. Examples are abbreviated or abstracted where appropriate.

Without proper safeguards, LLMs can generate harmful content and respond to malicious prompts with dire outcomes. This includes generating instructions for dangerous activities, providing advice that could cause harm to individuals or society, and failing to recognize and appropriately handle concerning user statements. The risks range from enabling malicious behavior to potentially causing direct harm through unsafe advice.¹

Figure 7-1, drawn from “SimpleSafetyTests: A Test Suite for Identifying Critical Safety Risks in Large Language Models” by Bertie Vidgen et al. (2023), shows a simple yet alarming example of harmful responses from an input prompt provided by some open source LLMs.

Figure 7-1. Responses from Mistral (7B), Dolly v2 (12B), and Llama2 (13B) to a harmful user prompt

Thus, when we deploy LLMs or build LLMBAs, we need to think about safety, and we need to do so at two distinct stages: safety evaluation during development and ...