Designing Large Language Model Applications

Chapter 7. Advanced Fine-Tuning Techniques

In the previous chapter, we presented the canonical way to fine-tune a typical LLM. In the real world, there are a wide variety of motivations for updating an LLM, and similarly there are multiple ways to update it. In this chapter, we will describe several advanced fine-tuning techniques and highlight the scenarios in which each technique would be suitable.

Why would you want to update the parameters of an LLM? We touched upon this in previous chapters but let’s go through it in more detail now:

Domain adaptation

The data that we work with belongs to a specialized domain that the LLM might not have been familiarized with during pre-training. In this case, we would like to update the model by training it on domain-specific data.

Task adaptation

We care about LLM performance on specific downstream tasks. To improve the LLM’s performance on these tasks, we can train it on task-specific data. This can be supervised or unsupervised.

Knowledge updating

We would like to keep the LLM’s knowledge up-to-date by continually training it on new data.

Controllability/steerability

We would like to control the behavior of the LLM, including making it more likely to follow user requests written in natural language, reject certain types of requests, and so on. Techniques to achieve this are collectively called alignment training. We will defer discussion of alignment training to Chapter 8.

In this chapter, we will learn techniques that can be ...