Improve CodeLlama’s Math Reasoning Capabilities with Prompt-Based Fine-Tuning

Contributor Name: Federico Castanedo

Difficulty level: Intermediate

We can improve an LLM’s ability to handle specific tasks (for example, math reasoning) by fine-tuning the model. Fine-tuning an LLM refers to the process of taking a pretrained model and further training it on a specific, often smaller, dataset to adapt it to a particular task or domain.

Supervised fine-tuning steps are usually resource-intensive because they involve having high-quality datasets curated by human evaluators and retraining the base LLM for several days.

Prompt-based fine-tuning is a cheap and quick method to adapt any LLM to a specific task and provide a better and clearer response by using just a few examples. This technique is known as few-shot prompting or in-context learning. With few-shot prompting, we provide the model with a few examples before asking for a specific answer to our question. Few-shot prompting helps an LLM understand the format of the response by looking at similar examples.

For reasoning tasks in particular, Wei et. al. recently proposed a prompt-based fine-tuning technique called Chain-of-Thought prompting that uses a series of intermediate reasoning steps within the examples provided in the prompt to improve a model’s reasoning capabilities. Their technique achieved a new state-of-the-art result on the challenging GSM8K dataset.

Inspired by this research, in this Shortcut, I am going to ...