Hands-On Small Language Models

Chapter 2. Selecting the Right Small Language Model

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In this chapter we will discuss how to compare different models. As with any machine learning endeavor, we must start with measurement. With the language models there are a few dimensions of measurement we could use. First, there is the quality of a language model as a language model. This only goes so far, especially for small language models which are supposed to be more specialized. The specialization is not just in terms of domains and topics, but also in terms of functionality. This specialization of functionality means that the general ability to produce valid sequences of general “language” is less important. For SLMs, we need to look towards task-specific metrics. The good part of this is that this will resemble classic machine learning measurement, and the bad part is that the more novel our task is, the less likely it is that there is a ready dataset for measurement.

Measuring Language Models

To understand how we will measure SLMs for our project, Theοros, we first need to understand the context ...