Large Language Models: The Hard Parts

Appendix. Tools for Local LLM Deployment

An expert is a man who has made all the mistakes which can be made, in a very narrow field.

Niels Bohr

This appendix covers three tools for local LLM deployment and one technique for making LLMs more accessible for local compute environments.

Ollama provides the easiest entry point, offering a Docker-like experience for downloading and running models with simple commands. llama.cpp gives you the underlying engine that powers much of the local LLM ecosystem, converting models to efficient formats and running them on consumer hardware. llamafile takes a unique approach by packaging models as single executable files that run anywhere without dependencies. Together, these tools represent different points on the spectrum between ease of use and fine-grained control.

Quantization is a technique for reducing model size and memory footprint while enhancing inference speed, but it comes with potential trade-offs in model quality. We will cover how to quantize, and also how to evaluate these trade-offs, using two popular measures: perplexity and KL divergence.

Ollama

Ollama is a lightweight, MIT-licensed, open source tool for running LLMs locally. It provides a simple interface for interacting with a wide range of LLMs. Ollama is designed to be easy to install and use, making it a popular choice for developers who want to run LLMs locally without the need for extensive setup or configuration.

Ollama’s key advantages include:

Streamlined model ...