Skip to Content
View all events

Working Locally with Open Source LLMs

Published by O'Reilly Media, Inc.

Beginner to intermediate content levelBeginner to intermediate

Leveraging LLMs with Ollama in the Terminal

Course outcomes

  • Introduce the fundamental concepts and benefits of LLMs of different sizes.
  • Teach participants how to use choose an LLM appropriate to their use case and that can be run on their hardware
  • Provide hands-on experience with running LLMs using Ollama and Jupyter notebooks.
  • Guide participants through the installation of Ollama to run LLMs in the command line and set up a simple RAG pipeline.
  • Explore various applications and use cases of local LLMs.
  • Discuss the challenges and considerations of using LLMs locally.

Running large language models (LLMs) on your own computer can provide significant benefits in terms of cost savings, enhanced privacy, and generally greater control. But with the number of LLMs growing every day, how do you choose the right one? How can you effectively run these models on your local machine? Join machine learning engineer Lisa Becker for this hands-on webinar where you will learn about small, powerful LLMs that are designed to run efficiently on personal hardware.

This course will guide you through the essential aspects of small LLMs, from understanding their benefits to practical implementation. You'll learn to set up and run various models using Jupyter notebooks and Ollama, a framework for LLM inference in C++.

What you’ll learn and how you can apply it

  • Understand the Pros and Cons of locally run LLMs
  • Understand and tune basic LLM parameters
  • Select the right LLM for specific use cases
  • Install, configure and run LLMs with Ollama and Jupyter Notebooks
  • Build a simple RAG pipeline

This live event is for you because...

  • Software engineers and developers
  • Data scientists and machine learning engineers
  • You want to be able to run open-source LLMs on your hardware

Prerequisites

  • Familiarity with Python programming, Jupyter and virtual environments
  • Basic understanding of LLMs

Course Set-up:

  • Windows or MacOS
  • Have Python and Jupyter installed
  • [Optional:] Have at least 1 document to use as custom knowledge base for the LLM ready, e.g. your CV in a .txt format
  • [Optional:] Have Ollama installed

Recommended preparation:

Recommended follow-up:

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Introduction (5 minutes)

  • Speaker and topic introduction
  • Course overview and learning objectives

What are language models? (30 minutes)

  • Definition and context of LLMs
  • Distinction between large LLMs and small LLMs
  • Benefits of small LLMs: efficiency, cost, privacy, and control
  • "Which of the following are LLMs?" (Engaging poll to identify real and fictional LLMs)
  • Overview and key features of smaller, open-source LLMs, such as gpt-oss (OpenAI), Gemma (Google), Mistral (Mistral AI), Llama (Meta), Phi (Microsoft), Deepseek, Qwen (Alibaba)

Setting up and running language models with Ollama (45 minutes)

  • Exercise/Lab: Identify which LLMs your computer can run
  • Choose an appropriate LLM
  • Walk-through of Ollama, showcasing locally running LLMs with some best practices and tips
  • Showcase running LLMs locally with best practices and tips
  • Perform a simple NLP task (text summarization)
  • Modify parameters for different use cases and performance tuning (like temperature, max tokens, top_k…)

Break (10 minutes)

Setting up and running RAG with Ollama and Jupyter Notebook (60 minutes)

  • Exercise/Lab: Setting up llama.cpp: Detailed installation guide and troubleshooting
  • Basic usage of llama.cpp:
  • Demonstrate a practical example, such as text generation or question answering

Retrieval-Augmented Generation (RAG) with llama.cpp

  • Introduction to RAG
  • Setting up a simple RAG pipeline
  • Running RAG examples

Wrap-up (15 minutes)

  • Comprehensive overview of various applications and use cases of LLMs
  • Discuss challenges and limitations of using small LLMs
  • Q&A

Your Instructor

Lisa Becker

Lisa Becker is a machine learning engineer specializing in natural language processing and large language models at Pure App. With a master’s degree in cognitive systems from the University of Potsdam, Lisa applies her expertise to domains that include speech recognition, sentiment analysis, and text generation, leveraging her technical and linguistic skills to enhance the user experience and engagement of Pure App. She develops and deploys NLP and LLM solutions that enable users to communicate more effectively, discover compatible matches, and receive personalized recommendations. She also contributes to the research and innovation of the app, exploring new ways of using AI to create meaningful and authentic connections. Lisa is a passionate advocate for diversity, equity, and inclusion and is dedicated to sharing her knowledge as a public speaker, blogger, and course instructor for O’Reilly. As a certified #IAmRemarkable facilitator, she empowers individuals to celebrate their achievements and break through societal biases.

Skills covered

  • Large Language Models (LLMs)
  • Generative AI
  • Unified Modeling Language (UML)