Effective Machine Learning Teams

Chapter 4. Effective Dependency Management in Practice

In the previous chapter, we laid out the principles for effective dependency management—can you recall the four principles?—and supporting tools. In this chapter, let’s have some fun and put them into practice.

In this chapter, you will learn:

• What “check out and go” looks like in practice

• How to use Docker, batect, and Poetry to create consistent, reproducible, and production-like runtime environments in each step of the ML delivery lifecycle

• How to automatically detect security vulnerabilities in your dependencies and automate dependency updates

The techniques in this chapter are what we use in our real-world projects to create reproducible, consistent, isolated, production-like runtime environments for our ML code. They help us effectively and securely manage dependencies and avoid dependency hell.

Let’s begin!

In Context: ML Development Workflow

In this section, you will see “check out and go” in action. In the code exercise, we’ll run through the following steps with the goal of training and serving a model that predicts the likelihood of a loan default:

1. Run a go script to install prerequisite dependencies on our host machine.

2. Create a Dockerized local development environment.

3. Configure our code editor to understand the project’s virtual environment, so that we can have a helpful coding assistant.

4. Run common tasks in the ML development lifecycle (e.g., train models, run tests, start API).

5. Train and ...