Effective Machine Learning Teams

Chapter 3. Effective Dependency Management: Principles and Tools

In this chapter and the next, we’ll tackle a challenge that every ML practitioner will no doubt encounter at many points in their career: dependency hell. Dependency hell is one of the common causes of the infamous “works on my machine” problem. As an ML practitioner who often deals with dependency hell, you might often be wishing for answers to the following questions:

• How can my teammates and I easily and reproducibly install dependencies whenever and wherever we need them—on a local machine, a notebook in the cloud, a distributed cluster, etc.—with minimal toil and troubleshooting?

• As our project’s dependencies grow ever larger, how can we optimize them so that installing dependencies doesn’t feel like we’re waiting to download the internet?

• How can we ensure that our project is not compromised by security vulnerabilities in its dependencies, and those dependencies’ dependencies?

By the end of Chapters 3 and 4, you will have answers to all these questions. You’ll be able to apply effective dependency management practices to your own projects as you learn:

• How to recognize an incomplete dependency management approach

• Principles and tools for effectively managing dependencies in ML projects

• When, why, and how to use containers

• How to simplify the use of Docker with batect, a command-line tool

These two chapters are suitable for novices who are new to dependency management in Python-based ML projects ...