Machine Learning Logistics

Chapter 6. Models in Production

...developing and deploying ML systems is relatively fast and cheap, but maintaining them over time is difficult and expensive

D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NIPS 2015

Modern machine learning systems are making it much easier to build a basic decisioning system. That ease, however, is a bit deceptive. Building and deploying a first decisioning system tends to go very well, and the results can be quite impressive for the right applications. Adding another system typically goes about as well. Before long, however, strange interactions can begin to appear in ways that should be impossible from a software engineering point of view. Changing one part of the system affects some other part of the system, even though tests in isolation might suggest that this is impossible.

The problem is that systems based on machine learning can have some very subtle properties that are very different from more traditional software systems. Partly, this difference comes from the fact that the outputs of machine learning systems have much more complex behaviors than typical software components. It also comes about, in part at least, because of the probabilistic nature of the judgments such systems are called upon to make.

This complexity and subtlety makes the management of such systems trickier than the management of traditional well-modularized software systems based on microservices. Complex machine learning systems can grow ...