Machine Learning Logistics

Chapter 8. Lessons Learned

The shape of the computing world has changed dramatically in the past few years with a dramatic emergence of machine learning as a tool to do new and exciting things. Revolution is in the air. Software engineers who might once have scoffed at the idea that they would ever build sophisticated machine learning systems are now doing just that. Look at Ian with his TensorChicken system. There are lots more Ians out there who haven’t yet started on that journey, but who soon will.

The question isn’t whether these techniques are taking off. The question is how well prepared you will be when you need to build one of these systems.

New Frontier

Machine learning, at scale, in practical business settings, is a new frontier, and it requires some rethinking of previously accepted methods of development, social structures, and frameworks. The emergence of the concept of DataOps—adding data science and data engineering skills to a DevOps approach—shows how team structure and communication change to meet the new frontier life. The rendezvous architecture is an example of the technical frameworks that are emerging to make it easier to manage machine learning logistics.

The old lessons and methods are still good, but they need to be updated to deal with the differences between effective machine learning applications and previous kinds of applications.

We have described a new approach that makes it easier to develop and deploy models, offers better model evaluation, ...