Machine Learning Logistics

Chapter 3. The Rendezvous Architecture for Machine Learning

Rendezvous architecture is a design to handle the logistics of machine learning in a flexible, responsive, convenient, and realistic way. Specifically, rendezvous provides a way to do the following:

• Collect data at scale from a variety of sources and preserve raw data so that potentially valuable features are not lost

• Make input and output data available to many independent applications (consumers) even across geographically distant locations, on premises, or in the cloud

• Manage multiple models during development and easily roll into production

• Improve evaluation methods for comparing models during development and production, including use of a reference model for baseline successful performance

• Have new models poised for rapid deployment

The rendezvous architecture works in concert with your organization’s global data fabric. It doesn’t solve all of the challenges of logistics and model management, but it does provide a pragmatic and powerful design that greatly improves the likelihood that machine learning will deliver value from big data.

In this chapter, we present in detail an explanation of what motivates this design and how it delivers the advantages we’ve mentioned. We start with the shortcomings of previous designs and follow a design path to a more flexible approach.

A Traditional Starting Point

When building a machine learning application, it is very common to want a discrete response system. In such a system, ...