Machine Learning Logistics

Chapter 7. Meta Analytics

I know who I WAS when I got up this morning, but I think I must have been changed several times since then.

Alice in Wonderland, by Lewis Carroll

Just as we need techniques to determine whether the data science that we used to create models was soundly applied to produce accurate models, we need additional metrics and analytics to determine whether the models are functioning as intended. The question of whether the models are working breaks down into whether the hardware is working correctly, whether the models are running, and whether the data being fed into the models is as expected. We need metrics and analytics techniques for all of these. We also need to be able to synthesize all of this information into simple alerts that do not waste our time (more than necessary).

The rendezvous architecture is designed to throw off all kinds of diagnostic information about how the models in the system are working. Making sense of all of that information can be difficult, and there are some simple tricks of the trade that are worth knowing. One major simplification is that because we are excluding the data science question of whether the models are actually producing accurate results, we can simplify our problem a bit by assuming that the models were working correctly to begin with—or at least as correctly as they could be expected to work. This means that our problem reduces to the problem of determining whether the models are working like they used to do. ...