Effective Machine Learning Teams

Preface

It was 9:25 p.m. and the soft glow of Dana’s computer screen glared into her bleary eyes as she logged on to continue fixing an error—red pipelines and countless open tabs filling her screen. She had eaten dinner and finished her everyday chores, but her mind wasn’t really there—it was in a few places, in fact.

It had been an intense day, scattered between long training runs and back-and-forth messages with the support team on customer queries about why the model denied their loan applications. She was in and out of the depths of debugging why the model’s performance just wouldn’t improve, despite various tweaks to the data and model architecture. The occasional stack traces only made things worse.

She was tired, and the tangled heap of uncommitted code changes sitting on her local machine added to the latent cognitive load that was bubbling over in her head. But she had to keep going—her team had already missed the initial release date by four months and the executives’ impatience was showing. What made things worse was a fear that her job might be on the line. One in ten employees in her company—several of whom she knew—were laid off in the latest round of cost-cutting measures.

Everyone on her team was well-meaning and capable, but they were getting bogged down every day in a quagmire of tedious testing, anxiety-laden production deployments, and wading through illegible and brittle code. After a few months of toil, they were all worn down. They were doing their level ...