Trustworthy AI

Chapter 4Transparent

Walter was the manager of BAM Inc.'s largest and most productive plant, based in Wichita, Kansas. He was a no‐nonsense leader who was prized for making the right decisions that kept employees safe, production flowing, and constraints managed. He was so good at it, in fact, that when the chief of operations informed him they would be deploying a new AI system to monitor equipment function, he said he didn't need it. After all, what could a machine do that he and his team could not? His plant was the gem in BAM Inc.'s global footprint.

Despite his refusal, the system was deployed anyway and soon Walter was seeing new data points and notices flagging issues that required attention. Some of the notifications proved to be almost spooky in their predictive capabilities. In one case, the yaw of a large secondary internal motor in a lathe was barely outside of its safe operating parameters, but it could have become a problem left unattended. But other notifications were just wrong, alerting on problems that simply did not exist.

Walter did not know much about AI, and aside from a three‐page memo from the corporate office, he had no idea how the new system functioned. And because of that and its inconsistencies, he did not trust it.

So when a notification told him a gear shaper machine would suffer a catastrophic failure within a few hours, he ignored it. It couldn't possibly be correct, right?

At the core of many challenges with achieving trustworthy AI is the ...