Trustworthy AI

Chapter 6Secure

BAM Inc. was known for taking information security seriously. The company dealt with trade secrets, proprietary information, and financial data from some of the largest companies in the world. It was because BAM Inc. was so reliably focused on information security that many customers elected to work with the enterprise. Given that, Chief Information Officer Masami was respected as a mission‐critical executive who always got the job done.

On an otherwise uneventful morning, Masami received a notice from a law enforcement body that cyber criminals had begun using a new exploit to attack back office AI that reviewed and settled vendor invoices. The tool looked for potential fraud while expediting payments, allowing human employees to work on more valuable tasks. The exploit, however, could mislead the AI and trick it into revealing sensitive financial information.

Masami read through the technical details of the law enforcement notice and realized the potential for harm. If the AI could be fooled, BAM Inc.'s stellar security reputation would suffer a staining blemish. She called in her team to begin identifying where their vulnerabilities were, how to patch them up, and given the threat, whether the AI system should be used at all.

Security is a necessary and pressing issue for all AI systems. We know from experience that valuable technological systems are attractive targets for bad actors. In 2020, the monetary loss from cybercrime was nearly $1 trillion, almost ...