Machine Learning for High-Risk Applications

Chapter 5. Security for Machine Learning

If “the worst enemy of security is complexity,” as Bruce Schneier claims, unduly complex machine learning systems are innately insecure. Other researchers have also released numerous studies describing and confirming specific security vulnerabilities for ML systems. And we’re now beginning to see how real-world attacks occur, like Islamic State operatives blurring their logos in online content to evade social media filters. Since organizations often take measures to secure valuable software and data assets, ML systems should be no different. Beyond specific incident response plans, several additional information security processes should be applied to ML systems. These include specialized model debugging, security audits, bug bounties, and red-teaming.

Some of the primary security threats for today’s ML systems include the following:

• Insider manipulation of ML system training data or software to alter system outcomes

• Manipulation of ML system functionality and outcomes by external adversaries

• Exfiltration of proprietary ML system logic or training data by external adversaries

• Trojans or malware hidden in third-party ML software, models, data, or other artifacts

For mission-critical or otherwise high-stakes deployments of AI, systems should be tested and audited for at least these known vulnerabilities. Textbook ML model assessment will not detect them, but newer model debugging techniques can help, especially when fine-tuned to ...