Responsible Machine Learning

Chapter 1. Introduction to Responsible Machine Learning

“Success in creating effective AI, could be the biggest event in the history of our civilization. Or the worst.”

Stephen Hawking

Machine learning (ML) systems can make and save money for organizations across industries, and they’re a critical aspect of many organization’s digital transformation plans. For these reasons (and others), ML investments were increasing rapidly before the COVID-19 crisis, and they’re expected to stay healthy as the situation unfolds. However, ML systems present risks for operators, consumers, and the general public. In many ways, this is similar to an older generation of transformational commercial technologies, like jetliners and nuclear reactors. Like these technologies, ML can fail on its own, or adversaries can attack it. Unlike some older transformational technologies, and despite growing evidence of ML’s capability to do serious harm, ML practitioners don’t seem to consider risk mitigation to be a primary directive of their work.¹

Common ML failure modes include unaccountable black-box mechanisms, social discrimination, security vulnerabilities, privacy harms, and the decay of system quality over time. Most ML attacks involve insider manipulation of training data and model mechanisms; manipulation of predictions or intellectual property extraction by external adversaries; or trojans hidden in third-party data, models, or other artifacts. When failures or attacks spiral out of control, they become full-blown AI incidents, creating significant adverse outcomes for the operator or the public. There have been over 1,000 reports of AI incidents to date.

While AI incidents are receiving more attention in the news and technology media of late, the hype around ML still seems to focus mostly on ML successes and not on ML risks. Subsequently, some decision makers and practitioners implement ML without a sober evaluation of its dangers. This report will cut through the hype to provide a high-level overview of ML’s emerging risk mitigation practices—often called “responsible machine learning.” This first chapter will give definitions of responsible AI and ML, and Chapters 2, 3, and 4 discuss viable ML risk mitigation steps for people, processes, and technologies, respectively. Chapter 5 closes this report with business-driven perspectives on risk and trust.

What Is Responsible Machine Learning?

What is responsible ML? It’s not strictly defined yet, and the authors of this report don’t seek to define it precisely. The concept of responsible ML needs time to evolve and grow with input from diverse practitioners, researchers, and decision makers. We hope that, like commercial aviation and energy production today, risk mitigation will eventually rise to the forefront of ML’s practice, and there will be no need to differentiate between the general practice of ML and the responsible practice of ML. So, instead of putting forward a single definition, we present several potential definitions and discuss a few key similarities and differences between them to increase community awareness of this vital concept.

Responsible Artificial Intelligence

Several researchers and organizations have put forward helpful related definitions, particularly for “Responsible Artificial Intelligence.” Given that ML is a subdiscipline of AI, and that the two terms are often used interchangeably, these definitions seem like an excellent place to start.

In her book, Responsible Artificial Intelligence (Springer), Virginia Dignum defines the eponymous concept: “Responsible Artificial Intelligence is about human responsibility for the development of intelligent systems along fundamental human principles and values, to ensure human-flourishing and well-being in a sustainable world.”

The Institute for Ethical AI & Machine Learning presents eight principles that “provide a practical framework to support technologists when designing, developing or maintaining systems that learn from data.” The principles include:

Human augmentation

Human review and assessment of risks

Bias evaluation

Understanding, documenting, and monitoring sociological discrimination

Explainability by justification

Transparency and explainability

Reproducible operations

Processes and outcomes should be reproducible

Displacement strategy

Consideration of the replacement of human jobs

Practical accuracy

Real-world accuracy in addition to test data accuracy

Trust by privacy

Addressing training data and consumer data privacy

Data risk awareness

Reasonable security precautions for data and models

Google has also put forward Responsible AI Practices. These include using human-centered design principles, using multiple assessment metrics for any AI system, examining raw data, understanding the limitations of selected approaches, and thorough testing and monitoring of AI systems. Google is just one many organizations to publicize such guidance, and a brief summary of the many posted responsible AI guidelines boils down to the use of transparent technical mechanisms that create appealable decisions or outcomes, perform reliably over time, exhibit minimal social discrimination, and are designed by humans with diverse experiences, both in terms of demographics and professional backgrounds.

The authors of this text recently put forward two additional relevant definitions. Both are visual definitions. One is a higher-level conceptual summary, and the other is geared toward frontline practitioners. The higher-level description uses a Venn diagram, presented in Figure 1-1, to portray responsible AI as a combination of several preexisting and evolving disciplines.

Figure 1-1. A responsible AI Venn diagram (courtesy of Benjamin Cox and H2O.ai).

Figure 1-1 claims that responsible AI is the combination of:

Ethical AI

Sociological fairness in ML predictions (i.e., whether one category of person is being weighed unequally or unfavorably)

Explainable AI

The ability to explain a model after it has been developed

Human-centered machine learning

Meaningful user interactions with AI and ML systems

Interpretable machine learning

Transparent model architectures and increasing how intuitive and comprehensible ML models can be

Secure AI

Debugging and deploying ML models with similar counter measures against insider and cyber threats, as seen in traditional software

Compliance

Aligning your ML systems with leading compliance guidance such as the EU GDPR, the Equal Credit Opportunity Act (ECOA), or the US Federal Reserve’s SR 11-7 guidance on model governance

In the next section, a more technical definition is presented as a workflow in Figure 1-2 and adapted from the recent paper, A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing. It specifically addresses details of Responsible ML.

A Responsible Machine Learning Definition

Most AI in the world today is likely based on ML. In trying to be as careful and realistic as possible, the Figure 1-2 workflow is designed specifically for today’s ML systems. It walks practitioners through the processes required to mitigate many known risks associated with ML. In addition to traditional ML workflow steps, this diagram emphasizes transparency, human review, model end-of-life issues, and the evaluation of multiple key performance indicators (KPIs), including fairness, privacy, and security.

The many other available definitions for responsible AI and ML touch on a wide variety of topics, including everything from environmental impact to future unemployment. Common themes running through most definitions include human consideration and review of risks, enabling effective human interaction with ML systems, enhanced transparency and the treatment of discrimination, privacy harms, and security vulnerabilities. Notably, both the Responsible Machine Learning Workflow paper and the Venn diagram in Figure 1-1, bring compliance and legality into the fold of responsible ML. Based on our experience as industry practitioners, we find that regulation and law can provide some of the clearest guidance for difficult ethical problems that arise in the implementation of ML systems. Moreover, legality is often the bottom-line concern for many high-stakes applications of ML. Compliance, legality, and regulation for ML, and several other concepts presented in the responsible AI and ML definitions will be discussed in the following chapters.

Figure 1-2. A responsible machine learning workflow diagram (adapted with permission of the authors).

¹ See also https://oreil.ly/9hzwC, https://oreil.ly/hFjRY, and https://oreil.ly/2T8Kt.