Chapter 1. Current View and Challenges of AI Adoption
While adopting AI is an important priority for businesses moving forward, considering its many potential applications, large numbers of organizations are not yet at a stage of their data strategy and digital transformation to even begin developing AI solutions.
To put this trend into context, in larger organizations there are three particularly troublesome hurdles related to data/AI that need to be watched closely:
-
Persistent problems in culture, policy, or technology that prevent the sharing of siloed data and analytics tools across departments—and that prevent collaboration in general
-
Technical debt in data infrastructure, which erodes trust and hinders collaboration
-
Lack of executive sponsorship for data-related projects
Overcoming these hurdles is table stakes before an organization can even begin work on AI projects. Not overcoming the hurdles is the path of least resistance, which may seem appealing, but it leads to “one-off” AI applications and a coin flip as to whether or not the solution will last in production. The persistence of these problems also results in distributed pockets of expertise across an organization, which end up working poorly together and lack the consolidated organizational support they need to sustain.
Another way to think about clearing these hurdles is in terms of layers: having robust data analytics practices is foundational for AI work, although having effective data infrastructure is foundational for analytics work in general. Although the previously listed challenges are generally IT considerations, each one also represents a kind of “people problem” in practice. In other words, we must be mindful about how people within an organization collaborate to make effective use of the organization’s data. We estimate that up to 50% of enterprise organizations are still struggling with these table stakes, largely due to staffing, business culture, cognitive bias, and so on—not due to lack of available technology. However, in terms of operationalizing AI, tackling these challenges is a necessary precondition.
Now that we’ve reviewed the current view of AI adoption in industry, let’s examine the specific challenges for both business stakeholders and technical stakeholders when it comes to AI projects. If we understand the challenges they face, then we can understand how they need to prepare their organizations for operationalizing AI.
Challenges for Business Stakeholders
The top challenges in AI tend to focus on people issues and their related business impact. First and foremost is how to staff a team with people who have the necessary skills through new hires, training current staff, or lateral moves in-house.
Staffing
Staffing AI projects entirely through new hires may be prohibitively expensive, since these required people (data scientists, for example) tend to be in high demand. That approach also limits people within the company from moving into AI projects and bringing along their domain expertise about the business. Good strategies for staffing AI projects will blend new hires with programs for upskilling existing staff, as well as promoting lateral moves in-house through mentoring programs.
One important role, often referred to as a “unicorn,” is the person who has both product management experience and familiarity with the AI technologies involved. These people become crucial for translating between available technology and the business use cases. Over time, the product management programs will incorporate more AI-specific skills, and more people coming from the technology side will advance into product roles. Meanwhile, this represents a staffing bottleneck.
Collaboration
As your AI projects move into production, multiple teams will need to coordinate closely, sharing responsibilities for oversight. How can those respective teams collaborate without having to learn one another’s disciplines in depth? For example, a given project might require a data science team, an operations team, and a risk team to review one another’s work. It would be prohibitive for each of those teams to learn one another’s respective disciplines; some cross-training can help build mutual trust and interdisciplinary collaboration, but there are limits on requiring people to wear too many hats at the same time. Finding a proper balance between abstracting another team’s contributions and obstructing the other teams is key.
Instead of finding out about possible limits or obstructing factors at the last minute, we can be proactive by engaging early with people across the organization who will be working on AI projects. They need to have a complete understanding about the data on which those projects will depend:
-
What is the scope of the data required, and does that data even exist within the organization?
-
Which team collects and manages that data?
-
Do we have access and permission to use it?
-
Which other teams use this data, and is there existing code or analysis that we could reuse?
-
Are there any known data quality issues, and can we trust this data source?
Resolving cross-departmental collaboration issues often requires business stakeholders to step in and set priorities overall. Such leadership may also clarify who to talk with about what, across the organization. Establishing this leadership from stakeholders often becomes the most difficult step.
Executive Support
Moving up an organizational level, far too often the executive ranks do not support the initiatives or even the kind of thinking required for successful AI projects. Executive staff and board members may think in terms of once-popular management ideologies such as Six Sigma and the 1990s era of business intelligence, which do not fit well with contemporary needs for AI. For example, one core principle in Six Sigma is to reduce process variation: to get rid of sources of uncertainty. The language of Six Sigma and related business philosophies tends to emphasize terms such as predictable and consistency and control, which are at odds with AI technologies that depend on probabilistic techniques and embrace “variation” in the data. Aversion to uncertainty is prevalent among executives. Meanwhile, data from that period of Six Sigma was generally organized in data warehouses and leveraged through business intelligence tools. Data that didn’t fit into SQL queries effectively didn’t matter, but that’s become legacy thinking.
Now, the world leverages machine learning. While SQL queries are still important for data management, reporting, and even getting training data ready for building models, we most definitely build machine learning and deep learning models using completely different techniques! The more valuable algorithms at their very core are stochastic in nature: the algorithms are based on probabilistic approaches—that is, parts are randomized—which cannot be predicted precisely. Nor do we necessarily want to reduce variance within the training data; in fact, that can become a way of introducing harmful bias.
As we noted, AI approaches tend to embrace uncertainty—for example, taking advantage of uncertainty in data to explore/exploit regions of opportunity. To those who still think in terms of earlier times, the priorities for AI will seem inverted: up is down, left is right. Unfortunately, all too often that confusion translates into missteps for organizational priorities.
An important point to keep in mind is that the enterprise transformations required to overcome these challenges—such as getting robust data infrastructure into place, with widespread adoption within an organization—are generally long-term projects that may take years to complete. Even if the top executive ranks give their full buy-in about priorities, the results won’t be immediately visible. Without long-term vision and commitment from the executive ranks, the foundational components become nonstarters for AI adoption. Meanwhile, the competition is probably leveraging AI and moving fast, gaining ground.
Challenges for Technical Stakeholders
In addition to the business challenges in adopting AI, there are significant challenges for the technical stakeholders. We’ve learned that the most effective practices for adopting AI are those that avoid producing “one-offs” and those that integrate their practices more broadly across the organization, as we’ll explore in later sections. For technical stakeholders, many of the challenges in adopting AI are related to people, processes, and platforms.
People: Training and Mentoring
From the perspective of the technical stakeholders, one of the most immediate concerns should be to get their people trained and prepared for the new roles required for AI work. To reinforce the upskilling and training, it’s also important to support mentoring programs internally so that people help one another learn and grow in their roles. In the next few sections, we’ll cover the steps needed to achieve an AI Center of Excellence, which addresses this need for training and mentoring.
Process: Unique Platforms and Processes
AI work often requires unique platforms and processes. Software engineers who work within integrated development environment (IDE) tools may feel bewildered by a flurry of new terms and techniques that now become potentially more crucial than stepping through code with a debugger: sampling, cross-validation, imbalanced data, dropout, learning rate, entropy, vanishing gradients, etc.
The typical processes used in AI development will lead to priorities, concerns, and means of evaluation that differ substantially from software engineering. This is where so much of the current MLOps dialogue in industry is coming from. Moreover, the best-of-breed tools for AI are evolving and changing rapidly. For example, among the deep learning frameworks, TensorFlow had been quite a buzzword in recent years, although PyTorch gets used more widely; even so, JAX usage is rising, and practitioners should probably be learning their way through HuggingFace libraries, too. For a technical stakeholder, embracing AI represents yet another set of steep learning curves to navigate, in order to make appropriate decisions at the organizational level.
Platform: Collaborative Data Infrastructure
Another hurdle faced by data science teams in many organizations is the matter of gaining access to the data needed for AI projects. Far too often, access to data gets hampered by outdated data management policies or data frameworks that are locked in silos. For example, one department may not be able to read another department’s datasets without authorization. Organizations must provide data platforms as infrastructure that allow for cross-departmental collaboration and data sharing. Often, that requires paying down infrastructure technical debt as a first step.
Other Challenges and Balancing Time-to-Market
Even when the necessary tools and practices are in place to begin developing AI solutions, there are still other areas of concern where the technical stakeholders must lead: trust, data privacy, fairness and bias, security, risks due to model drift, and so on. Given infinite time to research a problem and refine a solution for it, we could do wonders with AI, although we’ll probably never get to enjoy that situation. Instead, we have budgets, schedules, reviews, audits, and so on. The metric that puts all of these points into context is time-to-market. Technical stakeholders, therefore, need to focus on managing the time-to-market metrics in order to guide their teams toward success.
The Challenge of Trusted AI
Organizations that adopt AI are increasingly aware that they should be using it in a socially responsible, fair, and nondiscriminatory way. Adding to this awareness is a number of regulations being introduced by governing bodies in this space.
There are rising concerns, though, around how to ensure that AI can be trusted. Some of these concerns are fueled by organizations making news headlines in their use of AI in unfair, discriminatory ways. Let’s take a look at some of the challenges of trusted AI.
Bias and Fairness
When developing AI solutions, we need to ensure that privileged groups are not at a systematic advantage compared to nonprivileged groups. The definition of a privileged group depends on industry and use case. In general, features like age, gender, and ethnicity are particularly sensitive elements when defining privileged groups. Take the example of a credit risk use case. The artificial intelligence/machine learning (AI/ML) model classifies whether a certain credit application is risky or not. We need to ensure that the model outcomes (risky or not risky) are not biased toward a particular age, gender, or ethnicity.
Eliminating bias and ensuring fairness in our AI/ML models is not as simple as avoiding sensitive features for model training and scoring.
Implicit bias often becomes an issue when other features can act as proxies for sensitive features. Zip code, for example, is a particularly common proxy, but there can be other nonobvious proxies, especially when new features have been created through automated feature-engineering steps. Regardless of whether it is a direct or implicit bias scenario, we need to have the proper guardrails to ensure fair outcomes.
Drift
Most runtime environments will be able to monitor standard performance metrics used in an AI/ML model (for example, metrics like accuracy, AUC, precision, recall, and F1 score in a classification use case). The performance of a model can drift over time for various reasons. Model drift could be caused by changes in the relative weights or contributions of different features toward the outcome. Drift can also relate to the data that is being fed into the model. Data drift occurs when payload data (input data that is sent to an AI/ML model plus the model’s output) is considerably different from training data. For example, training data may not have included data ranges or combinations seen in real life. It is important to monitor for both model performance drift and data drift over time and take corrective action when thresholds are breached.
Robustness
We need our AI/ML model to be consistent in its behavior. This is especially true when executing models in environments different from original development, training, and testing environments. Models also need to be defended against adversarial attacks. For example, we need a mechanism to check if training data has been poisoned with malicious samples. Here again, we see the importance of monitoring the model over time.
Explainability
Traditional statistical models are relatively simple to interpret and explain. Interpretability and explainability become more difficult when working with models that use techniques like neural networks and gradient boosting. Explainability takes on different forms. It could be a global explanation of why a model behaves a certain way. We are seeing an increasing requirement for local explainability. This is where we need to provide an explanation for an individual transaction, either to the end user or to a governing or regulatory body. The organization should be able to play back the transaction, explain why the model produced a certain outcome, and then trace it all the way back through model version, engineered features, the datasets that features came from, and so on. There is also increasing demand for contrastive explanations, which support “what-if” analyses—for example, “At what point would an outcome have been different?”
All of these challenges need to be addressed as you operationalize AI, end-to-end and across the life cycle stages (which we’ll cover shortly). Emerging regulations around AI ethics are suggesting the need for a structured approach to detecting, monitoring, and correcting these issues to deliver trusted AI.
Regulations and Roles
Various roles in an organization need to be involved to ensure trusted AI. Traditionally, highly regulated industries like finance and health care have had model risk management (MRM) teams. For example, an MRM team in a US-based financial sector organization (like a retail bank) would look into ways to manage and mitigate the reputational or financial risk of using financial models, in compliance with the Federal Reserve’s SR 11-7 guidance. AI/ML models introduce additional challenges, as we previously discussed, and the MRM/validation teams need to work alongside the data science teams to address these challenges at the speed and scale at which model versions are produced.
Operationalizing AI in a trustworthy manner is not just the responsibility of a risk management or compliance team, but rather the responsibility of all the different roles and personas involved across the AI life cycle.
1 Mark Grover, “The Data Production-Consumption Gap”, Medium, December 1, 2020.
Get Operationalizing AI now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
