Enterprise Artificial Intelligence Transformation

Chapter 10Managing Model Risk

Artificial intelligence is just a new tool, one that can be used for good and for bad purposes and one that comes with new dangers and downsides as well. We know already that although machine learning has huge potential, data sets with ingrained biases will produce biased results – garbage in, garbage out.

Sarah Jeong, journalist specializing in information technology law

People may not notice AI in their day-to-day lives, but it is there. As we saw in Part II of this book, machine-learning-based programs now review many applications for mortgages. AI algorithms sort through resumes to find a small pool of appropriate candidates before job interviews are scheduled. AI systems curate content for every individual on Facebook. And phone calls to the customer-service departments of cable providers, utility companies, and banks, among other institutions, are answered by voice recognition systems based on AI.

This “invisible” AI, however, can make itself visible in some unintended and occasionally upsetting ways. Retail giant Target uses AI to understand what shoppers are buying and what to recommend to them, but the tactic backfired when Target sent coupons to a man's teenage daughter that featured nursery furniture and maternity clothing.¹ Sometime after storming into a Target store outside of Minneapolis and castigating the manager for encouraging her to get pregnant, the girl's angry father discovered that Target knew something he did not. “It ...