Azure AI Services at Scale for Cloud, Mobile, and Edge

Chapter 9. How Microsoft Runs Cognitive Services for Millions of Users

In the last two chapters, we looked at how important it is to use AI responsibly and with the best practices for machine learning that deliver a practical and responsible AI system. But what does the infrastructure and process for doing that look like in action?

The Azure Cognitive Services we covered in Chapter 4 run 24-7 in more than 30 Azure regions, underpinning features in Microsoft’s own applications as well as for large organizations like Airbus, Progressive Insurance, Uber, and Vodafone, powering apps for thousands of employees and millions of their customers.

There are more than 54 billion Cognitive Services transactions a month; the Speech services alone transcribe over 18 million hours of speech a month. Decision APIs power 6 million personalized experiences on Xbox every day, and over a billion images have been captioned in PowerPoint and Word with automatic alt text created by the Vision services. As Teams usage grew during the pandemic, so did Cognitive Services usage, because it powers live captioning in meetings and transcription of recorded meetings. The Speech services had to scale sevenfold to handle Teams caption needs, which consume 2 million core hours of compute a day on Azure.

And the different Cognitive Services are updated continuously with previews, new features, and fixes to any vulnerabilities in the underlying technology stack. If you want to run your own machine learning models, ...