Azure AI Services at Scale for Cloud, Mobile, and Edge

Preface

AI covers a wide range of techniques and approaches, and you’ll find it everywhere from smartphones to the factory floor.

As AI advances, these techniques are becoming more powerful—and more complex to implement. Increasingly, the most powerful AI systems are being driven by very large deep learning models, trained on huge amounts of data, using billions of parameters that are then customized to solve specific problems.

Building and training those very large models takes the resources of a very large company with a lot of technical expertise—and a huge investment in the infrastructure to run them on. Training the GPT-3 language generation model developed by OpenAI cost on the order of $4 million or more. Just the initial training run on the 45 TB of data used for GPT-3 would take maybe a month of continuous training and over a thousand high-end graphics processing unit (GPU) cards.

That means only a handful of organizations in the world can create and run these very large models, which Stanford’s Institute for Human-Centered Artificial Intelligence dubs foundation models because they’re so significant to the way AI is currently being developed and used—and because you can build on them to create new systems.¹

These include so-called large language models like GPT-3 but also extremely large machine learning models in many domains that rely on semi-supervised deep learning, self-supervised pretraining, transfer learning, and similar methods for creating large powerful models ...