Introduction to Generative AI, Second Edition

2 Training large language models: Learning at scale

This chapter covers

• How LLMs and multimodal models are trained
• Exploring efficient architectures, such as Mixture of Experts and sparse models
• Improving performance through post-training and inference-time techniques
• Emergent properties of LLMs

For decades, the digital economy has run on the currency of data. The digital economy of collecting and trading information about who we are and what we do online is worth trillions of dollars. As more of our daily activities have moved onto the internet, the mill has ever more grist to grind through. Large language models (LLMs) are inventions of the internet age, emulating human language by vacuuming up terabytes of text, image, and video data ...