Generative AI on Kubernetes

Chapter 7. Job Scheduling Optimization

While model training encompasses the entire LLM lifecycle (from pre-training to alignment to customization), the previous chapter focused on model customization, the most common and practical approach for organizations working with LLMs. It introduced different customization techniques and frameworks, like Kubeflow Trainer, to implement distributed customization jobs on Kubernetes. In particular, a platform administrator must address a new set of operational challenges that go beyond the basic configuration of a training job.

While Chapter 3 focused mainly on inference production workloads, significant overlap exists regarding GPU management in Kubernetes. Moreover, even just looking at the management of long-running jobs on Kubernetes, model customization workloads differ significantly from traditional Kubernetes applications in several critical ways:

• They are inherently resource intensive, requiring specialized hardware (GPUs) across multiple nodes for extended periods, sometimes days or even weeks.

• They exhibit strong interdependencies between components in a way that is not very common for Kubernetes workloads; for instance, all pods in a distributed training job must be scheduled together, using gang scheduling.

• They generate an impressive amount of data to be shared across the network, making network performance a critical bottleneck.

• They represent a considerable cost, both in terms of time and resources, so that a reliable ...