Managing Cloud Native Data on Kubernetes

Chapter 10. Machine Learning and Other Emerging Use Cases

In previous chapters, we covered traditional data infrastructure including databases, streaming platforms, and analytic engines with a focus on Kubernetes. Now it’s time to start looking beyond, exploring the projects and communities that are beginning to make cloud native their destination, especially concerning AI and ML.

Any time multiple arrows start pointing in the same direction, it’s worth paying attention. The directional arrows in data infrastructure all point to an overall macro trend of convergence on Kubernetes, supported by several interrelated trends:

• Common stacks are emerging for managing compute-intensive AI/ML workloads, including those that leverage specific hardware such as GPUs.

• Common data formats are helping to promote the efficient movement of data across compute, network, and storage resources.

• Object storage is becoming a common persistence layer for data infrastructure.

In this chapter, we will look at several emerging technologies that embody these trends, the use cases they enable, and how they contribute to helping you further manage the precious resources of compute, network, and storage. We have chosen a few projects that touch on different aspects of ML and using data—this is by no means an exhaustive survey of every technology in use today. We hear directly from the engineers working on each project and provide some details on how they fit into a cloud native data stack. You are ...