MLOps with Databricks

Chapter 2. Developing on Databricks

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo is available at https://github.com/mvechtomova/oreilly-mlops-with-databricks.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at [e-mail].

Databricks has significantly grown in popularity in the last years, and many data scientists and machine learning engineers struggle to find a way to develop their code on Databricks.

Notebooks are first-class citizens on a Databricks platform, and this is a to-go tool for data scientists to use during the experimentation phase. It is a perfect tool for that purpose and has significantly contributed to Python becoming the number one language on GitHub, but it does not help to promote best software engineering practices. As a result, code written by data scientists needs extensive refactoring afterwards:

• Creating functions, classes, and modules

• Separating tasks

• Separating configurations that may change in the future

• Adding unit testing

• Adding logging and documentation

• Packaging the project’s code

In my career, I have seen too ...