Building Recommendation Systems in Python and JAX

Chapter 5. Data Processing

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

In the trivial recommender that we defined in the introduction, we used a method get_availability; and in the MPIR, we used a method get_item_popularities. We hoped the choice of naming would provide sufficient context about what their function was; but we did not focus on the implementation details. Now we will start unpacking the details of some of this complexity, and see what the toolsets are for online and offline collectors.

Hydrating your system

Getting data into the pipeline is punnily referred to as hydration. There are a lot of water-theme naming conventions in Machine Learning and Data; one article that covers this is by Pardis Noorzad.

PySpark

Spark is an extremely general computing library, with APIs for Java, Python, SQL, and Scala. PySpark’s role in many ML pipelines is for data processing and transformation of the large scale datasets in your pipeline.

Let’s return to the data-structure we introduced for our recommendation problem; recall that ...