book

Building Recommendation Systems in Python and JAX

by Bryan Bischof, Hector Yee

December 2023

Intermediate to advanced

338 pages

8h 57m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Key Components of a Recommendation SystemCollectorRankerServerSimplest Possible RecommendersThe Trivial RecommenderMost-Popular-Item RecommenderA Gentle Introduction to JAXBasic Types, Initialization, and ImmutabilityIndexing and SlicingBroadcastingRandom NumbersJust-in-Time CompilationSummary
The User-Item MatrixUser-User Versus Item-Item Collaborative FilteringThe Netflix ChallengeSoft RatingsData Collection and User LoggingWhat to LogCollection and InstrumentationFunnelsBusiness Insight and What People LikeSummary
Zipf’s Laws in RecSys and the Matthew EffectSparsityUser Similarity for Collaborative FilteringPearson CorrelationRatings via SimilarityExplore-Exploit as a Recommendation System ϵ -greedyWhat Should ϵ Be?The NLP-RecSys RelationshipVector SearchNearest-Neighbors SearchSummary
Online Versus OfflineCollectorOffline CollectorOnline CollectorRankerOffline RankerOnline RankerServerOffline ServerOnline ServerSummary
Revision Control SoftwarePython Build SystemsRandom-Item RecommenderObtaining the STL Dataset ImagesConvolutional Neural Network DefinitionModel Training in JAX, Flax, and OptaxInput PipelineSummary
Hydrating Your SystemPySparkExample: User Similarity in PySparkDataLoadersDatabase SnapshotsData Structures for Learning and InferenceVector SearchApproximate Nearest NeighborsBloom FiltersFun Aside: Bloom Filters as the Recommendation SystemFeature StoresSummary
Architectures by Recommendation StructureItem-to-User RecommendationsQuery-Based RecommendationsContext-Based RecommendationsSequence-Based RecommendationsWhy Bother with Extra Features?Encoder Architectures and Cold StartingDeploymentModels as APIsSpinning Up a Model ServiceWorkflow OrchestrationAlerting and MonitoringSchemas and PriorsIntegration TestsObservabilityEvaluation in ProductionSlow FeedbackModel MetricsContinuous Training and DeploymentModel DriftDeployment TopologiesThe Evaluation FlywheelDaily Warm StartsLambda Architecture and OrchestrationLoggingActive LearningSummary

Tech StackData RepresentationBig Data FrameworksCluster FrameworksPySpark ExampleGloVE Model DefinitionGloVE Model Specification in JAX and FlaxGloVE Model Training with OptaxSummary
Bilinear Factor Models (Metric Learning)Feature-Based Warm StartingSegmentation Models and HybridsTag-Based RecommendersHybridizationLimitations of Bilinear ModelsCounting RecommendersReturn to the Most-Popular-Item RecommenderCorrelation MiningPointwise Mutual Information via Co-occurrencesSimilarity from Co-occurrenceSimilarity-Based RecommendationsSummary
Latent SpacesDot Product SimilarityCo-occurrence ModelsReducing the Rank of a Recommender ProblemOptimizing for MF with ALSRegularization for MFRegularized MF ImplementationWSABIEDimension ReductionIsometric EmbeddingsNonlinear Locally Metrizable EmbeddingsCentered Kernel AlignmentAffinity and p-salePropensity Weighting for Recommendation System EvaluationPropensitySimpson’s and Mitigating ConfoundingSummary
EnvironmentsOnline and OfflineUser Versus Item MetricsA/B TestingRecall and Precision@ kPrecision at kRecall at kR-precisionmAP, MMR, NDCGmAPMRRNDCGmAP Versus NDCG?Correlation CoefficientsRMSE from AffinityIntegral Forms: AUC and cAUCRecommendation Probabilities to AUC-ROCComparison to Other MetricsBPRSummary
Where Does Ranking Fit in Recommender Systems?Learning to RankTraining an LTR ModelClassification for RankingRegression for RankingClassification and Regression for RankingWARPk-order StatisticBM25Multimodal RetrievalSummary
Experimentation TipsKeep It SimpleDebug Print StatementsDefer OptimizationKeep Track of ChangesUse Feature EngineeringUnderstand Metrics Versus Business MetricsPerform Rapid IterationSpotify Million Playlist DatasetBuilding URI DictionariesBuilding the Training DataReading the InputModeling the ProblemFraming the Loss FunctionExercisesSummary
Hard RankingLearned AvoidsHand-Tuned WeightsInventory HealthImplementing AvoidsModel-Based AvoidsSummary
Diversification of RecommendationsImproving DiversityApplying Portfolio OptimizationMultiobjective FunctionsPredicate PushdownFairnessSummary
ShardingLocality Sensitive Hashingk-d TreesHierarchical k-meansCheaper Retrieval MethodsSummary
Markov ChainsOrder-Two Markov ChainOther Markov ModelsRNN and CNN ArchitecturesAttention ArchitecturesSelf-Attentive Sequential RecommendationBERT4RecRecency SamplingMerging Static and SequentialSummary
Multimodal RecommendationsGraph-Based RecommendersNeural Message PassingApplicationsRandom WalksMetapath and HeterogeneityLLM ApplicationsLLM RecommendersLLM TrainingInstruct Tuning for RecommendationsLLM RankersRecommendations for AISummary

Content preview from Building Recommendation Systems in Python and JAX

Part V. The Future of Recs

I’m hungry for more. What are companies doing in Prod?

We’ve come so far, but there’s so much more! RecSys moves fast, and it’s worth knowing the concepts you’ll see at next year’s conferences. These ideas are already proven to some extent—none of them are pure science fiction—but they haven’t settled into their final forms yet.

Before we get into these few very modern ideas, it’s also worth noting all the topics we haven’t covered in this book. Our most heinous omissions are probably reinforcement learning techniques and ideas related to conformal methods. Both are deeply important aspects of recommendation systems, but both also require a significantly different background and treatment, and thus are ill-suited to fit into the structure here. Additionally, neither is well introduced to the JAX ecosystem, and thus they are a much harder lift to scaffold.

When your scale is big enough, and the previous chapters are no longer buttering your toast, the following chapters will show you how to level up. At the time of writing, all these methods are in production at Fortune 500 companies valued at more than $10 billion. Learn these concepts and then go build the next TikTok.