Building a Recommender System with PySpark: Popularity Baselines, Collaborative Filtering and ALS at Scale
About the Project
This project began as a continuation of a recommendation-system project I developed as part of my capstone for DS-GA 1004: Big Data during my M.S. in Data Science at New York University’s Center for Data Science, Courant Institute.
The original project was a team effort built around a substantially larger dataset from GoodReads:
- 228,648,342 interactions/ ratings
- 2,360,655 books
- 876,145 users
Working at that scale meant that much of our attention was necessarily directed toward the big-data engineering problem: distributed processing, partitioning, Spark implementation, and working with a dataset too large to handle comfortably on a local machine.
However, in doing so, I realized that I had missed something I actually find more interesting: the mathematics behind the recommendation algorithms themselves.
In particular, I had implemented Alternating Least Squares (ALS), but had not spent enough time understanding the mathematical intuition behind how ALS learns a low-rank representation of the user-item interaction matrix, how regularization enters the optimization problem, or how this differs from other matrix factorization approaches.
So I decided to revisit the project over the summer using a much smaller dataset, with a deliberate shift in emphasis:
Less focus on “Can I process the data at scale?” and more focus on “Do I actually understand what the model is doing?”
Why Scale Down?
For this version, I use the MovieLens dataset, containing approximately:
- 100,000 ratings
- 3,600 tag applications
- 9,000 movies
- 600 users
- Last updated September 2018
The smaller dataset makes it possible to work through the mathematics and experiments much more explicitly while still retaining the structure of a genuine recommendation problem.
I continue to use PySpark, but run the project on the Databricks Free Edition rather than the much larger Google Dataproc/HPC environment used in the original project.
This was a deliberate choice rather than a requirement of the dataset. Although the MovieLens data could easily be processed locally in Python, keeping Spark in the workflow provides an opportunity to explore the distributed implementation of the algorithms and the practical considerations that accompany it, without allowing infrastructure to dominate the project.
The smaller scale therefore gives me the best of both worlds: a manageable environment for understanding the mathematics, while retaining the distributed-computing framework relevant to large-scale recommender systems.
Research Questions
The project is organized around three broad questions.
1. How should recommendation data be prepared?
Before building a recommender system, I examine the data-processing decisions that can substantially affect the validity and performance of the model.
This includes:
- Chronological train/validation/test splitting
- Avoiding temporal leakage
- Partitioning strategies in Spark
- CSV vs. Parquet
2. What is actually happening mathematically inside ALS?
The main focus of the project is understanding latent factor recommendation from first principles.
I explore:
- The user-movie interaction matrix
- The intuition behind a low-rank approximation
- Deriving the matrix-factorization objective
- Alternating optimization
- Regularization
- The relationship between ALS and Singular Value Decomposition (SVD)
- Explicit vs. implicit feedback
- How Spark implements ALS in practice
Rather than treating ALS as a black-box recommender, the goal is to connect the mathematical objective to the algorithm and finally to the implementation.
3. How do different recommendation approaches compare?
I build a progression of increasingly sophisticated recommendation approaches:
- Popularity baseline
- Item-based collaborative filtering
- User-based collaborative filtering
- Cosine similarity
- Explicit-feedback ALS
- Implicit-feedback ALS
The goal is not simply to find the model with the highest evaluation score, but to understand what each approach assumes about user behavior, what problem it is trying to solve, and where its limitations arise.
Similarity Search: Jaccard and MinHash
The project also includes a separate exploration of similarity search, focusing on the relationship between Jaccard similarity and MinHash.
Here, the central question is different:
How can we efficiently identify objects with similar behavioral footprints when the data are represented as sets?
I first derive and calculate Jaccard similarity explicitly, then show how MinHash provides a scalable approximation of Jaccard similarity. Importantly, this is treated separately from rating prediction: Jaccard/MinHash are used here to identify shared behavioral patterns, rather than to infer whether two users have identical preferences.
Project Roadmap
The project therefore moves through the following progression:
Data → Data Preparation → Baselines → Similarity-Based Collaborative Filtering → Matrix Factorization/ ALS Mathematics → Spark Implementation → Model Evaluation → Market Segmentation using Jaccard and Min Hash
The overall objective is not just to build a recommender system, but to use a manageable dataset to understand the mathematical, computational, and modeling decisions underneath it.