4. Collaborative Filtering with Cosine Similarity: Item-Based vs. User-Based
Before building ALS, this project implements the other major branch of collaborative filtering — memory-based CF, which makes predictions by directly comparing raw rows or columns of the interaction matrix, rather than learning a compressed latent representation.
Two variants were built, mirror images of each other:
- Item-based CF — find movies similar to a target movie (based on how users rated them), then predict a user’s rating using their own ratings on those similar movies.
- User-based CF — find users similar to a target user (based on how they rated movies in common), then predict using those similar users’ ratings on the target movie.
Both rely on the same similarity metric: cosine similarity, computed between rating vectors.
\[\text{cosine_sim}(x, y) = \frac{x \cdot y}{\|x\| \, \|y\|}\]For two movies, $x$ and $y$ are their rating vectors across users who rated both movies. For two users, $x$ and $y$ are their rating vectors across movies both users rated. The numerator is a dot product; the denominator normalizes by each vector’s magnitude — so cosine similarity captures the pattern of ratings (do these two rate things similarly, relative to their own scale) rather than raw magnitude alone.
Both models are trained on train only and evaluated on val/test, following the same split established in the data-splitting section.
4.1. Item-Based Collaborative Filtering
Step 1: Compute Item-Item Cosine Similarity
For every pair of movies, find the users who rated both, and compute cosine similarity across their shared ratings.
from pyspark.sql import functions as F
# self-join train on userId to get every pair of movies each user rated
r1 = train.select("userId", F.col("movieId").alias("movieId_1"), F.col("rating").alias("rating_1"))
r2 = train.select("userId", F.col("movieId").alias("movieId_2"), F.col("rating").alias("rating_2"))
pairs = r1.join(r2, on="userId").filter(F.col("movieId_1") < F.col("movieId_2"))
# aggregate: dot product + counts per movie pair
pair_stats = pairs.groupBy("movieId_1", "movieId_2").agg(
F.sum(F.col("rating_1") * F.col("rating_2")).alias("dot_product"),
F.count("*").alias("n_common_users")
)
# per-movie norm, computed once from train
movie_norms = train.groupBy("movieId").agg(
F.sqrt(F.sum(F.col("rating") * F.col("rating"))).alias("norm")
)
item_sim = pair_stats \
.join(movie_norms.withColumnRenamed("movieId","movieId_1").withColumnRenamed("norm","norm_1"), on="movieId_1") \
.join(movie_norms.withColumnRenamed("movieId","movieId_2").withColumnRenamed("norm","norm_2"), on="movieId_2") \
.withColumn("cosine_sim", F.col("dot_product") / (F.col("norm_1") * F.col("norm_2")))
Sanity check on the output: the top pairs by n_common_users showed sensible, well-supported similarities (cosine ~0.6–0.8, 170–195 shared raters). One standout pair — movieId 260 and 1196 (Star Wars-related titles) — came back with cosine similarity 0.81, backed by 170 common raters, exactly the kind of franchise-sequel result you’d expect to validate the approach.
Step 2: Predict Ratings via Similarity-Weighted Average
To predict a user’s rating for a target movie, take the movies they’ve already rated, weight each by its similarity to the target, and compute a weighted average:
\[\hat{r}_{u,\,target} = \frac{\sum_{j \in \text{rated}(u)} \text{sim}(target, j) \cdot r_{u,j}}{\sum_{j \in \text{rated}(u)} \text{sim}(target, j)}\]# make similarity symmetric, keep only positive similarities
sim_a = item_sim.select(F.col("movieId_1").alias("movieId"), F.col("movieId_2").alias("neighbor"), "cosine_sim")
sim_b = item_sim.select(F.col("movieId_2").alias("movieId"), F.col("movieId_1").alias("neighbor"), "cosine_sim")
sim_full = sim_a.union(sim_b).filter(F.col("cosine_sim") > 0)
# join val's target movies against each user's own rated movies, weighted by similarity
val_users_movies = val.select("userId", F.col("movieId").alias("target_movie"), F.col("rating").alias("actual_rating"))
user_ratings = train.select("userId", "movieId", "rating")
candidates = val_users_movies.join(user_ratings, on="userId").join(
sim_full.withColumnRenamed("movieId", "target_movie").withColumnRenamed("neighbor", "movieId"),
on=["target_movie", "movieId"], how="inner"
)
predictions = candidates.groupBy("userId", "target_movie", "actual_rating").agg(
F.sum(F.col("cosine_sim") * F.col("rating")).alias("weighted_sum"),
F.sum(F.col("cosine_sim")).alias("sim_sum")
).withColumn("pred_item_cf", F.col("weighted_sum") / F.col("sim_sum"))
Step 3: Handle Coverage Gaps and Evaluate
Not every (user, target movie) pair gets a prediction — if none of a user’s rated movies are positively similar to the target, no candidate row exists. These gaps fall back to global_mean, and predictions are clipped to the valid [0.5, 5.0] range before scoring:
val_scored = val.join(
predictions.select("userId", F.col("target_movie").alias("movieId"), "pred_item_cf"),
on=["userId", "movieId"], how="left"
)
n_missing = val_scored.filter(F.col("pred_item_cf").isNull()).count()
val_scored = val_scored.fillna({"pred_item_cf": global_mean}) \
.withColumn("pred_item_cf", F.when(F.col("pred_item_cf") > 5.0, 5.0)
.when(F.col("pred_item_cf") < 0.5, 0.5)
.otherwise(F.col("pred_item_cf")))
rmse_item_cf_val = evaluator.setPredictionCol("pred_item_cf").evaluate(val_scored)
Result: Val RMSE = 0.9500, with 760 / 10,358 val rows (7.3%) falling back to the global mean due to zero similarity coverage.
Elaborating why and how we need to handle coverage gaps
The Item-based CF modelled in this project, does not always produce a prediction for every (user, movie) pair. The prediction is built only from the user’s own rated movies that are positively similar to the target movie.
For example, suppose we want to predict User 7’s rating for Inception:
| userId | movieId | rating |
|---|---|---|
| 7 | Inception | 4.5 |
Suppose User 7 has rated Barbie and Grease in train, but their cosine similarities with Inception are:
| Pair | cosine_sim |
|---|---|
| Inception – Barbie | -0.12 |
| Inception – Grease | -0.05 |
Both similarities are negative. Since the implementation explicitly filters to positive similarities:
sim_full = sim_full.filter(F.col("cosine_sim") > 0)
neither movie contributes to the prediction. When the code joins the target movie with the user’s rated movies, no candidate row exists for (User 7, Inception) — there is simply no similarity-weighted signal from which to calculate a prediction.
Why We Drop Negative Similarities
It is important to note that dropping negative cosine similarities is a modeling choice, not a requirement of cosine similarity or item-based CF.
A negative cosine similarity is actually meaningful: it says that the rating patterns for the two movies tend to move in opposite directions. If Inception and Barbie have a strongly negative similarity, users who rate Barbie highly tend to rate Inception less favorably, and vice versa.
So intuitively, negative similarity does contain information. In our example:
| Pair | cosine_sim |
|---|---|
| Inception – Barbie | -0.12 |
| Inception – Grease | -0.05 |
If User 7 has rated Barbie and Grease, these negative similarities are telling us:
User 7’s demonstrated preferences are not aligned with Inception. There is evidence here against recommending Inception.
So why not simply include those negative similarities in the weighted-average prediction?
The problem is the prediction formula itself:
\(\hat r_{ui} = \frac{\sum_j s_{ij}r_{uj}}{\sum_j s_{ij}}\)
It is designed to be a weighted average, which works naturally when the weights are positive. Once negative similarities are allowed, two things can go wrong.
First, the denominator can become very small. Positive and negative similarities can cancel each other:
\(0.51 + (-0.50) = 0.01\)
A denominator close to zero makes the prediction extremely unstable: a tiny change in a similarity or rating can produce a very large change in the predicted rating.
Second, negative weights change the interpretation of the average. A highly rated movie with a negative similarity contributes a negative quantity to the numerator, and the negative denominator can then flip that contribution back into a positive rating. The result is no longer a clean weighted average of the user’s ratings.
So we make a deliberate trade-off:
sim_full = sim_full.filter(F.col("cosine_sim") > 0)
We keep only positive similarities, so every weight in the prediction is positive and the result remains a stable, interpretable weighted average.
The downside is that we can now encounter a coverage gap. If none of the movies User 7 has rated are positively similar to Inception, there is no valid weighted average to compute.
In that situation, the negative similarities still tell us something intuitively — “the movies this user likes are poorly aligned with Inception” — but rather than turning that negative evidence into a potentially unstable rating prediction, this implementation treats it as insufficient positive evidence and falls back to global_mean.
This is why the 7.3% coverage gap is not an inherent limitation of cosine similarity. It is partly a consequence of our decision to use only positive similarities in a simple similarity-weighted-average predictor.
However, if User 7 had also rated Interstellar, and Interstellar had a positive similarity with Inception, that pair would be retained and used as positive evidence in the prediction.
The fallback
When no candidate exists, the prediction is null after the join:
val_scored = val_scored.fillna({"pred_item_cf": global_mean})
That null is replaced with global_mean — the average rating across all of train. So the fallback is effectively saying: if collaborative information is unavailable, use the safest population-level estimate we have.
The clipping
This is a separate issue from coverage:
F.when(F.col("pred_item_cf") > 5.0, 5.0) \
.when(F.col("pred_item_cf") < 0.5, 0.5) \
.otherwise(...)
A similarity-weighted prediction is an arithmetic calculation and can technically fall outside the valid 0.5–5.0 rating range. Clipping simply brings any such prediction back within the valid range before scoring.
Summarising Explicit ALS result and coverage handling
Val RMSE = 0.9500
Out of 10,358 validation rows, 760 (7.3%) had zero positive-similarity coverage and therefore fell back to global_mean rather than receiving a genuine similarity-based prediction.
In short: Item-based CF in this model can fail to produce a prediction for a specific user–movie pair even when both have training data individually, because it filters train pairs for only positive cosine similarity with the target. In our case, this happened for 7.3% of validation rows, which were handled by falling back to the global mean.
4.2. User-Based Collaborative Filtering
The mirror image of item-based CF — same math, transposed onto the user axis.
Step 1: Compute User-User Cosine Similarity
u1 = train.select("movieId", F.col("userId").alias("userId_1"), F.col("rating").alias("rating_1"))
u2 = train.select("movieId", F.col("userId").alias("userId_2"), F.col("rating").alias("rating_2"))
user_pairs = u1.join(u2, on="movieId").filter(F.col("userId_1") < F.col("userId_2"))
user_pair_stats = user_pairs.groupBy("userId_1", "userId_2").agg(
F.sum(F.col("rating_1") * F.col("rating_2")).alias("dot_product"),
F.count("*").alias("n_common_movies")
)
user_norms = train.groupBy("userId").agg(
F.sqrt(F.sum(F.col("rating") * F.col("rating"))).alias("norm")
)
user_sim = user_pair_stats \
.join(user_norms.withColumnRenamed("userId","userId_1").withColumnRenamed("norm","norm_1"), on="userId_1") \
.join(user_norms.withColumnRenamed("userId","userId_2").withColumnRenamed("norm","norm_2"), on="userId_2") \
.withColumn("cosine_sim", F.col("dot_product") / (F.col("norm_1") * F.col("norm_2")))
Sanity check: top pairs showed cosine similarity in the 0.38–0.51 range, backed by 576–992 commonly-rated movies — strong statistical support, consistent with a well-formed similarity table.
Step 2 & 3: Predict and Evaluate
Same weighted-average and evaluation pattern, mirrored onto users instead of movies — for each (user, target movie) in val, find similar users who also rated that movie, weight their ratings by similarity:
sim_renamed = user_sim_full.select(
F.col("userId").alias("target_user"), F.col("neighbor_user").alias("similar_user"), "cosine_sim"
)
step1 = val_targets.join(sim_renamed, on="target_user")
user_candidates = step1.join(
movie_raters.withColumnRenamed("userId","similar_user").withColumnRenamed("movieId","target_movie"),
on=["similar_user","target_movie"]
)
# ... same weighted-average + fallback + clip + evaluate pattern as item-based CF
Result: Val RMSE = 1.0110, same 760/10,358 coverage gap (driven by the same val rows lacking any similarity coverage, regardless of axis).
4.4. Summary for Item Vs User based CF
| Model | Val RMSE |
|---|---|
| Popularity (weighted, C=3) | 0.9930 |
| User-based CF | 1.0110 |
| Item-based CF | 0.9500 |
Item-based CF meaningfully outperforms user-based CF — and even beats the tuned popularity baseline. User-based CF, by contrast, barely edges out (and on the final test set, slightly underperforms) plain popularity.
This isn’t a bug — it reproduces a well-documented finding in recommender systems research, most famously Amazon’s stated rationale for choosing item-to-item over user-to-user collaborative filtering at scale: item similarity is structurally more stable than user similarity. A movie’s “identity” — the pattern of who likes it — is reinforced by potentially hundreds of raters and doesn’t drift much. A user’s taste profile, by contrast, is inferred from a comparatively small, idiosyncratic set of ratings, making user-user cosine similarity a noisier signal to lean on.
This result sets up a natural comparison point for the ALS section that follows: does a model-based latent-factor approach (ALS) do meaningfully better than either memory-based CF variant, and if so, by how much?
4.5. Worked Example: What Actually Goes Into the Cosine Similarity Formula
Cosine similarity between two movies needs two “rating vectors” of equal length — but it’s worth being precise about how those vectors are actually built, since they’re not fixed-length lists indexed by every user in the dataset.
Take a tiny slice of train:
| userId | movieId | rating |
|---|---|---|
| 1 | Titanic | 5 |
| 1 | Notebook | 4 |
| 2 | Titanic | 3 |
| 2 | Matrix | 5 |
| 3 | Titanic | 4 |
| 3 | Notebook | 5 |
How pairs are generated
Self-joining train on userId pairs up every movie a user rated with every other movie that same user rated. User 1 contributes a (Titanic, Notebook) row; user 3 contributes another (Titanic, Notebook) row; user 2 — who never rated Notebook — contributes a (Titanic, Matrix) row instead, and no Titanic–Notebook row at all.
Grouping by (movieId_1, movieId_2) collects only the rows belonging to the same pair. For (Titanic, Notebook), only users 1 and 3 qualify:
| userId | rating (Titanic) | rating (Notebook) |
|---|---|---|
| 1 | 5 | 4 |
| 3 | 4 | 5 |
This is the numerator’s “vector” — and it’s automatically equal length on both sides, because it’s built from exactly the set of users who rated both movies. It isn’t a fixed 610-length array padded with zeros for non-raters; it’s only as long as however many users happen to overlap for that specific pair.
Numerator: the dot product, from shared raters only
\[x \cdot y = (5\times4) + (4\times5) = 20 + 20 = 40\]Denominator: the norm, from all of that movie’s raters
This is where things work differently. Each movie’s norm — $|x| = \sqrt{\sum x_i^2}$ — is computed from every rating that movie has in train, not just the raters it shares with whichever movie it’s being compared against.
Titanic was rated by users 1, 2, and 3 (5, 3, 4):
\[\|Titanic\| = \sqrt{5^2 + 3^2 + 4^2} = \sqrt{25+9+16} = \sqrt{50} \approx 7.07\]Notebook was rated by users 1 and 3 only (4, 5):
\[\|Notebook\| = \sqrt{4^2 + 5^2} = \sqrt{16+25} = \sqrt{41} \approx 6.40\]Notice Titanic’s norm includes user 2’s rating (3), even though user 2 never rated Notebook and therefore never appears in the Titanic–Notebook dot product at all. This is intentional, not an inconsistency: the norm represents a movie’s own stable, overall rating magnitude — a property that shouldn’t change depending on which other movie it’s being compared against.
Putting it together
\[\text{cosine_sim}(Titanic, Notebook) = \frac{\text{dot_product}} {\|Titanic\| \cdot \|Notebook\|} = \frac{40}{7.07 \times 6.40} = \frac{40}{45.25} \approx 0.884\]The numerator restricts to the overlap (you can’t multiply ratings from someone who didn’t rate both); the denominator uses each movie’s own complete, independently-computed magnitude. This is standard practice for pairwise cosine similarity over sparse data.
From similarity to a predicted rating
Once similarities are computed, predicting a rating is a similarity-weighted average of a user’s own ratings:
\[\hat r_{u,\,target} = \frac{\sum_{j \,\in\, \text{rated}(u)} \text{sim}(target, j) \times r_{u,j}}{\sum_{j \,\in\, \text{rated}(u)} \text{sim}(target, j)}\]Say Carol rated Titanic (4) and Notebook (5), and we want to predict her rating for Matrix, with $\text{sim}(Matrix,Titanic)=0.72$ and $\text{sim}(Matrix,Notebook)=0.55$:
\[\hat r_{Carol,\,Matrix} = \frac{(0.72\times4)+(0.55\times5)}{0.72+0.55} = \frac{2.88+2.75}{1.27} = \frac{5.63}{1.27} \approx 4.43\]The numerator is a similarity-weighted sum of Carol’s own ratings; the denominator is the sum of those same similarities, used purely to keep the result on the same 0.5–5.0 rating scale rather than growing unbounded.
The Same Logic, Transposed for User-Based CF
Everything above — pair generation, the shared-overlap numerator, the individually-computed norm denominator, and the similarity-weighted prediction — is repeated identically for user-based CF, just with the roles of user and movie swapped: pairs of users are formed from movies they both rated, each user’s norm is computed from all of their own ratings, and a target rating is predicted as a similarity-weighted average across similar users’ ratings for that movie, rather than a user’s own ratings across similar movies.