3. Popularity Baseline(s)
Before building anything genuinely personalized, every recommender system project needs a floor to beat — a model with zero understanding of individual taste, just a sense of what’s broadly liked or broadly watched. This section builds three variants of that floor, in increasing sophistication, and evaluates each one honestly against held-out data.
The Setup, Common to All Three
We have train, val, and test — split per-user and chronologically (see the Data Splitting section). Every popularity baseline in this section follows the same shape:
- Fit something using
trainonly. - Predict a value for every
(user, movie)pair inval. - Evaluate those predictions against
val’s actual ratings using RMSE.
One structural quirk worth naming upfront: none of these three models actually use userId at all. They predict the same value for a given movie regardless of who’s asking — that’s precisely what makes them non-personalized, and exactly why they’re the floor every later model (item/user CF, ALS) needs to beat.
Because the split is chronological, a small number of movies in val/test never appear in train at all — 901 of them, as established earlier. Every baseline below needs a fallback for these unknown movies.
In all three cases, that fallback is the global mean rating — the single average rating across all of train, computed once and reused throughout:
global_mean = train.agg(F.avg("rating")).first()[0]
3.1. Baseline 1: Naive Mean
The idea: predict, for any movie, whatever its average rating was in train, for every user.
When a movie appears in val or test, its predicted rating for all users will simply be the mean rating that movie achieved in the train set across all users who rated it there. If a movie in val or test was absent in train and therefore does not have a predicted mean assigned to it from training, then the prediction assigned to it is simply the mean rating of all the movies in the train set.
For every row in val, we look up that movie’s average rating in train and use that number as the prediction. If the movie never appeared in train, we fall back to global_mean.
Fit:
movie_avg = train.groupBy("movieId").agg(
F.avg("rating").alias("pred_naive")
)
Predict:
val_naive = (
val
.join(movie_avg, on="movieId", how="left")
.fillna({"pred_naive": global_mean})
)
Evaluate:
rmse_naive_val = evaluator.setPredictionCol(
"pred_naive"
).evaluate(val_naive)
Result: RMSE = 1.0156
The flaw is immediately visible when inspecting movie_avg: movies with very few, and therefore unreliable, ratings can have misleadingly extreme averages. A lone lucky (or unlucky) rating is therefore treated with exactly as much confidence as hundreds of ratings.
| title | n_ratings | pred_naive |
|---|---|---|
| Jane Eyre (1944) | 1 | 5.0 |
| Goal! The Dream Begins (Goal!) (2005) | 1 | 5.0 |
| My Man Godfrey (1957) | 1 | 5.0 |
| SORI: Voice from the Heart (2016) | 1 | 5.0 |
| Ooops! Noah is Gone… (2015) | 1 | 5.0 |
| Tickling Giants (2017) | 1 | 5.0 |
| Guy X (2005) | 1 | 5.0 |
| Marriage of Maria Braun (The Ehe der Maria Braun, Die) (1979) | 1 | 5.0 |
| My Sassy Girl (Yeopgijeogin geunyeo) (2001) | 1 | 5.0 |
| Continental Divide (1981) | 1 | 5.0 |
This motivates the next version.
3.2. Baseline 2: Weighted / Bayesian-Shrinkage Mean
The idea: still predict a single number per movie, but don’t fully trust movies with very few ratings. Instead, pull their average toward the global mean, proportional to how little data supports them.
The formula is:
\[\boxed{ \text{pred_weighted} = \frac{ C \cdot \text{global_mean} + n\_ratings \cdot \text{movie_avg} }{ C + n\_ratings } }\]Read it as a tug-of-war between two forces:
- The movie’s own average, weighted by how many ratings it actually has.
- The global mean, weighted by the constant
C.
When n_ratings is small compared with C, the global mean has more influence and the prediction stays close to global_mean.
When n_ratings is large compared with C, the movie’s own average dominates and the global mean’s influence becomes negligible.
C can therefore be thought of as the number of hypothetical average-rated votes we assume before fully trusting the movie’s observed average.
As an example, for C=20, notice how higher ratings were pulled down towards the mean, and lower ones were pulled up towards the mean, thereby shrinking (tightening) the entire ratings spread for predicted ratings.
Weighted mean of movies in the train set, sorted by descending view count, then ascending view count:
| movieId | movie_avg | n_ratings | pred_weighted |
|---|---|---|---|
| 318 | 4.421428571428572 | 280 | 4.360817939757031 |
| 858 | 4.2988505747126435 | 174 | 4.217759700655202 |
| 2959 | 4.252577319587629 | 194 | 4.183389635173408 |
| 260 | 4.219148936170213 | 235 | 4.163707380106312 |
| 1196 | 4.230366492146596 | 191 | 4.162300388280138 |
| 750 | 4.304597701149425 | 87 | 4.156498896515041 |
| 1197 | 4.251937984496124 | 129 | 4.1526535699805995 |
| 50 | 4.217032967032967 | 182 | 4.147254365975789 |
| 2571 | 4.192607003891051 | 257 | 4.143485133310864 |
| 48516 | 4.287356321839081 | 87 | 4.1424802049262555 |
For movies with only one rating:
| movieId | movie_avg | n_ratings | pred_weighted |
|---|---|---|---|
| 6835 | 5.0 | 1 | 3.5831134251004446 |
| 5746 | 5.0 | 1 | 3.5831134251004446 |
| 131724 | 5.0 | 1 | 3.5831134251004446 |
| 1473 | 4.0 | 1 | 3.5354943774813967 |
| 5764 | 4.5 | 1 | 3.559303901290921 |
Fit:
movie_stats = train.groupBy("movieId").agg(
F.avg("rating").alias("movie_avg"),
F.count("rating").alias("n_ratings")
)
movie_pop = movie_stats.withColumn(
"pred_weighted",
(
C * F.lit(global_mean)
+ F.col("n_ratings") * F.col("movie_avg")
) / (C + F.col("n_ratings"))
)
Predict + Evaluate: use the same join, fillna, and evaluation pattern as Baseline 1, but with pred_weighted as the prediction column.
Tuning C
C is a genuine hyperparameter, so I swept several candidate values against the validation set:
| C | Validation RMSE |
|---|---|
| 1 | 0.9959 |
| 2 | 0.9930 |
| 3 | 0.9930 |
| 5 | 0.9952 |
| 10 | 1.0015 |
| 20 | 1.0105 |
| 50 | 1.0243 |
| 100 | 1.0346 |
The minimum occurs at C = 3, with RMSE = 0.9930 — a modest but real improvement over the naive mean’s RMSE of 1.0156.
Inspecting movies with the fewest ratings also confirms that the shrinkage mechanism is working: their pred_weighted values now sit close to global_mean, rather than being pinned wherever their one or two lucky/unlucky ratings happened to land.
3.3. Baseline 3: Pure Popularity by Volume
The motivating question here is different:
What if “popular” shouldn’t mean “highly rated” at all — what if it should mean widely watched, regardless of whether people loved or merely tolerated it?
A blockbuster that a huge number of people watched, but rated only moderately, might still be worth recommending on the strength of sheer engagement volume alone.
The model’s output is therefore genuinely different from the first two approaches. It is never really predicting a rating — it is simply recommending every viewer the same hierarchy of movies, from most watched (most ratings) to least watched (fewest ratings).
So the “training stage” for this model is simply counting the number of ratings each movie received in train, and the “prediction stage” is recommending that sorted list to every user.
Fit:
count_baseline = train.groupBy("movieId").agg(
F.count("rating").alias("n_ratings")
)
The count itself acts as a measure of how strongly the movie should be recommended.
Evaluating the Count-Based Popularity Baseline
Since there is no rating being estimated, RMSE is not the natural evaluation metric here. The model produces a ranking signal — the number of ratings a movie received — rather than a predicted rating on the 0.5–5.0 star scale.
There are therefore two ways to evaluate it: one artificial approach that allows comparison with the earlier rating-based popularity baselines, and one that evaluates the model on its actual objective.
Option A — Faux RMSE
RMSE requires the actual and predicted values to sit on the same scale (0.5–5.0 stars). A raw count such as 250 ratings clearly isn’t a rating.
To force a comparison anyway, the rating count was min-max normalized onto the 0.5–5.0 range:
count_scaled = count_baseline.withColumn(
"pred_count_scaled",
0.5
+ (F.col("n_ratings") - min_n)
/ (max_n - min_n)
* (5.0 - 0.5)
)
For example, if Movie A received 220 ratings in the training set, its raw prediction for any user in val or test would simply be 220. The normalization artificially maps that count back onto the rating scale so that RMSE can be calculated.
Result: RMSE = 2.5647
This is dramatically worse than both rating-based baselines.
This isn’t a failure of the code. It is the expected outcome of forcing a ranking signal through a rating-prediction metric. Volume and quality are genuinely different signals: a movie that everyone watches doesn’t necessarily receive a high average rating.
The resulting RMSE therefore reveals a mismatch between the signal and the metric, rather than a problem with the implementation. The normalized count is not actually predicting how much a user will like a movie — it is only indicating how widely the movie has been watched.
This faux RMSE is therefore useful only as a rough comparison with the earlier mean-based popularity baselines; it should not be used to compare this model with other ranking-based methods.
Option B — Ranking-Based Evaluation with Precision@K
Since this model was never really predicting a rating, the fair evaluation is a ranking metric: Precision@K.
The idea is:
- Take the fixed top-K most-rated movies.
- Recommend those same K movies to every user.
- Check what fraction of those K recommendations each user actually went on to rate in
val(and test).
Precision@K = number of your K recommendations that the user actually rated / K
It does not check whether the order within those K recommendations was correct. Recommending the user’s #1 favorite movie in position 10 counts exactly the same as recommending it in position 1. It only asks: was the item in the recommended list at all?
For example, suppose we recommend 10 movies to a user. In val, that user actually rated 4 movies. If 2 of our 10 recommendations happen to match 2 of those 4 movies:
Precision@10 = 2 / 10 = 0.2 = 20%
This is not “2 out of 4 relevant movies found.” That would be Recall@10 = 2/4 = 0.5, which answers a different question.
The same calculation is performed for every user and then averaged across users.
top_k_movies = [
row.movieId
for row in count_baseline
.orderBy(F.desc("n_ratings"))
.limit(K)
.collect()
]
# Recommend the same top_k_movies to every user
# and check overlap against val's actual ratings
Result: Precision@10 = 0.0239
This means that, on average, 2.39% of the model’s Top 10 recommendations were actually rated by users in the held-out data.
This number needs an important qualification. Precision@10 has a structural ceiling for a majority of users in this dataset: 56.7% of users have fewer than 10 relevant movies in the val set at all. Even a flawless oracle recommender — one that only ever recommends movies the user will actually rate — cannot reach Precision@10 = 1.0 for these users simply because there aren’t even 10 items available in the numerator place for the vast majority of users in val.
The raw Precision@10 of 0.0239 should therefore be interpreted relative to a baseline rather than against an absolute ceiling of 1.0.
Against the random baseline of 0.0023, popularity achieves roughly a 10× lift:
| Model | Precision@10 |
|---|---|
| Random baseline | 0.0023 |
| Popularity (count-based ranking) | 0.0239 |
So, properly evaluated on its own terms, pure popularity-by-volume is a legitimately strong ranking baseline.
Does it care about order at all?
Plain Precision@K, as computed here, does not. If we wanted to reward a hit at position 1 more than a hit at position 10, we would need an order-sensitive metric such as NDCG (Normalized Discounted Cumulative Gain).
However, because 56.7% of users have fewer than 10 relevant movies in val, introducing additional order-sensitivity constraints here would push the evaluation further than the data comfortably supports. Precision@10 was therefore used as the simpler and more interpretable ranking metric.
Notable Result: Popularity Beats Implicit ALS (see chapter 5)
The count-based popularity baseline achieves Precision@10 = 0.0239, compared with 0.0177 for personalized implicit ALS.
So non-personalized popularity actually outperforms personalized implicit ALS on this metric.
This isn’t a failure of ALS. It is a useful finding about the strength of popularity in a top-K ranking task: popular movies are, almost by construction, more likely to overlap with what many users will go on to interact with.
It also does not mean personalization has no value. Precision@10 says nothing about recommendation diversity, performance on niche or long-tail items, or whether personalized recommendations are more useful to individual users beyond this particular metric.
It is therefore a legitimate, and somewhat humbling, finding rather than something to explain away.
3.4. Summary Table
| Baseline | Metric | Result |
|---|---|---|
| Naive mean | RMSE | 1.0156 |
| Weighted (Bayesian, C=3) | RMSE | 0.9930 |
| Count-scaled (forced onto rating scale) | RMSE | 2.5647 |
| Count-based ranking | Precision@10 | 0.0239 |
| Random baseline, for reference | Precision@10 | 0.0023 |
These three popularity approaches capture three distinct notions of “popular”:
- Naive mean: what rating does this movie typically receive?
- Weighted mean: what rating does this movie appear to deserve after accounting for how much evidence we have?
- Count-based popularity: how widely has this movie been watched?
Together, they establish the actual floor that the personalized models in the rest of the project need to contextualize and beat.