6. ALS Illustration in PySpark
Since I am a visual learner, this chapter is an attempt to help me build a mind map of the potential BTS happening in Spark as it builds our model and uses it to make predictions. Doesn’t add any new information to the modelling process or findings- feel free to skip if you’re already familiar with Spark!
6.1. How Spark handles Fit, Train, and the Cold-Start Drop, on Hypothetical Rows from our dataset
The math derivation earlier worked with \(A\) as a matrix — rows as users, columns as movies, cells as ratings. In Spark, the source data never actually looks like that. It’s a long-format DataFrame: one row per observed (user, movie) interaction, not a grid with empty cells for missing ones. Before walking through the code, it’s worth seeing that shape explicitly, since it’s the first thing that trips people up moving from the math to the implementation.
What \(A\) looks like in the math:
| Titanic | Avengers | Notebook | Matrix | |
|---|---|---|---|---|
| Alice | 5 | – | 4 | – |
| Bob | – | 5 | – | 4 |
What train actually looks like in Spark:
| userId | movieId | rating |
|---|---|---|
| Alice | Titanic | 5 |
| Alice | Notebook | 4 |
| Bob | Avengers | 5 |
| Bob | Matrix | 4 |
There is no row for (Alice, Avengers) — it’s simply absent, not present-with-a-null. This is the “missing values excluded from the loss” idea from the math section, expressed structurally: a missing rating isn’t a cell to skip inside a matrix, it’s a row that was never written into the table in the first place.
6.2. Explicit ALS
Step 1 — Configure and Fit
from pyspark.ml.recommendation import ALS
als = ALS(
userCol="userId",
itemCol="movieId",
ratingCol="rating",
rank=10,
maxIter=10,
regParam=0.1,
coldStartStrategy="drop",
seed=42
)
als_model = als.fit(train)
ALS(...) only configures the run — no training happens yet. userCol/itemCol/ratingCol map the long-format columns to their roles. rank is $k$, the latent factor count. maxIter is how many alternating rounds (solve $U$, then solve $M$, repeat) to run. regParam is $\lambda$ from the derivation. als.fit(train) is where the actual alternating least-squares optimization runs, producing a trained model.
Step 2 — What Training Actually Produced
Fitting populates two internal tables — literally $U$ and $M$ from the math, one row per user / per movie, each holding a length-$k$ vector:
als_model.userFactors.show()
als_model.itemFactors.show()
| id (user) | features |
|---|---|
| Alice | [1.2, -0.5] |
| Bob | [0.3, 0.9] |
| id (movie) | features |
|---|---|
| Titanic | [0.8, 0.1] |
| Avengers | [-0.2, 1.0] |
(illustrative 2-dimensional vectors, for readability — real runs use whatever rank was set to)
Step 3 — Predict via .transform()
val_preds = als_model.transform(val)
For every row in val, Spark looks up that row’s user vector from userFactors and movie vector from itemFactors, and computes their dot product — exactly $\hat A = UM^T$, evaluated one cell at a time rather than as a full matrix. That value becomes the new prediction column.
Example — row (userId=Alice, movieId=Titanic):
\[u_{Alice} \cdot m_{Titanic} = (1.2 \times 0.8) + (-0.5 \times 0.1) = 0.96 - 0.05 = 0.91\]Resulting table:
| userId | movieId | rating | prediction |
|---|---|---|---|
| Alice | Titanic | 5.0 | 0.91 |
(illustrative value only — real trained vectors land close to the true rating scale)
Step 4 — The Cold-Start Drop, Shown on an Actual Row
Say val contains a row for a movie that never appeared in train at all:
| userId | movieId | rating |
|---|---|---|
| Alice | UnseenMovie | 4.0 |
itemFactors simply has no row for UnseenMovie — there’s nothing to look up, so no dot product can be formed. With coldStartStrategy="drop", this row is silently removed from val_preds rather than producing a NaN:
n_dropped = val.count() - val_preds.count()
This directly measures how many rows hit exactly this scenario — the same mechanism behind the earlier item cold-start count (901 movies with zero training history in this project’s actual data).
Step 5 — Evaluate
rmse_als_val = evaluator.setPredictionCol("prediction").evaluate(val_preds)
Same RegressionEvaluator used throughout the project — compares the rating column (truth) against prediction (ALS’s dot-product output) across every surviving row, after the cold-start drop.
6.3. Implicit ALS — What Changes, Briefly
Implicit ALS follows the identical fit → transform → evaluate shape — only three things differ:
1. The input signal is presence, not a rating value:
train_implicit = train.select("userId", "movieId").withColumn("interaction", F.lit(1.0))
Every row just says “this interaction happened” — no rating column at all.
2. The estimator flips a flag and gains a new parameter:
als_implicit = ALS(
userCol="userId", itemCol="movieId", ratingCol="interaction",
rank=20, maxIter=10, regParam=0.1, alpha=40.0,
implicitPrefs=True,
coldStartStrategy="drop",
seed=42
)
implicitPrefs=True switches the internal loss to the confidence-weighted formulation. alpha controls how strongly an observed interaction is trusted relative to the many unobserved ones — a hyperparameter with no equivalent in explicit ALS.
3. userFactors/itemFactors still exist and coldStartStrategy="drop" still works exactly the same way — but .transform()’s output is an unbounded confidence/preference score, not a 0.5–5.0 rating, so RMSE no longer applies. Evaluation shifts to a ranking metric (Precision@K), covered in its own section.
Everything else — configure, fit, look up two vectors per row, dot-product them, drop rows with no matching vector — is identical to the explicit case.