Rutuja Jadhav
Website LinkedIn GitHub

5. ALS Implementation with PySpark

With the math established in the previous post, this section covers the actual build — training, tuning, and evaluating ALS on the MovieLens dataset in Databricks, for both the explicit and implicit formulations, and comparing the results against the popularity and collaborative filtering baselines established earlier.

The goal: determine whether a model-based, latent-factor approach (ALS) can meaningfully outperform simpler, non-personalized (popularity) and neighbor-based (item/user cosine similarity) approaches — and understand why, not just confirm that it does.

We reuse the train/val/test split from Chapter 1 (per-user, chronological, with the ceil()-guaranteed minimum representation in every split) throughout this section — no new splitting logic here.

5.1 Explicit ALS

5.1.1. Baseline Model

Spark’s ALS estimator implements exactly the alternating least-squares process derived in the math post — fixing one factor matrix, solving the other in closed form, and alternating.

python

from pyspark.ml.recommendation import ALS
from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(labelCol="rating", metricName="rmse")

als = ALS(
    userCol="userId", itemCol="movieId", ratingCol="rating",
    rank=10, maxIter=10, regParam=0.1,
    coldStartStrategy="drop",
    seed=42
)
als_model = als.fit(train)

coldStartStrategy="drop" handles the item cold-start problem identified in Chapter 1 (901 movies with zero training history) — without it, ALS returns NaN for any user/item unseen during training, which would corrupt the RMSE calculation. Dropping those rows is an evaluation-time fix, not a production one — a distinction covered later in this section.

Two hyperparameters matter here: rank (k, the number of latent factors) and regParam (\lambda, the regularization strength from the derivation). Both were swept against val — never test — same discipline applied to every model in this project:

python

ranks = [5, 10, 20, 50]
reg_params = [0.01, 0.05, 0.1, 0.2]

results = []
for rank in ranks:
    for reg in reg_params:
        model = ALS(
            userCol="userId", itemCol="movieId", ratingCol="rating",
            rank=rank, maxIter=10, regParam=reg,
            coldStartStrategy="drop", seed=42
        ).fit(train)
        rmse = evaluator.setPredictionCol("prediction").evaluate(model.transform(val))
        results.append((rank, reg, rmse))

Result pattern: regParam dominated the outcome almost entirely — RMSE improved monotonically as regParam increased from 0.01 toward 0.2, while rank barely mattered once regularization was adequate (all ranks converged to within 0.003 RMSE of each other at regParam=0.2). A follow-up sweep confirmed regParam=0.2 was a true local minimum, not a grid-edge artifact (values beyond 0.2 got worse again).

Why this pattern makes sense: the dataset is extremely sparse (98.3%), with a 20-rating-per-user floor. With so little signal per user, higher rank (more parameters to fit) mainly gave the model more room to overfit noise rather than capture real structure — regularization strength mattered far more than model capacity.

Final config: rank=5, regParam=0.2, Val RMSE = 0.9066.

5.1.3. 1.3 Error Distribution — Understanding What RMSE Actually Means

Beyond the aggregate RMSE, individual prediction errors were inspected directly:

python

val_preds.withColumn("abs_error", F.abs(F.col("rating") - F.col("prediction"))) \
    .select(
        F.avg("abs_error").alias("MAE"),
        F.expr("percentile_approx(abs_error, 0.5)").alias("median_error"),
        F.expr("percentile_approx(abs_error, 0.9)").alias("p90_error"),
        F.max("abs_error").alias("max_error")
    ).show()
MAE Median P90 Max
0.709 0.585 1.480 4.128

RMSE (0.9066) sitting noticeably above MAE (0.709) is the textbook signature of a small number of large outlier errors dragging up a squared-error metric — most predictions are quite close (median error 0.585), while a minority of ratings (likely reflecting genuine human rating inconsistency rather than model failure) incur large errors that RMSE penalizes disproportionately.

5.1.4 Final Test Evaluation

With hyperparameters locked in from val, the model — still fit on train only, no refit on train+val, for pipeline simplicity — was evaluated once on test:

Test RMSE = 0.9335 (val: 0.9066 — the expected small generalization gap, consistent with normal train→val→test degradation, not a red flag).

1.5 Explicit ALS Comparison Against Baselines

Model Val RMSE Test RMSE
Popularity (weighted, C=3) 0.9930 1.0180
User-based CF (cosine) 1.0110 1.0253
Item-based CF (cosine) 0.9500 0.9768
ALS (rank=5, reg=0.2) 0.9066 0.9335

ALS wins outright — an ~8.3% relative improvement over the popularity floor, and ~4.4% over item-based CF (the strongest baseline). The ranking held consistently between val and test, confirming the hyperparameter choices weren’t overfit to val.

Worth flagging: user-based CF underperformed even the simple popularity baseline. This reproduces a well-documented industry finding (e.g. Amazon’s rationale for choosing item-based over user-based CF) — item similarity is more stable than user similarity, since a movie’s identity is reinforced by many raters, while a user’s taste profile is a noisier, more idiosyncratic signal to compare directly.

5.2. Implicit ALS

5.2.1 Identifying Implicit Signals

MovieLens contains no organic implicit signal (views, clicks, watch-time). To demonstrate implicitPrefs mechanics honestly, ratings converted into binary labels — discarding the star value, keeping only “did this user interact with this movie at all.” This is explicitly an approximation, not a claim that rating presence behaves like real behavioral data.

Tags were considered and deliberately rejected as an implicit proxy — a user typing a tag like “boring” is still stating an opinion, just unstructured, closer to explicit feedback than to true implicit signal.

python

train_implicit = train.select("userId", "movieId").withColumn("interaction", F.lit(1.0))

5.2.2 Why RMSE Doesn’t Apply Here

Implicit ALS predicts an unbounded confidence/preference score, not a rating on a fixed scale — comparing it to actual stars via RMSE would compare incompatible things. The correct evaluation is a ranking metric: Precision@K — of the top-K movies recommended to a user, what fraction did they actually interact with in val/test.

Note on tooling: Spark’s built-in model.recommendForAllUsers(K) was blocked on Unity Catalog serverless compute. Precision@K was instead computed manually, wrapped into a reusable function so it could be applied to both the trained model and a random baseline for comparison.

# final model, refit with the best hyperparameters found during validation tuning
final_implicit_model = ALS(
    userCol="userId", itemCol="movieId", ratingCol="interaction",
    rank=20, maxIter=10, regParam=0.1, alpha=40.0,
    implicitPrefs=True, coldStartStrategy="drop", seed=42
).fit(train_implicit)

# ground truth for this final check: test's real (user, movie) interactions
test_actual = test.select("userId", "movieId").withColumn("actual_interaction", F.lit(1))

def precision_at_k_test(model, actual_df, K=10):
    # uf/itf = the trained model's user and item latent factor tables (U and M from the ALS math)
    uf = model.userFactors.withColumnRenamed("id","userId").withColumnRenamed("features","user_features")
    itf = model.itemFactors.withColumnRenamed("id","movieId").withColumnRenamed("features","item_features")

    # crossJoin pairs every user with every movie; dot(...) computes u_i . m_j for each pair —
    # the same reconstruction formula as A_hat = U M^T, evaluated one pair at a time
    cross = uf.crossJoin(itf).withColumn("score", dot(F.col("user_features"), F.col("item_features")))

    # rank each user's scored movies highest-to-lowest, keep only their top K — the model's recommendations
    w = Window.partitionBy("userId").orderBy(F.desc("score"))
    recs = cross.withColumn("rank_pos", F.row_number().over(w)) \
        .filter(F.col("rank_pos") <= K).select("userId", "movieId")

    # join recommendations against ground truth on BOTH userId and movieId — a match means
    # this specific user actually interacted with this specific recommended movie
    hits = recs.join(actual_df, on=["userId","movieId"], how="left") \
        .withColumn("is_hit", F.when(F.col("actual_interaction")==1, 1).otherwise(0))
        # no match -> actual_interaction is null -> is_hit falls back to 0

    # per-user hit rate (n_hits / K), then averaged into a single overall Precision@K
    return hits.groupBy("userId").agg(
        F.sum("is_hit").alias("n_hits"), F.count("*").alias("n_rec")
    ).withColumn("p", F.col("n_hits")/F.col("n_rec")).agg(F.avg("p")).first()[0]

test_precision = precision_at_k_test(final_implicit_model, test_actual, K=10)
print(f"[TEST] Implicit ALS Precision@10: {test_precision:.4f}")

# random_recs_df: K random movies "recommended" to each user, generated earlier as a floor to compare against —
# same join-and-average logic applied here, just against random picks instead of the model's ranked output
random_hits_test = random_recs_df.join(test_actual, on=["userId","movieId"], how="left") \
    .withColumn("is_hit", F.when(F.col("actual_interaction")==1, 1).otherwise(0))
random_test_precision = random_hits_test.groupBy("userId").agg(
    F.sum("is_hit").alias("n_hits"), F.count("*").alias("n_rec")
).withColumn("p", F.col("n_hits")/F.col("n_rec")).agg(F.avg("p")).first()[0]
print(f"[TEST] Random baseline Precision@10: {random_test_precision:.4f}")

Wrapping the logic into precision_at_k_test(...) is what let the exact same evaluation — build recommendations, join against ground truth, compute hit rate — be reused for two different inputs: the trained implicit ALS model, and a random baseline (random_recs_df, generated earlier by sampling K arbitrary movies per user). Comparing against a random baseline is essential here, since Precision@K has no fixed “good” value on its own — a model’s score is only meaningful relative to how much better it does than random guessing.

5.2.3 Grid Search — Including alpha

Implicit ALS introduces a third hyperparameter beyond rank/regParam: alpha, which controls the confidence gap between “interacted” and “did not interact” (c_{ui} = 1 + \alpha \cdot r_{ui}).

Result pattern: unlike explicit ALS, alpha dominated the outcome, not regParam. Low alpha values (1.0) underperformed even the simplest rank=5 explicit-style baseline; high alpha (40.0) combined with higher rank (20) produced the best results — the model needed strong confidence weighting to meaningfully separate observed interactions from the vast unobserved majority in this sparse, binary setting. A follow-up sweep (40, 60, 100) confirmed alpha=40 was a genuine peak, not a grid-edge effect.

Final config: rank=20, regParam=0.1, alpha=40.0.

5.2.4 Establishing a Random Baseline (Essential — Precision@K Alone Is Meaningless Without It)

python

random_recs_df = spark.createDataFrame([(u, mid, pos) for u in users for pos, mid in enumerate(random.sample(all_movie_ids, K), 1)], [...])
Precision@10  
Random baseline 0.0021 (val) / 0.0023 (test)
Implicit ALS 0.0233 (val) / 0.0177 (test)

An ~8-11x lift over random — a meaningful result, though it’s worth being precise about what “meaningful” means here: 56.7% of users have fewer than 10 relevant movies in val at all, meaning even a flawless oracle recommender is structurally capped below Precision\@10 = 1.0 for a majority of users. The numbers should be read relative to the random baseline and to each other, not against an absolute ceiling.

5.2.5 A Genuinely Notable Result: Popularity Beats Implicit ALS on Ranking

The count-based popularity baseline (Chapter on Popularity Baselines) — recommending the same fixed top-10 most-rated movies to every user, no personalization at all — scored:

Model Precision@10
Random baseline 0.0023
Popularity (count-based ranking) 0.0239
Implicit ALS 0.0177

Non-personalized popularity outperformed personalized implicit ALS on this metric. This isn’t a failure of ALS, and it’s a well-documented phenomenon in recsys research — broad appeal is a genuinely strong predictor at the top-K ranking task specifically, since popular items are, almost by construction, likely to overlap with what many individual users will go on to interact with. It doesn’t mean personalization has no value (it says nothing about diversity of recommendations, or performance on niche/long-tail items) — but it’s a legitimate, humbling finding, not a strawman to explain away.

5.3. Cold Start in Production — Beyond coldStartStrategy="drop"

coldStartStrategy="drop" solves an evaluation problem (NaN predictions corrupting RMSE) — it does not solve the product problem of what to actually recommend to a brand-new user or item in practice. Three real mechanisms matter here, discussed but not built into this project’s pipeline:

  1. Popularity fallback — the practical answer for a user/item with zero history at all, tying directly back to Part 1’s popularity baseline.
  2. Fold-in — a single closed-form solve (the exact per-row formula derived in the math post) using the existing, already-trained M, giving a new user a real vector in milliseconds without touching the rest of the model.
  3. Scheduled full retrain — periodically (nightly/weekly), all accumulated new users/items get folded into a proper joint retrain, refining U and M together rather than relying on fold-in’s approximation indefinitely.

5.4. Summary Table - Popularity Baseline, Item based CF, User based CF, Exp ALS and Imp ALS

Model Metric Val Test
Popularity (weighted, C=3) RMSE 0.9930 1.0180
User-based CF RMSE 1.0110 1.0253
Item-based CF RMSE 0.9500 0.9768
Explicit ALS RMSE 0.9066 0.9335
Random Precision@10 0.0021 0.0023
Implicit ALS Precision@10 0.0233 0.0177
Popularity (count ranking) Precision@10 — 0.0239

Explicit and implicit ALS answer genuinely different questions and can’t be compared on a shared metric — one predicts a rating value, the other predicts a ranking. Both, however, are best understood relative to their respective baselines rather than in isolation: explicit ALS’s win over popularity/CF is a real, personalization-driven improvement; implicit ALS’s loss to popularity-by-volume is an equally real, and equally informative, result about the limits of personalization on sparse, binarized interaction data.

This site uses Just the Docs, a documentation theme for Jekyll.