Forem: Alankrit Verma

Synthetic Population Testing for Recommendation Systems

Alankrit Verma — Sat, 04 Apr 2026 02:04:50 +0000

Offline evaluation is necessary for recommender systems. It is also not a full test of recommender quality. The missing layer is not only better aggregate metrics, but better ways to test how a model behaves for different kinds of users before launch.

TL;DR

In the last post, I argued that offline evaluation is useful but incomplete for recommendation systems.
After that, I built a small public artifact to make the gap concrete.
In the canonical MovieLens comparison, the popularity baseline wins Recall@10 and NDCG@10, but the candidate model does much better for Explorer and Niche-interest users and creates a very different behavioral profile.
I do not think this means “offline evaluation is wrong.”
I think it means a better pre-launch evaluation stack should include some form of synthetic population testing: explicit behavioral lenses, trajectory-aware diagnostics, and tests that make hidden tradeoffs visible before launch.

What Comes After “Offline Evaluation Is Not Enough”?

In the first post, I made a narrow claim:

offline evaluation is useful, but incomplete, because recommendation systems are interactive systems.

That argument matters, but by itself it leaves an obvious next question:

if aggregate offline metrics are not enough, what should be added to the evaluation stack?

I do not think the answer starts with a giant platform or a perfect user simulator.

I think the more practical place to start is smaller:

take the same baseline-vs-candidate comparison and test it through multiple behavioral lenses, not just one aggregate average.

That is what I built next.

The Artifact

The current artifact is a small public recommender behavior QA harness.

It compares:

one baseline recommender
one candidate recommender
one fixed evaluation setup

And it produces:

standard offline ranking metrics
bucket-level utility
behavioral diagnostics such as novelty, repetition, and catalog concentration
short trajectory traces that make model behavior easier to inspect

The canonical public run is intentionally narrow:

MovieLens 100K
Model A: popularity baseline
Model B: genre-profile recommender with a popularity prior
4 fixed buckets
one frozen report bundle

The point is not to claim that these two models define recommender evaluation. The point is to create one clean, reproducible proof that aggregate offline metrics can hide useful pre-launch information.

The Canonical Result

The canonical MovieLens run shows the core value in one comparison.

On aggregate offline ranking metrics, the popularity baseline wins:

Model	Recall@10	NDCG@10
Model A	0.088	0.057
Model B	0.058	0.036

If we stopped there, the conclusion would be straightforward: Model A looks better.

But the bucketed view tells a different story:

Bucket	Model A	Model B	Delta (B-A)
Conservative mainstream	0.519	0.532	0.012
Explorer / novelty-seeking	0.339	0.523	0.184
Niche-interest	0.443	0.722	0.279
Low-patience	0.321	0.364	0.043

That is the point.

Aggregate offline metrics say one thing. The segment-aware view says something more useful:

the baseline is better at recovering held-out positives
the candidate is much stronger for important user lenses
the behavioral profile of the system changes in ways the aggregate view compresses away

The behavioral diagnostics make that even clearer:

Model	Novelty	Repetition	Catalog concentration
Model A	0.395	0.279	1.000
Model B	0.678	0.664	0.717

This is worth pausing on, because not every behavioral metric moves in the same direction.

Model B is:

more novel
less catalog-concentrated
but also more repetitive in this diagnostic

That is not a bug in the framework. It is part of the point. Different recommendation strategies produce different behavioral signatures, and pre-launch evaluation should help make those signatures visible instead of collapsing everything into one average.

What “Synthetic Population Testing” Means Here

It is important to be precise about this phrase.

What I have today is not a rich simulation of realistic synthetic humans. There are no agent conversations, no generated personas with biographies, and no claim that the current system faithfully reproduces real user psychology.

What the artifact does have is a simpler and more controlled version of the same idea:

fixed behavioral lenses
explicit utility assumptions
short trajectory simulation under those assumptions

The four v1 buckets are:

Conservative mainstream
Explorer / novelty-seeking
Niche-interest
Low-patience

Each bucket values recommendation behavior differently. The evaluation then asks how the same two models behave when the user lens changes.

So when I say synthetic population testing here, I mean:

an early, lightweight form of synthetic population testing built from fixed behavioral lenses, not full synthetic-user simulation.

I think that still matters. It turns vague product intuition like “some users may prefer this model more than others” into an explicit, reproducible pre-launch test.

Why This Is Better Than Another Aggregate Metric

A natural response to the first post is to ask whether we simply need better aggregate metrics.

I do not think that is enough.

The problem is not only that a metric is imperfect. The deeper problem is that recommender quality is heterogeneous.

Different users are helped by different behaviors:

some want safer, familiar, high-exposure items
some benefit from more novelty and more variety
some have narrower tastes that require stronger matching to long-tail pockets
some degrade faster when sequences become stale

A single global score cannot represent all of that well.

That is why I think the next useful layer should look more like testing against a small synthetic population than inventing one more scalar.

Instead of asking only:

which model wins on average?

we should also ask:

which model wins for which behavioral lens?

where do the models differ most?

what kind of trajectory does each model produce?

This does not mean the current bucket lenses are perfect. It means they are often more informative than one collapsed aggregate average.

One Short Trajectory Example

The trajectory view matters because recommendation quality is not only one-step.

Here is one Explorer / novelty-seeking comparison from the canonical run:

Model A

Raiders of the Lost Ark -> Fargo -> Toy Story -> Return of the Jedi

Model B

Prophecy, The -> Cat People -> Wes Craven's New Nightmare -> Relic, The

The first sequence stays much closer to familiar, high-exposure titles. The second is much more tailored to a narrower taste profile and much more novel.

This is exactly the kind of difference that disappears when evaluation is reduced to one aggregate ranking score.

Why This Matters Before Launch

Pre-launch evaluation is about decisions, not just measurements.

If a team is deciding whether to ship a new recommender, the real question is usually not:

did one mean score go up?

It is closer to this:

who gets a better experience?
who gets a worse one?
does the candidate become more repetitive?
does it collapse toward head items?
does it create a healthier exploration profile?

Those are product and system questions, not only ranking-metric questions.

That is why I like this framing. It stays honest about what the artifact is doing. It is not trying to predict the full online future. It is trying to make hidden tradeoffs visible earlier, with a tool that is still small enough to run, inspect, and reason about.

What This Is, And What It Is Not

I think the strongest version of this argument is the honest one.

This artifact is:

a small public proof
a recommender-specific evaluation layer
a way to make segment-level and trajectory-level tradeoffs visible
a first wedge into broader testing for interactive systems

This artifact is not:

a proof that the candidate model is globally better
a replacement for offline evaluation
a replacement for online experiments
a full synthetic-human simulation framework

That distinction matters. If this work is useful, it will be useful because it is clear about what it adds, not because it overclaims.

A Better Evaluation Stack

The long-term picture I have in mind looks something like this:

Standard offline evaluation remains the first layer.
Segment-aware and trajectory-aware diagnostics become the second layer.
Richer synthetic population testing may become the next layer after that.
Online experiments still remain necessary for final validation.

That is a much more realistic stack than pretending a single aggregate metric can do the whole job.

In that stack, the current artifact sits at layer two. It adds explicit behavioral lenses and short trajectory diagnostics to the familiar offline comparison workflow.

That is why I think it matters, even in its current limited form.

It is not the final answer.

It is the first concrete artifact of the missing layer.

Conclusion

The first post argued that offline evaluation is not enough for recommendation systems.

This artifact is my first practical answer to what should come next.

Not a giant platform. Not a perfect simulation. Not a replacement for offline evaluation.

Just a small, reproducible evaluation harness that compares a baseline and a candidate through multiple behavioral lenses and shows tradeoffs that aggregate metrics compress away.

If offline evaluation is the first screen, then synthetic population testing, in some form, may be one of the next useful layers.

This v1 is a lightweight version of that idea.

If you want to see the public artifact, the canonical MovieLens demo lives in the limitation repo as a report, JSON result bundle, and supporting visuals.

Why Offline Evaluation Is Not Enough for Recommendation Systems?

Alankrit Verma — Sun, 29 Mar 2026 16:40:52 +0000

Why Offline Evaluation Is Not Enough for Recommendation Systems

Offline evaluation is essential for recommender systems. It is also easy to mistake for a fuller measure of quality than it really is.

TL;DR

Offline evaluation is useful, fast, and necessary for recommender systems.
But it is built on logged behavior generated under older exposure policies.
That makes it weak at judging policy shifts, novel items, cold start behavior, and longer interaction trajectories.
In a small MovieLens demo, the popularity baseline wins on aggregate offline ranking metrics, while a more personalized model does better for explorer, niche-interest, and low-patience user buckets.
The practical conclusion is not to replace offline evaluation, but to stop treating it as a full test of recommender quality.

Recommendation systems are interactive systems, but offline evaluation often treats them like static predictors.

1. The Testing Gap

We know how to test deterministic software. We are much less certain about how to test systems that influence the behavior they later observe.

Recommendation systems sit squarely in that second category. They do not just estimate what a user might click, watch, or purchase. They decide what the user gets a chance to see, and that choice helps shape the data that will later be treated as evidence.

Offline evaluation is one of the standard tools in recommender systems for good reason. It is practical, fast, and often highly informative. A team can compare candidate models on historical interaction data long before it is ready to send live traffic to a new ranking policy.

That usefulness, however, can make offline evaluation easy to over-interpret. A strong offline result often sounds like a strong statement about real recommendation quality. Sometimes it is. But the conclusion is narrower than it first appears.

Historical interaction logs are not simply records of user preference. They are records of user preference under a particular pattern of exposure. They reflect what earlier systems chose to rank, recommend, and repeat. In that sense, the data is policy-dependent from the beginning.

This matters because recommendation quality is not only about matching a fixed label. A recommender is an interactive system. Its outputs affect future inputs. Change the policy, and over time you may change what users discover, what they come to trust, what they ignore, and what they eventually consume.

Consider a movie recommender. One model may reliably surface popular, familiar titles. Another may be more personal and more willing to introduce niche films that fit a specific user's taste. If the historical logs were generated under a system that already emphasized mainstream titles, those logs may be much richer in evidence for the first model's choices than for the second model's.

That does not make offline evaluation wrong. It does mean the object being measured is more limited than many teams would like. Offline evaluation is useful, but insufficient.

The point of this article is narrow. It is not that offline evaluation should be discarded, and it is not a general argument about all machine learning systems. The claim is simpler: recommendation systems are interactive systems, and that fact places real limits on what historical replay can tell us.

2. What Offline Evaluation Is

Offline evaluation, in the recommender setting, means evaluating a model on historical logged interactions rather than on live user traffic. The usual pattern is straightforward: train on past user-item behavior, hold out a later slice of interactions, and ask whether the model ranks the held-out items highly for the relevant users.

In a movie recommendation system, the data might include watches, clicks, ratings, or add-to-list events. A model is trained on part of that history and then evaluated on interactions that were not shown during training. If a user later watched a particular film, one basic offline question is whether that film would have appeared near the top of the model's ranked list.

This setup supports the ranking-style metrics commonly used in recommender systems. Teams may report measures such as Recall@K, hit rate, or NDCG to summarize how well a model recovers held-out interactions. The exact metric matters, but the general logic is the same: use historical behavior as a proxy for whether the recommendations were good.

That approach is attractive because it gives a concrete and reproducible testing loop. Candidate models can be compared against the same held-out data. Regressions can be caught before launch. Incremental improvements can be measured without the cost and risk of online experimentation.

It is also important to be precise about what this evaluation is actually saying. Offline evaluation does not directly measure how users would respond to a new policy in a live environment. It measures how well a model aligns with historical interactions recorded under earlier exposure conditions.

That distinction is easy to blur because the workflow looks so familiar. We have training data, a test set, and a metric. But in recommendation systems, the labels are not independent of the system that helped generate them. The held-out watch or click is not just a fact about the user. It is also a fact about what the user was shown.

For now, that is enough of a working definition. Offline evaluation is historical replay over logged interactions, typically framed as a ranking problem, and used as a proxy for recommendation quality under observed conditions. It is a very useful proxy. The rest of the article asks where its boundaries are.

3. Where Offline Evaluation Breaks

The limitations of offline evaluation do not come from a single bad metric or a single avoidable mistake. They come from a more basic fact about recommender data: the data is generated under a policy. What users do in the logs depends in part on what earlier systems chose to show them.

That sounds obvious when stated directly. But it has deeper consequences than it first appears. If the evidence used for evaluation is itself shaped by older recommendation decisions, then offline evaluation is not observing some neutral ground truth about relevance. It is observing relevance through the filter of past exposure.

In a static prediction task, that distinction is often less severe. In recommendation, it sits near the center of the problem. A new recommender is rarely judged against untouched labels. It is judged against behavior recorded under an older recommender, with its own ranking habits, popularity biases, and coverage patterns.

We can state the issue in simple notation. Let pi_0 be the logging policy that generated the historical data, and let pi_1 be the new policy we want to evaluate. Offline replay uses observations gathered under pi_0 to estimate the quality of pi_1. If pi_1 behaves much like pi_0, that may be informative. If it changes exposure materially, the estimate becomes much less complete.

This is the core mismatch. The quantity we want is user response under the candidate policy. The quantity we usually observe is user response under the previous policy. The two overlap, but they are not the same object.

3.1 Exposure Bias

The first break is exposure bias. Users can only react to items they were actually shown.

That means an interaction log is not just a record of what users preferred. It is also a record of what the system made available. When an item receives no click, no watch, or no rating, that absence does not cleanly mean the item was irrelevant. In many cases it means the item was never placed in front of the user at all.

This matters immediately for offline evaluation. Suppose a movie platform has historically given heavy exposure to well-known studio releases and much lighter exposure to niche films. The resulting data will contain dense evidence for how users responded to the mainstream catalog and sparse evidence for how they would have responded to more specialized titles.

The bias here is structural rather than anecdotal. If observed feedback only exists for exposed items, then the support of the evaluation data is concentrated where the logging policy chose to spend attention. In compact form, observed reward is only available where pi_0(i | u, c) is nontrivial for user u, item i, and context c.

That is why historical replay is partial. It is not sampling uniformly from all relevant user-item pairs. It is sampling from the subset that earlier policies made visible. In a movie recommender, this can make “popular” look easier to measure than “personally relevant,” even when the latter is closer to the product goal.

3.2 Old-Policy Lock-In

Exposure bias becomes more consequential when a new policy differs from the old one in systematic ways. This is where old-policy lock-in appears.

In most offline evaluations, the labels used to assess a candidate model were generated under a different ranking policy. A held-out watch event looks like a simple target, but it is downstream of earlier recommendation decisions. The new model is therefore being judged with evidence produced by the system it may be trying to replace.

This creates an asymmetry. Models that resemble the old policy often enjoy richer and cleaner evidence in the historical logs. Models that shift probability mass toward less exposed regions of the catalog are evaluated in the parts of the space where the logs are thinnest.

Return to the movie example. If the old system strongly favored familiar blockbusters, then the held-out data will naturally contain many interactions with those titles. A candidate model that continues to rank them highly will line up well with the log. Another model that is more willing to surface quieter but well-matched films may look weaker offline, not necessarily because users dislike those recommendations, but because the old system rarely created opportunities to observe that preference.

This is one reason a better recommender can look worse offline. The issue is not only model accuracy. It is evaluation support. When performance is estimated on outcomes generated under pi_0, the comparison can systematically favor policies that stay close to pi_0.

That does not make all offline comparisons invalid. If two models differ only slightly, offline evaluation can still be highly useful. But when a candidate policy changes exposure patterns in meaningful ways, offline results should be read with more caution than the metric alone suggests.

3.3 Novel Items and Cold Start

The same logic becomes even sharper for new or rarely exposed content.

Offline evaluation is strongest where historical evidence is plentiful. It is weakest where exposure has been limited, recent, or absent. Unfortunately, those are often exactly the regions where recommendation systems are asked to do something valuable: introduce new items, expand coverage, and connect users to parts of the catalog they would not have reached on their own.

In a movie platform, consider a newly added independent film with very little interaction history. A model may have good reasons to recommend it to a narrow set of users based on metadata, embeddings, or nearby behavioral signals. But if the film barely appeared under the previous policy, then historical logs offer limited evidence for how good that recommendation would actually be.

The problem is not only that the item is new. The deeper issue is that offline replay inherits the conservatism of past exposure. It is much easier to validate recommendations for already visible inventory than for inventory the old policy neglected.

This creates a subtle but important pressure. Systems that stay near the historically exposed core of the catalog are easier to justify with offline evidence. Systems that broaden exposure toward the tail are often evaluated precisely where the data is least informative. Over time, that can make conservative recommendation strategies look more reliable than they really are, and exploratory strategies look less supported than they might deserve.

The claim is not that offline evaluation fails in every cold-start setting. It is that historical replay is structurally weak exactly where a recommender tries to broaden exposure. For recommenders, novelty is often where the evidence is thinnest.

3.4 Trajectory Blindness

Even if the exposure problem disappeared, there would still be another limitation. Recommendation quality is not purely one-step.

Most offline metrics compress evaluation into local ranking success. Did the model place the held-out item near the top? Did it recover the next watch? Did it improve a ranking score on observed interactions? Those are reasonable questions, but they are mostly questions about immediate alignment with historical events.

Users, however, experience recommendation systems as sequences. They return across sessions. They compare one recommendation to the previous one. They notice repetition. They develop trust or impatience. They learn whether the system helps them explore or merely loops them through slight variations of what it already knows how to sell.

This is where trajectory blindness enters. A recommender can look strong on one-step relevance and still create a poor multi-step experience.

Imagine a movie recommender that repeatedly serves highly similar popular thrillers because those titles have strong historical watch signals. In a one-step offline evaluation, this may look sensible. The recommendations are close to what users have previously consumed, and the metrics may reward that closeness. But over several sessions the user may experience the system as narrow, repetitive, and increasingly unhelpful.

Another model might trade a small amount of one-step certainty for a better sequence. It may alternate between reliable choices and occasional high-fit long-tail discoveries. That kind of quality often lives in the trajectory rather than in any single ranking event.

In notation, many offline metrics focus on something close to the quality of r_t at a single step. But recommender quality often depends on properties of the sequence (a_1, r_1), ..., (a_T, r_T): how concentrated the recommendations are, whether novelty appears at the right rate, whether boredom accumulates, and whether the system adapts well after earlier choices.

This is not an argument against ranking metrics. It is an argument about what they leave out. They summarize one-step fit to logged behavior. They do not, by themselves, tell us whether the interaction over time becomes richer, narrower, more repetitive, or more satisfying.

3.5 What This Means

Taken together, these limitations point to a single conclusion. Offline evaluation often treats recommendation as if it were a static prediction problem with fixed labels. In practice, recommendation is an interactive system problem.

The system chooses what to expose. Exposure shapes what users can respond to. Those responses become the data for future training and evaluation. Change the policy, and you may change the distribution of behavior itself.

Once that is clear, the goal of evaluation also becomes clearer. The question is not only whether a model can replay the past. It is whether it can support good interaction under a changed policy. Historical replay helps answer that question, but only in part.

4. Why It Still Matters

None of these limitations make offline evaluation disposable. They define its scope. That distinction matters.

Recommendation teams rely on offline evaluation because it solves real engineering problems well. It is fast, reproducible, and comparatively cheap. It allows model changes to be screened before they reach users. It supports regression testing, debugging, ablation work, and benchmarking across candidate approaches. In most practical settings, there is no credible evaluation stack that excludes it.

That remains true even after the critique above. A recommender team still needs a way to reject clearly weak models, validate implementation changes, and compare alternatives under a common protocol. Offline evaluation is often the first place where obvious failures become visible. If a ranking model cannot perform competitively in historical replay, it is usually hard to justify sending it to live traffic.

This is especially important because online tests are expensive in more than one sense. They consume time, user attention, and organizational focus. They are also constrained by risk. A platform may be willing to test a modest ranking change online, but not a model that already appears unstable or uncompetitive offline. Historical evaluation remains the practical filter through which many candidate models must pass.

The right conclusion, then, is not that offline evaluation should be replaced. It is that offline evaluation should be placed correctly. It is a strong tool for iteration and a weak tool for making broad claims about full recommender quality under changed exposure.

In other words, the critique is intentional. Offline evaluation is widely used because it earns its place. The mistake is not using it. The mistake is mistaking it for a complete test.

One compact way to summarize that balance is to separate what offline replay usually measures well from what it tends to leave undermeasured.

Evaluation aspect	What offline replay usually captures	What it tends to miss or undermeasure	Movie recommender example
Immediate relevance under existing exposure	Whether held-out watched items appear near the top of the ranked list	Whether that ranking would still look good under a materially different exposure policy	A familiar blockbuster appears in the top `K` because it was already heavily exposed
Performance under policy shift	Small improvements that stay near the old policy	Quality of recommendations in regions where the candidate policy differs most	A model that surfaces more niche dramas has little historical support where it differs from the old system
Novel or underexposed items	Some signal for items with enough prior exposure	Items that were new, rare, or historically under-shown	A newly added indie film receives little offline credit even if it fits the user well
Cold start behavior	Very coarse performance on sparse users or items	Early recommendation quality when interaction history is thin	A new documentary enters the catalog with too little evidence for replay to judge it fairly
Repetition over sessions	Little, unless explicitly measured	Accumulated sameness across repeated visits	The recommender keeps offering slight variations of the same thriller over multiple sessions
Novelty and exploration	Limited signal through held-out interactions	Whether the system introduces useful discovery at the right rate	A long-tail science-fiction recommendation may be good, but the old logs barely contain exposure to it
Segment-level differences	Aggregate averages over the evaluation set	Which user groups are helped or hurt by the new policy	Mainstream users may do well under Model A while exploration-seeking users do better under Model B
Trajectory-level user experience	Almost nothing in standard one-step metrics	Trust, boredom, fatigue, and satisfaction over sequences	A user keeps getting acceptable next picks but gradually disengages from repetition

5. Running Example: Model A vs. Model B

The structural issues above become easier to see with a simple running example. Consider a movie recommendation system with two candidate rankers.

Model A is conservative. It leans toward popular, broadly watched titles and tends to recommend within the historically dominant regions of the catalog. It is usually safe, usually familiar, and often repetitive.

Model B is more personalized. It still recommends mainstream films when they fit, but it is more willing to surface niche titles, less obvious matches, and items from thinner parts of the catalog when the user profile suggests they are a good fit.

Suppose the historical logs were generated under an earlier recommendation policy that behaved more like Model A. Popular titles received heavy exposure. Niche titles were shown less often. Over time, that policy produced abundant feedback on the mainstream catalog and much weaker evidence on long-tail items.

Now evaluate both models offline on held-out interactions from those logs.

Model A will often look strong for a simple reason: it aligns well with the exposure pattern that helped generate the data. It ranks many of the same kinds of items the old system already showed, so the held-out interactions contain ample opportunities to reward it.

Model B may be better calibrated to particular users, especially users with narrower tastes or stronger appetite for discovery. But if many of its most valuable recommendations lie in regions of the catalog that were rarely exposed before, the offline log may not give it much credit. The evidence needed to validate those choices was never fully collected.

This does not mean Model B is necessarily better overall. Some users may indeed prefer the safer behavior of Model A. That is part of the point. Recommendation quality is heterogeneous across users and across sessions, and a single aggregate score can hide that heterogeneity.

The difference becomes clearer over repeated interaction. Model A may continue to produce acceptable next-item recommendations while gradually narrowing the user's experience into a small, overexposed slice of the catalog. Model B may produce a slightly noisier immediate ranking while creating a better long-run sequence for users who value novelty or have specialized tastes.

This is the kind of divergence a later demo can make visible. Two models may look similar on an aggregate offline metric and still differ meaningfully in repetition, novelty, and which user groups they serve well.

A Small MovieLens Demo

To make that less abstract, I built a small comparison on MovieLens 100K. The setup is intentionally simple. Model A is a popularity baseline. Model B is a lightweight personalized recommender built from user genre profiles with a modest popularity prior. The point is not to produce the strongest possible recommender. The point is to see what different layers of evaluation say about the same pair of systems.

Aggregate view: on standard offline ranking metrics, Model A looks better.

Model	Recall@10	NDCG@10	Novelty	Repetition	Catalog concentration
Model A	0.088	0.057	0.395	0.675	1.000
Model B	0.058	0.036	0.678	0.693	0.717

If we stopped there, the conclusion would be straightforward: the popularity baseline wins offline.

But that is exactly the point of the article. Once the evaluation is widened beyond a single aggregate view, the picture changes.

Bucketed view: the same two models look quite different once we ask who is being served well.

Bucket	Model A utility	Model B utility	Delta (B-A)
Conservative mainstream	0.519	0.532	0.012
Explorer / novelty-seeking	0.339	0.523	0.184
Niche-interest	0.443	0.722	0.279
Low-patience	0.321	0.364	0.043

The bucketed results are more revealing than the aggregate ones. Explorer users and niche-interest users benefit much more from Model B. Low-patience users also do slightly better under Model B in the short-session simulation, even though the aggregate offline ranking metrics still prefer Model A.

The behavior diagnostics tell a related story. Model B is substantially more novel and much less concentrated in the most popular slice of the catalog. For explorer users, bucket-level novelty rises from 0.405 under Model A to 0.808 under Model B. For niche-interest users, mean bucket utility rises by 0.279. That is not a rounding error. It is a segment-level change that the aggregate offline metrics compress away.

What the demo says in one glance

Aggregate offline metrics favor Model A.
Explorer, niche-interest, and low-patience buckets do better under Model B.
Model B is much more novel and less concentrated in the most popular slice of the catalog.

Two short traces make the difference more tangible.

Explorer / novelty-seeking user

Model A: Raiders of the Lost Ark -> Fargo -> Toy Story -> Return of the Jedi
Model B: Prophecy, The -> Cat People -> Wes Craven's New Nightmare -> Relic, The

The first sequence stays close to familiar, high-exposure titles. The second is much more novel and much more tailored to a narrower taste profile.

Low-patience user

Model A: Star Wars -> Fargo -> Return of the Jedi -> Toy Story
Model B: Monty Python and the Holy Grail -> Full Monty -> American President -> Truth About Cats & Dogs

Here the difference is not just novelty. The second sequence moves through a less concentrated slice of the catalog rather than repeatedly returning to the same mainstream core.

This small demo does not prove that Model B is globally better. It does something more modest and more useful. It shows that the answer depends on what we mean by "better," which users we care about, and whether we look only at historical ranking recovery or also at the behavior a recommender produces over short trajectories.

6. A Better Direction, Briefly

If offline evaluation is necessary but incomplete, the natural response is not to discard it. The better response is to build a broader evaluation stack around it.

That broader stack should start from the failure modes already discussed. If logged exposure is policy-dependent, then evaluation should be more explicit about where the evidence is strong and where it is weak. If quality emerges over time, then some part of evaluation should examine sequences rather than only one-step ranking recovery.

In practice, this suggests a modest shift in emphasis. Instead of asking only for a single aggregate offline score, teams can also ask how models behave across user segments, how concentrated their recommendations become, how much novelty they introduce, and whether their behavior looks meaningfully different over short interaction traces.

For the movie example, that might mean comparing Model A and Model B not only on Recall@K or NDCG, but also on repetition, tail exposure, and bucket-level outcomes for users with different appetites for familiarity or exploration. None of these measurements solves the full problem. They simply make the evaluation better matched to the system being evaluated.

The same logic also motivates carefully designed simulated interaction or short trajectory-based testing. The point is not that such methods are already complete or universally trustworthy. The point is narrower: if recommenders shape future behavior, then some part of the evaluation stack should attempt to probe that interaction rather than treating historical replay as the whole story.

This is best understood as complement, not replacement. Offline evaluation remains the fast and reliable first layer. But serious evaluation of recommender quality likely needs additional layers that are more sensitive to exposure shifts, segment differences, and longer-run experience.

7. Conclusion

Offline evaluation remains one of the most useful tools in recommender systems. It is fast, practical, and deeply embedded in how teams iterate on models.

Its limitation is structural rather than procedural. The data it relies on is constrained by prior exposure and generated under earlier policies, so it provides only a partial test of recommender quality.

That matters most when a model changes what gets shown, expands beyond historically overexposed items, or affects the experience over repeated interaction. In those settings, replaying the past is not the same as evaluating the new system on its own terms.

Offline evaluation is indispensable, but it is not the whole test. Recommendation systems shape the behavior they later observe, so any serious evaluation stack should measure interaction, not just replay the past.

This demo is illustrative rather than definitive; its value is in showing how aggregate offline results can hide segment-level differences.

How GenAI Genesis Began

Alankrit Verma — Sat, 07 Mar 2026 05:12:44 +0000

TL;DR

Alankrit Verma came to the University of Toronto as a shy, math-driven student on scholarship who felt a deep responsibility to give back.

That instinct led him into student leadership through AMACSS, where he helped build a small experiment called AI Olympics with 39 participants.

That experiment revealed something bigger: students wanted a serious space to build, learn, and belong in AI.

So Alankrit and his co-founder Adib Fallahpour scaled that spark into GenAI Genesis — first as a cross-campus student hackathon, and eventually into one of Canada’s largest student AI hackathons.

Along the way, Alankrit helped lead the vision, website, sponsorships, partnerships, and long-term structure behind the event, including helping establish the GenAI Genesis Foundation so the mission could sustain beyond a single organizing cycle.

And now, in 2026, GenAI Genesis is entering its biggest and most ambitious chapter yet.

From a 39-person experiment to one of Canada’s largest student AI hackathons

“Some communities are joined. Others are built because you cannot stop thinking about the version that should exist.”

GenAI Genesis team with the cake. Surprise cake courtesy of Hasleen Kaur (Head of Finance 2025, Co-Chair 2026) and Ivan Semenov (Head of Operations 2025, Co-Chair 2026).

There are some things you plan carefully.

And then there are some things that begin so quietly, so casually, that you do not realize until much later that you were standing at the start of something much bigger.

GenAI Genesis was one of those things.

When I joined the University of Toronto, I was, in many ways, still a shy person.

I was not the loudest voice in every room. I was still figuring myself out, still trying to understand what kind of life I wanted to build, and what kind of contribution I wanted to make.

But I did know one thing with complete clarity: I had been given a rare opportunity, and I did not want to waste it.

Coming to this country and this university on a scholarship meant a lot to me. It gave me the ability to study freely, dream more freely, and imagine a future I may not otherwise have had. And from the beginning, that created a very deep feeling in me: I had to give back to the community that had given so much to me.

At that time, I cared about many things at once.

I cared about math.
I cared about building projects.
I cared about recognition, yes — but not just for ego. I wanted to build things that mattered.
I cared about real-world impact.
And I cared, very deeply, about community.

Math had been a big part of my identity for a long time. I had prepared seriously for the Euclid Mathematics Contest and scored 90/100, and that experience mattered to me for more than just the number. Euclid is one of those milestones that gives you credibility, but more importantly, it gave me confidence. It made me more ambitious. It made me believe that I could build something meaningful. And it made me want to create spaces where other students could feel that same sense of challenge, excitement, and possibility.

So when I came to U of T Scarborough, I started looking around and asking myself a simple question:

Where is that energy here?

And honestly, at the time, I did not see enough of it.

Especially in the computer science space, the student community was not really booming. This was the period after COVID, when many campus communities still felt quiet, fragmented, and difficult to revive. There was talent, but not enough momentum. Curiosity, but not enough structure. Ambition, but not enough spaces for it to gather.

And somewhere along the way, I quietly made it my mission to help fix that.

Not alone, of course. Communities are never built alone. But I wanted to be one of the people pushing hard in that direction.

That instinct led me to AMACSS — the Association of Mathematical and Computer Science Students, the Departmental Student Association for the CMS department at the University of Toronto Scarborough.

In my first year, I joined as a First-Year Representative Coordinator, where I represented first-year computer science and math students to the association, and the association back to them. I also coordinated a team of seven people, which turned out to be one of my first real lessons in leadership.

Leadership, I learned very quickly, is not just about taking initiative. It is about understanding people. It is about assigning responsibility thoughtfully. It is about getting buy-in. It is about leading with grace when everyone has different levels of energy, skill, confidence, and commitment.

I had always been someone who liked taking initiative, but AMACSS sharpened that instinct into something more deliberate.

And in that chapter of my life, the first version of GenAI Genesis quietly appeared.

Not as GenAI Genesis.

Not yet.

It started as something called AI Olympics.

Before Genesis, there was AI Olympics

AI Olympics was the first real experiment.

The original idea came from a mix of inspirations.

Part of it came from my love for mathematics competitions and the kind of intellectual excitement they create. Part of it came from online hackathons I had participated in, where I had seen how energizing it could be when people come together to build under time pressure. I remember thinking again and again:

Why do we not have something like this at our university too?

At one point, I was brainstorming with Katrina Best, who was the president at the time, about what might make for a strong event for first-year students. At first, we thought about doing something closer to a math contest. Then the idea evolved. I brainstormed with my team as well. Slowly, the concept shifted from Olympiad to something more build-oriented, more alive, more experimental.

That is where AI Olympics was born.

The name came from that same spirit. We wanted something that felt like an Olympiad, but more modern, more hands-on, and more builder-focused. “AI Olympics” felt close enough to that energy, and at the time, it captured exactly what we were trying to do.

It was a smaller hackathon-style event, around six to seven hours long, with 39 participants.

It was essentially the first-year team’s event through AMACSS, and leading it as First-Year Representative Coordinator made it feel especially personal.

Many of them were beginners. The vibe in the room was not “elite competition” in the intimidating sense. It was much more like collective learning. People were curious. People were experimenting. People were just starting to understand what they could build.

We taught participants how to use the tools. We gave them a website template they could plug their work into so they could build faster. We wanted to reduce friction and maximize momentum. We wanted them to feel like they could actually make something, even if they were just getting started.

And maybe one of my favorite little memories from that day is how we kept ordering coffee from Tim Hortons — not once, not twice, but three times — because people kept wanting more, and apparently everyone had collectively decided that vanilla was the flavor of innovation.

AI Olympics

Looking back, AI Olympics was small.

But it was not small in meaning.

Because it showed us something important:

People wanted a space to learn.
People wanted a space to build.
People wanted a space where AI felt exciting, approachable, social, and full of possibility.

The feedback made that obvious. People were interested in doing this again. They wanted to keep learning. They wanted to keep contributing. They wanted to build in public. They wanted more.

And that was the moment the idea stopped feeling like a one-off event and started feeling like the beginning of a much larger mission.

AI Olympics was the spark.

GenAI Genesis was the system we built around that spark.

The moment it stopped being small

Around that time, I was also working very closely with Adib Fallahpour, who is not just my co-founder, but also a good friend of mine.

I had first worked with Adib through my first-year team, and over time it became very clear that we were on the same wavelength in a lot of ways. He is a very kind person, a big thinker, and someone with strong vision. We both cared deeply about scaling this beyond its first version. We both felt that it should not remain a small campus event that people vaguely remembered. We wanted it to become something real.

I still remember a moment from second year when Adib and I were housemates. He came into my room, and we started discussing what this thing could actually become. Not just another event. Not just another student initiative. But a serious hackathon. Something with real scale. Something that could create a home for people interested in AI, machine learning, software, and ambitious building more broadly.

That conversation stayed with me.

Because from that point onward, this stopped being a nice idea and started becoming a serious project.

Like most ambitious student things, it began with a lot of conversations, a lot of hustle, and a slightly unreasonable amount of belief.

We first tried to define the idea on paper:

What exactly was GenAI Genesis?
What would it look like at scale?
What kind of experience were we trying to create?
What problem were we solving?

The problem, at least to us, felt clear.

At the time, there was not enough community in Toronto around this space — not the kind of entrepreneurial, energetic, builder-first AI ecosystem we wanted to see among students. There was talent, but not enough connected ambition. There were students interested in AI and ML, but not enough platforms bringing them together in a serious way.

So we decided to build one.

The name came together surprisingly quickly. We broke it into two parts: GenAI and Genesis. “Genesis” suggested beginning, emergence, evolution. And at the time, “GenAI” was the word in the air. The name reflected the moment, but the mission was always broader than just generative AI — it was about AI, machine learning, software, and the community around building them. Put together, it felt like a beginning worth naming.

We did not have a full team immediately. At first, we were figuring it out from scratch. Both Adib and I were part of Google Developer Student Club, and that gave us one starting point. We knew we could bring in people from there. Then we looked beyond Scarborough and started reaching across campuses, especially to St. George, where some of the strongest technical student communities already existed.

That is how collaborations started taking shape with groups like:

GDG
UTMIST
UofT AI
and later, CSSU

But I want to be careful and clear here, because this matters to the story.

Important distinction: GenAI Genesis was not a club-created event that I happened to be involved in.

Those groups mattered enormously, and their support helped the vision scale far beyond what we could have done alone in the early stages. They brought expertise, reach, operational support, and legitimacy. But the mission itself — the initial push, the insistence that this had to exist — came from us.

That distinction matters because founder stories can get flattened over time into partnerships, logos, and sponsor lists. But the truth is usually more human than that. It begins with a few people seeing a gap and deciding they are not willing to leave it empty.

The people who helped us scale it

Some of the most important early support came from people who believed in the idea and helped us take it seriously.

Richard, from UTMIST, played a crucial role in 2024. He was a senior to us and incredibly strong operationally. He helped us understand what it means to run something at scale, what it means to think through logistics properly, and how to turn energy into structure.

Nimit, through UofT AI, also played a very important role in helping the initiative come together. Both Richard and Nimit helped us build the cross-campus support that allowed GenAI Genesis to grow beyond its first form.

These collaborations mattered a lot. Not because the event “belonged” to those communities, but because they helped us bring the mission to the scale it deserved.

Sometimes scaling an idea is not about finding people who will take it over.

It is about finding people who understand it enough to help it rise.

A quick timeline

Year	What happened	Why it mattered
Winter 2023	We ran AI Olympics through AMACSS with 39 participants	It proved there was real demand for a build-first AI space
Winter 2024	We launched the first large-scale GenAI Genesis	The experiment became a serious institution
2025	We scaled dramatically with more sponsors, more prizes, and many more submissions	The hackathon became a recognized force in the student AI ecosystem
2026	We are taking it to our biggest scale yet	Bigger footprint, bigger ambition, bigger future

A tiny behind-the-scenes truth
Every row in that table was held together by a lot of invisible work: outreach, relationship management, budget stress, website iterations, venue uncertainty, and a hundred tiny decisions that never show up in a recap post.

2024: when the idea met reality

Getting Started with GenAI Genesis 2024

The 2024 edition was the moment things started to feel very real.

In winter 2024, we launched the first large-scale GenAI Genesis in downtown Toronto.

Up until that point, the idea had energy. It had promise. It had momentum. But 2024 was when it had to survive the test every ambitious student project eventually faces:

Could we actually execute this at scale?

That year taught me a lot.

And by “a lot,” I mean the kind of lessons that only appear when vision collides with logistics.

We had to learn how to:

work with a much larger team
coordinate across campuses
lead people with different styles, strengths, and expectations
manage conflict and disagreement without letting it fracture the mission
build trust with sponsors
make big promises responsibly

And in the middle of all that, I was deeply involved in the work itself.

From the very beginning until now, I have led the website side of GenAI Genesis. Tech has always been one of the areas I stayed especially close to. I was also heavily involved in sponsorships and partnerships — doing cold outreach, talking to organizations, building those relationships, and helping create the external support system that made the event possible.

One of the most memorable parts of that journey was our connection with Google, and how that relationship went from something that initially felt surreal to something that became a meaningful long-term thread in the GenAI Genesis story. There is a strange feeling when big names start trusting something you built. It is exciting, but it is also sobering. It makes you realize the stakes are now real.

The 2024 edition brought in support from names including Google, Knockri, Wombo, Vector Institute, the Academic Advising & Career Centre at UTSC, and the Rotman School of Management.

We had around 254 participants submit a project and awarded roughly $3,000 in prizes.
But what I remember most is not just the number.

I remember how much we had to figure out on the fly.

Venue booking was a huge hassle. A lot of things were fragile. Judging, especially, was something we did not have perfect prior experience with at that scale. And yet, when the time came, the team handled it with surprising grace. We made last-minute changes to make sure the judging process was fair, thoughtful, and well run. That moment stayed with me because it showed me something essential: even if we were new to this scale, we were capable of rising to it.

That was the year GenAI Genesis stopped feeling like a hopeful experiment.

It felt real.

GenAI Genesis 2024

genesis:

  v0: "AI Olympics"
  participants: 39
  then:
    - an experiment
    - a room full of beginners
    - a lot of coffee
    - a lot of belief
  now:
    - a cross-campus movement
    - a large-scale AI hackathon
    - a serious community
  constant:
    - vision
    - people
    - momentum

2025: scale changes everything

Then came 2025.

And 2025 felt different.

This was the year when GenAI Genesis started feeling less like an event and more like an ecosystem.

By then, we were no longer operating entirely from instinct. We had learned processes. We had built systems. We had a better understanding of what worked, what broke, what participants valued, and what scale actually requires. We planned earlier. We moved more formally. We operated with more clarity.

The leadership structure also evolved.

In the earlier chapter, the co-chair structure included me, Adib, Nimit, and Richard. By 2025, the co-chairs were me, Adib, and Matthew Tamura.

Matthew had already been involved as a strong contributor in 2024 through UTMIST and was someone I deeply appreciated — thoughtful, visionary, and strong at leading a team properly. In 2025, he stepped into a bigger leadership role with us, and that made a real difference.

We also worked hard to improve the participant experience in ways that went beyond the surface.

We brought in more sponsors.
We created more networking opportunities.
We designed stronger supporting events during the hackathon.
We sharpened logistics.
We elevated the experience.

And the scale reflected that.

In 2025, we had:

around $15,000 in awards
621 participants submitted a project
backing from Google, BWC, Cohere, AMD, CGI, RBC, Northeastern University, Edge.io Solutions, the Academic Advising & Career Centre, and the University of Toronto
support from partners including the United Nations Association in Canada, One Degree Cooler, Vector Institute, and Hack Canada

One of the most exciting moments that year was when AMD came in and supported us in a way that allowed participants to run more complex machine learning workloads on an AMD GPU-backed local cluster. That felt genuinely wild. It was one of those moments where you step back and realize the hackathon is not just getting bigger in numbers — it is getting more technically meaningful too.

From the outside, growth can look glamorous.

From the inside, it often looks like spreadsheets, calls, follow-ups, contingency planning, team alignment, venue negotiations, technical troubleshooting, partnership mapping, and a hundred open loops in your head at once.

People usually see the lights.

Founders remember the wiring.

And by 2025, I was exhausted. Truly.

But it was also the kind of exhaustion that comes from building something you care about so deeply that you keep choosing it, again and again, even when it would be easier not to.

There were many moments in those years where I could have spent my time doing something else for my résumé — some other project, some other opportunity, some other clean, convenient line on paper.

And again and again, I chose GenAI Genesis.

Because by then it was not just a project.

It was a commitment.

Some of the winners at GenAI Genesis 2025

What people see vs. what it takes

People usually experience a hackathon at the moment it becomes exciting.

Founders experience it in the months before that, when it is still fragile.

What goes into building a hackathon like this?

Not just posters and prize money.

It looks more like this:

sponsor outreach and partnership management
website design and technical infrastructure
judge and mentor coordination
cross-campus relationship building
team alignment across different working styles
planning future editions before the current one is even over
making sure the vision survives internal complexity
solving ten operational problems before breakfast
keeping something founder-led while still making it collaborative
doing a lot of invisible thinking about what the next step even is

A polished event always has a chaotic prequel.

And a surprising amount of inner work goes into making sure the chaos does not win.

What GenAI Genesis has meant to me

At one level, GenAI Genesis is about AI and machine learning.

But if I am being honest, it has never only been about AI.

It is about belonging.
It is about ambition.
It is about opportunity.

It is about building the kind of space I wish existed more abundantly when I first arrived.

A place where students do not just come to listen, collect swag, and leave. A place where they come to make things. To meet each other. To stretch. To take themselves seriously. To find their people. To realize that they are more capable than they thought.

That is what I wanted to create.

And I think that is why this has become more than just a hackathon to me.

It has become a community, a signal, a platform, and in some ways, living proof that if you build the right room, the right people will find each other inside it.

I have also learned a lot about myself through this.

I learned how passionate I am about the things I truly care about. I learned how much I care about my team. I learned that leadership is not something you perform; it is something you practice. I learned how much invisible thinking goes into visible outcomes. I learned that building something meaningful costs time, energy, sleep, and sometimes other opportunities.

But I also learned that some things are worth choosing repeatedly.

And this was one of them.

The foundation behind the future

As GenAI Genesis grew, it became increasingly important to make sure it could sustain itself beyond just the intensity of one year, one organizing cycle, or one group of students.

That is a big part of why I helped establish the GenAI Genesis Foundation as an NGO, along with four other directors.

That step mattered deeply to me.

Because if GenAI Genesis was going to keep growing properly, it needed more than momentum.

It needed structure.
It needed continuity.
It needed a long-term home.

Founding the Foundation was part of making sure that what we built would not just peak.

It would endure.

And I am very proud of that.

People I want to thank

No founder story is ever truly solo.

And GenAI Genesis certainly was not.

I started this with Adib Fallahpour, my co-founder, and I want to begin there. Thank you, Adib, for building this vision with me from the early days, for dreaming big, for caring deeply, and for helping turn a small experiment into something much larger than either of us could have reached alone.

I want to thank Richard, who helped us significantly in 2024 through UTMIST. Richard brought strong operational guidance at a time when we were still learning how to scale properly, and his support played an important role in helping us bring the hackathon to life at a bigger level.

I also want to thank Nimit, who helped us through UofT AI and contributed meaningfully to the growth of the initiative in its earlier large-scale chapter. Cross-campus support mattered a lot, and Nimit was part of that story.

For 2025, I want to thank Matthew Tamura, who joined me and Adib as a co-chair in 2025. Matthew brought a lot of clarity, vision, and leadership to that year, and I deeply appreciated building that edition alongside him.

And for 2026, I want to thank Hasleen Kaur and Ivan Semenov, who are co-chairing this year alongside me. Both are wonderful people and wonderful friends, and I am genuinely grateful to be building this chapter with them.

There are many people behind the scenes who have contributed to GenAI Genesis over the years — teammates, sponsors, organizers, mentors, judges, friends, and supporters — and I carry a lot of gratitude for all of them.

Communities may remember the banner.

But founders remember the people who helped hold it up.

And now, 2026

And now we arrive here.

What started as a 39-person experiment has grown into something far bigger, and in 2026, we are taking GenAI Genesis to its biggest scale yet.

This year, we are going much, much bigger.

We are preparing to bring together close to 1,000 people in person. We are building across three major spaces at the University of Toronto — Convocation Hall, Bahen, and Myhal — to create an experience that is bigger not just in attendance, but in ambition, energy, and depth.

This year feels different.

Not because the mission has changed, but because the scale has finally caught up to the size of the vision.

We are crossing into four digits.
We are building across multiple buildings.
We are thinking bigger than ever before.

And for me personally, this year is meaningful in another way too.

2026 will be my last year serving as Co-Chair of the hackathon. After this, I will be moving into more of an advisory role.

There is something beautiful about that.

Because one of the deepest measures of building something well is whether it can continue growing beyond the chapter where you are the one carrying it most directly.

That is what I want for GenAI Genesis.

I want it to outgrow any one person, any one year, any one team.

I want it to keep becoming a place where ambitious students find each other, where builders take themselves seriously, where new ideas are given room to breathe, and where community feels like a force multiplier rather than just a word on a poster.

So if you have been watching from the sidelines, this is your sign.

Join us on March 13, 14, and 15, 2026.

Come build with us.
Come meet the people shaping what comes next.
Come be part of something that started small, but refused to stay small.

And when you do, I hope you feel what I felt at the beginning of this whole journey:

That strange, beautiful energy that appears when ambitious people gather around an idea and decide to make it real.

That, in the end, is what GenAI Genesis has always been about.

Connect with me

If this story resonated with you, feel free to connect with me online, follow GenAI Genesis, or reach out.

I always love meeting people who care deeply about building communities, technology, and meaningful things.

See the 2026 event, follow the journey, or reach out if you want to build something meaningful together.