Forem: Francis Rudra D Cruze

I Thought My ML Model Was the Product. I Was Wrong.

Francis Rudra D Cruze — Fri, 03 Apr 2026 18:15:14 +0000

For almost 2 years, I genuinely believed that if the model works, the job is done.

Not because I was lazy. Because nobody ever told me otherwise. Every course, every lecture, every YouTube tutorial they all ended at the model. Accuracy looks good? Great. You're done. Here's your certificate.

I was building ML systems with this exact mindset. Train, evaluate, deploy. Move on to the next thing.

Then I started working on a real production system and everything broke.

The moment it actually clicked

The fraud detection model was performing well on paper. Precision was solid. The team was happy.

But fake orders were still going through.

I kept looking at the model. Tuning it. Re-evaluating it. The model was fine.

The problem was everything that came after the model. Nobody had built that part.

There was no decision layer. No threshold policy. No action logic. No feedback loop. Just a model outputting scores into a void, and a spreadsheet someone was manually checking twice a week.

That's when I understood the model was Layer 2. There were 3 more layers nobody had built.

What I wish someone told me at the start

The model is not the product.

The decision it enables is the product. The action it triggers is the product. The outcome it changes is the product.

Most ML education stops at prediction. But prediction without a decision layer is just a number floating in the air. A churn model that flags customers with no policy for what happens next is not a system. It is a very expensive guess.

Your churn model flags a customer at 0.87 probability. Now what?

Does someone call them? Does an email go out? Does nothing happen because nobody defined what "above threshold" actually means in terms of real action?

If you can't answer that before you deploy, you deployed a guess.

The part that really hurt to realize

I had been optimizing the wrong thing for 2 years.

Loss function going down. Accuracy going up. AUC-ROC looking clean. I thought I was building better and better systems.

But loss function is what the model learns from. Business objective is what the ops team cares about. These two things can point in completely different directions and nobody checks the alignment before deploy.

Low loss does not mean fewer fake orders reaching fulfillment. It just means the model got better at the thing you told it to get better at.

The thing you told it to get better at might not be the actual problem.

The thing nobody builds

I have worked in teams, done internships, built projects from scratch. In every single one of them, nobody ever talked about the feedback loop.

Build the model, ship it, move on to the next project. That was the full workflow.

But if your system has no feedback loop, it is not a system. It is a one-time guess that gets more wrong every day as the world changes around it. You find out something is broken only when a campaign tanks or a manager asks why the numbers look off.

The most dangerous model is not the one that fails loudly. It is the one that gives confidently wrong answers for 6 months because nobody closed the loop.

What actually changed how I think

Before you write a single line of code, write one sentence.

This model will help [who] make [what decision] so that [what business outcome happens].

If you cannot write that sentence, stop. You are not ready to model yet.

The model is the easy part, to be very honest. The hard part is everything before and after it. What data actually represents the problem. What decision follows each output tier. What action fires in the real world. What outcome gets recorded so the system can learn.

Most ML is Layer 2 only. Prediction and nothing else.

The actual value lives in the other layers. Nobody teaches those.

I wrote a full breakdown of the 5-layer framework on DEV what each layer means, where most systems collapse, and the one question you need to answer before you deploy anything.

Click Here if you want to read the whole thing.

ML Alone Is Just Numbers. Here's the 5-Layer Framework That Actually Ships.

Francis Rudra D Cruze — Thu, 02 Apr 2026 18:14:50 +0000

I used to think if the model works, the job is done.

Like literally train, evaluate, deploy, done. That was the whole workflow in my head. Nobody ever told me otherwise, not in any course, not in any academic lecture I sat through.

To be very honest, it took me embarrassingly long to realize that's not engineering. That's just hoping the world doesn't change.

The thing nobody actually tells you

Every ML course stops at the model. Accuracy looks good? Here's your certificate, you're done.

Nobody ever asked okay but what happens after the prediction?

And that gap, that's where real systems either make money or quietly burn it. Most of the time nobody even notices until something breaks badly.

The 5 Layers (and why most ML is stuck at Layer 2)

This is the framework that completely broke my old way of thinking.

Layer 1 — Data

Not just "clean your data." I mean is this data actually representative of what's gonna happen in production? Is the distribution gonna shift in 3 months?

The orders you never caught and never labeled? Your model has never seen them. So it's only ever good at catching the fraud it already knows. New patterns? Invisible. Completely invisible.

Most people spend 80% of time here and still get it wrong because they're cleaning without asking why the data looks the way it does.

Layer 2 — Prediction

This is where 90% of ML education lives. Loss function. Accuracy. AUC-ROC. Hyperparameter tuning and all of that.

Here's the thing a 91% accurate model that nobody acts on is worth exactly zero. A 74% model connected to a real decision layer is worth millions.

The model is not the product. I had to learn this the hard way.

Layer 3 — Decision

This is where most deployed systems quietly collapse and nobody talks about it.

Your churn model flags a customer. Now what? What's the threshold? What action fires above 0.5? What fires above 0.8?

If you can't answer what decision follows each output tier, you're not ready to deploy. Full stop.

Lead C — 90% reply rate. Sounds amazing right. Expected value is $450.

Lead B — 30% reply rate. Expected value is $14,400.

Probability picked the wrong lead. Expected value picked the right one. That gap is real money left on the table, and nobody taught this in any course I took, to be very honest.

Layer 4 — Action

Okay so the decision fired. What actually happens in the real world?

Does an alert go out? Does an email trigger? Does a $5 voucher get sent to a customer with $4 expected lifetime value — losing money on every single send?

EV = P × Value − Cost. That's it. Run this before every single intervention. At 10,000 customers that's $10,000 burned because nobody did 30 seconds of math before wiring up the action layer.

The action layer is where your ML system actually touches the real world. It deserve more than an afterthought.

Layer 5 - Feedback

This is the one that nobody builds. And it's the one that make everything else actually matter.

If your ML system has no feedback loop it's not a system. It's a one-time guess that gets more wrong every day as the world changes around it.

The most dangerous model is not the one that fails loudly. It's the one that gives confidently wrong customer segments for 6 months because the underlying data distribution shifted and nobody closed the loop. You find out when the campaign numbers tank and everyone just look confused.

In every job I worked, every project I built — nobody ever talked about the feedback loop. Build the model, ship it, move on to the next thing. That's not engineering. That's just hoping.

Why accuracy doesn't actually tell you anything useful

Most ML engineers can explain gradient descent in 10 minutes. Almost none can explain why their loss function was the right choice for the actual business problem.

That gap is where bad models get shipped.

Loss function is what your model learns from. Business objective is what your ops team cares about. These can point in completely different directions and nobody checks the alignment before deploy.

Low loss doesn't mean fewer fake orders reaching fulfillment. Check the alignment before you deploy.

The honest thing I wish someone told me earlier

Before you write a single line of code, write one sentence

"This model will help [who] make [what decision] so that [what business outcome happens]."

If you can not write that sentence, stop. You are not ready to model yet.

Customer segmentation without a policy is just art. You have 5 beautiful clusters. Now what do you do with each one? If you can't answer that before you run K-Means, you're not ready to run K-Means.

Gist

Layer 1 (Data) — representative, not just clean
Layer 2 (Prediction) — model output, the part everyone optimizes for
Layer 3 (Decision) — what action follows which output? what's the policy?
Layer 4 (Action) — what actually happens in the real world?
Layer 5 (Feedback) — does the system learn from outcomes? does anything close the loop?

Most production ML Layer 2 only.

The other 4 layers is where the actual business value lives.

Nobody teaches the other 4.