Forem: Manish Podiyal

The Data Refinery: Why Apache Spark is the Engine Behind Real-World Big Data Use Cases

Manish Podiyal — Mon, 04 May 2026 13:00:45 +0000

You've likely heard that "Data is the new oil". But raw oil is useless without a refinery. In the world of Big Data, Apache Spark is that refinery.

Whether it's millisecond-level fraud detection or processing terabytes of logs, Spark's ability to handle massive scale with in-memory speed is why it remains a core skill for every ML & Data Engineer.

Here are 5 real-world problems and exactly how Spark solves them:

1. Stopping Credit Card Fraud in Real-Time
The Problem: Banks need to flag fraudulent transactions in under 500ms before the "Swipe" is even finished.
The Spark Solution: Use Structured Streaming to ingest Kafka feeds, join them with historical user profiles in Cassandra, and run a pre-trained MLlib model to score the risk instantly.

2. Predicting Machine Failure Before it Happens
The Problem: Unexpected factory downtime costs millions. How do we predict a pump failure using "noisy" IoT sensor data?
The Spark Solution: Aggregate high-frequency vibration and temp data into Data Frames, calculate rolling averages for feature engineering, and train a Random Forest regressor to predict the machine's "Remaining Useful Life."

3. Personalizing Your Shopping Feed
The Problem: Static "Top Sellers" lists don't convert. Users want recommendations based on their specific behavior.
The Spark Solution: Leverage the ALS (Alternating Least Squares) algorithm in Spark to process a massive user-item matrix across a distributed cluster, serving up hyper-relevant "You might also like" items.

4. Unifying a Messy Data Lake
The Problem: Data is trapped in silos-SQL, JSON, CSV and it's too big for one server to clean.
The Spark Solution: Build a robust ETL pipeline using Spark SQL to de-duplicate millions of records, mask PII for compliance, and save the result into an optimized Delta Lake format.

5. Hunting for Cyber Threats in Terabytes of Logs
The Problem: Finding one malicious IP in a mountain of server logs is like finding a needle in a haystack.
The Spark Solution: Use Spark's distributed Regex and windowing functions to scan billions of log lines simultaneously, flagging spikes in failed logins or suspicious geographic traffic patterns.

The Takeaway:
Spark isn't just a tool - it's a "Unified engine" for batch, streaming, and ML. If you aren't using it to solve these scale problems, you might be leaving performance on the table.

TDD is Like Cooking: Why I Taste My Code as I Go

Manish Podiyal — Tue, 24 Feb 2026 13:10:32 +0000

I have a passion for software development and cooking is one of my favorite hobby. At first, they seem different. One is digital, the other is delicious! But they actually follow the same philosophy.

What is TDD/BDD anyway?
Some people think Test/Behavior-Driven Development (TDD/BDD) means writing every test case before you write any code. That’s not true.

It’s actually about a cycle:

Red: Write a small test that fails.
Green: Write just enough code to make it pass.
Refactor: Clean up the code.
Repeat: Start over from Step 1.

Here is how that looks in my kitchen:

1. The "Taste Profile" (BDD)
Before I start, I decide what the dish should be.

The Goal: "A spicy tomato soup with a hint of garlic."
The Initial State: Right now, I just have a pot of plain water.
The Result: If I taste it now, it fails. It’s not spicy, garlicky or even soup yet! This is my "failing case" that guides me.

2. Add and Test (TDD)
I don’t throw all the spices or salt in at once. I go step-by-step:

Step 1: I add chili. I taste it. Still not spicy enough? (Test Fails).
Step 2: I add more chili. I taste it again. Now it's spicy! (Test Passes).
Step 3: Now I add the garlic. I repeat the process.

3. Cleaning the Kitchen (Refactoring)
In code, we refactor to make it clean. In cooking, I might strain the soup to make it smooth or adjust the heat. I am not changing the flavor (the behavior); I am just making the quality better.

Why does this work?

No Surprises: I know exactly when the "flavor" is right.
Safety Net: If I add too much salt, I know immediately because I am testing (tasting) as I go.
Evolution: The recipe evolves with the dish, just like code evolves with the project.

My Advice
Don't just build it and hope for the best at the end. Taste your code as you go!

Cooking isn't just a hobby; it's a sandbox for software engineering

Manish Podiyal — Tue, 17 Feb 2026 09:18:47 +0000

> "Cooking isn't just a hobby; it's a sandbox for engineering. From 'debugging' flavors to architecting 'protein-rich' algorithms, the journey from following a recipe to creating your own patterns perfectly mirrors the evolution of an engineer's mindset."

Over the years in software development, I've realized that my kitchen and my IDE aren't that different.

When you start, you're just looking for a "stable build." You want to cook something that tastes good and more importantly - something you can debug. When a dish is too salty or a function throws an error, you analyze the "logs", adjust the ingredients, and try again. Every "failed" meal is just a bug fix that makes you a better Engineer and a better Cook.

From Documentation to Intuition

After years of daily commits and daily meals, something amazing happens: the documentation disappears.

You no longer need a step-by-step guide (or a spoon-fed recipe) to know how much salt or "syntax" to add. You stop just "following" and start architecting. You begin to recognize design patterns in flavors. You can invent your own "custom libraries" in the kitchen, creating recipes that are:

1. Optimized: No wasted movements or ingredients.
2. Balanced: Perfectly handled dependencies between spice and heat.
3. High Performance: Revenue-generating (or at least, protein-rich and fuel-efficient!)

Reaching "Pro" Status

This is what reaching the "Pro" level feels like. You’re no longer just following a script; you're writing the source code for a healthier, more creative life. You're becoming a better version of yourself - one commit and one meal at a time.

Time to ship the next meal. सुभुक्ताम् (Subhuktam) and Happy coding!