Forem: Narayan

🔥 Single Biggest Idea Behind Polars Isn't Rust — It's LAZY 🔥 Part(2/5)

Narayan — Fri, 07 Nov 2025 05:41:15 +0000

If you're still processing data in sequential steps (Pandas-style), you're missing out on 90% of Polars' performance gains.

This is the core difference: Eager vs. Lazy. Understanding this makes the Expression API click.

❌ 𝐓𝐇𝐄 𝐄𝐀𝐆𝐄𝐑 (𝐏𝐀𝐍𝐃𝐀𝐒) 𝐖𝐀𝐘: 𝐄𝐱𝐞𝐜𝐮𝐭𝐞 𝐈𝐦𝐦𝐞𝐝𝐢𝐚𝐭𝐞𝐥𝐲

Every line runs instantly, creating a new DataFrame in memory at each step.

import pandas as pd
df = pd.read_csv("large_file.csv") # 1. Loads ALL columns
df['doubled'] = df['new_val'] * 2 # 2. Creates new copy
df = df.groupby('category').sum() # 3. Final compute

🚨 Result: Huge memory footprint, wasted I/O, no query optimization.

✅ 𝐓𝐇𝐄 𝐋𝐀𝐙𝐘 (𝐏𝐎𝐋𝐀𝐑𝐒) 𝐖𝐀𝐘: 𝐏𝐥𝐚𝐧 𝐅𝐢𝐫𝐒𝐭, 𝐄𝐱𝐞𝐜𝐮𝐭𝐞 𝐎𝐧𝐜𝐄

Polars records all operations, builds an optimized plan, and only runs when you call .collect(). This unlocks QUERY OPTIMIZATION.

🎯 The Two-Step Dance:

Step 1: Define the Plan (LazyFrame). Nothing runs yet.
import polars as pl
q = (
pl.scan_csv("large_file.csv")
.filter(pl.col("value") > 100)
.with_columns(
(pl.col("new_val") * 2).alias("doubled")
)
.group_by("category")
.agg(pl.col("doubled").sum())
)

Step 2: Execute the Optimized Plan.
result = q.collect()

🧠 𝐖𝐇𝐀𝐓 𝐓𝐇𝐄 𝐐𝐔𝐄𝐑𝐘 𝐎𝐏𝐓𝐈𝐌𝐈Z𝐄𝐑 𝐃𝐎𝐄𝐒

Polars applies transformations to your plan:

Projection Pushdown: Only read the columns you use.
Predicate Pushdown: Filter rows while reading the CSV (skip rows at the source).
Expression Fusion: Combine multiple operations into a single, efficient kernel (no intermediate copies).

💰 𝐑𝐄𝐀𝐋-𝐖𝐎𝐑𝐋𝐃 𝐈𝐌𝐏𝐀𝐂𝐓 (10𝐆𝐁 𝐂𝐒𝐕 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤)

Metric	Pandas (Eager)	Polars (Lazy)
Time	~8 minutes	~45 seconds (10x faster)
Memory	12GB peak	2GB peak (6x less)

Why the difference? Polars only loaded what it needed, filtered while reading, and fused operations.

🔑 𝐊𝐄𝐘 𝐓𝐀𝐊𝐄𝐀𝐖𝐀𝐘

Lazy evaluation is why Polars is fast. The speedups come from:

Loading only what you need.
Filtering at the source.
Fusing operations.

DataEngineering #Polars #Python #DataScience #DataAnalytics

💥 Polars vs. Pandas: Why Your Next ETL Pipeline Should Run on Rust (Part 1/5)

Narayan — Tue, 14 Oct 2025 20:57:10 +0000

If you're a Data Engineer, you've seen the struggle: Pandas is brilliant for analysis, but when you hit the scaling, multi-threading, or memory wall in production, it falls short.

I've been doing a deep dive into Polars as part of my own "learning in public" journey. It's not just "faster Pandas"; it's a complete shift in how data processing is handled, built on a Rust core to solve our toughest ETL problems.

This post shares my findings and conviction that Polars is the future of performant, single-node data engineering.

🚀 The Core Difference: Rust and Arrow
Polars' superior performance isn't magic—it's architecture. It leverages the best features of modern systems engineering:

Engine Core: Rust, a blazingly fast and memory-safe systems language. This is where the speed comes from, allowing Polars to execute code efficiently and in parallel.

Memory Model: Apache Arrow, which uses a columnar format. This means Polars only loads the columns it needs, uses less memory overall, and enables zero-copy sharing between processes.

💡 Code Simplicity: The Functional API
Polars encourages a clean, functional style that reduces bugs and improves readability.

In Pandas, you often mutate the DataFrame (df[col] = ...). In Polars, you build an execution plan using chained methods and expressions, which is key to its optimization engine.

The Polars method is a declarative instruction that the Rust optimizer can rearrange and fuse for maximum performance. This is critical for maintainable, high-speed pipelines.

This is Part 1 of a 5-part miniseries documenting my deep dive and journey into Polars for scalable ETL.

👉 Question: What size dataset (rows/GBs) was the tipping point that made you start looking beyond Pandas? Share your experience below!

Two Years of Microsoft Fabric: Game Changer or Still Leveling Up? 🚀

Narayan — Wed, 03 Sep 2025 21:59:41 +0000

Microsoft Fabric has been making waves for two years now, and after diving deep with more than 6 months of hands-on experience, I’ve got some thoughts. Is it the unified data dream, or does it still have some growing pains? Let’s break it down.

✅ The Superpowers: What Fabric Nails

Unified Platform & OneLake 🤝

This is Fabric’s biggest win. OneLake as a "single pane of glass" isn't just marketing hype; it genuinely cuts down on data duplication and massively improves data quality. Imagine all your data, from raw to refined, living in one place, accessible by all your tools.

Efficiency Gains ⚡️

Remember the days of juggling Azure Data Factory, Synapse, and Power BI separately? With Fabric, those days are gone. The integration is seamless, making the entire data workflow feel like one continuous, efficient process. It’s a huge time-saver.

BI + AI Integration 🧠

With Copilot in Power BI, turning natural language into analytics feels incredibly powerful. It makes the entire analytics and data engineering process more fluid and accessible, bringing BI and AI closer together than ever before.

⚠️ The Road Ahead: What's Still Evolving

Scale 📈

Fabric handles big volumes, but under extreme, highly concurrent workloads, I've seen some hiccups. It's not a deal-breaker for most, but it’s something to watch and plan for, especially with mission-critical pipelines.

Governance 🛡️

While the basics are there, the fine-grained controls and enterprise-level audit trails that large organizations need are still maturing. For a company with strict regulatory requirements, this is a key area that needs more polish.

Adoption 🧭

Momentum is building, but most orgs are still in pilot mode or in limited production. Fully embracing Fabric requires a significant cultural shift and a reskilling effort for data teams, which naturally slows down widespread adoption.

👉 My Take: Early Growth, Powerful Potential
Fabric is a game-changer for teams already deep in the Microsoft ecosystem. However, it's not yet the “one platform to rule them all.” It’s powerful, but still in its early growth phase. The potential is massive, but the journey to full enterprise maturity is still underway.

💬 What's your take? Let's discuss!
Would you trust Fabric with production-grade pipelines today, or wait for more polish? Share your experience below! 👇

🚀 Synthetic Data: The Next Frontier for Data Engineers

Narayan — Wed, 20 Aug 2025 15:57:02 +0000

Hey data community, let’s talk about a topic that's quickly becoming essential. We all know the struggle: a world hungry for data and AI, but with privacy rules (HIPAA, GDPR) that make it tough to get what we need.

That's where synthetic data comes in. Forget about "fake data"—think of it as a meticulously engineered copycat of the real thing. It looks, feels, and acts just like production data, but without any of the sensitive info. This lets us build, test, and innovate without the usual legal headaches.

📊 What Are We Actually Using This For?

If you're wondering where the real work is happening, here’s a quick look at where data teams are putting their effort:

Testing Pipelines: This is a big one. We can now run our ETL jobs on massive, realistic datasets to catch bugs and fine-tune performance before they ever touch live data.

Training AI: AI models need tons of data, and synthetic data provides an unlimited, ethical source. It's how we're building better, less-biased models for drug discovery and other critical tasks.

Running Simulations: Ever wanted to test a new business strategy? Now we can run realistic simulations in a secure environment, saving tons of time and money.

💻 A Peek Under the Hood

This isn't just theory. We’re using powerful tools to get it done. Libraries like SDV are perfect for learning the statistical DNA of a dataset and generating a twin. Then, for enterprise-level scaling, platforms like Gretel.ai and Mostly AI take it to the next level.

Take a look at this quick example, which has data generated using Faker. See how the synthetic data isn't just random—it captures the relationships from the original data, like how age might correlate with a medical diagnosis. That's the real magic.

This isn’t just a cool new toy. The ability to build, manage, and scale these pipelines is quickly becoming a core skill for any data engineer. It's a fundamental shift in our field, and it’s happening now.

So, for those of you out there who have run a proof-of-concept with synthetic data, what was your biggest win or a lesson you learned the hard way? Share your thoughts below! 👇

🌟 Back After 3 Years: My First Full Stack App in 3 Days

Narayan — Thu, 07 Aug 2025 04:26:14 +0000

It’s been more than 3 years since I last posted, and I’m quietly excited to share a big step: I built my first full-stack app, MediTrack, in just three days during my company’s Vibe Coding challenge. As a data engineer with no app dev experience, this was a big deal for me, and I’m eager to tell you about it.

What’s Vibe Coding? 🛠️
It’s a short, creative coding event where developers explore tools like Cursor, Replit, Gemini Code Assist, GitHub Copilot, and frameworks like Windsurf. The goal is to learn, build something useful, and share it with others.

I used Windsurf, which made building an app feel surprisingly doable.

💊 Why MediTrack?

Missed a pill lately? I built MediTrack to help people—seniors, caregivers, or anyone—stay on top of their medication schedules with ease.

⚙️ My Tech Stack

MediTrack runs on Windsurf, a beginner-friendly full-stack framework with:

Next.js 📱: For seamless frontend and backend.

Prisma 📊: Easy database setup.

TailwindCSS + shadcn/ui 🎨: For a clean, simple design.

Clerk 🔒: For secure logins.

Windsurf handled the complex parts, letting me focus on features.

🚀 What I Built in 3 Days

MediTrack is packed with practical features:

Secure Logins 🔐: Dashboards for patients or caregivers.

Manage Meds ✍️: Add, edit, or delete meds with number-only fields for quantity, refills, and prescribed amounts.

Smart Suggestions 💡: Auto-suggests tablets, capsules, or syrups for common meds via dropdowns.

Auto-Formatting 🖌️: Medicine names in Title Case, dosage times as HH:MM.

Confirm Changes ✅: Prompts when editing names or switching Taken/Missed status.

Track Doses 📋: Monitor quantity, refills, and prescriptions.

Weekly Reports 📧: Summaries sent via SMS or email.

User-Friendly 😊: Clear interface for first-timers.

Mobile Support 📲: Works great on phones and tablets.

📚 What I Learned

This project taught me a lot:

You don't need to be a design expert to build a solid app.
Tools like Windsurf simplify complex tasks.
AI tools help, but your ideas make the difference.

🌱 Why This Matters

After three years away, building MediTrack was a personal win. Vibe Coding pushed me to step out of my data engineering comfort zone, and I’m proud of what I created. It’s rewarding to make something that could help people.

Thinking of trying Windsurf or Copilot? Start with a small project—it’s worth it. Got an idea you’re working on? I’d love to hear about it in the comments! 💬

Disclaimer: This was part of my company’s Vibe Coding challenge. All thoughts are my own.