DEV Community

Bharathvaj
Bharathvaj

Posted on • Originally published at bharathvaj.com

How Generators improves performance ?

In modern software systems, we often work with large datasets — think server logs, customer records, telemetry streams, or product catalogs. Loading everything into memory just to loop over it can easily bring your system to its knees.

The smarter way? Use iterators or generators that process data one piece at a time.


🌍 Real-World Use Case: Processing a Large Log File

Imagine you're analyzing a log file with 1 million entries — stored line-by-line. You want to extract specific events or count errors. Let’s see how to do this efficiently in Python and JavaScript.


❌ Memory-Heavy: Read Entire File in Memory

const fs = require("fs");

const lines = fs.readFileSync("server.log", "utf-8").split("\n");

for (const line of lines) {
  if (line.includes("ERROR")) {
    console.log("Found an error:", line);
  }
}
Enter fullscreen mode Exit fullscreen mode

This loads the entire file into memory, which may be fine for small logs — but dangerous for huge files.

✅ Memory-Efficient: Async Generator

const fs = require("fs");
const readline = require("readline");

async function* readLogLines(filePath) {
  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({ input: fileStream });

  for await (const line of rl) {
    yield line; // yield one line at a time
  }
}

// Process the log
(async () => {
  for await (const line of readLogLines("server.log")) {
    if (line.includes("ERROR")) {
      console.log("Found an error:", line);
    }
  }
})();
Enter fullscreen mode Exit fullscreen mode

While this reads line one after another, making it efficient.

In Python,

def read_log_lines(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line.strip()  # Yield one line at a time

# Process the log
for line in read_log_lines("server.log"):
    if "ERROR" in line:
        print("Found an error:", line)

Enter fullscreen mode Exit fullscreen mode

✅ Key Takeaways

Technique Memory Usage Suitable For
Full Load High Small datasets
Generator/Stream Low Large files, real-time streams, infinite data

🧠 Summary

  • Iterators and generators let you process large data without blowing up memory.
  • Use them when working with:
    • Big files (e.g., logs, exports)
    • API pagination
    • Sensor data
    • Infinite sequences (e.g., timers, streams)
  • And best of all — you still get to use the familiar for loop syntax.

📌 Pro Tip: Always ask: Do I really need the entire dataset in memory? If not, generators might be your best friend.

Dev Diairies image

User Feedback & The Pivot That Saved The Project

🔥 Check out Episode 3 of Dev Diairies, following a successful Hackathon project turned startup.

Watch full video 🎥

Top comments (0)

Build gen AI apps that run anywhere with MongoDB Atlas

Build gen AI apps that run anywhere with MongoDB Atlas

MongoDB Atlas bundles vector search and a flexible document model so developers can build, scale, and run gen AI apps without juggling multiple databases. From LLM to semantic search, Atlas streamlines AI architecture. Start free today.

Start Free

👋 Kindness is contagious

Dive into this thoughtful piece, beloved in the supportive DEV Community. Coders of every background are invited to share and elevate our collective know-how.

A sincere "thank you" can brighten someone's day—leave your appreciation below!

On DEV, sharing knowledge smooths our journey and tightens our community bonds. Enjoyed this? A quick thank you to the author is hugely appreciated.

Okay