Forem: Ajit Kumar

From AWS Key Leak to evnx: The Origin Story of a Developer's Safety Net

Ajit Kumar — Wed, 01 Apr 2026 08:30:06 +0000

How an AWS key leak, years of .env frustrations, and a poorly formatted LLM prompt accidentally built a developer tool.

Part 1: The Accident — An AWS Key Leak That Changed Everything

Every developer has that story. The one they hesitate to tell in interviews, wince at in retrospectives, and quietly reference whenever someone asks why security tooling matters. This is mine.

About a year ago, I was deep in a refactor of an Apache Airflow project at work — my first Airflow project, born from early-career learning and grown organically into a system with 10+ DAGs and a sprawling network of helper Python files. The project had been running for over a year. Scrapers, ETL tasks, one-off experiments that graduated to production — each added in its own folder, each with its own .env file.

At the end of a long day of refactoring, I just wanted to save my work. No ceremony. A quick git add ., a commit, and a push. The project didn't have pre-commit hooks at the time. Within minutes, things started breaking. Services went down. I went digging through backend code for bugs I'd introduced that day, and found nothing. Then I opened my email.

GitHub. AWS. Key revocation notices.

I had accidentally committed one of the nested .env files — one that contained AWS credentials and keys for several backend services. Everything connected to those keys was now broken. I had to explain the incident to my supervisor. I had to trace the blast radius. I had to rotate credentials and restore services one by one.

It was embarrassing. More than that, it was a moment that stuck with me at a deeper level than any code review comment ever had.

The immediate fix was pragmatic: I added pre-commit with secret scanning and moved on. But the underlying problem — the real one — quietly settled into the back of my mind, waiting.

Part 2: Years of `.env` Pain Across Projects

The AWS incident was acute, but the chronic pain had been building for years. Working across 7–8 projects at my previous job, plus personal and consulting work, I kept running into the same categories of friction around .env management — just in different flavors each time.

The .env vs .env.example drift problem. The idea is simple: commit a .env.example with all the keys but no real values, so other developers know what they need. In practice, as a project matures over months, keys get added to .env and never make it into .env.example. New team members clone the repo, fill out the example file, and then hit cryptic runtime errors because the example was out of date. What seems like a one-line discipline problem compounds into real onboarding pain.

Sharing and syncing across a distributed team. Software is a team sport, and every team member needs a working .env.local, .env.staging, or environment-specific variant. Add remote team members to the mix — especially part-timers with high turnover — and you have a constant cycle of sharing, revoking, rotating, and re-syncing credentials. For me, managing this for a team where people were frequently entering and leaving was one of the most draining non-coding tasks in the development workflow.

The wild west of .env formats. There's no formal specification for .env files. Every tool interprets them slightly differently. python-dotenv will happily parse quoted values like KEY="value" and strip the quotes for you. Docker's --env-file parser is stricter and will choke on the same syntax when building images. And then there are the migrations: for one project, I needed to move 30+ variables from a .env file into AWS Secrets Manager, which required a JSON format. Another project needed those same variables pushed into GitHub Actions secrets. Each migration meant careful, highly concentrated copy-paste — the kind where one missed variable or one formatting mistake could mean a broken deployment or a security gap. Nobody should be doing this manually.

All of it — the drift, the syncing, the format fragmentation — was death by a thousand cuts. Manageable, individually. Maddening, collectively.

Part 3: The Poorly Written Prompt That Started It All

Here's where the story takes an unexpected turn.

Sometime after the AWS incident, sitting with all of these accumulated frustrations about .env files, I did what many developers do: I opened an LLM and started typing. I wanted a comprehensive resource — a tutorial covering what .env files are, how to use them correctly, their limitations, and how different languages and ecosystems handle them.

My mistake: I forgot to specify the output format. I assumed it would come back as Markdown.

The LLM returned a beautifully structured HTML document.

I read through it. It was actually quite good. And in a moment of "well, it's just one file," I decided to host it. Bought a domain. Deployed the single HTML page. That became dotenv.space.

I added Google Analytics and Plausible, mostly out of curiosity. Within days, organic search traffic was arriving. People were landing on the page. The data made one thing clear: there was genuine demand for better .env tooling, not just documentation.

That weekend, I decided to dust off my Rust skills — about a year of on-and-off learning — and build a CLI that could actually solve the problems I had been living with. The first working version of what would become evnx came together in three to four days, with LLM assistance bridging the gaps in my Rust knowledge. I iterated on it, fixed bugs, used it on real projects, then did something that revealed the real state of the code: I read it line by line.

Even as a Rust beginner, I could see the issues. The structure wasn't right. The abstractions were leaking. So I started over — not from scratch, but from understanding. The second iteration was better. The third was something I felt good enough to ship.

Part 4: From `dotenv-cli` to `evnx` — A Name Born From Distribution

As the CLI matured through iterations, I turned my attention to distribution — a completely new problem space for me. I wanted the tool to be accessible everywhere developers already were: Cargo for Rust users, pip for Python users, npm for JavaScript developers, Homebrew for macOS, Scoop for Windows, winget for Windows package management, and GitHub Actions for CI/CD pipelines.

Each distribution channel has its own conventions, its own packaging requirements, its own review process. Working through all of them was an education in itself.

When it came to naming, I ran into a constraint: the name I originally had in mind — envX — was already taken across distribution channels. I needed something close, something that captured the spirit of the project.

The solution: evnx. Phonetically adjacent, practically distinct, and backronymed into something meaningful — Environment Variables Next Generation.

The name felt right. It acknowledged the lineage while staking out new territory.

Part 5: Where Things Stand Now

I'm currently in the gap between my last role and whatever comes next — a rare window of time that I've chosen to spend building evnx into something genuinely useful for other developers.

I use it myself. It solves the real problems that led to its creation: validating .env files, keeping .env.example in sync, scanning for accidentally committed secrets, converting between formats (JSON, YAML, Kubernetes, Docker, GitHub Actions, and more), and integrating into CI/CD pipelines via GitHub Actions.

The origin wasn't glamorous. It started with embarrassment, accumulated through years of low-grade frustration, and got catalyzed by a formatting oversight in an LLM prompt. But that's how a lot of useful tools get built — not from grand vision, but from lived pain that finally crosses the threshold of tolerance.

If any of this sounds familiar, I'd love for you to try evnx, file issues, suggest features, and help shape what it becomes.

Get Started with evnx

🌐 Official Website

evnx.dev — Documentation, installation guides, and full feature reference. Start here if you're new to evnx.

📦 Main Repository

github.com/urwithajit9/evnx — The core CLI source code. Open issues, submit pull requests, and follow development here.

Status:
✅ CI Passing •
📦 Latest Release •
🦀 crates.io •
🐍 PyPI •
📦 npm •
📄 MIT License •
🔧 GitHub Action

🧪 Test Repository

github.com/urwithajit9/evnx-test — Sample .env files, test fixtures, and integration test cases. Useful if you want to explore evnx commands against realistic inputs before running it on your own project.

⚙️ GitHub Actions Integration

github.com/urwithajit9/evnx-action — The official evnx GitHub Action. Add .env validation and secret scanning directly to your CI/CD pipeline with a few lines of YAML. Prevent leaks before they reach your repository.

🍺 Homebrew Tap (macOS / Linux)

github.com/urwithajit9/homebrew-evnx — Install evnx on macOS or Linux via Homebrew:

brew tap urwithajit9/evnx
brew install evnx

🪟 Scoop Bucket (Windows)

github.com/urwithajit9/scoop-evnx — Install evnx on Windows via Scoop:

scoop bucket add evnx https://github.com/urwithajit9/scoop-evnx
scoop install evnx

evnx is open source under the MIT License. Contributions, feedback, and stars are all deeply appreciated.

Python 3 & Security: A Deep-Dive Tutorial

Ajit Kumar — Fri, 13 Mar 2026 05:40:09 +0000

Assessment Focus: Python 3 built-ins · Functional tools · Modern Python 3.13 · Security fundamentals
Level: Beginner → Advanced — structured for systematic reading and practice
Time to complete: ~90 minutes reading + practice

Python 3.13 — What Changed and Why It Matters
Input & Safe Data Entry
Core Iteration Built-ins — zip, enumerate, map, filter
Aggregation Built-ins — sum, any, all, min, max
Sorting — sorted, sort, key functions
Lambda Functions & Functional Style
Comprehensions — list, dict, set, generator
Exception Handling — try, except, else, finally
Custom Exceptions & Exception Chaining
Context Managers & Resource Safety
String Security — Formatting, Sanitisation, Validation
Injection Attacks — SQL, Command, Path Traversal
Password & Secret Handling
Cryptography Fundamentals in Python
Authentication — JWT, Sessions, Tokens
Timing Attacks & Side Channels
Dangerous Python Patterns to Avoid
Practice Problems with Full Solutions
Assessment Cheat Sheet

1. Python 3.13 — What Changed and Why It Matters

Python 3.13 (released October 2024) introduced several changes that directly affect how you write safe, idiomatic code. These are the ones most likely to appear in an assessment testing "modern Python" knowledge.

1.1 The New Interactive Interpreter (REPL)

The REPL was completely rewritten using PyPy's pyrepl library:

# Python 3.13 REPL features:
# ✦ Multi-line editing — navigate up/down within a block
# ✦ Syntax highlighting in the terminal
# ✦ Exit with 'exit' (no parentheses needed, though exit() still works)
# ✦ F1 brings up interactive help browser
# ✦ F2 strips output/prompts for clean copying
# ✦ F3 toggles "paste mode" — prevents auto-indentation issues

# How to check your Python version in code:
import sys
print(sys.version)          # '3.13.0 (main, Oct 7 2024, ...)'
print(sys.version_info)     # sys.version_info(major=3, minor=13, ...)
print(sys.version_info >= (3, 13))  # True

1.2 input() — The Security-Critical Change ⚠️

This is the most important Python 3.13 change for security. In all Python versions prior to 3.13, input() in certain contexts could be manipulated. Python 3.13 reinforces the guarantee that input() always returns a plain string — never evaluates it.

# In Python 2, input() was equivalent to eval(raw_input())
# This was a critical security vulnerability:
# user types: __import__('os').system('rm -rf /')
# Python 2 would EXECUTE it!

# Python 3 (all versions including 3.13):
user_input = input("Enter a value: ")
# Always returns a str, NEVER evaluates
# Even if user types: 2 + 2
# You get the STRING "2 + 2", not the integer 4

# Python 3.13 makes this explicit in the docs and adds stricter
# safeguards in the REPL — input() in all modes is now truly raw

# ✅ Safe pattern for numeric input
def get_integer(prompt: str) -> int | None:
    raw = input(prompt)           # always a string
    try:
        return int(raw)           # explicit conversion, may raise ValueError
    except ValueError:
        print(f"'{raw}' is not a valid integer")
        return None

# ✅ Safe pattern with validation
def get_positive_number(prompt: str) -> float:
    while True:
        raw = input(prompt).strip()
        try:
            value = float(raw)
            if value <= 0:
                raise ValueError("Must be positive")
            return value
        except ValueError as e:
            print(f"Invalid input: {e}. Try again.")

# ❌ NEVER do this — eval on input is always wrong
result = eval(input("Enter expression: "))   # arbitrary code execution!

1.3 Free-Threaded Mode (Experimental in 3.13)

# Python 3.13 introduces experimental "free-threaded" mode (--disable-gil)
# This allows true parallel threads without the GIL
# Build with: --disable-gil flag, check with:

import sys
print(sys._is_gil_enabled())   # False in free-threaded build, True otherwise

# Security implication: without GIL, concurrent code needs explicit locks
# previously "safe" singleton patterns break without threading.Lock

# Check at runtime so code is forward-compatible:
import threading
_lock = threading.Lock()

def thread_safe_operation():
    with _lock:
        # critical section — safe in both GIL and free-threaded mode
        pass

1.4 Improved Error Messages

Python 3.13 error messages are significantly better. Knowing what they mean speeds up debugging:

# NameError — now suggests did you mean?
pint = 3.14
# print(pitn)  → NameError: name 'pitn' is not defined. Did you mean: 'pint'?

# AttributeError — now suggests similar attributes
my_list = [1, 2, 3]
# my_list.appnd(4)  → AttributeError: 'list' object has no attribute 'appnd'.
#                      Did you mean: 'append'?

# TypeError — clearer messages about argument counts
def greet(name, greeting):
    return f"{greeting}, {name}!"
# greet("Alice")  → TypeError: greet() missing 1 required positional argument: 'greeting'

# Python 3.13 also shows the exact location of the error with a caret (^)
# x = {'a': 1}
# x['b']['c']  → KeyError with helpful traceback pointing to 'b'

1.5 Type Parameter Syntax (PEP 695, stabilised in 3.13)

# Old style (still valid):
from typing import TypeVar
T = TypeVar('T')
def first(lst: list[T]) -> T:
    return lst[0]

# New style — Python 3.12+ (3.13 polishes this):
def first[T](lst: list[T]) -> T:       # T declared inline
    return lst[0]

type Vector = list[float]               # type alias statement
type Matrix[T] = list[list[T]]          # generic type alias

# These are documentation/tooling hints — still not enforced at runtime
# Use mypy or pyright for static type checking

✏️ Quick Check 1

Q: A junior developer writes result = eval(input("Enter formula: ")). What's the exact security risk?

A: This is Remote Code Execution (RCE). A user can type any Python expression and it will be executed with the full permissions of the running process. For example: __import__('os').system('cat /etc/passwd') or __import__('subprocess').call(['rm','-rf','/']). Even on internal tools, this violates the principle of least privilege and creates an insider threat vector.

2. Input & Safe Data Entry

2.1 The Full input() Contract

# input() always:
# 1. Displays the prompt string to stdout (no newline)
# 2. Reads one line from stdin
# 3. Strips the trailing newline
# 4. Returns a str — NEVER evaluates, NEVER converts

name = input("Name: ")         # "Alice\n" → "Alice"
print(type(name))              # <class 'str'>

# Edge cases
empty = input("Press Enter: ") # user just presses Enter → ""
space = input("Enter: ")       # user types "  " → "  " (spaces preserved!)

2.2 Robust Input Validation Patterns

import re
from typing import Callable, TypeVar

T = TypeVar('T')

def validated_input(
    prompt: str,
    converter: Callable[[str], T],
    validator: Callable[[T], bool] | None = None,
    error_msg: str = "Invalid input",
    max_attempts: int = 3
) -> T:
    """Generic validated input — try up to max_attempts times."""
    for attempt in range(1, max_attempts + 1):
        raw = input(prompt).strip()
        try:
            value = converter(raw)
            if validator is not None and not validator(value):
                raise ValueError(error_msg)
            return value
        except (ValueError, TypeError) as e:
            remaining = max_attempts - attempt
            print(f"  Error: {e}. {remaining} attempt(s) remaining.")
    raise RuntimeError(f"Failed after {max_attempts} attempts")

# Usage examples:
age   = validated_input("Age: ", int, lambda x: 0 < x < 150, "Age must be 1–149")
score = validated_input("Score (0–100): ", float, lambda x: 0 <= x <= 100)
email = validated_input(
    "Email: ",
    str,
    lambda s: bool(re.fullmatch(r'[^@\s]+@[^@\s]+\.[^@\s]+', s)),
    "Not a valid email address"
)

2.3 Security-First Input Rules

import html
import unicodedata

# Rule 1: Strip and limit length BEFORE processing
def safe_input(prompt: str, max_length: int = 256) -> str:
    raw = input(prompt)
    stripped = raw.strip()
    if len(stripped) > max_length:
        raise ValueError(f"Input too long (max {max_length} chars)")
    return stripped

# Rule 2: Normalise Unicode to prevent homograph attacks
# "аdmin" (Cyrillic 'а') looks identical to "admin" (Latin 'a')
def normalise_unicode(s: str) -> str:
    return unicodedata.normalize('NFKC', s)  # normalises lookalike chars

print(normalise_unicode("аdmin") == "admin")   # True — detected!

# Rule 3: HTML-escape before rendering in web contexts
user_name = "<script>alert('xss')</script>"
safe_name = html.escape(user_name)
# "&lt;script&gt;alert(&#x27;xss&#x27;)&lt;/script&gt;"

# Rule 4: Whitelist, don't blacklist
def is_safe_username(s: str) -> bool:
    # Allow ONLY alphanumeric + underscore, 3–32 chars
    return bool(re.fullmatch(r'[a-zA-Z0-9_]{3,32}', s))

# Blacklist approach is always incomplete — attackers find bypasses
def is_safe_username_bad(s: str) -> bool:
    return '<' not in s and '>' not in s   # misses %3C, &#60;, etc.

3. Core Iteration Built-ins — zip, enumerate, map, filter

These are the workhorses of Pythonic code. They all return lazy iterators — they don't compute results until you iterate.

3.1 zip — Parallel Iteration

# zip pairs elements from multiple iterables
names  = ["Alice", "Bob", "Carol"]
scores = [95, 87, 91]
grades = ["A", "B+", "A-"]

# Basic zip — stops at shortest iterable
for name, score in zip(names, scores):
    print(f"{name}: {score}")

# zip returns an iterator — materialise with list() if needed
pairs = list(zip(names, scores))    # [("Alice",95), ("Bob",87), ("Carol",91)]

# zip with * operator — unzip (transpose)
pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
nums, letters = zip(*pairs)         # (1,2,3) and ('a','b','c')

# zip_longest — don't stop at shortest (from itertools)
from itertools import zip_longest
a = [1, 2, 3, 4]
b = ['x', 'y']
list(zip_longest(a, b, fillvalue=None))  # [(1,'x'),(2,'y'),(3,None),(4,None)]

# zip three or more iterables
for name, score, grade in zip(names, scores, grades):
    print(f"{name}: {score} ({grade})")

# zip for dict construction
keys   = ['host', 'port', 'db']
values = ['localhost', 5432, 'mydb']
config = dict(zip(keys, values))
# {'host': 'localhost', 'port': 5432, 'db': 'mydb'}

# Security use: zip for parallel processing with validation
def process_user_data(usernames: list, passwords: list) -> list[dict]:
    if len(usernames) != len(passwords):
        raise ValueError("Mismatched username/password counts")  # security check!
    return [
        {'username': u.strip(), 'hashed': hash_password(p)}
        for u, p in zip(usernames, passwords)
        if u.strip()   # skip empty usernames
    ]

3.2 enumerate — Index + Value

fruits = ["apple", "banana", "cherry", "date"]

# Without enumerate — error-prone
for i in range(len(fruits)):
    print(i, fruits[i])    # fragile — easy to have off-by-one errors

# With enumerate — clean and idiomatic
for i, fruit in enumerate(fruits):
    print(i, fruit)         # 0 apple, 1 banana, ...

# Custom start index
for i, fruit in enumerate(fruits, start=1):
    print(f"{i}. {fruit}")  # 1. apple, 2. banana, ...

# enumerate returns (index, value) tuples — useful for list of dicts
records = [{'name': 'Alice', 'score': 95}, {'name': 'Bob', 'score': 87}]
for idx, record in enumerate(records):
    print(f"Record {idx}: {record['name']} scored {record['score']}")

# Security use: enumerate for audit logging with position context
def validate_config_lines(lines: list[str]) -> list[str]:
    errors = []
    for line_num, line in enumerate(lines, start=1):
        line = line.strip()
        if not line or line.startswith('#'):
            continue   # skip empty lines and comments
        if '=' not in line:
            errors.append(f"Line {line_num}: missing '=' in '{line}'")
        key, _, value = line.partition('=')
        if not key.strip():
            errors.append(f"Line {line_num}: empty key")
        if not value.strip():
            errors.append(f"Line {line_num}: empty value for key '{key.strip()}'")
    return errors

3.3 map — Apply a Function to Every Element

# map(function, iterable) → lazy iterator
numbers = [1, 4, 9, 16, 25]
roots   = list(map(lambda x: x ** 0.5, numbers))  # [1.0, 2.0, 3.0, 4.0, 5.0]

# map with a named function
words      = ["  hello  ", "  world  ", "  python  "]
stripped   = list(map(str.strip, words))   # ['hello', 'world', 'python']
uppercased = list(map(str.upper, words))

# map with multiple iterables — zip-like, stops at shortest
a = [1, 2, 3]
b = [10, 20, 30]
sums = list(map(lambda x, y: x + y, a, b))   # [11, 22, 33]

# map vs list comprehension — map is faster for simple named functions,
# comprehensions are faster for inline expressions and more readable
# Generally prefer comprehensions for clarity:
stripped_lc = [w.strip() for w in words]   # preferred

# Security use: sanitise a batch of user inputs
raw_inputs = ["  Alice  ", "<Bob>", "Carol\n", "  dave@example.com  "]
cleaned    = list(map(lambda s: html.escape(s.strip()), raw_inputs))

3.4 filter — Keep Elements Matching a Predicate

import html

numbers = [-3, -1, 0, 1, 2, 3, 4, 5]

# filter(predicate, iterable) — keeps elements where predicate is True
positives = list(filter(lambda x: x > 0, numbers))   # [1,2,3,4,5]
non_zero  = list(filter(None, numbers))               # [nonzero elements] — filter(None,...) removes falsy

# Equivalent comprehension (preferred for clarity)
positives_lc = [x for x in numbers if x > 0]

# filter with a named function
words = ["hello", "", "world", "  ", "python", None, "!"]
# Keep only truthy non-empty strings
valid = list(filter(lambda w: w and w.strip(), words))   # ['hello','world','python','!']

# Chaining filter + map — pipeline style
data    = ["  admin  ", "", "  Bob  ", "   ", "charlie"]
cleaned = list(map(str.upper, filter(lambda s: s.strip(), data)))
# ['ADMIN', 'BOB', 'CHARLIE']

# Security use: filter out suspicious inputs before processing
ALLOWED_PATTERN = re.compile(r'^[a-zA-Z0-9_@.\-]+$')

def filter_safe_inputs(inputs: list[str]) -> list[str]:
    def is_safe(s: str) -> bool:
        s = s.strip()
        return bool(s) and bool(ALLOWED_PATTERN.match(s))
    return list(filter(is_safe, inputs))

suspicious = ["alice", "bob; DROP TABLE users;--", "carol", "../../etc/passwd"]
safe_only  = filter_safe_inputs(suspicious)   # ['alice', 'carol']

✏️ Practice Problem 3

Problem: You have a list of log entries as strings:

"2024-01-15 ERROR user=admin ip=192.168.1.1 msg=login_failed"
"2024-01-15 INFO  user=alice ip=10.0.0.5 msg=login_ok"
"2024-01-15 ERROR user=root ip=203.0.113.5 msg=login_failed"

Using filter, map, and enumerate, extract all ERROR entries, parse each into a dict, and return a list of (line_number, parsed_dict) tuples.

Solution:

logs = [
    "2024-01-15 ERROR user=admin ip=192.168.1.1 msg=login_failed",
    "2024-01-15 INFO  user=alice ip=10.0.0.5 msg=login_ok",
    "2024-01-15 ERROR user=root ip=203.0.113.5 msg=login_failed",
    "2024-01-15 WARN  user=bob ip=172.16.0.1 msg=rate_limited",
]

def parse_log_line(line: str) -> dict:
    parts  = line.split()
    date   = parts[0]
    level  = parts[1].strip()
    fields = {}
    for part in parts[2:]:
        if '=' in part:
            k, _, v = part.partition('=')
            fields[k] = v
    return {'date': date, 'level': level, **fields}

# Step 1: enumerate gives us (line_number, line)
# Step 2: filter keeps only ERROR lines
# Step 3: map parses each into a dict

numbered = list(enumerate(logs, start=1))                           # (linenum, line) tuples

error_entries = list(filter(
    lambda pair: 'ERROR' in pair[1],
    numbered
))   # [(1, "...ERROR..."), (3, "...ERROR...")]

parsed_errors = list(map(
    lambda pair: (pair[0], parse_log_line(pair[1])),
    error_entries
))

for line_num, entry in parsed_errors:
    print(f"Line {line_num}: {entry}")
# Line 1: {'date': '2024-01-15', 'level': 'ERROR', 'user': 'admin', 'ip': '192.168.1.1', 'msg': 'login_failed'}
# Line 3: {'date': '2024-01-15', 'level': 'ERROR', 'user': 'root', 'ip': '203.0.113.5', 'msg': 'login_failed'}

4. Aggregation Built-ins — sum, any, all, min, max

4.1 sum — Flexible Accumulation

# Basic sum — works on any numeric iterable
numbers = [1, 2, 3, 4, 5]
total   = sum(numbers)          # 15

# start parameter — initial value (crucial for non-zero starts)
sum(numbers, start=100)         # 115
sum(numbers, 0)                 # 15 (positional, but named is clearer)

# Summing non-integers — as long as + is defined
from decimal import Decimal
prices  = [Decimal('9.99'), Decimal('4.99'), Decimal('14.99')]
total   = sum(prices, Decimal('0'))   # Decimal('29.97')

# Sum of a generator expression — memory efficient
total_sq = sum(x**2 for x in range(1000))   # no intermediate list!

# Flatten one level with sum + start=[]  (works but is slow — use itertools.chain)
nested  = [[1,2], [3,4], [5,6]]
flat    = sum(nested, start=[])   # [1,2,3,4,5,6]  — but O(n²)! avoid for large lists
# Better:
import itertools
flat    = list(itertools.chain.from_iterable(nested))   # O(n)

# Security use: count failed attempts from events list
events = [
    {'type': 'login_fail', 'user': 'admin'},
    {'type': 'login_ok', 'user': 'alice'},
    {'type': 'login_fail', 'user': 'root'},
    {'type': 'login_fail', 'user': 'root'},
]
fail_count = sum(1 for e in events if e['type'] == 'login_fail')   # 3
# Or: sum(e['type'] == 'login_fail' for e in events)  — bools are ints (True=1, False=0)

4.2 any and all — Logical Reductions

# any(iterable) — True if ANY element is truthy (short-circuits on first True)
# all(iterable) — True if ALL elements are truthy (short-circuits on first False)

flags = [True, False, True, True]
print(any(flags))   # True  — at least one True
print(all(flags))   # False — not ALL are True

# Edge cases — always know these
any([])   # False — no elements to be True
all([])   # True  — vacuously true (no element is False)

# With generator expressions — lazy, short-circuits
numbers = range(1, 1_000_000)
has_even  = any(n % 2 == 0 for n in numbers)   # stops at n=2
all_pos   = all(n > 0 for n in numbers)         # stops at first n ≤ 0 (never, so scans all)

# Security uses — validation pipelines
def validate_password(password: str) -> tuple[bool, list[str]]:
    checks = [
        (len(password) >= 12,              "at least 12 characters"),
        (any(c.isupper() for c in password), "at least one uppercase letter"),
        (any(c.islower() for c in password), "at least one lowercase letter"),
        (any(c.isdigit() for c in password), "at least one digit"),
        (any(c in '!@#$%^&*()_+-=' for c in password), "at least one special character"),
    ]
    failures = [msg for passed, msg in checks if not passed]
    return (len(failures) == 0, failures)

ok, errors = validate_password("MyPass1!")
print(ok, errors)   # False, ['at least 12 characters']

ok, errors = validate_password("MySecurePass1!")
print(ok, errors)   # True, []

# Check multiple permissions at once
def can_access(user: dict, resource: str) -> bool:
    required_permissions = ['authenticated', 'active', f'can_read_{resource}']
    return all(user.get(p, False) for p in required_permissions)

# Validate a list of IP addresses
import ipaddress
def all_ips_valid(ip_list: list[str]) -> bool:
    def is_valid_ip(ip: str) -> bool:
        try:
            ipaddress.ip_address(ip)
            return True
        except ValueError:
            return False
    return all(is_valid_ip(ip) for ip in ip_list)

4.3 min and max — With Custom Keys

students = [
    {'name': 'Alice', 'score': 95, 'age': 20},
    {'name': 'Bob',   'score': 87, 'age': 22},
    {'name': 'Carol', 'score': 91, 'age': 21},
]

# Default min/max on numbers
print(min(3, 1, 4, 1, 5, 9))   # 1
print(max([3, 1, 4, 1, 5, 9])) # 9

# With key function — transforms each element before comparison
top_scorer    = max(students, key=lambda s: s['score'])   # Alice (95)
youngest      = min(students, key=lambda s: s['age'])     # Alice (20)
name_first    = min(students, key=lambda s: s['name'])    # Alice (alphabetical)

# default parameter — avoids ValueError on empty iterable
winner = max([], default=None)    # None instead of ValueError

# Security use: find the most suspicious IP (most failed attempts)
login_attempts = [
    {'ip': '192.168.1.1', 'failures': 3},
    {'ip': '203.0.113.5', 'failures': 47},
    {'ip': '10.0.0.5',    'failures': 1},
]
most_suspicious = max(login_attempts, key=lambda x: x['failures'])
print(most_suspicious['ip'])   # '203.0.113.5'

5. Sorting — sorted, sort, key functions

5.1 sorted vs .sort()

# sorted() — returns a NEW sorted list, works on any iterable
nums = [3, 1, 4, 1, 5, 9, 2, 6]
asc  = sorted(nums)              # [1,1,2,3,4,5,6,9] — nums unchanged
desc = sorted(nums, reverse=True)  # [9,6,5,4,3,2,1,1]

# .sort() — in-place, returns None, only on lists
nums.sort()              # modifies nums in place
nums.sort(reverse=True)  # in-place descending

# GOTCHA: sorted() returns a list even from non-list iterables
sorted_str  = sorted("python")    # ['h','n','o','p','t','y'] — sorted chars
sorted_set  = sorted({3,1,2})     # [1,2,3]
sorted_dict = sorted({'b':2,'a':1})  # ['a','b'] — sorts KEYS only

# Stability — Python's sort is stable (TimSort)
# Elements with equal keys maintain their original relative order
data = [('Bob', 90), ('Alice', 90), ('Carol', 85)]
sorted(data, key=lambda x: x[1])
# [('Carol',85), ('Bob',90), ('Alice',90)] — Bob before Alice preserved!

5.2 The key Parameter — Transformation Before Comparison

# key= receives a function applied to each element ONCE before comparison
# The key values are compared, not the original elements

# Case-insensitive sort
words = ["Banana", "apple", "Cherry", "date"]
sorted(words, key=str.lower)    # ['apple', 'Banana', 'Cherry', 'date']

# Sort by string length, then alphabetically (multi-key sort)
sorted(words, key=lambda w: (len(w), w.lower()))
# ['date', 'apple', 'Banana', 'Cherry']

# Sort dicts by a field
users = [
    {'name': 'Carol', 'last_login': '2024-01-10', 'role': 'admin'},
    {'name': 'Alice', 'last_login': '2024-01-15', 'role': 'user'},
    {'name': 'Bob',   'last_login': '2024-01-12', 'role': 'user'},
]
by_login  = sorted(users, key=lambda u: u['last_login'], reverse=True)
# Most recent login first: Alice → Bob → Carol

# Sort by role then name (admin first, then alphabetical)
by_role   = sorted(users, key=lambda u: (u['role'] != 'admin', u['name']))
# Carol (admin) first, then Alice, Bob

# operator.itemgetter — faster than lambda for dict/tuple access
from operator import itemgetter, attrgetter
by_name   = sorted(users, key=itemgetter('name'))   # faster than lambda u: u['name']

# attrgetter for objects
from dataclasses import dataclass

@dataclass
class User:
    name: str
    score: int
    active: bool

user_list = [User('Bob', 70, True), User('Alice', 90, False), User('Carol', 80, True)]
active_by_score = sorted(
    filter(attrgetter('active'), user_list),   # filter active users
    key=attrgetter('score'),
    reverse=True
)   # [Carol(80), Bob(70)]

5.3 Security Relevance of Sorting

# Sorting for security reporting
def top_n_attackers(events: list[dict], n: int = 10) -> list[dict]:
    """Return top N IPs by number of failed attempts."""
    from collections import Counter
    failure_counts = Counter(
        e['ip'] for e in events if e.get('event') == 'login_failure'
    )
    # Sort by count descending, then IP for determinism
    return sorted(
        [{'ip': ip, 'failures': count} for ip, count in failure_counts.items()],
        key=lambda x: (-x['failures'], x['ip'])
    )[:n]

# Priority queue for processing security alerts
alerts = [
    {'severity': 3, 'msg': 'Port scan detected'},
    {'severity': 1, 'msg': 'Failed login'},
    {'severity': 5, 'msg': 'Root shell spawned'},
    {'severity': 2, 'msg': 'Unusual outbound traffic'},
]
priority_order = sorted(alerts, key=lambda a: -a['severity'])
# Severity 5 first — critical alerts processed first

6. Lambda Functions & Functional Style

6.1 Lambda Syntax and Scope

# lambda args: expression
# Single expression only — no statements, no assignments, no return keyword

double   = lambda x: x * 2            # ← function object
add      = lambda x, y: x + y
identity = lambda x: x

# Default arguments
greet = lambda name, greeting="Hello": f"{greeting}, {name}!"

# Immediately Invoked Lambda (unusual but valid)
result = (lambda x, y: x ** y)(2, 10)   # 1024

# Lambdas are closures — they capture surrounding variables by reference!
multipliers = [lambda x, n=n: x * n for n in range(5)]
# Note: n=n captures the current value (default arg trick)
# Without n=n: all lambdas would capture n=4 (the final loop value)

# ❌ Common closure bug:
bad_fns = [lambda x: x * n for n in range(5)]
bad_fns[0](1)   # 4! Not 0 — n=4 (final loop value) for all of them

# ✅ Capture loop variable correctly:
good_fns = [lambda x, n=n: x * n for n in range(5)]
good_fns[0](1)  # 0
good_fns[3](1)  # 3

6.2 Lambda with Built-ins

data = [{'id': 3, 'val': 9}, {'id': 1, 'val': 25}, {'id': 2, 'val': 4}]

# sorted with lambda key
sorted(data, key=lambda d: d['val'])       # by val ascending

# max/min with lambda key
max(data, key=lambda d: d['id'])           # id=3

# map with lambda
list(map(lambda d: d['val'] ** 0.5, data)) # [3.0, 5.0, 2.0]

# filter with lambda
list(filter(lambda d: d['val'] > 5, data)) # [{'id':3,'val':9}, {'id':1,'val':25}]

# Chained functional pipeline:
result = list(
    map(
        lambda d: {**d, 'sqrt_val': round(d['val'] ** 0.5, 2)},
        filter(lambda d: d['val'] > 0, data)
    )
)

6.3 When NOT to Use Lambda

# ❌ Too complex — unreadable
process = lambda x: (x.strip().lower() if isinstance(x, str) else str(x)).replace(' ', '_')

# ✅ Use a named function — self-documenting
def normalise_field(x) -> str:
    """Convert to lowercase string with spaces as underscores."""
    if not isinstance(x, str):
        x = str(x)
    return x.strip().lower().replace(' ', '_')

# ❌ Lambda assigned to a name (PEP 8 violation — use def instead)
square = lambda x: x**2       # linters will flag this

# ✅ Named function
def square(x): return x ** 2

# Lambda is best as an inline anonymous key/transform passed to another function
# Not as a named reusable function

7. Comprehensions — list, dict, set, generator

7.1 List Comprehensions

# [expression for item in iterable if condition]

squares        = [x**2 for x in range(10)]
even_squares   = [x**2 for x in range(10) if x % 2 == 0]

# Nested comprehension — matrix operations
matrix    = [[1,2,3],[4,5,6],[7,8,9]]
flat      = [cell for row in matrix for cell in row]   # [1,2,3,4,5,6,7,8,9]
transposed = [[row[i] for row in matrix] for i in range(3)]

# Walrus operator (:=) in comprehension (Python 3.8+)
# Compute and keep only non-None results without calling the function twice
import re
lines = ["error: foo", "warning: bar", "info: baz", "error: qux"]
error_msgs = [
    m.group(1)
    for line in lines
    if (m := re.match(r'error: (.+)', line))
]
# ['foo', 'qux']

7.2 Dict and Set Comprehensions

# Dict comprehension: {key_expr: value_expr for item in iterable if condition}
word    = "mississippi"
freq    = {char: word.count(char) for char in set(word)}
# {'m':1, 'i':4, 's':4, 'p':2}

# Invert a dict (assumes unique values)
original  = {'a': 1, 'b': 2, 'c': 3}
inverted  = {v: k for k, v in original.items()}   # {1:'a', 2:'b', 3:'c'}

# Filter a dict
scores    = {'Alice': 95, 'Bob': 60, 'Carol': 82, 'Dave': 45}
passing   = {name: score for name, score in scores.items() if score >= 70}
# {'Alice': 95, 'Carol': 82}

# Normalize dict keys (security: prevent key confusion attacks)
raw_config = {'  Host ': 'localhost', 'PORT': '5432', 'Db_Name': 'mydb'}
clean_config = {k.strip().lower(): v.strip() for k, v in raw_config.items()}
# {'host': 'localhost', 'port': '5432', 'db_name': 'mydb'}

# Set comprehension
unique_domains = {email.split('@')[1].lower() for email in [
    'Alice@Example.com', 'bob@gmail.com', 'carol@EXAMPLE.COM'
]}
# {'example.com', 'gmail.com'}

7.3 Generator Expressions — Lazy Evaluation

# Generator expressions compute values ONE AT A TIME on demand
# Critical for large datasets — use O(1) memory regardless of size

# Materialise with list() or iterate directly
gen = (x**2 for x in range(10))   # NO computation yet
next(gen)   # 0 — computes first value
next(gen)   # 1 — computes second value

# Pass directly to functions that accept iterables
total = sum(x**2 for x in range(1_000_000))   # O(1) memory!
maximum = max(len(line) for line in open('file.txt'))  # line by line

# Generator with filter
safe_inputs = (
    s.strip()
    for s in raw_inputs
    if s.strip() and len(s.strip()) <= 256
)

# When to use list vs generator:
# - list comprehension: when you need the result multiple times, or need len()
# - generator expression: when you only iterate once, especially for large data

8. Exception Handling — try, except, else, finally

This is one of the most important Python topics for writing robust, secure code. Sloppy exception handling is a major source of security vulnerabilities.

8.1 The Full try/except/else/finally Anatomy

def read_config(path: str) -> dict:
    try:
        # ── Code that might raise exceptions ──
        with open(path, 'r') as f:
            content = f.read()
        config = parse_config(content)
        return config

    except FileNotFoundError:
        # ── Specific exception — handle gracefully ──
        print(f"Config file not found: {path}")
        return {}

    except PermissionError:
        # ── Different specific exception ──
        raise RuntimeError(f"Cannot read config: permission denied") from None

    except (ValueError, KeyError) as e:
        # ── Multiple exceptions in one handler ──
        raise ValueError(f"Malformed config in {path}: {e}") from e

    except Exception as e:
        # ── Catch-all for truly unexpected errors ──
        # Log it, then re-raise — don't silently swallow
        import logging
        logging.exception(f"Unexpected error reading {path}")
        raise   # re-raise the SAME exception with traceback preserved

    else:
        # ── Runs ONLY if no exception was raised in try ──
        # Perfect for code that should only run on success
        # NOT protected by the excepts above it
        log_successful_config_load(path)

    finally:
        # ── ALWAYS runs — exception or not ──
        # Use for guaranteed cleanup
        cleanup_temp_resources()

8.2 Exception Hierarchy — Know What You're Catching

# Built-in exception hierarchy (abbreviated):
# BaseException
# ├── SystemExit           ← sys.exit()
# ├── KeyboardInterrupt    ← Ctrl+C
# ├── GeneratorExit        ← generator .close()
# └── Exception            ← Almost everything else
#     ├── StopIteration    ← end of iteration
#     ├── ArithmeticError
#     │   ├── ZeroDivisionError
#     │   └── OverflowError
#     ├── LookupError
#     │   ├── IndexError   ← list[999]
#     │   └── KeyError     ← dict['missing']
#     ├── TypeError        ← wrong type for operation
#     ├── ValueError       ← right type, wrong value
#     ├── OSError (IOError, EnvironmentError)
#     │   ├── FileNotFoundError
#     │   ├── PermissionError
#     │   └── ConnectionError
#     ├── RuntimeError
#     └── AttributeError

# ❌ WRONG — catching BaseException catches SystemExit and Ctrl+C!
try:
    risky()
except BaseException:
    pass   # Ctrl+C now broken — user can't interrupt!

# ❌ WRONG — bare except has same problem
try:
    risky()
except:
    pass

# ✅ RIGHT — catch Exception if you want "everything except system signals"
try:
    risky()
except Exception as e:
    handle(e)

# ✅ BEST — catch SPECIFIC exceptions
try:
    value = int(user_input)
except ValueError:
    print("Not a valid integer")

8.3 Exception Patterns for Secure Code

# Pattern 1: Fail-safe defaults
def get_user_role(user_id: int, db) -> str:
    try:
        user = db.query(user_id)
        return user.role
    except (DatabaseError, KeyError, AttributeError):
        # On ANY error, return most restrictive role
        return 'anonymous'   # fail-safe: deny access rather than grant it

# Pattern 2: Specific exception for each failure mode
def authenticate(username: str, password: str) -> dict:
    try:
        user = find_user(username)       # may raise UserNotFoundError
        verify_password(password, user.hash)  # may raise InvalidCredentials
        if not user.active:
            raise AccountDisabledError(f"Account {username} is disabled")
        return generate_token(user)
    except (UserNotFoundError, InvalidCredentials):
        # SAME error message for both — prevents username enumeration!
        raise AuthenticationError("Invalid username or password") from None
    except AccountDisabledError:
        raise  # re-raise as-is for specific handling by caller

# Pattern 3: Never swallow exceptions silently
def bad_auth(username, password):
    try:
        return do_auth(username, password)
    except:
        return None   # ← SILENT FAILURE: bug? attack? we'll never know

def good_auth(username, password):
    try:
        return do_auth(username, password)
    except AuthError:
        raise   # propagate to caller
    except Exception as e:
        logging.error(f"Unexpected auth error: {type(e).__name__}: {e}")
        raise RuntimeError("Authentication service unavailable") from e

# Pattern 4: Context-safe exception handling
import contextlib

# Suppress a specific exception (use sparingly)
with contextlib.suppress(FileNotFoundError):
    os.remove(temp_file)   # OK if file doesn't exist — we wanted it gone anyway

8.4 The else Clause — Often Forgotten

# The else clause runs only when try completes WITHOUT raising
# This is subtly different from putting code at the end of try

# Without else — misleading: the except catches setup AND parse errors
try:
    data = fetch_data()
    result = parse(data)   # if this raises, it looks like a fetch error
except NetworkError as e:
    handle_network_error(e)

# With else — clear separation of concerns
try:
    data = fetch_data()     # only network errors caught here
except NetworkError as e:
    handle_network_error(e)
else:
    result = parse(data)    # parse errors propagate naturally (not caught)
    process(result)

9. Custom Exceptions & Exception Chaining

9.1 Designing a Custom Exception Hierarchy

# Base exception for your application — all app errors inherit from this
class AppError(Exception):
    """Base exception for all application errors."""
    def __init__(self, message: str, code: str | None = None):
        super().__init__(message)
        self.code = code
        self.message = message

    def __str__(self):
        if self.code:
            return f"[{self.code}] {self.message}"
        return self.message

# Specific error categories
class ValidationError(AppError):
    """Raised when user input fails validation."""
    pass

class AuthenticationError(AppError):
    """Raised when authentication fails."""
    def __init__(self, message: str = "Invalid credentials"):
        super().__init__(message, code="AUTH_FAILED")

class AuthorizationError(AppError):
    """Raised when authenticated user lacks permission."""
    def __init__(self, resource: str, action: str):
        super().__init__(
            f"Permission denied: cannot {action} on {resource}",
            code="AUTHZ_DENIED"
        )
        self.resource = resource
        self.action   = action

class DatabaseError(AppError):
    """Raised when database operations fail."""
    pass

class RateLimitError(AppError):
    """Raised when rate limit is exceeded."""
    def __init__(self, retry_after: int = 60):
        super().__init__(
            f"Rate limit exceeded. Try again in {retry_after}s.",
            code="RATE_LIMITED"
        )
        self.retry_after = retry_after

# Usage
def get_user(user_id: int) -> dict:
    if not isinstance(user_id, int) or user_id <= 0:
        raise ValidationError(f"Invalid user_id: {user_id!r}", code="INVALID_ID")
    user = db.find(user_id)
    if user is None:
        raise AuthenticationError()
    if not user['active']:
        raise AuthorizationError(resource=f"user/{user_id}", action="access")
    return user

9.2 Exception Chaining — raise ... from ...

# Python 3: exceptions chain automatically (implicit chaining)
try:
    int("abc")
except ValueError:
    raise RuntimeError("Config parsing failed")
# RuntimeError shows "During handling of the above exception, another exception occurred"
# The original ValueError is accessible as __context__

# Explicit chaining: raise X from Y — use when you want to document the cause
def load_config(path: str) -> dict:
    try:
        with open(path) as f:
            return json.load(f)
    except FileNotFoundError as e:
        raise AppError(f"Config not found: {path}") from e   # e is the __cause__
    except json.JSONDecodeError as e:
        raise ValidationError(f"Malformed JSON in {path}: {e}") from e

# Suppress chaining: raise X from None — hides the original
# Use when the original exception contains sensitive information
def authenticate(username: str, password: str):
    try:
        user = db.query("SELECT * FROM users WHERE username=?", (username,))
        if user is None:
            raise AuthenticationError()
    except DatabaseError as e:
        # The DatabaseError might reveal internal schema details
        # raise AuthenticationError() from e  ← LEAKS db info in traceback
        raise AuthenticationError() from None   # clean — hides db error

# Accessing chained exceptions
try:
    load_config("missing.json")
except AppError as e:
    print(e)           # [None] Config not found: missing.json
    print(e.__cause__)  # [Errno 2] No such file or directory: 'missing.json'

9.3 Exception Groups (Python 3.11+)

# ExceptionGroup lets you raise/handle multiple exceptions at once
# Useful for parallel operations that can fail independently

import asyncio

# Raising an ExceptionGroup
def validate_form(data: dict) -> None:
    errors = []
    if not data.get('email'):
        errors.append(ValueError("Email is required"))
    if not data.get('username') or len(data['username']) < 3:
        errors.append(ValueError("Username must be at least 3 characters"))
    if errors:
        raise ExceptionGroup("Form validation failed", errors)

# Handling with except* (Python 3.11+)
try:
    validate_form({'username': 'ab'})
except* ValueError as eg:
    for exc in eg.exceptions:
        print(f"Validation error: {exc}")

10. Context Managers & Resource Safety

10.1 The with Statement Contract

# The context manager protocol: __enter__ and __exit__
# __enter__: called at start, return value is bound to 'as' variable
# __exit__: ALWAYS called — even on exception, return, or Ctrl+C

# File handling — the canonical example
with open('secret.txt', 'r') as f:
    data = f.read()
# f.close() is GUARANTEED even if read() raises an exception

# Multiple context managers (Python 3.10+ — parenthesised form)
with (
    open('input.txt', 'r') as src,
    open('output.txt', 'w') as dst,
    database.transaction() as tx
):
    dst.write(src.read())
    tx.commit()

# Both are equivalent to nested withs:
with open('input.txt', 'r') as src:
    with open('output.txt', 'w') as dst:
        dst.write(src.read())

10.2 contextlib — Building Context Managers

import contextlib
import threading

# @contextmanager — simplest way to build a context manager
@contextlib.contextmanager
def managed_temp_file(suffix: str = '.tmp') -> str:
    import tempfile, os
    path = tempfile.mktemp(suffix=suffix)
    try:
        yield path          # everything BEFORE yield is __enter__
    finally:
        # everything in finally is __exit__ — ALWAYS runs
        if os.path.exists(path):
            os.remove(path)
            print(f"Cleaned up {path}")

with managed_temp_file('.json') as path:
    with open(path, 'w') as f:
        json.dump({'secret': 'data'}, f)
    process(path)
# file deleted automatically even if process() raises

# Context manager for timing
@contextlib.contextmanager
def timer(label: str):
    import time
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        print(f"{label}: {elapsed:.4f}s")

with timer("Database query"):
    results = db.query(sql)

# Context manager for acquiring/releasing locks
@contextlib.contextmanager
def locked(lock: threading.Lock, timeout: float = 5.0):
    acquired = lock.acquire(timeout=timeout)
    if not acquired:
        raise RuntimeError("Could not acquire lock within timeout")
    try:
        yield
    finally:
        lock.release()

10.3 Security-Critical Resource Patterns

import contextlib, secrets, os

# Pattern: ensure sensitive data is cleared from memory
@contextlib.contextmanager
def sensitive_buffer():
    """Clears sensitive bytes from memory on exit."""
    data = bytearray(256)   # mutable — we can zero it out
    try:
        yield data
    finally:
        # Zero out the buffer (prevents cold-boot/memory inspection attacks)
        for i in range(len(data)):
            data[i] = 0

with sensitive_buffer() as buf:
    # read password into buf
    pass
# buf is now all zeros

# Pattern: atomic file write (prevent partial writes on crash)
@contextlib.contextmanager
def atomic_write(path: str, mode: str = 'w'):
    """Write to a temp file, then atomically rename to target."""
    tmp_path = path + f'.{secrets.token_hex(4)}.tmp'
    try:
        with open(tmp_path, mode) as f:
            yield f
        os.replace(tmp_path, path)   # atomic on POSIX systems
    except Exception:
        with contextlib.suppress(FileNotFoundError):
            os.remove(tmp_path)
        raise

with atomic_write('/etc/app/config.json') as f:
    json.dump(new_config, f, indent=2)
# Readers always see a complete file — never a partial write

11. String Security — Formatting, Sanitisation, Validation

11.1 f-strings and Format Security

# f-strings (Python 3.6+) — fast, readable, but NOT safe for untrusted templates
name   = "Alice"
score  = 95.678
status = True

# Basic f-string
print(f"Hello, {name}!")

# Format specifiers
print(f"Score: {score:.2f}")        # 95.68 — 2 decimal places
print(f"Score: {score:08.2f}")      # 00095.68 — padded
print(f"Hex: {255:#010x}")          # 0x000000ff
print(f"Big number: {1_000_000:,}") # 1,000,000

# Expressions inside f-strings
print(f"{'PASS' if status else 'FAIL'}")
print(f"{name.upper()!r}")          # 'ALICE' — !r for repr(), !s for str(), !a for ascii()

# Self-documenting (Python 3.8+) — variable name + value
x = 42
print(f"{x = }")    # x = 42 (includes variable name!)

# ❌ CRITICAL SECURITY: NEVER use f-strings with user-controlled templates
user_template = input("Enter template: ")   # user enters: {config.__class__.__dict__}
# NEVER: eval(f"{user_template}")  or  format_map with attacker-controlled keys

# ✅ Safe template system — use string.Template with $ substitution
from string import Template
t = Template("Hello, $name! Your score is $score.")
result = t.safe_substitute(name=user_name, score=95)  # safe_substitute ignores unknown keys
# User CANNOT access variables beyond what you provide

11.2 String Methods for Validation

# Know all the string methods used in security validation:

s = "Hello, World! 123"

# Character classification (per-char boolean)
"abc".isalpha()        # True — all letters
"123".isdigit()        # True — all digits (0-9)
"abc123".isalnum()     # True — letters and digits
"  ".isspace()         # True — all whitespace
"HELLO".isupper()      # True
"hello".islower()      # True
"Hello World".istitle() # True — title case

# Safer: use re for precise control
import re
re.fullmatch(r'[a-zA-Z]{2,50}', name)    # only letters, 2-50 chars
re.fullmatch(r'\d{4}-\d{2}-\d{2}', date) # YYYY-MM-DD date format
re.fullmatch(r'[^@\s]+@[^@\s]+\.[^@\s]+', email)

# Strip methods
"  hello  ".strip()     # "hello"
"  hello  ".lstrip()    # "hello  "
"  hello  ".rstrip()    # "  hello"
"xxhelloxx".strip('x')  # "hello" — strips specified chars from both ends

# Case methods
"hello".upper()         # "HELLO"
"HELLO".lower()         # "hello"
"hello world".title()   # "Hello World"
"Hello World".swapcase() # "hELLO wORLD"
"hello WORLD".casefold() # "hello world" — more aggressive than lower() for Unicode

# Membership
"hello".startswith("he")    # True
"hello".endswith("lo")      # True
"hello".startswith(("he", "wo"))  # True — tuple of prefixes

# Partitioning
"user=alice".partition('=')     # ('user', '=', 'alice')
"host:port:db".split(':', 1)    # ['host', 'port:db'] — split once only!
"host:port:db".split(':')       # ['host', 'port', 'db']

11.3 Encoding and Unicode Security

# Encoding
"hello".encode('utf-8')          # b'hello'
b"hello".decode('utf-8')         # "hello"
"café".encode('utf-8')           # b'caf\xc3\xa9'

# Always specify encoding explicitly
with open('file.txt', 'r', encoding='utf-8') as f:
    data = f.read()

# Unicode normalisation — CRITICAL for security
import unicodedata

# Homograph attack: different Unicode, same visual appearance
admin1 = "admin"       # all ASCII
admin2 = "\u0430dmin"  # Cyrillic 'а' in first position
print(admin1 == admin2)   # False! Different bytes, same look in many fonts

# Normalise before comparing
def safe_compare(a: str, b: str) -> bool:
    return unicodedata.normalize('NFKC', a) == unicodedata.normalize('NFKC', b)

# NFKC normalisation collapses many lookalike characters
print(safe_compare("admin", "ａｄｍｉｎ"))   # True — fullwidth chars normalised

# HTML escaping (prevent XSS)
import html
user_input = "<script>alert('xss')</script>"
safe       = html.escape(user_input)         # &lt;script&gt;alert(&#x27;xss&#x27;)&lt;/script&gt;
unsafe     = html.unescape(safe)             # back to original (only do this for display)

# URL encoding
from urllib.parse import quote, unquote, urlencode
safe_url = quote("hello world/path")      # "hello%20world%2Fpath"
safe_qs  = urlencode({'name': 'alice', 'q': 'hello world'})

12. Injection Attacks — SQL, Command, Path Traversal

12.1 SQL Injection — The Most Common Critical Vulnerability

import sqlite3

# ❌ VULNERABLE — string interpolation / concatenation
def get_user_bad(username: str) -> dict | None:
    conn = sqlite3.connect("app.db")
    query = f"SELECT * FROM users WHERE username = '{username}'"
    # If username = "'; DROP TABLE users; --"
    # Query becomes: SELECT * FROM users WHERE username = ''; DROP TABLE users; --'
    result = conn.execute(query).fetchone()
    return result

# ❌ Also vulnerable — .format() and % formatting
query = "SELECT * FROM users WHERE id = %s" % user_id    # WRONG
query = "SELECT * FROM users WHERE id = {}".format(user_id)  # WRONG

# ✅ SAFE — parameterized queries (placeholders)
def get_user_safe(username: str) -> dict | None:
    conn = sqlite3.connect("app.db")
    # The ? placeholder is ALWAYS safe — driver handles escaping
    cursor = conn.execute(
        "SELECT id, username, role FROM users WHERE username = ?",
        (username,)    # always a tuple, even for single param
    )
    row = cursor.fetchone()
    if row:
        return dict(zip([d[0] for d in cursor.description], row))
    return None

# ✅ Multiple parameters
def get_active_users_by_role(role: str, min_age: int) -> list:
    conn = sqlite3.connect("app.db")
    cursor = conn.execute(
        "SELECT username, email FROM users WHERE role = ? AND age >= ? AND active = 1",
        (role, min_age)
    )
    return cursor.fetchall()

# Column and table names CANNOT be parameterized — use allowlisting instead
ALLOWED_COLUMNS = {'username', 'email', 'created_at'}
ALLOWED_TABLES  = {'users', 'posts', 'comments'}

def dynamic_query(table: str, column: str) -> str:
    if table not in ALLOWED_TABLES:
        raise ValueError(f"Invalid table: {table!r}")
    if column not in ALLOWED_COLUMNS:
        raise ValueError(f"Invalid column: {column!r}")
    return f"SELECT {column} FROM {table}"   # safe because we allowlisted

12.2 Command Injection

import subprocess, shlex, os

# ❌ CRITICALLY VULNERABLE — shell=True with user input
def ping_bad(host: str) -> str:
    result = os.system(f"ping -c 1 {host}")
    # host = "google.com; cat /etc/passwd"  → executes BOTH commands!

# ❌ Also vulnerable
def ping_bad2(host: str) -> str:
    result = subprocess.run(f"ping -c 1 {host}", shell=True, capture_output=True)

# ✅ SAFE — list form, no shell
def ping_safe(host: str) -> subprocess.CompletedProcess:
    # Validate input first
    import re
    if not re.fullmatch(r'[a-zA-Z0-9.\-]{1,253}', host):
        raise ValueError(f"Invalid hostname: {host!r}")

    # List form: OS passes args directly to execve, bypassing the shell
    return subprocess.run(
        ["ping", "-c", "1", host],   # each arg is a separate list element
        capture_output=True,
        text=True,
        timeout=5              # always set timeout for external commands!
    )

# ✅ If you MUST use shell=True, use shlex.quote (POSIX systems only)
def grep_safe_but_fragile(pattern: str, filepath: str) -> str:
    # shlex.quote wraps the argument in single quotes and escapes internal quotes
    safe_pattern  = shlex.quote(pattern)
    safe_filepath = shlex.quote(filepath)
    result = subprocess.run(
        f"grep {safe_pattern} {safe_filepath}",
        shell=True, capture_output=True, text=True
    )
    return result.stdout
    # Still prefer the list form — shlex.quote has edge cases on Windows

12.3 Path Traversal

import os
from pathlib import Path

BASE_DIR = Path("/var/app/uploads").resolve()

# ❌ VULNERABLE — naive path join
def serve_file_bad(filename: str) -> bytes:
    path = os.path.join("/var/app/uploads", filename)
    # filename = "../../etc/passwd" → "/var/app/uploads/../../etc/passwd" → "/etc/passwd"
    with open(path, 'rb') as f:
        return f.read()

# ✅ SAFE — resolve and validate the final path
def serve_file_safe(filename: str) -> bytes:
    # Resolve symlinks and normalise the path
    requested = (BASE_DIR / filename).resolve()

    # The final resolved path MUST be inside BASE_DIR
    if not str(requested).startswith(str(BASE_DIR)):
        raise PermissionError(f"Access denied: {filename!r}")

    # Additional validation
    if not requested.is_file():
        raise FileNotFoundError(f"File not found: {filename!r}")

    with open(requested, 'rb') as f:
        return f.read()

# Test cases:
# serve_file_safe("report.pdf")         → OK
# serve_file_safe("../../etc/passwd")   → PermissionError
# serve_file_safe("/etc/shadow")        → PermissionError (absolute path attack)

# Using pathlib (cleaner):
def serve_file_pathlib(filename: str) -> bytes:
    safe_path = BASE_DIR / Path(filename).name  # .name strips directory components!
    # Path("../../etc/passwd").name == "passwd" — only the filename!
    if not safe_path.exists():
        raise FileNotFoundError(filename)
    return safe_path.read_bytes()

13. Password & Secret Handling

13.1 Why You Must NEVER Use SHA-256 for Passwords

import hashlib, time

# The fundamental problem: SHA-256 is DESIGNED to be fast
# Performance test (illustrative):
password = "hunter2"

start = time.perf_counter()
for _ in range(1_000_000):
    hashlib.sha256(password.encode()).hexdigest()
elapsed = time.perf_counter() - start
print(f"SHA-256: 1M hashes in {elapsed:.2f}s")
# Result: ~0.5s on a laptop CPU
# Modern GPUs: BILLIONS of SHA-256 hashes per second
# Dictionary attack on a stolen SHA-256 password hash: seconds to minutes

# Password hashing requirements:
# 1. SLOW — intentionally high work factor
# 2. SALTED — unique random value per password (prevents rainbow tables)
# 3. ADAPTIVE — work factor can be increased as hardware improves
# These properties are provided by: bcrypt, scrypt, Argon2

# ❌ WRONG password storage methods:
hashlib.md5(password.encode())          # fast + broken cryptography
hashlib.sha256(password.encode())       # fast (see above)
hashlib.sha256((password + "salt").encode())  # fast — salt doesn't fix speed

13.2 bcrypt — Industry Standard

import bcrypt   # pip install bcrypt

password = "MySecurePassword!123"

# Hash — slow by design (12 rounds = ~250ms on modern hardware)
# rounds: each increment doubles the work — 12 is a good default (2^12 iterations)
hashed = bcrypt.hashpw(
    password.encode('utf-8'),
    bcrypt.gensalt(rounds=12)
)
# b'$2b$12$...' — 60-byte string containing: algorithm, rounds, salt, hash
print(type(hashed))   # bytes
print(len(hashed))    # 60

# Verify — constant-time comparison built in
def check_password(plain: str, hashed_pw: bytes) -> bool:
    return bcrypt.checkpw(plain.encode('utf-8'), hashed_pw)

print(check_password("MySecurePassword!123", hashed))   # True
print(check_password("wrong_password", hashed))          # False

# Storage: store the full hashed bytes (or decode to str for varchar columns)
stored_hash = hashed.decode('utf-8')  # safe to store as string
retrieved   = stored_hash.encode('utf-8')  # convert back for checkpw

13.3 Python's hashlib.scrypt (Built-in)

import hashlib, os

# scrypt — built into Python 3.6+ standard library (no extra package!)
password = b"MySecurePassword!123"
salt     = os.urandom(32)   # ALWAYS use a random salt, never hardcode

# Parameters:
# n = CPU/memory cost (must be power of 2, min 2^14 for interactive)
# r = block size (8 is standard)
# p = parallelisation factor
hashed = hashlib.scrypt(password, salt=salt, n=2**14, r=8, p=1, dklen=64)
# Store salt + hashed together (both needed for verification)

def hash_password_scrypt(plain: str) -> tuple[bytes, bytes]:
    salt   = os.urandom(32)
    hashed = hashlib.scrypt(plain.encode(), salt=salt, n=2**14, r=8, p=1, dklen=64)
    return salt, hashed

def verify_password_scrypt(plain: str, salt: bytes, stored_hash: bytes) -> bool:
    computed = hashlib.scrypt(plain.encode(), salt=salt, n=2**14, r=8, p=1, dklen=64)
    return hmac.compare_digest(computed, stored_hash)   # constant-time!

13.4 Secrets Module — Cryptographically Secure Randomness

import secrets, random

# ❌ WRONG for security purposes
token_bad  = random.randint(0, 2**32)     # NOT cryptographically random!
token_bad2 = str(hash(time.time()))       # predictable, collision-prone

# ✅ CORRECT — secrets module uses OS CSPRNG (urandom)
token_hex  = secrets.token_hex(32)        # 64 hex chars = 256 bits of entropy
token_b64  = secrets.token_urlsafe(32)    # URL-safe base64 = ~43 chars
token_raw  = secrets.token_bytes(32)      # 32 raw bytes

# Use cases:
session_id   = secrets.token_urlsafe(32)  # session token
reset_token  = secrets.token_hex(32)      # password reset link token
csrf_token   = secrets.token_hex(16)      # CSRF protection
api_key      = secrets.token_urlsafe(48)  # API key

# secrets.choice — for generating passwords/pins
import string
alphabet = string.ascii_letters + string.digits + "!@#$%^&*"
# Pick 16 characters at random — each pick uses CSPRNG
password = ''.join(secrets.choice(alphabet) for _ in range(16))

# secrets.compare_digest (wraps hmac.compare_digest) — constant-time comparison
def safe_compare_tokens(a: str, b: str) -> bool:
    return secrets.compare_digest(a, b)   # resists timing attacks

13.5 Environment Variables & Secret Management

import os
from functools import lru_cache

# ❌ NEVER hardcode secrets
DATABASE_URL = "postgresql://admin:password123@db.internal/myapp"
API_KEY      = "sk_live_abcdef123456"

# ✅ Read from environment
DATABASE_URL = os.environ["DATABASE_URL"]     # raises KeyError if missing — GOOD
API_KEY      = os.getenv("API_KEY", "")       # returns "" if missing

# ✅ Validate at startup — fail fast if config is incomplete
def load_config() -> dict:
    required = ["DATABASE_URL", "SECRET_KEY", "REDIS_URL"]
    missing  = [key for key in required if not os.getenv(key)]
    if missing:
        raise EnvironmentError(f"Missing required environment variables: {missing}")
    return {key: os.environ[key] for key in required}

# ✅ Never log secrets
import logging
secret_key = os.environ["SECRET_KEY"]
logging.info(f"Using key: {secret_key[:8]}...")   # log only prefix!
# logging.info(f"Key: {secret_key}")  ← NEVER DO THIS

# ✅ Use python-dotenv for local development only
# from dotenv import load_dotenv
# load_dotenv()   # loads .env file — .env should be in .gitignore!

14. Cryptography Fundamentals in Python

14.1 HMAC — Message Authentication Code

import hmac, hashlib, secrets

# HMAC: proves a message came from someone with the shared key AND wasn't modified
# It does NOT encrypt — the message is visible, but authenticated

key     = secrets.token_bytes(32)   # shared secret key
message = b"user_id=42;role=admin;expires=1700000000"

# Create MAC
mac = hmac.new(key, message, hashlib.sha256).digest()   # 32 bytes

# Verify on receipt — MUST use compare_digest, never ==
def verify_mac(key: bytes, message: bytes, received_mac: bytes) -> bool:
    expected = hmac.new(key, message, hashlib.sha256).digest()
    return hmac.compare_digest(expected, received_mac)  # constant-time!

# Tamper-evident cookies (simplified):
def sign_cookie(data: str, key: bytes) -> str:
    message = data.encode()
    mac     = hmac.new(key, message, hashlib.sha256).hexdigest()
    return f"{data}.{mac}"

def verify_cookie(cookie: str, key: bytes) -> str | None:
    if '.' not in cookie:
        return None
    data, received_mac = cookie.rsplit('.', 1)
    expected_mac = hmac.new(key, data.encode(), hashlib.sha256).hexdigest()
    if hmac.compare_digest(expected_mac, received_mac):
        return data
    return None   # tampered!

14.2 Symmetric Encryption — Fernet (AES-128-CBC)

from cryptography.fernet import Fernet, InvalidToken   # pip install cryptography

# Generate a key — store this securely (env var, secrets manager)
key = Fernet.generate_key()   # 32 random bytes, base64-encoded
# b'gAAAAAB...' — safe to store as a string

f = Fernet(key)

# Encrypt — returns ciphertext (bytes)
plaintext  = b"This is sensitive PII data"
ciphertext = f.encrypt(plaintext)
# Each call produces different ciphertext (random IV) — safe!

# Decrypt
try:
    decrypted = f.decrypt(ciphertext)
    print(decrypted)   # b"This is sensitive PII data"
except InvalidToken:
    print("Decryption failed — data tampered or wrong key!")

# ✅ Fernet provides:
# 1. Confidentiality — AES-128-CBC encryption
# 2. Integrity — HMAC-SHA256 authentication (detects tampering)
# 3. Freshness — includes a timestamp (optional TTL enforcement)

# With TTL (reject tokens older than 60 seconds):
try:
    decrypted = f.decrypt(ciphertext, ttl=60)
except InvalidToken:
    print("Token expired or invalid")

# Rotating keys — key rotation without losing old data
from cryptography.fernet import MultiFernet

old_key = Fernet.generate_key()
new_key = Fernet.generate_key()
mf = MultiFernet([Fernet(new_key), Fernet(old_key)])
# Encrypts with new_key, decrypts with either

14.3 Hashing for Integrity (Non-Password Use)

import hashlib

# SHA-256 IS appropriate for:
# - File integrity verification (is this download uncorrupted?)
# - Content-addressable storage
# - Generating deterministic IDs from content
# NOT appropriate for: passwords (see section 13)

def file_checksum(filepath: str) -> str:
    """Compute SHA-256 hash of a file efficiently."""
    h = hashlib.sha256()
    with open(filepath, 'rb') as f:
        # Read in chunks — handles large files without loading into memory
        for chunk in iter(lambda: f.read(65536), b''):
            h.update(chunk)
    return h.hexdigest()

# Verify a downloaded file
def verify_download(filepath: str, expected_sha256: str) -> bool:
    actual = file_checksum(filepath)
    return hmac.compare_digest(actual, expected_sha256.lower())

# Content-based cache key
def cache_key(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()[:16]  # first 16 hex chars = 64-bit key

# SHA-3 (Keccak) — alternative to SHA-2, different design
sha3_hash = hashlib.sha3_256(b"data").hexdigest()

15. Authentication — JWT, Sessions, Tokens

15.1 JWT Deep Dive

import jwt   # pip install PyJWT
import time
import os
from datetime import datetime, timezone, timedelta

SECRET_KEY = os.environ.get("JWT_SECRET", "change-me-in-production")

# JWT structure: base64url(header).base64url(payload).signature
# Header: {"alg":"HS256","typ":"JWT"}
# Payload: {"sub":"user_123","iat":...,"exp":...}
# Signature: HMAC-SHA256(header.payload, secret)
# CRITICAL: payload is NOT encrypted — only signed — anyone can decode it!

def create_access_token(user_id: str, role: str = "user") -> str:
    now = datetime.now(timezone.utc)
    payload = {
        "sub":  user_id,               # subject — who this token is for
        "role": role,                  # custom claim
        "iat":  now,                   # issued at
        "exp":  now + timedelta(hours=1),   # expiration — ALWAYS set this!
        "jti":  secrets.token_hex(8),  # JWT ID — enables revocation
    }
    return jwt.encode(payload, SECRET_KEY, algorithm="HS256")

def verify_access_token(token: str) -> dict:
    try:
        payload = jwt.decode(
            token,
            SECRET_KEY,
            algorithms=["HS256"],   # ALWAYS specify allowed algorithms!
            # If you omit algorithms=, an attacker can set alg=none in the header
            # and forge any token!
        )
        return payload
    except jwt.ExpiredSignatureError:
        raise AuthenticationError("Token has expired")
    except jwt.InvalidTokenError as e:
        raise AuthenticationError(f"Invalid token: {e}")

# Token refresh pattern
def create_refresh_token(user_id: str) -> str:
    payload = {
        "sub":  user_id,
        "type": "refresh",    # mark as refresh token
        "iat":  datetime.now(timezone.utc),
        "exp":  datetime.now(timezone.utc) + timedelta(days=30),
    }
    return jwt.encode(payload, SECRET_KEY, algorithm="HS256")

15.2 JWT Vulnerabilities and Mitigations

# Vulnerability 1: Algorithm confusion (alg=none)
# An attacker strips the signature and sets alg=none in the header
# MITIGATION: always specify algorithms=["HS256"] in decode()

# Vulnerability 2: HS256 with public RSA key
# If you switch from HS256 to RS256, an attacker might send an HS256 token
# signed with the PUBLIC key (which they can see!)
# MITIGATION: be explicit about which algorithms are accepted per token type

# Vulnerability 3: No expiration
# A stolen token works forever
# MITIGATION: always set exp claim, keep it short (1h for access, 30d for refresh)

# Vulnerability 4: Sensitive data in payload
# JWT payload is just base64 — anyone with the token can read it
bad_payload = {
    "sub": "user_123",
    "password_hash": "$2b$12$...",  # ← NEVER DO THIS
    "ssn": "123-45-6789",           # ← NEVER DO THIS
    "credit_card": "4111111111111111"  # ← NEVER DO THIS
}

# Vulnerability 5: Missing issuer/audience validation
# A token from service A might be replayed against service B
payload = {
    "sub": "user_123",
    "iss": "https://auth.myapp.com",   # issuer
    "aud": "api.myapp.com",            # intended audience
}
jwt.decode(token, key, algorithms=["HS256"],
           issuer="https://auth.myapp.com",
           audience="api.myapp.com")   # validates both claims!

16. Timing Attacks & Side Channels

16.1 How Timing Attacks Work

import time, hmac, secrets

# The vulnerability: early-exit string comparison reveals information
# through execution time differences

# ❌ VULNERABLE comparison
def check_api_key_bad(submitted: str, real_key: str) -> bool:
    if len(submitted) != len(real_key):
        return False   # ← returns FAST for wrong-length input — leaks length!
    for a, b in zip(submitted, real_key):
        if a != b:
            return False   # ← returns faster when early mismatch found
    return True

# An attacker sends millions of requests with different prefixes,
# measures response time, and determines the correct key byte by byte:
#
# "aaaa..." → 0.001ms (fails at pos 0)
# "baaa..." → 0.001ms (fails at pos 0)
# "zaaa..." → 0.002ms (pos 0 matches! fails at pos 1)
# "zbaa..." → 0.002ms (fails at pos 1)
# ...
# Eventually reconstructs the full key through timing measurements

# ✅ SAFE — constant-time comparison regardless of mismatch position
def check_api_key_safe(submitted: str, real_key: str) -> bool:
    return hmac.compare_digest(submitted, real_key)

# Equivalent:
def check_api_key_safe2(submitted: str, real_key: str) -> bool:
    return secrets.compare_digest(submitted, real_key)

# How compare_digest works (simplified):
# It XORs all byte pairs and accumulates the result
# ALWAYS processes ALL bytes — no early exit
# The result is 0 only if ALL bytes match

16.2 User Enumeration — An Information Side Channel

# If your app says "user not found" vs "wrong password",
# attackers can enumerate which usernames exist!

# ❌ VULNERABLE — reveals whether username exists
def login_bad(username: str, password: str) -> dict:
    user = db.find_user(username)
    if user is None:
        return {"error": "User not found"}        # leaks: this username doesn't exist
    if not bcrypt.checkpw(password.encode(), user.password_hash):
        return {"error": "Incorrect password"}    # leaks: this username EXISTS
    return {"token": generate_token(user)}

# ✅ SAFE — same error message for both failure cases
def login_safe(username: str, password: str) -> dict:
    user = db.find_user(username)

    # ALWAYS run bcrypt.checkpw — even when user doesn't exist
    # This prevents timing differences too (bcrypt is slow)
    dummy_hash = b"$2b$12$00000000000000000000000000000000000000000000000000000000"
    if user is None:
        bcrypt.checkpw(password.encode(), dummy_hash)   # waste equal time
        raise AuthenticationError("Invalid username or password")

    if not bcrypt.checkpw(password.encode(), user.password_hash):
        raise AuthenticationError("Invalid username or password")   # SAME message!

    return {"token": generate_token(user)}

# Same principle for password reset:
# ❌ "Email not found" vs "Reset email sent"
# ✅ Always: "If that email is registered, you'll receive a reset link"

16.3 Rate Limiting and Brute Force Protection

import time
from collections import defaultdict
import threading

class RateLimiter:
    """Token bucket rate limiter — thread-safe."""

    def __init__(self, max_requests: int, window_seconds: int):
        self.max    = max_requests
        self.window = window_seconds
        self._data  = defaultdict(list)   # ip → list of timestamps
        self._lock  = threading.Lock()

    def is_allowed(self, key: str) -> tuple[bool, int]:
        """Returns (allowed, seconds_until_reset)."""
        now = time.time()
        cutoff = now - self.window

        with self._lock:
            # Remove expired timestamps
            self._data[key] = [t for t in self._data[key] if t > cutoff]

            if len(self._data[key]) >= self.max:
                oldest = self._data[key][0]
                retry_after = int(self.window - (now - oldest)) + 1
                return False, retry_after

            self._data[key].append(now)
            return True, 0

# Usage in a login endpoint:
limiter = RateLimiter(max_requests=5, window_seconds=300)  # 5 per 5 minutes

def protected_login(ip: str, username: str, password: str) -> dict:
    allowed, retry_after = limiter.is_allowed(ip)
    if not allowed:
        raise RateLimitError(retry_after=retry_after)
    return login_safe(username, password)

17. Dangerous Python Patterns to Avoid

17.1 eval, exec, and compile — Never on Untrusted Input

# eval() executes arbitrary Python expressions
# exec() executes arbitrary Python statements (including imports, class definitions)
# compile() prepares code objects for exec() or eval()

# ❌ CATASTROPHICALLY DANGEROUS
eval(input("Enter expression: "))
exec(user_provided_code_string)

# ❌ Trying to "sandbox" eval — this does NOT work
eval(expr, {"__builtins__": {}})
# Attackers bypass this with:
# ().__class__.__bases__[0].__subclasses__()  — accesses all classes via MRO
# Can still reach file system, network, etc.

# ✅ Safe alternatives for common eval use cases:
# Math expressions:
import ast
def safe_eval_math(expr: str) -> float:
    tree = ast.parse(expr, mode='eval')
    # Whitelist allowed node types
    ALLOWED = {
        ast.Expression, ast.BinOp, ast.UnaryOp, ast.Num, ast.Constant,
        ast.Add, ast.Sub, ast.Mult, ast.Div, ast.Pow, ast.USub
    }
    for node in ast.walk(tree):
        if type(node) not in ALLOWED:
            raise ValueError(f"Forbidden expression: {type(node).__name__}")
    return eval(compile(tree, '<expr>', 'eval'))

safe_eval_math("2 + 2 * 3")    # 8 — OK
# safe_eval_math("__import__('os')")  → ValueError: Forbidden

# Literal data structures only:
import ast
data = ast.literal_eval("{'key': [1, 2, 3], 'flag': True}")   # safe
# Only handles: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None

17.2 Pickle — Deserialisation of Untrusted Data

import pickle, json

# ❌ CRITICAL VULNERABILITY — pickle can execute arbitrary code
class Exploit:
    def __reduce__(self):
        import os
        return (os.system, ("echo OWNED && cat /etc/passwd",))

payload = pickle.dumps(Exploit())
# pickle.loads(payload)  → executes os.system("echo OWNED...")!

# Any user-controlled data run through pickle.loads is RCE
# This includes: uploaded files, cached values from Redis, cookies, API payloads

# ✅ Use JSON for data exchange
safe_data = json.loads(user_provided_json)   # executes no code

# ✅ If you MUST use pickle, sign it first
import hmac, hashlib

def safe_pickle_dumps(obj: object, key: bytes) -> bytes:
    data = pickle.dumps(obj)
    mac  = hmac.new(key, data, hashlib.sha256).digest()
    return mac + data   # prepend MAC

def safe_pickle_loads(signed_data: bytes, key: bytes) -> object:
    mac, data = signed_data[:32], signed_data[32:]
    expected  = hmac.new(key, data, hashlib.sha256).digest()
    if not hmac.compare_digest(mac, expected):
        raise ValueError("Pickle data tampered or invalid key")
    return pickle.loads(data)   # only safe because we verified the MAC

# Alternatives to pickle: json, msgpack, protobuf, orjson

17.3 Other Dangerous Patterns

import yaml   # PyYAML

# ❌ yaml.load on untrusted data — similar to pickle!
data = yaml.load(user_input, Loader=yaml.Loader)   # DANGEROUS — can call constructors

# ✅ Use safe loader
data = yaml.safe_load(user_input)   # only basic Python types, no constructors
data = yaml.load(user_input, Loader=yaml.SafeLoader)  # explicit

# ❌ assert for security checks — disabled with -O flag!
assert user.is_authenticated, "Not authenticated"  # COMPLETELY BYPASSED with python -O

# ✅ Use proper exceptions
if not user.is_authenticated:
    raise AuthenticationError("Authentication required")

# ❌ Logging sensitive data
import logging
logging.debug(f"Processing request: user={user.username} password={password}")
logging.info(f"Auth token: {token}")

# ✅ Log what's needed, redact the rest
logging.info(f"Auth attempt for user: {user.username}")
logging.debug(f"Token prefix: {token[:8]}...")

# ❌ Wildcard imports in security-sensitive code
from crypto_utils import *   # What did you import? Can attackers control it?

# ✅ Explicit imports
from crypto_utils import encrypt, decrypt, verify_mac

# ❌ Mutable default arguments (a Python gotcha)
def add_to_blacklist(ip: str, blacklist: list = []) -> list:  # shared between calls!
    blacklist.append(ip)
    return blacklist

# ✅ Use None and create fresh default each call
def add_to_blacklist(ip: str, blacklist: list | None = None) -> list:
    if blacklist is None:
        blacklist = []
    blacklist.append(ip)
    return blacklist

18. Practice Problems with Full Solutions

Problem A — Secure User Registration Pipeline

Problem: Build a complete register_user() function that validates all inputs using built-ins (any, all, filter, map), hashes the password, and raises specific custom exceptions.

import re, bcrypt, secrets
from dataclasses import dataclass

class RegistrationError(Exception):
    def __init__(self, message: str, fields: list[str]):
        super().__init__(message)
        self.fields = fields

@dataclass
class NewUser:
    username: str
    email:    str
    password_hash: bytes

def register_user(username: str, email: str, password: str, confirm: str) -> NewUser:
    errors = []

    # Validate username
    if not re.fullmatch(r'[a-zA-Z0-9_]{3,32}', username):
        errors.append("username: 3–32 alphanumeric chars/underscores only")

    # Validate email (simplified)
    if not re.fullmatch(r'[^@\s]+@[^@\s]+\.[^@\s]+', email.strip()):
        errors.append("email: invalid format")

    # Validate password using any() and all()
    password_rules = [
        (len(password) >= 12,                          "at least 12 characters"),
        (any(c.isupper() for c in password),           "at least one uppercase letter"),
        (any(c.islower() for c in password),           "at least one lowercase letter"),
        (any(c.isdigit() for c in password),           "at least one digit"),
        (any(c in '!@#$%^&*()_+-=' for c in password),"at least one special character"),
    ]
    pw_errors = [msg for ok, msg in password_rules if not ok]
    if pw_errors:
        errors.append(f"password: requires {', '.join(pw_errors)}")

    # Confirm password matches
    if password != confirm:
        errors.append("confirm_password: passwords do not match")

    if errors:
        raise RegistrationError("Registration failed", fields=errors)

    # Hash password — only after all validation passes
    hashed = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt(rounds=12))

    return NewUser(
        username=username.strip(),
        email=email.strip().lower(),
        password_hash=hashed
    )

# Test it:
try:
    user = register_user("alice_99", "alice@example.com", "MyPass!2024ok", "MyPass!2024ok")
    print(f"Registered: {user.username}")
except RegistrationError as e:
    for field_error in e.fields:
        print(f"  ✗ {field_error}")

Problem B — Log Parser with Functional Built-ins

Problem: Parse a list of access log entries and produce a security report: top-5 IPs with most 4xx errors, sorted by count descending.

from collections import Counter

logs = [
    "192.168.1.1 - GET /login 200",
    "203.0.113.5 - POST /login 401",
    "203.0.113.5 - POST /login 401",
    "203.0.113.5 - GET /admin 403",
    "10.0.0.1 - GET /data 200",
    "203.0.113.5 - GET /etc/passwd 404",
    "192.168.1.1 - GET /api 401",
    "198.51.100.1 - POST /login 401",
    "198.51.100.1 - POST /login 401",
    "203.0.113.5 - GET /wp-admin 404",
]

def parse_entry(line: str) -> dict:
    parts = line.split()
    return {
        'ip':     parts[0],
        'method': parts[2],
        'path':   parts[3],
        'status': int(parts[4]),
    }

def security_report(logs: list[str], top_n: int = 5) -> list[dict]:
    # 1. Parse all entries
    parsed = list(map(parse_entry, logs))

    # 2. Filter to 4xx status codes only
    client_errors = list(filter(lambda e: 400 <= e['status'] < 500, parsed))

    # 3. Count errors per IP
    ip_counts = Counter(e['ip'] for e in client_errors)

    # 4. Sort by count descending, then IP for stability
    top_ips = sorted(
        ip_counts.items(),
        key=lambda pair: (-pair[1], pair[0])   # negative count → descending
    )[:top_n]

    # 5. Build report with enumerate for rank
    return [
        {'rank': rank, 'ip': ip, 'error_count': count}
        for rank, (ip, count) in enumerate(top_ips, start=1)
    ]

report = security_report(logs)
for entry in report:
    print(f"#{entry['rank']}: {entry['ip']} — {entry['error_count']} errors")
# #1: 203.0.113.5 — 4 errors
# #2: 192.168.1.1 — 1 errors
# #3: 198.51.100.1 — 2 errors  ← wait, let's sort correctly

Problem C — Exception Chain: Database Config Loader

Problem: Write a load_database_config() function that reads from an env var, parses it, and uses proper exception chaining to wrap lower-level errors in application-level errors.

import os, re
from urllib.parse import urlparse

class ConfigError(Exception):
    pass

class DatabaseConfig:
    def __init__(self, host, port, dbname, user, password):
        self.host     = host
        self.port     = port
        self.dbname   = dbname
        self.user     = user
        self.password = password

    def __repr__(self):
        return f"DatabaseConfig(host={self.host!r}, port={self.port}, db={self.dbname!r}, user={self.user!r})"

def load_database_config() -> DatabaseConfig:
    # Step 1: Get environment variable
    try:
        url_str = os.environ["DATABASE_URL"]
    except KeyError as e:
        raise ConfigError("DATABASE_URL environment variable is not set") from e

    # Step 2: Parse the URL
    try:
        url = urlparse(url_str)
        if url.scheme not in ('postgresql', 'postgres', 'mysql'):
            raise ValueError(f"Unsupported scheme: {url.scheme!r}")

        host     = url.hostname
        port     = url.port or (5432 if 'postgres' in url.scheme else 3306)
        dbname   = url.path.lstrip('/')
        user     = url.username
        password = url.password

        # Validate required components
        missing = [name for name, val in [
            ('host', host), ('dbname', dbname), ('user', user), ('password', password)
        ] if not val]

        if missing:
            raise ValueError(f"Missing components: {', '.join(missing)}")

    except ValueError as e:
        raise ConfigError(f"Invalid DATABASE_URL format: {e}") from e

    # Step 3: Validate port range
    if not (1 <= port <= 65535):
        raise ConfigError(f"Invalid port number: {port}")

    return DatabaseConfig(host=host, port=port, dbname=dbname, user=user, password=password)

# Test:
os.environ["DATABASE_URL"] = "postgresql://admin:secret@db.local:5432/myapp"
try:
    cfg = load_database_config()
    print(cfg)
except ConfigError as e:
    print(f"Config error: {e}")
    if e.__cause__:
        print(f"  Caused by: {e.__cause__}")

Problem D — Custom Context Manager: Audit Logging

Problem: Write a context manager audit_action() that logs the start, end, duration, and success/failure of any operation, writing to a tamper-evident audit log.

import contextlib, time, hmac, hashlib, json, os
from datetime import datetime, timezone

AUDIT_KEY = secrets.token_bytes(32)   # in production: load from secure store

def sign_record(record: dict, key: bytes) -> str:
    """Create a signed audit record."""
    payload = json.dumps(record, sort_keys=True, default=str).encode()
    mac = hmac.new(key, payload, hashlib.sha256).hexdigest()
    return json.dumps({**record, '_mac': mac}, default=str)

@contextlib.contextmanager
def audit_action(action: str, user_id: str, resource: str, log_file: str = "audit.log"):
    record = {
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'action':    action,
        'user_id':   user_id,
        'resource':  resource,
        'status':    'started',
    }
    start = time.perf_counter()

    try:
        yield   # run the protected code

        record['status']   = 'success'
        record['duration'] = round(time.perf_counter() - start, 4)

    except Exception as e:
        record['status']      = 'failure'
        record['error_type']  = type(e).__name__
        record['error_msg']   = str(e)
        record['duration']    = round(time.perf_counter() - start, 4)
        raise   # re-raise — don't swallow

    finally:
        # Always write the audit log entry
        signed = sign_record(record, AUDIT_KEY)
        with open(log_file, 'a') as f:
            f.write(signed + '\n')

# Usage:
with audit_action("file_download", user_id="u123", resource="/reports/q4.pdf"):
    # simulate file serving
    time.sleep(0.01)
    print("File served")

try:
    with audit_action("admin_panel", user_id="u456", resource="/admin"):
        raise PermissionError("Insufficient privileges")
except PermissionError as e:
    print(f"Blocked: {e}")

Problem E — Functional Pipeline: Security Event Processor

Problem: Using zip, enumerate, sorted, filter, map, any, all, and sum, build a pipeline that scores and prioritises security events.

events = [
    {'type': 'login_fail',    'ip': '203.0.113.5', 'user': 'root',  'count': 50},
    {'type': 'login_ok',      'ip': '10.0.0.1',    'user': 'alice', 'count': 1},
    {'type': 'port_scan',     'ip': '203.0.113.5', 'user': None,    'count': 1000},
    {'type': 'data_exfil',    'ip': '192.168.1.50','user': 'bob',   'count': 3},
    {'type': 'login_fail',    'ip': '198.51.100.1','user': 'admin', 'count': 12},
    {'type': 'config_change', 'ip': '10.0.0.5',    'user': 'carol', 'count': 1},
]

SEVERITY = {
    'login_fail':    2,
    'port_scan':     3,
    'data_exfil':    5,
    'config_change': 3,
    'login_ok':      0,
}
HIGH_VALUE_USERS = {'root', 'admin', 'carol'}

def score_event(e: dict) -> int:
    base     = SEVERITY.get(e['type'], 1)
    count_factor = min(e['count'] // 10, 10)   # cap at 10
    user_bonus   = 2 if e.get('user') in HIGH_VALUE_USERS else 0
    return base + count_factor + user_bonus

# 1. Filter out benign events
suspicious = list(filter(lambda e: SEVERITY.get(e['type'], 1) > 0, events))

# 2. Score and annotate each event using map
scored = list(map(lambda e: {**e, 'score': score_event(e)}, suspicious))

# 3. Sort by score descending
prioritised = sorted(scored, key=lambda e: -e['score'])

# 4. Enumerate with rank for the report
report = [(rank, e) for rank, e in enumerate(prioritised, start=1)]

# 5. Summary stats using sum and any/all
total_score  = sum(e['score'] for e in scored)
has_critical = any(e['score'] >= 10 for e in scored)
all_internal = all(e['ip'].startswith('10.') for e in scored)

print("=== SECURITY EVENT REPORT ===")
for rank, event in report:
    print(f"  #{rank} [{event['score']:2d}] {event['type']:<15} ip={event['ip']:<15} user={event.get('user','N/A')}")

print(f"\nTotal risk score: {total_score}")
print(f"Critical events present: {has_critical}")
print(f"All internal: {all_internal}")

# 6. Group by IP using zip and dict
ips     = [e['ip'] for e in scored]
scores  = [e['score'] for e in scored]
from collections import defaultdict
ip_risk = defaultdict(int)
for ip, score in zip(ips, scores):
    ip_risk[ip] += score

worst_ip = max(ip_risk, key=ip_risk.get)
print(f"\nHighest risk IP: {worst_ip} (total score {ip_risk[worst_ip]})")

19. Assessment Cheat Sheet

Built-in Functions — Signatures and Returns

Function	Signature	Returns	Notes
`zip`	`zip(*iterables)`	lazy iterator of tuples	stops at shortest
`enumerate`	`enumerate(it, start=0)`	lazy iterator of (int, item)	start defaults to 0
`map`	`map(fn, *iterables)`	lazy iterator	multiple iterables = zip-like
`filter`	`filter(fn, iterable)`	lazy iterator	`filter(None, it)` removes falsy
`sum`	`sum(it, start=0)`	number	`start=[]` flattens (slow!)
`any`	`any(iterable)`	bool	short-circuits on first True
`all`	`all(iterable)`	bool	short-circuits on first False
`min`	`min(it, key=None, default=...)`	element	raises on empty without default
`max`	`max(it, key=None, default=...)`	element	key transforms before comparing
`sorted`	`sorted(it, key=None, reverse=False)`	new list	stable, works on any iterable
`reversed`	`reversed(seq)`	lazy iterator	requires sequence (has len+index)

Exception Handling Rules

try:
    ...          # might raise
except Specific as e:
    ...          # handle one type
except (A, B) as e:
    ...          # handle multiple
except Exception as e:
    ...          # catch-all (NOT BaseException!)
    raise        # re-raise without losing traceback
else:
    ...          # runs ONLY if try succeeded — not protected by excepts
finally:
    ...          # ALWAYS runs — cleanup goes here

Exception Chaining

Syntax	Effect	Use when
`raise X from Y`	Sets `X.__cause__ = Y`	Y is the documented root cause
`raise X from None`	Suppresses chain	Original reveals sensitive info
`raise X` (inside except)	Sets `X.__context__ = Y`	Automatic implicit chaining
`raise` (bare)	Re-raises current	Want to log but propagate

Security Quick Reference

Risk	Vulnerable Code	Safe Code
SQL Injection	`f"WHERE name='{name}'"`	`cursor.execute("WHERE name=?", (name,))`
Command Injection	`os.system(f"ls {path}")`	`subprocess.run(["ls", path])`
Path Traversal	`open(base + user_path)`	Resolve + verify prefix
Password Storage	`sha256(password)`	`bcrypt.hashpw(pw, gensalt(12))`
Random Tokens	`random.randint(...)`	`secrets.token_hex(32)`
Code Execution	`eval(user_input)`	`ast.literal_eval(...)`
Pickle RCE	`pickle.loads(untrusted)`	`json.loads(untrusted)`
Token Comparison	`a == b`	`hmac.compare_digest(a, b)`
User Enumeration	Different error per case	Same error always
Secret Storage	Hardcoded in source	`os.environ["SECRET"]`

Python 3.13 Specifics

Feature	Details
`input()`	Always returns `str`, never evaluates — reinforced in 3.13
New REPL	Multi-line edit, F1=help, F2=clean copy, F3=paste mode
Free-threaded	Experimental `--disable-gil` build; use explicit locks
Error messages	`NameError`/`AttributeError` now suggest corrections
`type X = ...`	Type alias statement (stabilised from 3.12)
`def fn[T](...)`	Inline type parameter syntax

Lambda Patterns

# ✅ Use lambda for: short, inline, anonymous key/transform functions
sorted(data, key=lambda d: (d['priority'], d['name']))
filter(lambda x: x > 0, data)
max(users, key=lambda u: u.score)

# ❌ Avoid lambda for: complex logic, named reusable functions, multi-line
square = lambda x: x**2   # use def instead

# Closure gotcha — capture loop variable
# ❌ Bug:    [lambda x: x*n for n in range(5)]
# ✅ Fixed:  [lambda x, n=n: x*n for n in range(5)]

Common Mistakes in Assessments

any([]) returns False; all([]) returns True — know these edge cases
zip stops at shortest — use zip_longest if you want all elements
map/filter return iterators — wrap in list() if you need to index or iterate multiple times
sorted() returns a list; .sort() returns None — don't do x = lst.sort()
lambda in a loop captures the variable, not its value — use default argument n=n
Bare except: catches SystemExit and Ctrl+C — always use except Exception:
else in try/except runs only when NO exception occurred — not the same as code after try block
raise from None suppresses exception chain — know when to use it
input() always returns str — explicit conversion required: int(input("N: "))
hmac.compare_digest not == for all security-sensitive comparisons

PyTorch Tensors & Matrix Operations: A Deep-Dive Tutorial

Ajit Kumar — Fri, 13 Mar 2026 04:45:42 +0000

Assessment Focus: Python 3 · PyTorch · Matrix manipulations (not model training)
Level: Beginner → Advanced — read end-to-end before your test
Time to complete: ~90 minutes reading + practice

What is PyTorch and Why Does It Matter?
Tensor Fundamentals — Creation & Inspection
Data Types, Devices & Casting
Indexing, Slicing & Boolean Masking
Reshaping, Views & Squeezing
Element-wise Operations & Reductions
Broadcasting — The Silent Multiplier
Matrix Multiplication — The Core Operation
Batched Matrix Operations
Einstein Summation (einsum)
Linear Algebra — linalg Module
Advanced Indexing — gather & scatter
Memory Layout, Strides & Contiguity
Practice Problems with Full Solutions
Assessment Cheat Sheet

1. What is PyTorch and Why Does It Matter?

PyTorch is a numerical computation library built on top of C++/CUDA, exposing a Python API that feels like NumPy but with two killer features:

GPU acceleration — tensors can live on CUDA devices and perform operations at GPU speed
Autograd — automatic differentiation through any computation graph (used for training, but not the focus of matrix-only assessments)

For this assessment, think of PyTorch as NumPy with superpowers: the same mental model of n-dimensional arrays, but with broadcasting, batched operations, and linear algebra built in at native speed.

import torch
import torch.nn.functional as F

# Confirm version
print(torch.__version__)   # e.g. 2.2.0
print(torch.cuda.is_available())   # True if GPU available

Key mindset: Every operation in PyTorch either returns a new tensor (functional) or modifies in-place (trailing underscore, e.g. add_). Always know which one you're calling.

2. Tensor Fundamentals — Creation & Inspection

2.1 Ways to Create Tensors

Understanding how a tensor is created determines whether it shares memory with Python/NumPy data or owns its own memory.

import torch
import numpy as np

# ── From Python data ────────────────────────────────────────────
a = torch.tensor([1.0, 2.0, 3.0])        # copies data → always safe
b = torch.tensor([[1, 2], [3, 4]])         # 2D from nested list

# ── Sharing memory with NumPy ───────────────────────────────────
arr = np.array([1.0, 2.0, 3.0])
t   = torch.from_numpy(arr)       # SHARES memory! mutation propagates both ways
t2  = torch.as_tensor(arr)        # same — shares when possible
t3  = torch.tensor(arr)           # COPIES — fully independent

arr[0] = 99.0
print(t[0])   # tensor(99.) — changed!
print(t3[0])  # tensor(1.)  — unchanged

# ── Structured tensors ──────────────────────────────────────────
zeros    = torch.zeros(3, 4)               # all 0.0, shape (3,4)
ones     = torch.ones(2, 3, 4)            # all 1.0, shape (2,3,4)
eye      = torch.eye(4)                   # 4×4 identity matrix
empty    = torch.empty(3, 3)              # uninitialized (garbage values!)
full     = torch.full((2, 3), fill_value=7.0)  # fill with 7.0

# ── Sequence tensors ────────────────────────────────────────────
arange   = torch.arange(0, 10, step=2)   # [0, 2, 4, 6, 8]
linspace = torch.linspace(0, 1, steps=5) # [0.0, 0.25, 0.5, 0.75, 1.0]

# ── Random tensors ──────────────────────────────────────────────
torch.manual_seed(42)                   # reproducibility
uniform  = torch.rand(3, 3)             # Uniform[0, 1)
normal   = torch.randn(3, 3)            # Normal(0, 1)
randint  = torch.randint(0, 10, (3, 3)) # integers in [0, 10)
perm     = torch.randperm(8)            # random permutation of [0..7]

# ── Tensors like another tensor (same shape/dtype/device) ───────
x = torch.randn(2, 3)
torch.zeros_like(x)
torch.ones_like(x)
torch.rand_like(x)
torch.full_like(x, fill_value=5.0)

2.2 Inspecting a Tensor

Every tensor carries metadata you should query habitually:

t = torch.randn(3, 4, 5)

print(t.shape)       # torch.Size([3, 4, 5]) — like a tuple
print(t.size())      # same as .shape
print(t.size(0))     # 3 — size of dimension 0
print(t.ndim)        # 3 — number of dimensions
print(t.dtype)       # torch.float32
print(t.device)      # device(type='cpu') or device(type='cuda', index=0)
print(t.numel())     # 60 — total element count (3 * 4 * 5)
print(t.is_contiguous())  # True (freshly created tensors are contiguous)
print(t.stride())    # (20, 5, 1) — elements to skip to reach next in each dim

✏️ Quick Check 2.2

Q: What does torch.empty(3, 3) contain?

A: Uninitialized memory — whatever bytes happen to be at those addresses. It's the fastest creation method (no fill), but never use its values without first writing to the tensor. Use torch.zeros() when you need a defined starting state.

3. Data Types, Devices & Casting

3.1 The dtype Hierarchy

# Floating point (most ML operations use these)
torch.float16  # fp16 — half precision (2 bytes)
torch.float32  # fp32 — DEFAULT for tensors (4 bytes)
torch.float64  # fp64 — double precision (8 bytes)
torch.bfloat16 # brain float — better range than fp16, same size

# Integer
torch.int8     # (1 byte, signed)
torch.int16    # (2 bytes)
torch.int32    # (4 bytes)
torch.int64    # (8 bytes) — DEFAULT for integer tensors

# Boolean
torch.bool     # True/False (1 byte)

# Explicit dtype at creation
t = torch.ones(3, dtype=torch.float64)
i = torch.tensor([1, 2, 3], dtype=torch.int32)

# Casting (creates a new tensor with converted values)
t_fp32  = t.float()           # → float32
t_fp16  = t.half()            # → float16
t_fp64  = t.double()          # → float64
t_int   = t.long()            # → int64
t_bool  = t.bool()            # → bool

# Generic cast
t.to(torch.float16)
t.to(dtype=torch.int32)

3.2 Devices

cpu_tensor = torch.randn(3, 3)                           # on CPU (default)
gpu_tensor = torch.randn(3, 3, device='cuda')            # on GPU
gpu_tensor = torch.randn(3, 3, device='cuda:0')          # explicit GPU index

# Moving between devices
on_gpu = cpu_tensor.to('cuda')        # CPU → GPU
on_cpu = gpu_tensor.to('cpu')         # GPU → CPU
on_gpu = cpu_tensor.cuda()            # shorthand
on_cpu = gpu_tensor.cpu()             # shorthand

# Best practice: use device variable so code works on CPU or GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
t = torch.randn(3, 3, device=device)

Assessment rule: All tensors in an operation must be on the same device and use compatible dtypes. Mixing CPU + GPU, or int + float in some operations, raises errors.

3.3 Mixed Operations & Promotion

a = torch.tensor([1.0])    # float32
b = torch.tensor([2.0], dtype=torch.float64)

# PyTorch promotes to the higher-precision type
c = a + b   # float64 — float32 was promoted

# Integer + float → float
d = torch.tensor([1]) + torch.tensor([1.5])  # float32

✏️ Quick Check 3

Q: You have a = torch.randn(3) (float32) and b = torch.randint(0, 5, (3,)) (int64). What happens when you compute a + b?

A: PyTorch promotes int64 to float32, so the result is float32. The rule is: if one operand is floating point, the result is floating point.

4. Indexing, Slicing & Boolean Masking

This is one of the most tested areas because the distinction between views (shared memory) and copies (independent memory) is subtle and important.

4.1 Basic Indexing and Slicing

t = torch.arange(24).reshape(2, 3, 4)
# Shape: (2, 3, 4) — think: 2 matrices, each 3 rows × 4 cols
# t[batch, row, col]

# ── Single-element indexing ─────────────────────────────────────
t[0]          # first matrix → shape (3, 4)
t[-1]         # last along dim 0 → shape (3, 4)
t[0, 1]       # row 1 of first matrix → shape (4,)
t[0, 1, 2]    # scalar element → tensor(6)
t[0][1][2]    # equivalent but slower (three indexing ops)

# ── Slicing (Python-style, half-open intervals) ─────────────────
t[:, 0, :]    # all batches, row 0, all cols → shape (2, 4)
t[..., -1]    # last column across all dims → shape (2, 3)
t[:, 1:3, :]  # rows 1 and 2 from both batches → shape (2, 2, 4)
t[:, ::2, :]  # every other row → shape (2, 2, 4)

# The Ellipsis (...) fills in as many : as needed
# t[..., 2] ≡ t[:, :, 2] for a 3D tensor

4.2 Views vs Copies — Critical Distinction

t = torch.arange(12, dtype=torch.float32).reshape(3, 4)
# [[0,1,2,3],[4,5,6,7],[8,9,10,11]]

# ── SLICES return VIEWS (shared memory) ────────────────────────
s = t[0, :]     # first row — this is a VIEW
s[0] = 999.0    # modifies t as well!
print(t[0, 0])  # tensor(999.) — YES, t changed

# Verify they share storage
print(t.storage().data_ptr() == s.storage().data_ptr())  # True

# ── Get an independent copy ─────────────────────────────────────
c = t[0, :].clone()    # COPY — fully independent
c[0] = -1.0
print(t[0, 0])  # tensor(999.) — t unchanged

# ── .detach() vs .clone() ──────────────────────────────────────
# .detach() — shares memory, removes from autograd graph
# .clone()  — copies memory, preserves grad_fn
# .clone().detach() — copy + remove from graph (common pattern)

4.3 Fancy Indexing (Always Returns a Copy)

t = torch.arange(20).reshape(4, 5)

# Index with a list of indices — picks specific rows
rows = t[[0, 2, 3]]           # rows 0, 2, 3 → shape (3, 5)

# Index with parallel lists for each dimension
rows_idx = torch.tensor([0, 1, 3])
cols_idx = torch.tensor([2, 4, 1])
t[rows_idx, cols_idx]   # picks (0,2), (1,4), (3,1) — shape (3,)
                         # t[0,2]=2, t[1,4]=9, t[3,1]=16

# These ALWAYS produce a copy — safe to mutate

4.4 Boolean Masking

t = torch.tensor([[1., -2., 3.], [-4., 5., -6.]])

# Create a boolean mask
mask = t > 0
# tensor([[True, False, True], [False, True, False]])

# Apply mask — returns 1D tensor of selected values
positives = t[mask]     # tensor([1., 3., 5.])

# Count True values
mask.sum()              # tensor(3)
mask.float().mean()     # tensor(0.5) — 50% positive

# Conditional replacement (in-place)
t[t < 0] = 0.0         # ReLU!
print(t)
# tensor([[1., 0., 3.], [0., 5., 0.]])

# torch.where — vectorised ternary (NOT in-place, returns new tensor)
result = torch.where(mask, t, torch.zeros_like(t))

✏️ Practice Problem 4

Problem: Given t = torch.arange(1, 26).reshape(5, 5).float(), write a single expression that:

Selects the bottom-right 3×3 submatrix
Sets all values in that submatrix above 20 to 0

Solution:

t = torch.arange(1, 26).reshape(5, 5).float()
sub = t[2:, 2:]      # bottom-right 3×3: [[13,14,15],[18,19,20],[23,24,25]]

# sub is a VIEW — mutations propagate to t!
sub[sub > 20] = 0.0

# t is now:
# [[ 1,  2,  3,  4,  5],
#  [ 6,  7,  8,  9, 10],
#  [11, 12, 13, 14, 15],
#  [16, 17, 18, 19, 20],
#  [21, 22,  0,  0,  0]]

Key insight: The slice t[2:, 2:] is a view, so sub[sub > 20] = 0.0 modifies t directly. This is intentional and efficient — no copy needed.

5. Reshaping, Views & Squeezing

Shape manipulation is the connective tissue of tensor math — you'll do it constantly to align tensors for operations.

5.1 reshape vs view

t = torch.arange(12, dtype=torch.float32)   # shape (12,)

# reshape — safe, handles non-contiguous, may or may not share memory
r1 = t.reshape(3, 4)      # (3, 4)
r2 = t.reshape(2, 2, 3)   # (2, 2, 3)
r3 = t.reshape(4, -1)     # (4, 3) — -1 infers the size

# view — faster but requires contiguous memory; always shares memory
v = t.view(3, 4)          # fine — t is contiguous
t_T = t.reshape(3, 4).T   # transpose is non-contiguous
# t_T.view(12)             # RuntimeError! non-contiguous
t_T.contiguous().view(12) # fix: make contiguous first, then view

# When in doubt: use reshape (safe), not view

5.2 Adding and Removing Dimensions

t = torch.tensor([1., 2., 3.])   # shape (3,)

# unsqueeze — add a dimension of size 1
t.unsqueeze(0)    # (1, 3) — new batch dimension
t.unsqueeze(1)    # (3, 1) — new column dimension
t.unsqueeze(-1)   # (3, 1) — same as unsqueeze(1) for 1D

# squeeze — remove dimensions of size 1
x = torch.randn(1, 3, 1, 4)
x.squeeze()         # (3, 4) — removes ALL size-1 dims
x.squeeze(0)        # (3, 1, 4) — removes only dim 0
x.squeeze(2)        # (1, 3, 4) — removes only dim 2
x.squeeze(-1)       # (1, 3, 1) — no wait: that removes dim 3 → (1, 3, 1)?
# Careful: squeeze(dim) only removes that dim IF its size is 1; no-op otherwise

# None indexing — alternative unsqueeze
t[None, :]    # (1, 3) — same as t.unsqueeze(0)
t[:, None]    # (3, 1) — same as t.unsqueeze(1)

5.3 Concatenation and Stacking

a = torch.randn(3, 4)
b = torch.randn(3, 4)

# cat — join along an EXISTING dimension
torch.cat([a, b], dim=0)    # (6, 4) — stack vertically
torch.cat([a, b], dim=1)    # (3, 8) — stack horizontally

# stack — create a NEW dimension
torch.stack([a, b], dim=0)  # (2, 3, 4) — batch of 2 matrices
torch.stack([a, b], dim=1)  # (3, 2, 4)
torch.stack([a, b], dim=2)  # (3, 4, 2)

# vstack/hstack — convenience wrappers
torch.vstack([a, b])   # (6, 4) — same as cat(dim=0)
torch.hstack([a, b])   # (3, 8) — same as cat(dim=1)

# split — inverse of cat
parts = torch.split(torch.cat([a, b], dim=0), split_size_or_sections=3, dim=0)
# (tensor(3,4), tensor(3,4)) — splits evenly

# chunk — split into N equal pieces
chunks = torch.chunk(torch.cat([a, b], dim=0), chunks=2, dim=0)

5.4 Transpose and Permute

t = torch.randn(2, 3, 4)

# transpose — swap exactly two dimensions
t.transpose(0, 1)   # (3, 2, 4)
t.transpose(1, 2)   # (2, 4, 3)
t.T                 # full reversal of dims — only well-defined for 2D!

# permute — reorder ALL dimensions at once
t.permute(2, 0, 1)  # (4, 2, 3) — dim 2 goes to position 0, etc.
t.permute(0, 2, 1)  # (2, 4, 3) — swap dims 1 and 2

# IMPORTANT: transpose and permute return non-contiguous tensors
print(t.transpose(0,1).is_contiguous())  # False
# Fix with .contiguous() when needed:
t.permute(2, 0, 1).contiguous()

✏️ Practice Problem 5

Problem: You receive a batch of images in NHWC format (batch, height, width, channels) with shape (8, 32, 32, 3). PyTorch convolutions expect NCHW format (batch, channels, height, width). Convert it.

Solution:

images_nhwc = torch.randn(8, 32, 32, 3)
# Axes:                     N   H   W   C
# Desired:                  N   C   H   W
# So we move axis 3 (C) to position 1

images_nchw = images_nhwc.permute(0, 3, 1, 2)
print(images_nchw.shape)   # torch.Size([8, 3, 32, 32])

# Or equivalently:
images_nchw = images_nhwc.transpose(1, 3).transpose(2, 3)

# After permute, you likely need contiguous for convolution:
images_nchw = images_nchw.contiguous()

6. Element-wise Operations & Reductions

6.1 Arithmetic Operations

All arithmetic operators work element-wise and require tensors of compatible shapes (or broadcastable shapes — covered in section 7).

a = torch.tensor([1., 2., 3., 4.])
b = torch.tensor([2., 2., 2., 2.])

# Standard arithmetic — all return NEW tensors
a + b          # [3., 4., 5., 6.]
a - b          # [-1., 0., 1., 2.]
a * b          # [2., 4., 6., 8.]  — ELEMENT-WISE multiply (not dot!)
a / b          # [0.5, 1., 1.5, 2.]
a ** 2         # [1., 4., 9., 16.] — element-wise power
a // b         # [0., 1., 1., 2.]  — floor division
a % b          # [1., 0., 1., 0.]  — modulo

# In-place variants (trailing underscore — modifies a!)
a.add_(b)      # a += b
a.mul_(2)      # a *= 2
a.sub_(1)      # a -= 1

# Functional API (equivalent, often needed for clarity)
torch.add(a, b)
torch.mul(a, b)
torch.div(a, b)
torch.pow(a, 2)

# Scalar operations — broadcast the scalar to all elements
a + 10
a * 0.5
a ** 0.5       # element-wise square root via **, equivalent to torch.sqrt(a)

6.2 Math Functions

t = torch.tensor([-2., -1., 0., 1., 2.])

torch.abs(t)             # [2., 1., 0., 1., 2.]
torch.sqrt(torch.abs(t)) # requires non-negative input
torch.exp(t)             # e^t element-wise
torch.log(t + 3)         # ln — watch out: log(0) = -inf, log(-x) = nan
torch.log2(t + 3)
torch.log10(t + 3)
torch.sin(t); torch.cos(t); torch.tanh(t)
torch.sigmoid(t)         # 1/(1+e^(-t)) — same as F.sigmoid
torch.relu(t)            # max(0, t) — equivalent to t.clamp(min=0)
torch.clamp(t, min=-1, max=1)  # clip values to [-1, 1]
torch.floor(t); torch.ceil(t); torch.round(t)
torch.sign(t)            # -1., -1., 0., 1., 1.
torch.reciprocal(t + 3)  # 1 / (t+3) element-wise

# Useful for numerical stability
torch.clamp(x, min=1e-8)  # prevent log(0)

6.3 Reduction Operations

Reductions collapse dimensions. Understanding the dim parameter is crucial.

t = torch.tensor([[1., 2., 3.],
                   [4., 5., 6.]])
# Shape: (2, 3)

# Global reductions — collapse everything to a scalar
t.sum()           # tensor(21.)
t.mean()          # tensor(3.5)
t.max()           # tensor(6.) — the value
t.min()           # tensor(1.)
t.std()           # tensor(1.8708...)
t.var()           # tensor(3.5)
t.prod()          # tensor(720.)  — product of all elements
t.norm()          # Frobenius norm √(sum of squares)

# Reduce along a dimension — 'dim' specifies which axis to collapse
t.sum(dim=0)      # tensor([5., 7., 9.]) — shape (3,): sum each column
t.sum(dim=1)      # tensor([6., 15.])   — shape (2,): sum each row
t.sum(dim=1, keepdim=True)  # tensor([[6.],[15.]]) — shape (2,1): keeps dims!

# argmax / argmin — return INDEX of max/min, not the value
t.argmax()        # tensor(5) — flat index (row-major)
t.argmax(dim=0)   # tensor([1, 1, 1]) — row index of max in each column
t.argmax(dim=1)   # tensor([2, 2])   — col index of max in each row

# Returns both value and index
vals, idxs = t.max(dim=1)   # vals=[3.,6.], idxs=[2, 2]
vals, idxs = t.min(dim=0)   # vals=[1.,2.,3.], idxs=[0,0,0]

# Cumulative reductions
t.cumsum(dim=1)   # cumulative sum along rows
t.cumprod(dim=0)  # cumulative product along columns

# Any / All — boolean reductions
(t > 3).any()     # tensor(True)  — any element > 3?
(t > 0).all()     # tensor(True)  — all elements > 0?
(t > 3).any(dim=1)  # tensor([False, True]) — per row

✏️ Practice Problem 6

Problem: Given a matrix scores of shape (100, 10) representing 100 students' scores on 10 tests, compute:

Each student's average score
Each test's average score
Number of students who passed (average ≥ 60)
Normalise the scores so each student has zero mean and unit std

Solution:

torch.manual_seed(0)
scores = torch.randint(30, 100, (100, 10)).float()

# 1. Each student's average — collapse the test dimension
student_avg = scores.mean(dim=1)    # shape (100,)

# 2. Each test's average — collapse the student dimension
test_avg    = scores.mean(dim=0)    # shape (10,)

# 3. Count passing students
passed = (student_avg >= 60).sum()  # scalar tensor
print(f"Passed: {passed.item()}")

# 4. Normalise each student's scores independently
# Need keepdim=True so broadcasting works across the 10-test dimension
mean = scores.mean(dim=1, keepdim=True)   # (100, 1)
std  = scores.std(dim=1, keepdim=True)    # (100, 1)
std  = torch.clamp(std, min=1e-8)          # avoid division by zero
normalised = (scores - mean) / std         # (100, 10) — broadcast!

# Verify: each student now has approx mean=0, std=1
print(normalised.mean(dim=1)[:5])  # near zero
print(normalised.std(dim=1)[:5])   # near one

7. Broadcasting — The Silent Multiplier

Broadcasting is PyTorch's mechanism to perform operations between tensors of different shapes without copying data. Mastering it eliminates a huge class of shape errors.

7.1 The Broadcasting Rules

Two shapes are broadcast-compatible if, aligning from the right (trailing dimensions), every pair of sizes is either:

Equal, or
One of them is 1 (the size-1 dimension expands to match the other)

If one tensor has fewer dimensions, it's treated as if it has leading size-1 dimensions prepended.

Shape (4, 3)  +  Shape (3,)
Right-align:    (4, 3)
                (   3)   ← padded to (1, 3) implicitly
               ─────────
Dim 0:  4 vs 1 → expand (1,3) row to fill 4 rows
Dim 1:  3 vs 3 → equal
Result: (4, 3)  ✓

Shape (4, 1)  +  Shape (1, 3)
Dim 0: 4 vs 1 → expand to 4
Dim 1: 1 vs 3 → expand to 3
Result: (4, 3)  ✓

Shape (2, 3)  +  Shape (4, 3)
Dim 0: 2 vs 4 → NEITHER is 1 → ERROR ✗

7.2 Practical Broadcasting Examples

# ── Example 1: add a bias to every row ─────────────────────────
weight = torch.randn(4, 5)    # 4 neurons, each with 5 inputs
bias   = torch.randn(5)       # one bias per input (not realistic but illustrative)
# Actually: bias per OUTPUT neuron
bias   = torch.randn(4)       # 4 biases, one per output
output = weight.sum(dim=1) + bias   # (4,) + (4,) — trivial, same shape

# Real neural layer: (batch, in_features) @ (in_features, out_features) + (out_features,)
X = torch.randn(32, 10)         # 32 samples, 10 features
W = torch.randn(10, 4)          # weights
b = torch.randn(4)              # one bias per output neuron
out = X @ W + b    # (32,4) + (4,) — broadcast adds b to every row of X@W

# ── Example 2: column-wise normalisation ───────────────────────
data = torch.randn(100, 5)       # 100 samples, 5 features
col_mean = data.mean(dim=0)      # (5,)     — mean of each feature
col_std  = data.std(dim=0)       # (5,)
normalised = (data - col_mean) / col_std   # (100,5) - (5,) → broadcast!

# ── Example 3: pairwise distances ──────────────────────────────
# Points: A shape (N, D), B shape (M, D)
# Distance matrix: shape (N, M)
A = torch.randn(10, 3)   # 10 points in 3D
B = torch.randn(7, 3)    # 7 points in 3D

# A.unsqueeze(1): (10, 1, 3)
# B.unsqueeze(0): (1, 7, 3)
# Subtraction:    (10, 7, 3)
diff = A.unsqueeze(1) - B.unsqueeze(0)      # (10, 7, 3)
dist = torch.sqrt((diff ** 2).sum(dim=2))   # (10, 7) — distance matrix

# ── Example 4: outer product via broadcasting ───────────────────
a = torch.tensor([1., 2., 3.])    # (3,)
b = torch.tensor([10., 20.])      # (2,)
outer = a.unsqueeze(1) * b.unsqueeze(0)   # (3,1) * (1,2) → (3,2)
# [[10,20],[20,40],[30,60]]

7.3 Common Broadcasting Mistakes

a = torch.randn(3, 4)
b = torch.randn(4, 3)

# This will FAIL — shapes are not broadcast-compatible in the wrong orientation
# a + b   # RuntimeError!

# Fix 1: Transpose one
a + b.T   # (3,4) + (3,4) ✓

# Fix 2: Explicit reshape
a + b.reshape(3, 4)  # if that's what you meant

# Mistake: missing unsqueeze
col_mean = a.mean(dim=1)   # (3,) — mean of each ROW
# a - col_mean   # tries (3,4) - (3,) → broadcast makes (3,) → (1,3) → ERROR
# Fix: keep dimension
col_mean = a.mean(dim=1, keepdim=True)   # (3, 1)
a - col_mean    # (3,4) - (3,1) → (3,4) ✓ subtracts row mean from each row

✏️ Practice Problem 7

Problem: You have embeddings E of shape (vocab_size=1000, embed_dim=64) and a query vector q of shape (64,). Compute the dot product (similarity score) of the query against every embedding, producing a score vector of shape (1000,).

Solution:

E = torch.randn(1000, 64)
q = torch.randn(64)

# Method 1: Matrix-vector multiply
scores = E @ q    # (1000, 64) @ (64,) → (1000,)

# Method 2: Explicit sum reduction (broadcasting)
scores = (E * q).sum(dim=1)  # (1000,64)*(64,) broadcast → (1000,64) → sum → (1000,)

# Method 3: Using einsum (elegant and explicit)
scores = torch.einsum('vd,d->v', E, q)   # v=vocab, d=dim

# All three produce identical results
# Now find the top-5 most similar vocabulary items
topk_scores, topk_indices = torch.topk(scores, k=5)
print(topk_scores)    # 5 highest scores
print(topk_indices)   # their indices in the vocabulary

8. Matrix Multiplication — The Core Operation

This is the central operation of all linear algebra in PyTorch. Know every variant cold.

8.1 The Full Operator Taxonomy

A = torch.randn(3, 4)
B = torch.randn(4, 5)

# ── 2D matrix multiplication ────────────────────────────────────
C = torch.mm(A, B)           # (3,5) — 2D ONLY, most explicit
C = torch.matmul(A, B)       # (3,5) — works for any ndim
C = A @ B                    # (3,5) — operator alias for matmul

# ── Matrix-vector product ───────────────────────────────────────
v = torch.randn(4)
torch.mv(A, v)               # (3,) — matrix × vector, 2D only
A @ v                        # (3,) — matmul auto-handles (4,) → column vec

# ── Vector dot product (1D only) ────────────────────────────────
a = torch.randn(4)
b = torch.randn(4)
torch.dot(a, b)              # scalar — ONLY for 1D vectors
a @ b                        # scalar — same

# ── Outer product ───────────────────────────────────────────────
torch.outer(a, b)            # (4, 4)
a.unsqueeze(1) @ b.unsqueeze(0)  # same: (4,1) @ (1,4) → (4,4)

# ── matmul is the universal operator ───────────────────────────
# 1D × 1D   → scalar (dot product)
# 2D × 2D   → matrix multiply
# 2D × 1D   → matrix-vector (treats 1D as column vector, result is 1D)
# 1D × 2D   → vector-matrix (treats 1D as row vector, result is 1D)

8.2 matmul Broadcasting Rules

torch.matmul (and @) batch-broadcasts when tensors have more than 2 dimensions. The last two dimensions are the matrix dimensions; all leading dimensions are broadcast.

# Batch of matrices × single matrix
bA = torch.randn(10, 3, 4)   # 10 matrices of shape (3,4)
B  = torch.randn(4, 5)       #  1 matrix  of shape (4,5)
torch.matmul(bA, B)          # (10,3,5) — B broadcast to each of 10 batches

# Batch × Batch (broadcast)
bA = torch.randn(10, 3, 4)
bB = torch.randn(10, 4, 5)
torch.matmul(bA, bB)          # (10,3,5) — one matmul per batch element

# Higher-rank broadcasting
bA = torch.randn(2, 10, 3, 4)
bB = torch.randn(   10, 4, 5)   # implicit (1,10,...) prepended
torch.matmul(bA, bB)            # (2,10,3,5) — broadcast over dim 0

8.3 Understanding the Shapes

The most common source of bugs is shape confusion. Develop this instinct:

(M, K) × (K, N) → (M, N)
      ^   ^
      These must match — the "inner dimensions"

For batched: (B, M, K) × (B, K, N) → (B, M, N)
             (*, M, K) × (*, K, N) → (*, M, N)
             (all leading dims broadcast)

# Worked example: multi-head attention shapes
# Q: (batch, seq_len, d_k)
# K: (batch, seq_len, d_k)
# Attention scores: (batch, seq_len, seq_len)

B, S, d_k = 4, 16, 32
Q = torch.randn(B, S, d_k)
K = torch.randn(B, S, d_k)

# Q @ K^T — K must be transposed to (B, d_k, S)
scores = Q @ K.transpose(-2, -1)   # (4, 16, 16)
# Scale by sqrt(d_k) for numerical stability
scores = scores / (d_k ** 0.5)

8.4 Performance Tips

# ── Use @ or matmul, NOT manual loops ───────────────────────────
A = torch.randn(1000, 1000)
B = torch.randn(1000, 1000)

# SLOW — pure Python loop
result = torch.zeros(1000, 1000)
for i in range(1000):
    result[i] = (A[i].unsqueeze(0) @ B).squeeze()

# FAST — single BLAS call
result = A @ B    # milliseconds vs seconds

# ── Reuse transposed tensors ─────────────────────────────────────
# If you need A.T multiple times, store it
At = A.T.contiguous()   # make contiguous once
for _ in range(100):
    result = At @ B     # faster than (A.T @ B) * 100 times

# ── Choose dtype based on precision needs ───────────────────────
A16 = A.half()   # float16 — 2x faster matmul on GPU, 2x less memory
B16 = B.half()
result = A16 @ B16   # fast but less precise

✏️ Practice Problem 8

Problem: Implement linear regression prediction from scratch using only tensor operations:

Input X has shape (N=100, features=5)
Weights W has shape (5,) and bias b is a scalar
Compute predictions y_hat of shape (N,)
Compute Mean Squared Error loss as a scalar

Solution:

torch.manual_seed(42)
N, F = 100, 5

X = torch.randn(N, F)       # design matrix
W = torch.randn(F)           # weight vector
b = torch.tensor(0.5)        # bias
y = torch.randn(N)           # true labels

# Predictions: X @ W + b
y_hat = X @ W + b            # (100,5)@(5,) → (100,) + scalar → (100,)

# MSE: mean over all samples
residuals = y_hat - y        # (100,)
mse = (residuals ** 2).mean()   # scalar

# Equivalent:
mse = torch.nn.functional.mse_loss(y_hat, y)

print(f"y_hat shape: {y_hat.shape}")   # torch.Size([100])
print(f"MSE: {mse.item():.4f}")

9. Batched Matrix Operations

When you have collections of matrices to operate on simultaneously, batched operations are critical.

9.1 bmm vs matmul

# bmm — STRICTLY 3D (batch, M, K) × (batch, K, N)
bA = torch.randn(8, 3, 4)
bB = torch.randn(8, 4, 5)
torch.bmm(bA, bB)          # (8, 3, 5) ✓

# bmm does NOT broadcast — all three must have same batch size
bC = torch.randn(1, 4, 5)
# torch.bmm(bA, bC)        # ERROR if batch dim != 8

# matmul — handles broadcast over any leading dims
torch.matmul(bA, bC)       # (8, 3, 5) — bC broadcast to 8 batches ✓
torch.matmul(bA, bB)       # (8, 3, 5) ✓ — works like bmm when 3D

9.2 Batched Applications

# ── Batched covariance matrices ──────────────────────────────────
# Given B batches of N samples with D features: (B, N, D)
# Compute covariance: (B, D, D)
data = torch.randn(4, 100, 8)   # 4 batches, 100 samples, 8 features

mean = data.mean(dim=1, keepdim=True)   # (4, 1, 8)
centered = data - mean                   # (4, 100, 8)

# Covariance: (1/N) * X^T X
# centered.transpose(-2,-1) : (4, 8, 100)
# centered                   : (4, 100, 8)
cov = torch.bmm(centered.transpose(-2, -1), centered) / 100   # (4, 8, 8)

# ── Applying different transforms to different batches ───────────
transforms = torch.randn(8, 4, 4)   # 8 different 4×4 transform matrices
points     = torch.randn(8, 4, 100)  # 8 batches of 100 4D points
transformed = torch.bmm(transforms, points)   # (8, 4, 100)

# ── Solving multiple linear systems simultaneously ───────────────
# Ax = b for A: (B, N, N), b: (B, N, K) → solve for x: (B, N, K)
A = torch.randn(5, 4, 4)
A = A @ A.transpose(-2,-1) + torch.eye(4)  # make positive definite
b = torch.randn(5, 4, 3)
x = torch.linalg.solve(A, b)   # (5, 4, 3) — solves all 5 systems

✏️ Practice Problem 9

Problem: You have a batch of 16 square matrices, each 6×6 (shape (16, 6, 6)). Compute the trace (sum of diagonal elements) of each matrix.

Solution:

batch = torch.randn(16, 6, 6)

# Method 1: diagonal + sum
# torch.diagonal returns shape (6, 16) by default — need dim1, dim2
diags = torch.diagonal(batch, dim1=-2, dim2=-1)   # (16, 6)
traces = diags.sum(dim=-1)                          # (16,)

# Method 2: einsum — trace is contraction of diagonal indices
traces = torch.einsum('bii->b', batch)   # b=batch, i=diagonal index

# Method 3: Using linalg (most readable)
traces = torch.vmap(torch.trace)(batch)  # PyTorch 2.0+ vmap

print(traces.shape)   # torch.Size([16])
print(traces[:3])

10. Einstein Summation (einsum)

torch.einsum is the most expressive way to write tensor contractions. Once you understand the notation, it replaces nested loops, transposes, and complex matmul chains with a single readable string.

10.1 The Einsum Grammar

The notation is: 'input_subscripts -> output_subscripts'

Each letter represents one dimension
Repeated letters on the left are contracted (summed over)
Letters appearing only in the output are kept
Letters in the input but missing from the output are summed away

'ij,jk->ik'
 ^^  ^^  ^^
 A   B   C

i → output dimension (rows of A / rows of C)
j → contracted dimension (must match: cols of A == rows of B)
k → output dimension (cols of B / cols of C)
Result: C[i,k] = Σ_j A[i,j] * B[j,k]

10.2 Einsum for Common Operations

import torch

A = torch.randn(3, 4)
B = torch.randn(4, 5)
a = torch.randn(3)
b = torch.randn(3)
v = torch.randn(4)

# ── Linear Algebra ──────────────────────────────────────────────
torch.einsum('ij,jk->ik', A, B)     # (3,5) — matrix multiply
torch.einsum('ij,j->i', A, v)       # (3,)  — matrix-vector multiply
torch.einsum('i,i->', a, b)         # scalar — dot product
torch.einsum('i,j->ij', a, b)       # (3,3) — outer product
torch.einsum('ij->ji', A)           # (4,3) — transpose
torch.einsum('ii->', A[:min(A.shape):])  # scalar — trace (square submatrix)
torch.einsum('ij->', A)             # scalar — sum all elements
torch.einsum('ij->i', A)            # (3,)  — row sums
torch.einsum('ij->j', A)            # (4,)  — column sums

# ── Element-wise / Hadamard ─────────────────────────────────────
C = torch.randn(3, 4)
torch.einsum('ij,ij->ij', A, C)    # (3,4) — element-wise multiply (A*C)
torch.einsum('ij,ij->', A, C)      # scalar — Frobenius inner product (A*C).sum()

# ── Batched operations ──────────────────────────────────────────
bA = torch.randn(8, 3, 4)
bB = torch.randn(8, 4, 5)
torch.einsum('bij,bjk->bik', bA, bB)   # (8,3,5) — batched matmul

# ── Diagonal extraction ─────────────────────────────────────────
sq = torch.randn(4, 4)
torch.einsum('ii->i', sq)              # (4,) — diagonal elements

# ── Bilinear form: x^T A y ─────────────────────────────────────
x, y = torch.randn(4), torch.randn(4)
torch.einsum('i,ij,j->', x, sq, y)    # scalar

# ── Multi-head attention score computation ─────────────────────
# Q: (batch, heads, seq, d_k)  K: (batch, heads, seq, d_k)
B, H, S, d = 2, 4, 16, 32
Q = torch.randn(B, H, S, d)
K = torch.randn(B, H, S, d)
# Attention scores: (batch, heads, seq_q, seq_k)
scores = torch.einsum('bhqd,bhkd->bhqk', Q, K)   # (2,4,16,16)

10.3 Understanding Contraction Step by Step

# Let's trace 'ij,jk->ik' by hand for small tensors
A = torch.tensor([[1., 2.],
                   [3., 4.]])   # (2,2) — i=2, j=2
B = torch.tensor([[5., 6., 7.],
                   [8., 9., 10.]])  # (2,3) — j=2, k=3

result = torch.einsum('ij,jk->ik', A, B)
# result[0,0] = A[0,0]*B[0,0] + A[0,1]*B[1,0] = 1*5 + 2*8 = 21
# result[0,1] = A[0,0]*B[0,1] + A[0,1]*B[1,1] = 1*6 + 2*9 = 24
# result[0,2] = A[0,0]*B[0,2] + A[0,1]*B[1,2] = 1*7 + 2*10 = 27
# result[1,0] = A[1,0]*B[0,0] + A[1,1]*B[1,0] = 3*5 + 4*8 = 47
print(result)
# tensor([[21., 24., 27.],
#          [47., 54., 61.]])

✏️ Practice Problem 10

Problem: Using einsum, implement:

Batch dot products: given A (shape (B, D)) and B (shape (B, D)), compute the dot product of each corresponding pair → shape (B,)
Hadamard product summed: element-wise product of two (M, N) matrices, then sum → scalar
Tensor contraction: for T of shape (A, B, C) and M of shape (C, D), contract the last dim → shape (A, B, D)

Solution:

B_size, D = 8, 16
A_t = torch.randn(B_size, D)
B_t = torch.randn(B_size, D)
M_t = torch.randn(3, 4)
N_t = torch.randn(3, 4)
T   = torch.randn(5, 6, 7)
M   = torch.randn(7, 9)

# 1. Batch dot products
batch_dots = torch.einsum('bd,bd->b', A_t, B_t)
print(batch_dots.shape)   # (8,)

# Verify against loop
for i in range(B_size):
    assert torch.allclose(batch_dots[i], torch.dot(A_t[i], B_t[i]))

# 2. Hadamard product summed (Frobenius inner product)
fro = torch.einsum('ij,ij->', M_t, N_t)
print(fro.shape)   # scalar = torch.Size([])

# 3. Tensor-matrix contraction over last dim
result = torch.einsum('abc,cd->abd', T, M)
print(result.shape)   # (5, 6, 9)

# Verify with matmul (equivalent)
result2 = (T.reshape(-1, 7) @ M).reshape(5, 6, 9)
print(torch.allclose(result, result2))   # True

11. Linear Algebra — linalg Module

torch.linalg is the modern (PyTorch 1.9+) home for all linear algebra operations. Prefer it over older scattered functions.

11.1 Norms

t = torch.tensor([[3., 4.], [1., 2.]])

# Vector norms (when applied to a 1D tensor or with dim specified)
torch.linalg.norm(torch.tensor([3., 4.]))      # L2 norm: 5.0  (√(9+16))
torch.linalg.norm(t, ord=2)                    # spectral norm (max singular value)
torch.linalg.norm(t, ord='fro')               # Frobenius norm: √(9+16+1+4) = √30
torch.linalg.norm(t, ord=1)                    # max column sum
torch.linalg.norm(t, ord=float('inf'))         # max row sum

# Per-row or per-column norms (using dim parameter)
torch.linalg.norm(t, dim=1)       # L2 norm of each row: [5.0, 2.236]
torch.linalg.norm(t, dim=0)       # L2 norm of each column: [3.162, 4.472]

# L1 norm of rows (used in normalisation)
torch.linalg.norm(t, ord=1, dim=1)   # sum of abs values per row

# Normalise rows to unit L2 norm
normed = t / torch.linalg.norm(t, dim=1, keepdim=True)
print(torch.linalg.norm(normed, dim=1))   # [1., 1.] — unit vectors

11.2 Determinant and Inverse

A = torch.tensor([[4., 7.], [2., 6.]], dtype=torch.float32)

# Determinant
det = torch.linalg.det(A)            # tensor(10.)  (4*6 - 7*2)
logdet = torch.linalg.slogdet(A)     # (sign, log|det|) — numerically stable

# Inverse — use only when you need the matrix explicitly
A_inv = torch.linalg.inv(A)
print(A @ A_inv)                      # should be close to identity

# BETTER: when solving Ax = b, never invert!
b = torch.tensor([1., 0.])
x = torch.linalg.solve(A, b)          # more stable and faster than inv(A) @ b

# Batched versions work the same way
batch_A = torch.randn(10, 4, 4)
batch_b = torch.randn(10, 4, 2)
batch_A = batch_A @ batch_A.transpose(-2,-1) + torch.eye(4)  # positive definite
x = torch.linalg.solve(batch_A, batch_b)   # (10, 4, 2)

11.3 Eigendecomposition

# For symmetric/Hermitian matrices (more stable)
A_sym = torch.randn(4, 4)
A_sym = A_sym @ A_sym.T            # A_sym is now symmetric positive semi-definite

eigenvalues = torch.linalg.eigvalsh(A_sym)          # real eigenvalues (sorted!)
eigenvalues, eigenvectors = torch.linalg.eigh(A_sym) # both

# For general matrices
eigenvalues_complex = torch.linalg.eigvals(A_sym)    # may return complex
eigenvalues_c, eigenvectors_c = torch.linalg.eig(A_sym)

# Reconstruction: A = V @ diag(Λ) @ V^{-1}
# For symmetric: A = V @ diag(Λ) @ V.T  (V is orthogonal)
A_reconstructed = eigenvectors @ torch.diag(eigenvalues) @ eigenvectors.T
print(torch.allclose(A_sym, A_reconstructed, atol=1e-5))   # True

11.4 Singular Value Decomposition (SVD)

SVD is one of the most important decompositions. It works for any matrix (not just square), and is the foundation of PCA, low-rank approximation, and pseudo-inverse.

A = U @ diag(S) @ Vh

A:  (M, N)  — the original matrix
U:  (M, M)  — left singular vectors (orthogonal columns)
S:  (min(M,N),) — singular values (non-negative, sorted descending)
Vh: (N, N)  — right singular vectors (Vh is V^H, the conjugate transpose of V)

A = torch.randn(5, 3)   # (5, 3)

# Full SVD
U, S, Vh = torch.linalg.svd(A)
print(U.shape)   # (5, 5)
print(S.shape)   # (3,) — min(5,3) = 3
print(Vh.shape)  # (3, 3)

# Reconstruct A
A_recon = U[:, :3] @ torch.diag(S) @ Vh    # need only first 3 cols of U
print(torch.allclose(A, A_recon, atol=1e-5))  # True

# Truncated SVD (economy/thin) — more efficient
U, S, Vh = torch.linalg.svd(A, full_matrices=False)
print(U.shape)   # (5, 3) — economy
print(S.shape)   # (3,)
print(Vh.shape)  # (3, 3)

# Reconstruct directly
A_recon = U @ torch.diag(S) @ Vh
print(torch.allclose(A, A_recon, atol=1e-5))  # True

# ── Low-rank approximation (keep top k singular values) ───────────
def low_rank_approx(A, k):
    U, S, Vh = torch.linalg.svd(A, full_matrices=False)
    return U[:, :k] @ torch.diag(S[:k]) @ Vh[:k, :]

rank2_approx = low_rank_approx(A, k=2)
print("Frobenius error:", torch.linalg.norm(A - rank2_approx, ord='fro'))

# ── Pseudo-inverse (for non-square or singular matrices) ──────────
A_pinv = torch.linalg.pinv(A)   # (3, 5) — right pseudo-inverse
print(A_pinv.shape)
# Satisfies: A @ A_pinv @ A ≈ A

11.5 QR and Cholesky Decompositions

# QR Decomposition: A = Q @ R
# Q: orthogonal, R: upper triangular
A = torch.randn(5, 3)
Q, R = torch.linalg.qr(A)
print(Q.shape)  # (5, 3) — economy QR by default
print(R.shape)  # (3, 3)
print(torch.allclose(A, Q @ R, atol=1e-5))   # True

# Q is orthogonal: Q.T @ Q ≈ I
print(torch.allclose(Q.T @ Q, torch.eye(3), atol=1e-5))   # True

# Cholesky: A = L @ L.T (for positive definite A)
A = torch.randn(4, 4)
A = A @ A.T + torch.eye(4)   # make positive definite
L = torch.linalg.cholesky(A)
print(torch.allclose(L @ L.T, A, atol=1e-5))   # True

✏️ Practice Problem 11 — PCA from Scratch

Problem: Implement PCA (Principal Component Analysis) using SVD to project a data matrix X of shape (200, 10) down to 3 principal components.

Solution:

torch.manual_seed(0)
X = torch.randn(200, 10)   # 200 samples, 10 features

# Step 1: Center the data (subtract column means)
mean = X.mean(dim=0)           # (10,)
X_centered = X - mean          # (200, 10) — broadcast

# Step 2: Compute SVD of the centred data
# Covariance structure is encoded in the right singular vectors Vh
U, S, Vh = torch.linalg.svd(X_centered, full_matrices=False)
# U:  (200, 10)
# S:  (10,)      — singular values (√eigenvalues of covariance * N)
# Vh: (10, 10)   — rows are principal components

# Step 3: Project onto top-3 principal components
# Principal components are the ROWS of Vh (right singular vectors)
components = Vh[:3, :]    # (3, 10) — top 3 PCs
X_projected = X_centered @ components.T   # (200, 10) @ (10, 3) → (200, 3)

print(f"Original:  {X.shape}")        # (200, 10)
print(f"Projected: {X_projected.shape}")  # (200, 3)

# Explained variance ratio
total_var = (S ** 2).sum()
explained = (S[:3] ** 2) / total_var
print(f"Variance explained: {explained.tolist()}")
print(f"Cumulative: {explained.cumsum(0).tolist()}")

# Reconstruction from top-3 (lossy)
X_reconstructed = X_projected @ components + mean   # (200, 3) @ (3, 10) + (10,)
reconstruction_error = torch.linalg.norm(X - X_reconstructed, ord='fro')
print(f"Reconstruction error: {reconstruction_error:.4f}")

12. Advanced Indexing — gather & scatter

These operations are fundamental for NLP (token embedding lookups, masked attention), RL (action selection), and any problem requiring non-uniform selection or writing.

12.1 torch.gather — Selective Read

gather collects values from input at specified index positions along a dimension.

output[i][j][k] = input[i][index[i][j][k]][k]   (for dim=1)

Crucially: output has the same shape as index.

t = torch.tensor([[10., 20., 30.],
                   [40., 50., 60.],
                   [70., 80., 90.]])   # (3, 3)

# For each row, select specific columns
idx = torch.tensor([[0, 2],     # row 0: pick columns 0 and 2
                    [1, 1],     # row 1: pick columns 1 and 1 (same)
                    [2, 0]])    # row 2: pick columns 2 and 0
result = torch.gather(t, dim=1, index=idx)
# result[0] = [t[0,0], t[0,2]] = [10, 30]
# result[1] = [t[1,1], t[1,1]] = [50, 50]
# result[2] = [t[2,2], t[2,0]] = [90, 70]
print(result)
# tensor([[10., 30.],
#          [50., 50.],
#          [90., 70.]])

12.2 Practical gather — One-Hot, Token Lookup

# ── Token embedding lookup (what nn.Embedding does internally) ─
vocab_size, embed_dim = 1000, 64
embeddings = torch.randn(vocab_size, embed_dim)   # (V, D)
token_ids  = torch.randint(0, vocab_size, (8, 16)) # (B, S) batch of token sequences

# Method 1: fancy indexing (simple, works)
embedded = embeddings[token_ids]   # (8, 16, 64)

# ── Gathering logits for specific classes ──────────────────────
# batch_size=8, num_classes=10
logits  = torch.randn(8, 10)
targets = torch.tensor([3, 7, 1, 0, 9, 4, 2, 6])   # one target per sample

# Gather the logit corresponding to the true class
# index must match logits.ndim — need shape (8, 1)
idx     = targets.unsqueeze(1)           # (8, 1)
correct_logits = torch.gather(logits, dim=1, index=idx)  # (8, 1)
correct_logits = correct_logits.squeeze(1)               # (8,)

# ── NLL loss from scratch ──────────────────────────────────────
log_probs = torch.log_softmax(logits, dim=1)   # (8, 10)
nll = -torch.gather(log_probs, 1, targets.unsqueeze(1)).squeeze()  # (8,)
loss = nll.mean()   # scalar

12.3 torch.scatter_ and torch.scatter — Selective Write

scatter_ is the inverse of gather: instead of reading from specific positions, it writes to them. Note the trailing underscore — it's in-place by default.

output[i][index[i][j]][k] = src[i][j][k]   (for dim=1)

# ── One-hot encoding ──────────────────────────────────────────
batch_size, num_classes = 8, 5
labels = torch.tensor([0, 3, 1, 4, 2, 0, 3, 1])   # (8,)

one_hot = torch.zeros(batch_size, num_classes)      # (8, 5)
one_hot.scatter_(1, labels.unsqueeze(1), 1.0)       # write 1.0 at label positions
print(one_hot)
# tensor([[1,0,0,0,0],
#          [0,0,0,1,0],
#          [0,1,0,0,0], ...])

# ── Scatter add (accumulate) ──────────────────────────────────
# Count occurrences of each token in a batch of sequences
vocab_size = 10
tokens = torch.tensor([3, 1, 3, 5, 1, 2, 3])   # token IDs
counts = torch.zeros(vocab_size)
counts.scatter_add_(0, tokens, torch.ones_like(tokens, dtype=torch.float))
print(counts)   # [0,2,1,3,0,1,0,0,0,0] — token 3 appears 3 times

# ── Functional scatter (non-in-place) ─────────────────────────
src = torch.ones(2, 3)
idx = torch.tensor([[0, 1, 2], [0, 1, 4]])
out = torch.zeros(2, 5)
out.scatter_(1, idx, src)   # writes src values at idx positions

12.4 torch.index_select, torch.index_put

t = torch.randn(5, 4)

# index_select — select rows or columns by index tensor
selected_rows = torch.index_select(t, 0, torch.tensor([0, 2, 4]))  # (3, 4)
selected_cols = torch.index_select(t, 1, torch.tensor([1, 3]))     # (5, 2)

# index_put — batch assignment
indices = (torch.tensor([0, 2, 4]), torch.tensor([1, 3, 0]))
values  = torch.tensor([99., 99., 99.])
t.index_put_(indices, values)   # t[0,1]=99, t[2,3]=99, t[4,0]=99

✏️ Practice Problem 12

Problem: Implement a "masked softmax" without any loops:

Input: logits of shape (B=4, S=8) — attention logits
mask of shape (B=4, S=8) — boolean, True = position to IGNORE
Compute softmax over logits, setting ignored positions to 0 in the output

Solution:

B, S = 4, 8
logits = torch.randn(B, S)
mask   = torch.zeros(B, S, dtype=torch.bool)
mask[:, -2:] = True   # ignore last 2 positions in every sequence

# Fill ignored positions with -infinity BEFORE softmax
# This makes them contribute 0 after exp(x) / sum(exp(x))
masked_logits = logits.masked_fill(mask, float('-inf'))  # (B, S)

# Numerically stable softmax
attention = torch.softmax(masked_logits, dim=-1)   # (B, S)

# Verify: masked positions are 0
print(attention[:, -2:])   # should be [[0,0],[0,0],[0,0],[0,0]]

# Verify: non-masked rows sum to 1
print(attention.sum(dim=-1))   # [1., 1., 1., 1.]

# Alternative: torch.where approach
NEG_INF = torch.full_like(logits, float('-inf'))
masked_logits2 = torch.where(mask, NEG_INF, logits)
attention2 = torch.softmax(masked_logits2, dim=-1)
print(torch.allclose(attention, attention2))   # True

13. Memory Layout, Strides & Contiguity

Understanding this makes you dangerous in code reviews and debugging sessions.

13.1 How Tensors Are Stored

A tensor's data is a flat 1D array in memory. The strides tell PyTorch how many elements to skip to advance by 1 in each dimension.

t = torch.arange(12).reshape(3, 4)

Memory:  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Shape:   (3, 4)
Strides: (4, 1)

t[i, j] is at memory offset: i * 4 + j * 1
t[0,0]=0, t[0,1]=1, t[1,0]=4, t[2,3]=11 ✓

t = torch.arange(12).reshape(3, 4)
print(t.stride())    # (4, 1) — C-contiguous (row-major)

t_T = t.T           # transpose — SWAP strides, no copy
print(t_T.stride())  # (1, 4) — Fortran-contiguous (column-major)
print(t_T.is_contiguous())   # False

# Strides for a sliced tensor
s = t[::2, ::2]     # every other row and column
print(s.stride())    # (8, 2) — skipping over elements

# Make contiguous (allocates new memory, rewrites data)
s_c = s.contiguous()
print(s_c.stride())  # (2, 1) — back to C-contiguous

13.2 Why Contiguity Matters

t = torch.randn(4, 4)
t_T = t.T

# Some operations require contiguous memory
# t_T.view(16)   # RuntimeError: non-contiguous
t_T.reshape(16)  # OK — reshape handles non-contiguous with an internal copy

# Storage_offset: where in the underlying storage does the tensor start
s = t[1:]          # starts at element 4 in storage
print(s.storage_offset())   # 4

# Checking if two tensors share storage
print(t.storage().data_ptr() == t_T.storage().data_ptr())  # True — shared!
print(t.storage().data_ptr() == t_T.contiguous().storage().data_ptr())  # False — copy

13.3 Performance Implications

# Contiguous tensors are cache-friendly — sequential memory access
t = torch.randn(1000, 1000)

# Fast: row-major access matches C-contiguous layout
row_sums = t.sum(dim=1)    # iterates along contiguous dimension last

# Operations that force contiguous copies are expensive
# Avoid in tight loops:
t_T = t.T.contiguous()    # forces copy
# Do instead: plan your layout so you don't need to transpose repeatedly

# float16 vs float32 memory
t32 = torch.randn(1000, 1000)           # 4 MB
t16 = t32.half()                         # 2 MB — half the memory
t16 = t32.to(torch.float16)             # equivalent

# In-place ops save memory (no intermediate tensor)
t.add_(1.0)     # modifies t, no new allocation
t.relu_()       # same

14. Practice Problems with Full Solutions

These problems are calibrated to the style and difficulty of your assessment.

Problem A — Shape Surgery

Problem: Given x of shape (B=4, C=3, H=8, W=8) (a batch of images), produce a tensor where each image's pixels are flattened into a row, resulting in shape (4, 192).

x = torch.randn(4, 3, 8, 8)

# Method 1: reshape
out = x.reshape(4, -1)         # (4, 192)

# Method 2: flatten (cleaner API)
out = x.flatten(start_dim=1)   # flatten all dims except batch

# Method 3: view (only if contiguous)
out = x.contiguous().view(4, -1)

print(out.shape)   # torch.Size([4, 192])

Problem B — Efficient Softmax from Scratch

Problem: Implement a numerically stable softmax (without using torch.softmax). For input x of shape (B, C), compute softmax over the class dimension.

def stable_softmax(x):
    # Subtract max for numerical stability: softmax(x) = softmax(x - c)
    x_max = x.max(dim=1, keepdim=True).values   # (B, 1)
    x_shifted = x - x_max                        # (B, C) — broadcast

    exp_x = torch.exp(x_shifted)                 # (B, C)
    sum_exp = exp_x.sum(dim=1, keepdim=True)      # (B, 1)
    return exp_x / sum_exp                        # (B, C) — broadcast

x = torch.randn(4, 10)
out = stable_softmax(x)

# Verify: rows sum to 1
print(out.sum(dim=1))   # [1., 1., 1., 1.]

# Verify: all values non-negative
print((out >= 0).all())  # True

# Verify: matches PyTorch's softmax
print(torch.allclose(out, torch.softmax(x, dim=1), atol=1e-6))  # True

Problem C — Pairwise Cosine Similarity Matrix

Problem: Given a matrix A of shape (N, D) where each row is a vector, compute the full (N, N) cosine similarity matrix without any Python loops.

def cosine_similarity_matrix(A):
    # Normalise each row to unit L2 norm
    norms = torch.linalg.norm(A, dim=1, keepdim=True)   # (N, 1)
    norms = torch.clamp(norms, min=1e-8)                  # avoid division by zero
    A_norm = A / norms                                     # (N, D)

    # Cosine similarity = dot product of unit vectors = A_norm @ A_norm.T
    return A_norm @ A_norm.T   # (N, N)

N, D = 10, 16
A = torch.randn(N, D)
sim = cosine_similarity_matrix(A)

print(sim.shape)             # (10, 10)
print(sim.diagonal())        # all 1.0 — each vector is identical to itself
print(sim.min(), sim.max())  # values in [-1, 1]

# Verify diagonal is exactly 1.0
print(torch.allclose(sim.diagonal(), torch.ones(N), atol=1e-5))  # True

Problem D — Sliding Window Sum (Strided Trick)

Problem: Given a 1D signal x of length N=12, compute the sum of every window of size k=4 without a Python loop, producing output of length N-k+1=9.

x = torch.arange(1., 13.)   # [1,2,3,4,5,6,7,8,9,10,11,12]
k = 4

# Method 1: unfold + sum (most elegant)
windows = x.unfold(dimension=0, size=k, step=1)   # (9, 4)
window_sums = windows.sum(dim=1)                   # (9,)
print(window_sums)
# [10, 14, 18, 22, 26, 30, 34, 38, 42]

# Method 2: cumulative sum trick — O(N) not O(N*k)
cumsum = torch.cat([torch.zeros(1), x.cumsum(0)])   # (13,)
window_sums2 = cumsum[k:] - cumsum[:-k]              # (9,)
print(torch.allclose(window_sums, window_sums2))    # True

# What does unfold give us?
print(windows)
# tensor([[ 1,  2,  3,  4],
#          [ 2,  3,  4,  5],
#          ...
#          [ 9, 10, 11, 12]])

Problem E — Masked Fill and Causal Attention Mask

Problem: Create a causal (lower-triangular) mask for a sequence of length S=6 and apply it to an attention score matrix, setting future positions to -inf so softmax ignores them.

S = 6

# Create causal mask — True where we should MASK (upper triangle, excluding diagonal)
causal_mask = torch.triu(torch.ones(S, S, dtype=torch.bool), diagonal=1)
print(causal_mask.int())
# [[0,1,1,1,1,1],
#  [0,0,1,1,1,1],
#  [0,0,0,1,1,1],
#  [0,0,0,0,1,1],
#  [0,0,0,0,0,1],
#  [0,0,0,0,0,0]]

# Apply to attention scores
attention_scores = torch.randn(2, 4, S, S)   # (batch, heads, S, S)

# Broadcast mask: (S, S) → applied to all batches and heads automatically
masked_scores = attention_scores.masked_fill(causal_mask, float('-inf'))

# Compute attention weights
attention_weights = torch.softmax(masked_scores, dim=-1)   # (2,4,S,S)

# Verify: upper triangle is 0 (softmax of -inf)
print(attention_weights[0, 0])   # lower triangular structure
print((attention_weights[0, 0] > 0).int())   # should be lower triangular

Problem F — Batch Matrix Inversion with Validation

Problem: Given a batch of B=5 random 4×4 matrices, make them invertible (positive definite), compute their inverses, and verify A @ A_inv ≈ I for each.

B, N = 5, 4

# Generate random positive definite matrices
A_raw = torch.randn(B, N, N)
A = torch.bmm(A_raw, A_raw.transpose(-2, -1)) + torch.eye(N)   # (B,N,N)

# Compute batch inverse
A_inv = torch.linalg.inv(A)   # (B, N, N)

# Verify: A @ A_inv should be identity for each batch element
product = torch.bmm(A, A_inv)   # (B, N, N)
identity = torch.eye(N).expand(B, N, N)

max_error = (product - identity).abs().max()
print(f"Max deviation from identity: {max_error:.2e}")   # should be ~1e-6

# Better approach for Ax=b: don't invert, just solve
b = torch.randn(B, N, 2)
x = torch.linalg.solve(A, b)   # (B, N, 2) — numerically superior
print(torch.allclose(torch.bmm(A, x), b, atol=1e-5))   # True

Problem G — Custom Distance Matrix (Assessment Style)

Problem: Implement L2 (Euclidean) pairwise distance between two sets of points without any Python loops:

A: shape (N=50, D=8)
B: shape (M=30, D=8)
Output: (N, M) where out[i,j] = ||A[i] - B[j]||_2

def pairwise_l2(A, B):
    # ||a - b||^2 = ||a||^2 + ||b||^2 - 2 * a.b
    # This avoids the (N, M, D) intermediate tensor from naive subtraction

    A_sq = (A ** 2).sum(dim=1, keepdim=True)   # (N, 1)
    B_sq = (B ** 2).sum(dim=1)                  # (M,) → need (1, M)

    # (N,1) + (1,M) - 2*(N,M) → (N,M)
    sq_dist = A_sq + B_sq.unsqueeze(0) - 2 * (A @ B.T)

    # Clamp to avoid negative values from floating point errors near 0
    return torch.sqrt(torch.clamp(sq_dist, min=0.0))

N, M, D = 50, 30, 8
A = torch.randn(N, D)
B = torch.randn(M, D)

dist = pairwise_l2(A, B)
print(dist.shape)   # (50, 30)

# Verify one distance manually
i, j = 3, 7
manual = torch.linalg.norm(A[i] - B[j])
print(torch.allclose(dist[i, j], manual, atol=1e-5))   # True

# Find nearest neighbour for each point in A
nn_indices = dist.argmin(dim=1)   # (N,) — index in B closest to each point in A
nn_dists   = dist.min(dim=1).values
print(nn_indices.shape, nn_dists.shape)   # (50,) (50,)

Problem H — Full Einsum Challenge

Problem: Use einsum only (no @, mm, matmul) to:

Compute W @ X.T where W: (4,3), X: (5,3) → (4,5)
Compute the batch outer product of rows of A: (B,D) → (B,D,D)
Compute a weighted sum: weights: (B,S), values: (B,S,D) → (B,D)

W = torch.randn(4, 3)
X = torch.randn(5, 3)
A = torch.randn(8, 6)
B_size, S, D_size = 4, 10, 16
weights = torch.softmax(torch.randn(B_size, S), dim=1)
values  = torch.randn(B_size, S, D_size)

# 1. W @ X.T: (4,3) × (3,5) → (4,5)
result1 = torch.einsum('ij,kj->ik', W, X)   # j is contracted, k appears in output
print(result1.shape)   # (4, 5)
print(torch.allclose(result1, W @ X.T))     # True

# 2. Batch outer product: for each b, outer(A[b], A[b]) → (B,D,D)
result2 = torch.einsum('bi,bj->bij', A, A)   # no contraction — pure outer
print(result2.shape)   # (8, 6, 6)
# result2[b] = outer product of A[b] with itself

# 3. Weighted sum: for each batch, sum_s weight[b,s]*value[b,s,:] → (B,D)
result3 = torch.einsum('bs,bsd->bd', weights, values)
print(result3.shape)   # (4, 16)
# Verify against manual:
result3_manual = (weights.unsqueeze(-1) * values).sum(dim=1)  # (B,1)*(B,S,D).sum(1)
print(torch.allclose(result3, result3_manual, atol=1e-6))   # True

15. Assessment Cheat Sheet

A one-stop reference for the most common operations you'll need in 60 minutes.

Tensor Creation

Operation	Code	Output Shape
From list	`torch.tensor([[1,2],[3,4]])`	`(2,2)`
Zeros	`torch.zeros(3,4)`	`(3,4)`
Identity	`torch.eye(N)`	`(N,N)`
Range	`torch.arange(0, 10, 2)`	`(5,)`
Linspace	`torch.linspace(0, 1, 5)`	`(5,)`
Normal	`torch.randn(B, D)`	`(B,D)`
Like	`torch.zeros_like(x)`	same as `x`

Shape Operations

Operation	Code	Notes
Reshape	`t.reshape(2, -1)`	-1 = infer
View	`t.view(N)`	must be contiguous
Transpose (2D)	`t.T` or `t.t()`	shares memory
Transpose (any)	`t.transpose(1, 2)`	swap two dims
Permute	`t.permute(2,0,1)`	reorder all dims
Add dim	`t.unsqueeze(0)`	insert size-1
Remove dim	`t.squeeze(0)`	remove size-1
Flatten	`t.flatten(1)`	flatten from dim
Cat	`torch.cat([a,b], 0)`	join existing dim
Stack	`torch.stack([a,b], 0)`	create new dim

Indexing

Pattern	Code	Notes
Slice (view)	`t[1:3, :]`	shares memory
Bool mask (copy)	`t[t > 0]`	1D result
Fancy (copy)	`t[[0,2,4]]`	new tensor
Clone	`t.clone()`	independent copy
Masked fill	`t.masked_fill(mask, -inf)`	in-place variant: `masked_fill_`
Where	`torch.where(c, a, b)`	vectorised ternary

Matrix Operations

Operation	Code	Dims	Notes
Matrix multiply	`A @ B`	2D+	universal
2D only	`torch.mm(A, B)`	2D	explicit
Batched	`torch.bmm(A, B)`	3D	no broadcast
Batch broadcast	`torch.matmul(A, B)`	any	broadcasts
Einsum	`torch.einsum('ij,jk->ik', A, B)`	any	most expressive
Outer product	`torch.outer(a, b)`	1D	returns 2D
Dot product	`torch.dot(a, b)`	1D	scalar

Reductions

Operation	Code	Notes
Sum all	`t.sum()`	scalar
Sum axis	`t.sum(dim=1)`	collapses dim 1
Keep dim	`t.sum(dim=1, keepdim=True)`	for broadcasting
Argmax	`t.argmax(dim=1)`	index of max
Topk	`torch.topk(t, k=5)`	values + indices
Norm	`torch.linalg.norm(t, dim=1)`	L2 per row

Linear Algebra

Operation	Code	Notes
Inverse	`torch.linalg.inv(A)`	square only
Solve Ax=b	`torch.linalg.solve(A, b)`	prefer over inv
SVD	`torch.linalg.svd(A, full_matrices=False)`	returns U,S,Vh
Eigenvalues	`torch.linalg.eigvalsh(A)`	symmetric
QR	`torch.linalg.qr(A)`	orthogonal + upper tri
Det	`torch.linalg.det(A)`	scalar
Norm	`torch.linalg.norm(A, ord='fro')`	Frobenius
Pseudo-inv	`torch.linalg.pinv(A)`	for non-square

Advanced Indexing

Operation	Code	Notes
Gather	`torch.gather(t, dim=1, index=idx)`	output = idx.shape
Scatter	`out.scatter_(1, idx, src)`	in-place write
Scatter add	`out.scatter_add_(0, idx, src)`	accumulate
Index select	`t.index_select(0, idx)`	select rows/cols

Common Broadcasting Patterns

# Row-wise operation: subtract column mean
mean = t.mean(dim=0)           # (C,)
t - mean                        # (N,C) - (C,) ✓ (right-aligns)

# Col-wise operation: subtract row mean
mean = t.mean(dim=1, keepdim=True)  # (N,1) — keepdim is CRITICAL
t - mean                             # (N,C) - (N,1) → (N,C) ✓

# 3D broadcast: add bias to each batch
bias = torch.randn(C)           # (C,)
t = torch.randn(B, N, C)
t + bias                        # (B,N,C) + (C,) ✓

# Outer product via unsqueeze
a = torch.randn(M)   # (M,)
b = torch.randn(N)   # (N,)
a.unsqueeze(1) * b.unsqueeze(0)   # (M,1) * (1,N) → (M,N) ✓

Critical Gotchas to Avoid in the Assessment

a * b is element-wise, NOT matrix multiply — use a @ b for matmul
t.T is non-contiguous — if the next op needs contiguity, call .contiguous() first
torch.dot is 1D only — use @ or einsum for higher dimensions
reshape vs view: view fails on non-contiguous tensors, reshape doesn't
Slices are views — mutating a slice mutates the original tensor
keepdim=True in reductions — critical when you need to broadcast the result back
bmm does NOT broadcast — use matmul or @ for broadcasting batch operations
Float/int mixing — always cast before mixed operations to avoid type errors
squeeze() with no args removes ALL size-1 dims — be specific: squeeze(0)
gather output shape = index shape — size the index tensor as the output you want

Happy studying! Focus on the Practice Problems (section 14) — work through each one without looking at the solution first, then check your answer. That's the fastest way to build the pattern recognition you'll need under time pressure.

API Testing in VS Code Using `api.http` (REST Client) — A Practical Guide for Experienced Developers

Ajit Kumar — Wed, 25 Feb 2026 08:07:08 +0000

If you’ve spent years using tools like Postman, you already understand the value of structured API testing, collections, environments, and documentation. But as projects mature, you may start wanting something:

Lightweight
Version-controlled
Close to the codebase
Zero context switching

That’s where the REST Client extension in VS Code with .http files becomes a serious alternative.

For a 4–5 year experienced developer, this isn’t about replacing Postman. It’s about:

Keeping API examples in Git
Reducing tool overhead
Treating API calls as documentation
Making backend repos self-testable

This guide walks you from installation to advanced environment setup, including two major issues most developers hit:

--- vs ### request separation and JSON parsing errors
Why rest-client.environment.json sometimes doesn’t work and how .vscode/settings.json solved it

1️⃣ Why Use `api.http` Instead of Postman?

For experienced developers, the real advantages are:

✅ Version Control Friendly

.http files live in your repository. API examples evolve with the backend.

✅ Lightweight

No separate app. No syncing accounts. No collections export.

✅ Documentation + Testing in One

Your api.http becomes living documentation.

✅ CI-Friendly

You can later convert these into automated tests.

2️⃣ Installing REST Client in VS Code

Open Extensions (Ctrl + Shift + X)
Search for:

REST Client

Install:

REST Client by Huachao Mao

Reload VS Code.

3️⃣ Creating Your First `api.http`

Create a file:

api.http

Let’s use a free public API for demo:

We’ll use:

https://jsonplaceholder.typicode.com

Example 1 — Simple GET Request

### Get all posts
GET https://jsonplaceholder.typicode.com/posts

Click “Send Request” above it.

You’ll see the response in a new VS Code tab.

That’s it. No GUI required.

Example 2 — POST Request

### Create new post
POST https://jsonplaceholder.typicode.com/posts
Content-Type: application/json

{
  "title": "API Testing in VS Code",
  "body": "Using REST Client instead of Postman",
  "userId": 1
}

Clean. Readable. Commit-ready.

4️⃣ Separating Multiple Requests (CRITICAL)

When you have multiple requests in one file, you must separate them using:

###

Correct:

### Get posts
GET https://jsonplaceholder.typicode.com/posts

###

### Get single post
GET https://jsonplaceholder.typicode.com/posts/1

🚨 BIG ISSUE #1 — `---` vs `###`

Many developers coming from Markdown or other tools use:

---

This is WRONG for REST Client.

If you write:

POST https://example.com/api
Content-Type: application/json

{
  "name": "Ajit"
}
---

VS Code will send this entire block:

{
  "name": "Ajit"
}
---

And your backend (for example, Django REST Framework) will throw:

JSON parse error - Extra data

Because:

--- is treated as part of the JSON body.

✅ Rule:

Only ### separates requests.

Nothing else works.

5️⃣ Using Variables (Cleaner & Reusable)

Instead of hardcoding URLs:

@baseUrl = https://jsonplaceholder.typicode.com

### Get posts
GET {{baseUrl}}/posts

Works immediately.

But this doesn’t solve environment-specific values like tokens.

6️⃣ Setting Up Environments (Local & Production)

There are two approaches:

❌ Attempt 1 — `rest-client.environment.json` (Did Not Work Initially)

You create:

rest-client.environment.json

In project root:

{
  "local": {
    "baseUrl": "http://127.0.0.1:8000",
    "authToken": "real-token"
  },
  "prod": {
    "baseUrl": "https://api.example.com",
    "authToken": "prod-token"
  }
}

And use:

GET {{baseUrl}}/api/users
Authorization: Token {{authToken}}

Expected behavior:
Bottom-right environment selector appears.

Problem:
It didn’t appear.

Why?

Because in some workspace setups:

Multi-root workspace
Custom VS Code config
REST Client not auto-detecting root file

The extension didn’t auto-load the file.

✅ Final Working Solution — `.vscode/settings.json`

Inside project root:

.vscode/settings.json

Add:

{
  "rest-client.environmentVariables": {
    "local": {
      "baseUrl": "http://127.0.0.1:8000",
      "authToken": "your-real-token"
    },
    "prod": {
      "baseUrl": "https://api.example.com",
      "authToken": "prod-token"
    }
  }
}

Reload VS Code.

Now:

Open .http
Bottom-right shows environment selector
Choose local or prod

Variables resolve correctly.

Problem solved.

⚠️ Important Security Note

.vscode/settings.json now contains tokens.

That means:

You MUST add this to `.gitignore`:

.vscode/

Otherwise you risk committing:

API tokens
Secrets
Environment configs

Safer production pattern:

.vscode/settings.json          (ignored)
.vscode/settings.example.json  (committed template)

Example template:

{
  "rest-client.environmentVariables": {
    "local": {
      "baseUrl": "http://127.0.0.1:8000",
      "authToken": "paste-token-here"
    }
  }
}

7️⃣ Final Clean Project Structure

project/
│
├── api.http
├── .vscode/
│   └── settings.json   (ignored)
├── .gitignore
└── README.md

.gitignore:

.vscode/

8️⃣ When Should You Still Use Postman?

Use Postman when:

You need automated test scripting
Team collaboration on collections
Complex OAuth flows
Mock servers

Use api.http when:

You want repo-based documentation
Backend development speed matters
You want zero tool switching
You prefer Git-tracked API specs

9️⃣ Final Thoughts

For experienced developers, api.http with REST Client feels like:

curl, but readable
Postman, but version-controlled
Documentation, but executable

It removes friction.

And once you fix:

--- vs ###
Environment configuration quirks

It becomes a powerful, lightweight part of your backend workflow.

Stop Clicking Around: API Testing with `.http` Files (From Beginner to Pro)

Ajit Kumar — Mon, 16 Feb 2026 10:12:24 +0000

Modern API development does not require heavyweight GUI tools. If you are building APIs with Django REST Framework or any backend stack, you can test, document, and version your APIs using simple .http files — directly inside Visual Studio Code.

This tutorial walks from beginner to advanced usage of the .http approach, compares it with tools like Postman, HTTPie, and Swagger, and shows how to build a clean, executable API workflow.

Why `.http` Files?

A .http file is a plain text file containing HTTP requests.

Example:

GET http://127.0.0.1:8000/api/blogs/
Accept: application/json

With the REST Client extension in VS Code, you can:

Click “Send Request”
See status code
View headers
Inspect JSON response
Measure response time

All without leaving your editor.

This gives you:

Executable documentation
Version-controlled API tests
Zero external dependency
Simple text-based workflow

Part 1 — Beginner Level

Step 1: Install REST Client

Open VS Code → Extensions → Install:

REST Client (by Huachao Mao)

Step 2: Create `api.http`

Create a file:

api.http

Add:

@baseUrl = http://127.0.0.1:8000

### List Blogs
GET {{baseUrl}}/api/blogs/
Accept: application/json

Start your server:

python manage.py runserver

Click Send Request.

You’ll see:

200 OK
Response body
Headers
Timing

That’s it.

You just tested your API with a text file.

Part 2 — Intermediate Usage

Now let’s add CRUD operations.

### Create Blog
POST {{baseUrl}}/api/blogs/
Content-Type: application/json

{
  "title": "API Testing Simplified",
  "content": "Using .http files improves workflow."
}

### Get Blog by ID
GET {{baseUrl}}/api/blogs/1/

### Update Blog
PUT {{baseUrl}}/api/blogs/1/
Content-Type: application/json

{
  "title": "Updated Title",
  "content": "Updated content."
}

### Delete Blog
DELETE {{baseUrl}}/api/blogs/1/

Now your file:

Documents endpoints
Stores example payloads
Acts as test suite
Becomes team reference

Part 3 — Advanced Techniques

1️⃣ Environment Switching

At the top:

@baseUrl = http://127.0.0.1:8000

Change to:

@baseUrl = https://staging.myapp.com

All requests update automatically.

2️⃣ Variables from Response

You can extract response data:

### Create Blog
POST {{baseUrl}}/api/blogs/
Content-Type: application/json

{
  "title": "Dynamic Test",
  "content": "Testing extraction"
}

Then reference:

GET {{baseUrl}}/api/blogs/{{createBlog.response.body.id}}/

Now requests become chained.

3️⃣ Authentication Headers

@token = your_jwt_token_here

GET {{baseUrl}}/api/secure-endpoint/
Authorization: Bearer {{token}}

4️⃣ Expected Response Documentation

You can document expected behavior:

# Expected 201 Created
# Metadata should auto-generate

POST {{baseUrl}}/api/blogs/
Content-Type: application/json

{
  "title": "AI",
  "content": "AI is powerful."
}

This becomes living API documentation.

How It Compares to Other Tools

1️⃣ Postman

Postman is powerful:

Pros:

GUI
Team collaboration
Automated test scripts
Cloud sync

Cons:

Requires account
Harder to version-control cleanly
Collections stored outside codebase
Heavy UI

.http advantage:

Lives in repo
Lightweight
No login
Git diff-friendly

2️⃣ HTTPie

HTTPie is CLI-based.

Example:

http GET http://127.0.0.1:8000/api/blogs/

Pros:

Beautiful CLI output
Fast
Scriptable

Cons:

Not self-documenting
Not clickable
Harder for beginners

.http sits between curl and Postman — readable and interactive.

3️⃣ Swagger / OpenAPI

Swagger auto-generates API documentation.

Pros:

Interactive UI
Schema validation
Industry standard

Cons:

Generated, not curated
Harder to add narrative examples
Not ideal for manual test workflows

Best practice:

Use Swagger for:

Public documentation
API contracts

Use .http for:

Development testing
Internal API examples
Regression testing

Why `.http` is Excellent for Teams

Stored in Git
Reviewable via pull request
Encourages documented APIs
Works offline
Zero SaaS dependency
Reproducible environments

Your repository now contains:

api.http

Which acts as:

API spec
Test runner
Example library
Debug tool

Scaling the Pattern

For large systems:

/api-tests/
    blog.http
    auth.http
    analytics.http

Organize per domain.

For multi-version APIs:

/api-tests/v1/

When Not to Use `.http`

Load testing (use k6 or Locust)
Automated CI assertions (use pytest + requests)
Public API documentation portal (use OpenAPI)

It is a developer-first tool, not an enterprise API gateway.

Final Recommendation

For solo developers, small teams, backend-focused engineers:

.http + VS Code is the fastest, cleanest API testing workflow available.

It removes friction.
It keeps documentation close to code.
It scales from beginner to advanced usage.

And it avoids unnecessary tooling overhead.

The Complete Guide to Ollama: Run Large Language Models Locally

Ajit Kumar — Mon, 16 Feb 2026 09:19:48 +0000

Imagine having the power of GPT-4 or Claude running entirely on your laptop—no internet required, no API costs, and complete privacy. Until recently, this seemed like a distant dream reserved for tech giants with massive server farms. But the landscape has changed dramatically. Thanks to Ollama, anyone with a modern computer can now run sophisticated AI models locally, whether you're coding on a plane at 35,000 feet, analyzing sensitive documents that can never touch the cloud, or simply experimenting with AI without watching your API bill climb. This comprehensive guide will walk you through everything you need to know about Ollama—from your first installation to building production-ready AI-powered applications that respect your privacy and your wallet.

Part A: Setting Up Ollama

Introduction to Ollama

Ollama is a powerful, open-source tool that enables you to run large language models (LLMs) locally on your own machine. Think of it as Docker for AI models—it packages everything you need to run models like Llama, Mistral, CodeLlama, and dozens of others into easy-to-use containers that work seamlessly on your computer.

Why use Ollama?

Privacy: Your data never leaves your machine. Perfect for sensitive code, personal documents, or proprietary information.
No API costs: Run unlimited queries without paying per token.
Offline capability: Work without internet connectivity once models are downloaded.
Speed: No network latency—responses are generated locally.
Experimentation: Try different models and configurations freely.
Integration: Works with various tools and development environments.

Installation

Ollama supports macOS, Linux, and Windows. The installation process is straightforward for all platforms.

macOS

Download the official installer from ollama.com:

# Or use Homebrew
brew install ollama

Linux

Run the installation script:

curl -fsSL https://ollama.com/install.sh | sh

This script supports most Linux distributions including Ubuntu, Debian, Fedora, and CentOS.

Windows

Download the Windows installer from ollama.com or use the preview version. Ollama on Windows runs natively and supports the same commands as other platforms.

Docker Installation

For containerized deployment:

docker pull ollama/ollama
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Testing Your Installation

After installation, verify Ollama is working:

# Check version
ollama --version

# Check if the service is running
ollama serve

You should see output indicating the Ollama server is running on http://localhost:11434.

To test with a simple model:

# Pull and run a small model
ollama run llama3.2

# You should get an interactive prompt
# Try: "Hello, how are you?"

If you see a response from the model, congratulations—Ollama is working correctly!

Updating Ollama

Keeping Ollama updated ensures you have the latest features, bug fixes, and security patches.

Why Update?

New model support: Access to the latest model architectures and versions
Performance improvements: Faster inference and better memory management
Bug fixes: Resolved issues and stability improvements
New features: Enhanced CLI commands and API capabilities
Security patches: Important security updates

How to Update

macOS (Homebrew):

brew update
brew upgrade ollama

macOS (Installer):
Download and run the latest installer from ollama.com—it will update your existing installation.

Linux:

curl -fsSL https://ollama.com/install.sh | sh

The script automatically detects and updates your existing installation.

Windows:
Download and run the latest installer.

Docker:

docker pull ollama/ollama:latest
docker stop ollama
docker rm ollama
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

After updating, verify the new version:

ollama --version

Understanding Ollama Models

Ollama supports a vast ecosystem of models, each optimized for different tasks. Here's a breakdown of the major categories:

General Purpose Models

Llama 3.2 (1B, 3B) - Meta's latest efficient models

Best for: General conversation, reasoning, lightweight tasks
Size: 1B parameters (740MB), 3B parameters (2GB)
Command: ollama run llama3.2 or ollama run llama3.2:1b

Llama 3.1 (8B, 70B, 405B) - Meta's powerful flagship models

Best for: Complex reasoning, long context (128K tokens)
Size: 8B (4.7GB), 70B (40GB), 405B (231GB)
Command: ollama run llama3.1 or ollama run llama3.1:70b

Mistral (7B) - Efficient and capable model

Best for: General purpose tasks with good speed/quality balance
Size: 4.1GB
Command: ollama run mistral

Mixtral (8x7B, 8x22B) - Mixture of Experts models

Best for: High-quality outputs with efficient inference
Size: 26GB (8x7B), 80GB (8x22B)
Command: ollama run mixtral

Coding Models

CodeLlama (7B, 13B, 34B, 70B) - Specialized for code

Best for: Code generation, completion, debugging
Size: 7B (3.8GB), 13B (7.4GB), 34B (19GB), 70B (39GB)
Command: ollama run codellama

DeepSeek Coder (6.7B, 33B) - Strong coding performance

Best for: Multiple programming languages, code explanation
Size: 6.7B (3.8GB), 33B (19GB)
Command: ollama run deepseek-coder

Qwen 2.5 Coder (1.5B, 7B, 32B) - Alibaba's coding model

Best for: Code generation with multilingual support
Command: ollama run qwen2.5-coder

Specialized Models

Llama 3.2 Vision (11B, 90B) - Multimodal understanding

Best for: Image analysis, visual question answering
Command: ollama run llama3.2-vision

Phi-3 (3.8B, 14B) - Microsoft's small language models

Best for: Efficient deployment, edge devices
Size: 3.8B (2.3GB), 14B (7.9GB)
Command: ollama run phi3

Gemma 2 (2B, 9B, 27B) - Google's open models

Best for: Safety-focused applications
Command: ollama run gemma2

Neural Chat (7B) - Fine-tuned for conversations

Best for: Interactive dialogue
Command: ollama run neural-chat

How to Choose a Model

For general use on consumer hardware: llama3.2:3b or mistral
For coding tasks: codellama:7b or deepseek-coder:6.7b
For best quality (with powerful GPU): llama3.1:70b or mixtral:8x22b
For resource-constrained systems: phi3:3.8b or llama3.2:1b
For vision tasks: llama3.2-vision

Starting and Stopping Ollama

Starting Ollama

The Ollama server needs to be running to handle model requests.

On macOS and Linux:

# Start in foreground (see logs)
ollama serve

# Or run as background service (automatically starts on boot)
# On macOS, Ollama runs as a LaunchAgent
# On Linux with systemd:
systemctl start ollama

On Windows:
Ollama runs as a service automatically after installation. Access it via the system tray icon.

Verify it's running:

curl http://localhost:11434
# Should return: "Ollama is running"

Stopping Ollama

On macOS:

# If running in foreground, press Ctrl+C
# If running as service:
launchctl unload ~/Library/LaunchAgents/com.ollama.ollama.plist

On Linux:

# If running in foreground, press Ctrl+C
# If running as systemd service:
systemctl stop ollama

On Windows:
Right-click the Ollama icon in the system tray and select "Quit Ollama."

Docker:

docker stop ollama

Serving a Specific Model

Once Ollama is running, you can interact with models in several ways.

Interactive Mode

Run a model and chat directly in the terminal:

ollama run llama3.2

This starts an interactive session. Type your prompts and press Enter. Type /bye to exit.

Pull a Model Without Running

Download a model for later use:

ollama pull codellama:13b

List Downloaded Models

See which models you have locally:

ollama list

Output shows model name, size, and last modified date.

Run with Custom Parameters

Control model behavior with parameters:

ollama run llama3.2 --verbose

# Or set parameters in the prompt
ollama run llama3.2 "Write a haiku" --temperature 0.8 --top-p 0.9

API Mode (Serving to Other Applications)

Ollama exposes an OpenAI-compatible API on http://localhost:11434:

# Generate a completion
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}'

This API allows integration with countless tools and applications.

Remove Models You Don't Need

Free up disk space:

ollama rm codellama:70b

Quick Reference Commands

# Installation & Updates
ollama --version              # Check version
curl -fsSL https://ollama.com/install.sh | sh  # Install/update

# Model Management
ollama pull <model>           # Download a model
ollama list                   # List local models
ollama rm <model>             # Delete a model
ollama show <model>           # Show model information

# Running Models
ollama run <model>            # Interactive mode
ollama run <model> "prompt"   # One-off generation
ollama serve                  # Start API server

# API Testing
curl http://localhost:11434   # Check if running
curl http://localhost:11434/api/tags  # List models via API

Part B: Real-World Use Cases

Now that you have Ollama set up, let's explore powerful ways to use it in your daily workflow.

Use Case 1: Local Coding with Claude Code and AI IDEs

One of the most powerful applications of Ollama is integrating local models with development tools. This gives you AI-powered coding assistance that's private, fast, and free.

Using Ollama with Claude Code

Claude Code is a command-line agentic coding tool. While it typically uses Anthropic's API, you can configure it to work with local models via Ollama for certain tasks.

Setup:

Install Claude Code:

npm install -g @anthropic-ai/claude-code

Configure to use Ollama as a secondary tool for code completion:

# You can use Ollama's OpenAI-compatible endpoint
export OLLAMA_API_BASE=http://localhost:11434/v1

Workflow:

Use Claude Code with Anthropic's API for complex agentic tasks (planning, multi-file refactoring)
Use local Ollama models for quick completions, code explanations, or offline work
Combine both: Claude for architecture, Ollama for implementation details

Using Ollama with Cursor

Cursor is a popular AI-powered IDE. Configure it to use Ollama:

Open Cursor Settings
Navigate to Models
Add custom model endpoint: http://localhost:11434/v1
Select your preferred model (e.g., codellama:13b)

Benefits:

No API costs for code completions
Faster responses (no network latency)
Private code never leaves your machine

Using Ollama with Continue.dev

Continue is a VS Code / JetBrains extension that brings AI assistance to your IDE:

Install the Continue extension
Configure ~/.continue/config.json:

{
  "models": [
    {
      "title": "CodeLlama Local",
      "provider": "ollama",
      "model": "codellama:13b",
      "apiBase": "http://localhost:11434"
    }
  ]
}

Use Cmd/Ctrl+L to chat with your local model
Highlight code and ask questions, generate tests, or refactor

Using Ollama with Tabby (Self-Hosted Code Completion)

Tabby provides GitHub Copilot-like code completion:

# Run Tabby with Ollama backend
docker run -it --gpus all \
  -p 8080:8080 \
  -v ~/.tabby:/data \
  tabbyml/tabby \
  serve --model TabbyML/CodeLlama-7B --device cuda

Configure your IDE to point to http://localhost:8080.

Best Practices for Local Coding Models

Choose the right model size: 7B models for real-time completions, 13B+ for deeper analysis
Use code-specific models: CodeLlama, DeepSeek Coder, or Qwen Coder perform better than general models
Combine tools: Use local models for most tasks, cloud models for complex reasoning
Fine-tune prompts: Local models benefit from clear, specific instructions
Monitor resource usage: Large models need significant RAM/VRAM

Use Case 2: Document Ingestion and RAG Workflows

Retrieval-Augmented Generation (RAG) is a technique where you feed your own documents to an LLM, enabling it to answer questions based on your specific data. Ollama is perfect for building private RAG systems.

What is RAG?

Traditional LLMs only know what they were trained on. RAG extends their knowledge by:

Ingesting your documents (PDFs, text files, web pages)
Chunking them into smaller pieces
Embedding these chunks into vector representations
Storing embeddings in a vector database
Retrieving relevant chunks when you ask a question
Generating answers using both the retrieved context and the LLM

Simple RAG with Ollama and Python

Here's a basic RAG implementation:

# Install dependencies
# pip install ollama chromadb langchain pypdf --break-system-packages

import ollama
import chromadb
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.document_loaders import PyPDFLoader

# 1. Load and chunk documents
loader = PyPDFLoader("your_document.pdf")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = text_splitter.split_documents(documents)

# 2. Create embeddings and store in vector DB
client = chromadb.Client()
collection = client.create_collection("my_docs")

for i, chunk in enumerate(chunks):
    # Use Ollama to generate embeddings
    embedding = ollama.embeddings(
        model="nomic-embed-text",
        prompt=chunk.page_content
    )["embedding"]

    collection.add(
        ids=[str(i)],
        embeddings=[embedding],
        documents=[chunk.page_content]
    )

# 3. Query the system
def ask_question(question):
    # Get embedding for question
    question_embedding = ollama.embeddings(
        model="nomic-embed-text",
        prompt=question
    )["embedding"]

    # Find relevant chunks
    results = collection.query(
        query_embeddings=[question_embedding],
        n_results=3
    )

    # Build context from retrieved chunks
    context = "\n\n".join(results["documents"][0])

    # Generate answer using Ollama
    response = ollama.chat(
        model="llama3.2",
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}"
        }]
    )

    return response["message"]["content"]

# Use it
answer = ask_question("What are the main points in the document?")
print(answer)

Advanced RAG with Ollama and Open Source Tools

Using Ollama with LangChain:

from langchain.llms import Ollama
from langchain.chains import RetrievalQA
from langchain.vectorstores import FAISS
from langchain.embeddings import OllamaEmbeddings

# Initialize Ollama
llm = Ollama(model="llama3.2")
embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Create vector store
vectorstore = FAISS.from_documents(chunks, embeddings)

# Create QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    chain_type="stuff"
)

# Ask questions
result = qa_chain.run("What does the document say about X?")

Using Ollama with AnythingLLM:

AnythingLLM is a full-stack RAG application with a beautiful UI:

Install: Download from anythingllm.com
Configure Ollama as LLM provider in settings
Upload documents via the UI
Start chatting with your documents

Using Ollama with Quivr (Your Second Brain):

Quivr is a RAG application for personal knowledge management:

# Clone and setup
git clone https://github.com/QuivrHQ/quivr
cd quivr
docker-compose up

# Configure to use Ollama in .env file
LLM_PROVIDER=ollama
OLLAMA_API_BASE=http://localhost:11434

Ingestion Workflow Best Practices

1. Document Preprocessing:

Clean text (remove headers, footers, page numbers)
Extract tables and images separately if needed
Use OCR for scanned documents (tesseract-ocr)

2. Chunking Strategies:

Fixed size: Simple, works for uniform content (500-1000 tokens)
Semantic: Split by paragraphs or sections (better context)
Recursive: LangChain's RecursiveCharacterTextSplitter (balanced)

3. Embedding Models:

nomic-embed-text: Best all-around, 137M parameters
mxbai-embed-large: Highest quality, 335M parameters
all-minilm: Fastest, lightweight option

4. Retrieval Optimization:

Use hybrid search (keyword + semantic)
Re-rank results with a cross-encoder
Adjust number of retrieved chunks (typically 3-5)

5. Prompt Engineering:

Context: [retrieved chunks]

Based ONLY on the context above, answer the following question.
If the answer is not in the context, say "I don't have enough information."

Question: {user_question}

Answer:

Production RAG Considerations

Vector Database: Use Qdrant, Weaviate, or Pinecone for scale
Caching: Cache embeddings to avoid recomputation
Monitoring: Track retrieval accuracy and response quality
Updates: Implement incremental updates when documents change
Multi-modal: Use vision models for PDFs with images/charts

Performance Tips for Ollama

GPU Acceleration: Ollama automatically uses GPU if available (NVIDIA, AMD, or Apple Silicon)
Memory Management: Larger models need more RAM/VRAM (check with ollama show <model>)
Concurrent Requests: Ollama handles multiple requests, but memory is shared
Model Quantization: Use quantized models (Q4, Q5) for better performance with acceptable quality loss

Conclusion

Ollama opens up a world of possibilities for running powerful AI models on your own hardware. From coding assistance to document analysis, from privacy-preserving workflows to cost-free experimentation, the applications are endless.

Key Takeaways:

Ollama makes running local LLMs accessible and practical
Choose models based on your hardware and use case
Integration with development tools enhances productivity
RAG workflows unlock the power of your private data
Local models provide privacy, speed, and zero API costs

Next Steps:

Install Ollama and test a few models
Integrate with your favorite IDE or coding tool
Build a simple RAG system with your documents
Explore the Ollama model library at ollama.com/library
Join the community at github.com/ollama/ollama

Happy local LLM exploration! 🚀

Why Your Deep Learning Job Dies After SSH Logout — A Practical Guide to Persistent Linux Sessions

Ajit Kumar — Mon, 16 Feb 2026 05:28:43 +0000

You SSH into a remote server.
You start a long-running deep learning training job:

python train.py

Everything works. GPU utilization is high. Logs are printing.

Then your laptop sleeps.
Or WiFi drops.
Or you close the terminal.

You reconnect to the server…

Your training job is gone.

But on another server, your training continues even after you disconnect.

Why?

This is not random behavior. It is deterministic Unix session management.

Let’s break it down precisely.

The Root Cause: Controlling Terminals and `SIGHUP`

When you SSH into a Linux server:

SSH creates a login session.
A shell process (e.g., bash) is started.
That shell is attached to a controlling terminal (TTY).
Any command you run becomes a child process of that shell.

Process tree example:

sshd
 └── bash
      └── python train.py

When your SSH session ends:

The terminal disappears.
The shell receives a SIGHUP (Hang Up signal).
The shell forwards SIGHUP to its child processes.
Your Python training process exits.

This is default POSIX behavior.

The operating system is doing exactly what it is designed to do.

Why It Works on Another Server

If training survives logout on another machine, one of these is true:

It was started inside tmux
It was started inside screen
It was launched using nohup
It was disowned
It was started as a systemd service
The SSH daemon is configured differently (rare but possible)

The most common reason: tmux is being used.

Real-World Scenario (Deep Learning Context)

Imagine this:

You launch a 3-day transformer training run.
You’re using expensive GPU hardware.
Model checkpoints save every 2 hours.
Your internet drops after 4 hours.

Without proper session handling:

Training dies immediately.
GPU memory is released.
You lose progress since the last checkpoint.
Compute time is wasted.
Experiment reproducibility is impacted.

In production ML workflows, this is unacceptable.

Proper Solutions (Ranked by Maturity)

1️⃣ tmux — The Research Standard

tmux creates a persistent terminal multiplexer. Your training process attaches to tmux — not directly to SSH.

Install:

sudo apt install tmux

Start session:

tmux new -s training

Run training:

python train.py

Detach safely:

Ctrl + B, then D

Logout freely. Training continues.

Reconnect later:

tmux attach -t training

Why this works:

tmux owns the controlling TTY.
SSH disconnection does not kill tmux.
Your process remains alive.

This is the recommended workflow for ML engineers and researchers.

2️⃣ nohup — Minimal Fix

nohup python train.py > train.log 2>&1 &

What it does:

Ignores SIGHUP
Redirects output
Runs in background

Good for:

Simple batch jobs
Fire-and-forget scripts

Downside:

No interactive recovery
Harder process management

3️⃣ disown — Shell-Level Detachment

python train.py &
disown

This removes the process from the shell job table.

Less robust than tmux but works.

4️⃣ systemd — Production-Grade

For structured environments:

Create a service file:

/etc/systemd/system/ml-training.service

Then:

sudo systemctl daemon-reload
sudo systemctl start ml-training

Benefits:

Auto restart on failure
Resource control
Logging via journalctl
Reproducible startup
Can run at boot

This is appropriate for:

Production ML pipelines
Enterprise GPU clusters
Persistent inference services

How to Diagnose Your Current Setup

Before disconnecting, check:

ps -o pid,ppid,cmd -p <python_pid>

If the parent process (PPID) is your shell → it will die when SSH disconnects.

Check if you are inside tmux:

echo $TMUX

If empty → you are not using tmux.

Server Admin Configuration Considerations

From the system administration side, there are several configurations that impact user sessions.

1️⃣ SSH Daemon Settings

File:

/etc/ssh/sshd_config

Relevant options:

ClientAliveInterval
ClientAliveCountMax
TCPKeepAlive

These control idle timeout behavior — but they do NOT prevent SIGHUP when a session closes.

They only affect when the connection is dropped.

2️⃣ systemd User Session Settings

Modern Linux distributions use systemd-logind.

File:

/etc/systemd/logind.conf

Key option:

KillUserProcesses=yes

If enabled:

All user processes are killed when the session ends.
Even tmux sessions may die.

To allow persistent processes:

KillUserProcesses=no

After change:

sudo systemctl restart systemd-logind

This setting is critical on shared GPU servers.

3️⃣ Resource Policies

Admins may enforce:

cgroups limits
SLURM job scheduler policies
Idle job cleanup scripts

On managed clusters, processes outside scheduler control may be automatically terminated.

In HPC environments, users should launch jobs via:

sbatch
srun
qsub
Kubernetes job manifests

Not direct SSH shells.

When Admin Intervention Is Required

If:

tmux sessions still die on logout
background processes terminate unexpectedly
training stops even with nohup

Then the admin should verify:

KillUserProcesses setting
PAM session configuration
Custom logout hooks
Cluster job scheduler enforcement
cgroup cleanup policies

On well-configured ML servers, tmux should work without special privileges.

Best Practice for ML Engineers

For experimentation:

Always use tmux
Enable periodic checkpointing
Log to file
Monitor GPU usage via nvidia-smi

For production:

Use systemd or a scheduler
Configure restart policies
Use structured logging
Implement health checks

Final Takeaway

Your training stops because:

It is attached to a shell that receives SIGHUP when SSH disconnects.

This is expected behavior.

The solution is not “keep SSH open.”

The solution is:

Decouple your process from the SSH session.
Use tmux, nohup, or systemd appropriately.
Ensure systemd-logind is not killing user processes.

Once configured correctly, your deep learning jobs should survive network instability, laptop sleep, and terminal closure — exactly as they should in a robust ML workflow.

From Zero to Cached: Building a High-Performance Housing Portal with Django, Next.js, and Redis- Part 7 - Deployment

Ajit Kumar — Fri, 13 Feb 2026 08:57:03 +0000

Part 7: Deployment — From localhost:3000 to production.yoursite.com

In Part 1, we containerized everything. In Part 2, we built the database. In Part 3, we cached the API. In Part 4, we optimized queries. In Part 5, we built the UI. In Part 6, we integrated the CDN. Today, we deploy to production.

If you're jumping in here, you need the full context. Part 1 through Part 6 built a complete, optimized housing portal that runs on localhost. If you're continuing from Part 6, you have a working app with images, caching, and all the features — but only you can access it.

Today we make it public. By the end of this post, you'll understand deployment architectures, choose the right hosting strategy for your needs, and have a live site accessible from anywhere. We'll cover four complete deployment paths — from completely free (for demos) to enterprise-grade AWS infrastructure.

The Deployment Challenge

Deploying a full-stack application isn't like deploying a simple website. We have:

A Next.js frontend — needs Node.js, build process, edge caching
A Django backend — needs Python, Gunicorn, environment variables
A PostgreSQL database — needs persistence, backups, connection pooling
A Redis cache — needs memory, persistence configuration
A Cloudinary CDN — already deployed, just needs credentials
Static files — CSS, JS, images that need to be served efficiently
Environment variables — different for dev, staging, production
SSL certificates — HTTPS is mandatory for production
Domain names — custom domains or free subdomains

There isn't one "right" way to deploy all of this. The right choice depends on your budget, traffic, technical comfort, and requirements.

This post shows you four complete deployment architectures and helps you choose:

Part A: The Architecture Decision Tree

Before we deploy anything, we need to understand the options.

The Four Deployment Paths

Path 0: The Free Tier (Demo/Portfolio)

Cost: $0/month
Complexity: Low-Medium
Best for: This blog series, demos, portfolio projects, MVPs
Tradeoff: Cold starts (30s delay after inactivity), usage limits

Path 1: Railway Only (Simple)

Cost: ~$20/month
Complexity: Low
Best for: Side projects, MVPs, small apps
Tradeoff: Higher cost at scale, less control

Path 2: Vercel + Railway (Optimal)

Cost: ~$15-35/month
Complexity: Medium
Best for: Production apps, startups, SaaS
Tradeoff: More moving parts to manage

Path 3: AWS DIY (Enterprise)

Cost: ~$90+/month
Complexity: High
Best for: Enterprise, compliance needs, learning DevOps
Tradeoff: Requires infrastructure knowledge

Decision Criteria

Choose Path 0 (Free Tier) if:

✅ Following this blog series and don't want to spend money
✅ Building a portfolio project or demo
✅ Testing an MVP with < 10 daily users
✅ Cold starts (30s first load) are acceptable
✅ You plan to upgrade when you get traction

Choose Path 1 (Railway Only) if:

✅ Want the simplest deployment possible
✅ Have a small budget ($20/month)
✅ Don't want to manage infrastructure
✅ Need automatic SSL and deployments
✅ Okay with vendor lock-in

Choose Path 2 (Vercel + Railway) if:

✅ Need production-grade performance
✅ Want global edge caching for frontend
✅ Can manage multiple services
✅ Need 99.9% uptime
✅ This is a real product with users

Choose Path 3 (AWS DIY) if:

✅ Need full control of infrastructure
✅ Have compliance requirements (HIPAA, SOC2)
✅ Want to learn DevOps
✅ Have high traffic (100k+ requests/day)
✅ Can manage servers

📊 [Architecture Decision Tree]

Architecture Patterns Comparison

Let's visualize each architecture before we dive into implementation.

Pattern 1: The Monolith (Railway Only)

Everything runs on one platform. Simple but less optimized.

User → Railway (Frontend + Backend + DB + Redis)

📊 [Monolith Architecture]

Pattern 2: The Modern Split (Vercel + Railway)

Frontend on edge (Vercel), backend on PaaS (Railway). Optimal performance.

User → Vercel Edge (Next.js) → Railway (Django + DB + Redis)

📊 [Modern Split Architecture]

Pattern 3: The Cloud DIY (AWS)

Self-managed infrastructure. Maximum control, maximum complexity.

User → CloudFront → ALB → EC2 (Django) → RDS (PostgreSQL)
                                        → ElastiCache (Redis)

📊 [AWS Architecture]

Cost vs Complexity Matrix

Deployment Path	Monthly Cost	Setup Time	Maintenance	Performance	Scalability
Path 0: Free Tier	$0	2 hours	None	3/5 ⭐	Limited
Path 1: Railway	$20	1 hour	Minimal	4/5 ⭐	Medium
Path 2: Vercel+Railway	$15-35	3 hours	Low	5/5 ⭐	High
Path 3: AWS DIY	$90+	8 hours	High	5/5 ⭐	Very High

For this tutorial series, Path 0 (Free Tier) is perfect for learning. When you're ready to launch, Path 2 (Vercel + Railway) is the sweet spot.

Part B: Production Readiness — Preparing the Code

Before we deploy anywhere, we need to prepare our code for production. The settings that work in development are not safe for production.

Step 1: Create Production Django Settings

We'll use a split settings pattern: settings/base.py, settings/development.py, settings/production.py.

Create backend/core/settings/ directory:

mkdir -p backend/core/settings

Move existing settings to base:

mv backend/core/settings.py backend/core/settings/base.py

Create backend/core/settings/__init__.py:

"""
core/settings/__init__.py

Import the correct settings based on DJANGO_ENV environment variable.
Defaults to development if not set.
"""

import os

env = os.environ.get('DJANGO_ENV', 'development')

if env == 'production':
    from .production import *
elif env == 'staging':
    from .staging import *
else:
    from .development import *

Create backend/core/settings/development.py:

"""
core/settings/development.py

Development-specific settings.
Imports from base and overrides as needed.
"""

from .base import *

DEBUG = True

ALLOWED_HOSTS = ['*']

# Use console email backend in development
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

# Show SQL queries in debug toolbar
DEBUG_TOOLBAR_CONFIG = {
    'SHOW_TOOLBAR_CALLBACK': lambda request: DEBUG,
}

Create backend/core/settings/production.py:

"""
core/settings/production.py

Production settings with security hardening.
"""

from .base import *
import os

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

# Must be set in environment variables
ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS', '').split(',')
CSRF_TRUSTED_ORIGINS = os.environ.get('CSRF_TRUSTED_ORIGINS', '').split(',')

# Security settings
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
X_FRAME_OPTIONS = 'DENY'
SECURE_HSTS_SECONDS = 31536000  # 1 year
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True

# Use a strong secret key from environment
SECRET_KEY = os.environ.get('SECRET_KEY')
if not SECRET_KEY:
    raise ValueError('SECRET_KEY environment variable must be set in production')

# Database connection pooling
DATABASES['default']['CONN_MAX_AGE'] = 600  # 10 minutes

# Static files (WhiteNoise)
MIDDLEWARE.insert(1, 'whitenoise.middleware.WhiteNoiseMiddleware')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

# CORS settings for production
CORS_ALLOWED_ORIGINS = os.environ.get('CORS_ALLOWED_ORIGINS', '').split(',')

# Logging configuration
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'root': {
        'handlers': ['console'],
        'level': 'INFO',
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': False,
        },
    },
}

Install WhiteNoise for static file serving:

cd backend
pip install whitenoise
pip freeze > requirements.txt

Step 2: Create Next.js Environment Files

Create frontend/.env.production:

# Production environment variables
# Update these with your actual production URLs

NEXT_PUBLIC_API_URL=https://your-backend.railway.app
# or
NEXT_PUBLIC_API_URL=https://api.yourdomain.com

Create frontend/.env.local.example:

# Example environment variables
# Copy this to .env.local and fill in your values

NEXT_PUBLIC_API_URL=http://localhost:8000

Update frontend/lib/api.ts to use environment variables:

/**
 * lib/api.ts
 * 
 * Updated to use environment variables for production.
 */

const getBaseURL = () => {
  // In production, use the environment variable
  if (process.env.NEXT_PUBLIC_API_URL) {
    return process.env.NEXT_PUBLIC_API_URL;
  }

  // Development: differentiate server vs client
  if (typeof window === 'undefined') {
    return 'http://backend:8000';
  }
  return 'http://localhost:8000';
};

export const API_BASE_URL = getBaseURL();

// ... rest of the file stays the same

Step 3: Update Dependencies

Create backend/requirements/base.txt:

Django==5.0.1
djangorestframework==3.14.0
django-cors-headers==4.3.1
django-filter==23.5
psycopg2-binary==2.9.9
redis==5.0.1
django-redis==5.4.0
cloudinary==1.37.0
django-cloudinary-storage==0.3.0
gunicorn==21.2.0
whitenoise==6.6.0

Create backend/requirements/production.txt:

-r base.txt

# Production-specific packages
sentry-sdk==1.39.2

Create backend/requirements/development.txt:

-r base.txt

# Development-specific packages
django-debug-toolbar==4.2.0

Update backend/requirements.txt:

-r requirements/production.txt

Step 4: Create .gitignore

Ensure these are in .gitignore:

# Environment variables
.env
.env.local
.env.production
.env.staging

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
venv/
ENV/

# Django
*.log
db.sqlite3
staticfiles/
mediafiles/

# Next.js
.next/
out/
node_modules/

# IDEs
.vscode/
.idea/
*.swp
*.swo

Step 5: Generate Production Secrets

Generate a secure SECRET_KEY for Django:

python -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())'

Save this — you'll need it in environment variables.

Part C: Path 0 — The Free Tier Deployment (RECOMMENDED FOR THIS TUTORIAL)

This is the perfect path for following this blog series without spending money. We'll use 100% free tiers.

What we're deploying:

Frontend: Vercel (free tier)
Backend: Render.com (free tier)
Database: Neon.tech (free PostgreSQL, 0.5GB)
Redis: Upstash (free tier, 10k requests/day)
Images: Cloudinary (already set up, free tier)

Total cost: $0/month

Tradeoffs to accept:

Backend sleeps after 15 minutes of inactivity (30-second cold start on first request)
Database limited to 0.5GB (~50,000 property records)
Redis limited to 10,000 requests per day
Free subdomains only (no custom domain)

This is perfect for demos, portfolios, and learning deployments.

Step 1: Deploy Database (Neon.tech)

Go to neon.tech
Sign up with GitHub (no credit card required)
Click "Create a project"
Name: housing-portal-db
Region: Choose closest to you (us-east-1, eu-central-1, etc.)
Click "Create project"

You'll get a connection string like:

postgres://username:password@ep-long-string.us-east-1.aws.neon.tech/neondb?sslmode=require

Save this — you'll need it for the backend.

Neon automatically handles:

✅ SSL connections
✅ Automatic backups
✅ Connection pooling
✅ Scales to zero (no usage = no resources used)

Step 2: Deploy Redis (Upstash)

Go to upstash.com
Sign up with GitHub
Click "Create Database"
Name: housing-portal-redis
Type: Regional
Region: Choose same as Neon (for lower latency)
Eviction: No eviction
Click "Create"

You'll get a connection string like:

redis://default:password@us1-host.upstash.io:port

Or REST URL (alternative if Redis protocol doesn't work):

https://us1-host.upstash.io

Save this connection string.

Step 3: Deploy Backend (Render.com)

Go to render.com
Sign up with GitHub
Click "New +" → "Web Service"
Connect your GitHub repository
Select the housing-caching-demo repository
Configure:

Settings:

Name: housing-portal-backend
Region: Same as database
Branch: main
Root Directory: backend
Runtime: Python 3
Build Command: pip install -r requirements.txt && python manage.py collectstatic --noinput && python manage.py migrate
Start Command: gunicorn core.wsgi:application --bind 0.0.0.0:$PORT
Instance Type: Free

Environment Variables:
Click "Add Environment Variable" for each:

DJANGO_ENV=production
SECRET_KEY=<your-generated-secret-key>
DEBUG=False
ALLOWED_HOSTS=<will-fill-after-deploy>.onrender.com
CSRF_TRUSTED_ORIGINS=https://<will-fill-after-deploy>.onrender.com
DATABASE_URL=<neon-postgres-url>
REDIS_URL=<upstash-redis-url>
CLOUDINARY_CLOUD_NAME=<your-cloudinary-name>
CLOUDINARY_API_KEY=<your-cloudinary-key>
CLOUDINARY_API_SECRET=<your-cloudinary-secret>
CORS_ALLOWED_ORIGINS=https://your-app.vercel.app

Note: You'll update ALLOWED_HOSTS and CORS_ALLOWED_ORIGINS after you know the actual URLs.

Click "Create Web Service"

Render will:

Build your app (takes 2-3 minutes)
Run migrations
Start the server

You'll get a URL like: https://housing-portal-backend.onrender.com

Test it:

curl https://housing-portal-backend.onrender.com/api/properties/cached/

If you get JSON back, it works! If you get a 403 or CORS error, that's expected — we haven't configured CORS yet.

Now go back and update environment variables:

ALLOWED_HOSTS=housing-portal-backend.onrender.com
CSRF_TRUSTED_ORIGINS=https://housing-portal-backend.onrender.com

Step 4: Deploy Frontend (Vercel)

Go to vercel.com
Sign up with GitHub
Click "Add New..." → "Project"
Import your housing-caching-demo repository
Configure:

Settings:

Framework Preset: Next.js
Root Directory: frontend
Build Command: npm run build (default)
Output Directory: .next (default)

Environment Variables:
Click "Add" for each environment:

NEXT_PUBLIC_API_URL=https://housing-portal-backend.onrender.com

Click "Deploy"

Vercel will:

Install dependencies
Build the Next.js app
Deploy to edge network globally

You'll get a URL like: https://housing-portal-abc123.vercel.app

Test it:
Open that URL in your browser. You should see your housing portal!

Step 5: Connect Frontend to Backend (CORS)

Now update the backend CORS settings to allow requests from Vercel:

Go back to Render.com → your backend service → Environment → Edit CORS_ALLOWED_ORIGINS:

CORS_ALLOWED_ORIGINS=https://housing-portal-abc123.vercel.app

Click "Save Changes" → Render will redeploy.

Test again:
Refresh your Vercel URL. The property listings should load.

Step 6: Understanding Cold Starts

The First Request:
Since Render's free tier sleeps after 15 minutes of inactivity, the first request will:

Wake up the server (15-30 seconds)
Connect to database
Return response

Subsequent Requests:
Instant (as long as you keep using it within 15 minutes).

How to explain this to users:
Add a note in your README or on the site:

"This is a demo deployed on free hosting. The first load may take 30 seconds as the server wakes up. Subsequent loads are instant."

Step 7: Verifying the Deployment

Check each component:

Frontend: https://your-app.vercel.app — should load the homepage
Backend API: https://your-backend.onrender.com/api/properties/cached/ — should return JSON
Database: Properties should display (proves DB connection works)
Redis: Navigate between pages quickly (proves cache works)
Images: Property cards should show images (proves Cloudinary works)

Open DevTools and check:

Network tab → No CORS errors
No console errors
Images load from res.cloudinary.com

📊 [Exercise / Showcase Free Tier Deployment - Live Site]

Instructions: Capture a screenshot of your deployed site on Vercel showing property listings loading successfully.

Step 8: Free Tier Limits and Monitoring

Neon.tech:

0.5GB storage (check usage in dashboard)
Unlimited queries
Auto-scales to zero (saves resources when not in use)

Upstash:

10,000 requests per day
Dashboard shows usage
Resets daily

Render.com:

750 hours/month (31 days = 744 hours — enough for 1 service)
512MB RAM
Sleeps after 15 min inactivity

Vercel:

Unlimited bandwidth (fair use)
Unlimited deployments
100GB bandwidth/month (more than enough)

Cloudinary:

25GB storage
25GB bandwidth/month
Unlimited transformations

When you'll hit limits:

Upstash: ~400 users per day (assuming 25 requests each)
Neon: ~50,000 properties (way more than needed for demo)
Never hitting Vercel/Cloudinary limits for a demo

Step 9: Migration Path (When to Upgrade)

Free Tier is fine when:

< 50 daily active users
Demo/portfolio project
MVP validation
Cold starts are acceptable

Upgrade to Railway ($5/month) when:

50-500 daily active users
Can't tolerate cold starts
Need custom domain
Exceed Redis 10k requests/day

Upgrade to Vercel + Railway ($15-35/month) when:

500+ daily active users
Need 99.9% uptime
Global audience (Vercel edge network)
This is a real business

Step 10: Adding a Custom Domain (Optional)

Even on free tiers, you can add a custom domain if you have one.

Vercel:

Go to your project → Settings → Domains
Add your domain (e.g., housing.yourdomain.com)
Update DNS:
- Type: CNAME
- Name: housing
- Value: cname.vercel-dns.com
Vercel automatically provisions SSL

Render:

Go to your service → Settings → Custom Domains
Add api.yourdomain.com
Update DNS:
- Type: CNAME
- Name: api
- Value: your-service.onrender.com
Render automatically provisions SSL

Update environment variables:

Render: ALLOWED_HOSTS=api.yourdomain.com
Render: CSRF_TRUSTED_ORIGINS=https://api.yourdomain.com
Vercel: (no change needed)

Path 0 Complete! You now have a fully deployed housing portal for $0/month. This is perfect for following the tutorial, building your portfolio, or validating an MVP.

Part D: Path 1 — Railway Only (Simple)

If you need to avoid cold starts and want everything in one place, Railway is the simplest paid option.

What we're deploying:

Frontend: Railway
Backend: Railway
Database: Railway PostgreSQL
Redis: Railway Redis

Cost: ~$20/month ($5/service × 4 services)

Pros:

No cold starts
Everything in one dashboard
Automatic SSL
Easy scaling
Preview deployments

Step 1: Create Railway Account

Go to railway.app
Sign up with GitHub
Click "New Project"
Select "Deploy from GitHub repo"
Choose housing-caching-demo

Step 2: Add Services

Railway will create one service by default. We need 4 total.

Add PostgreSQL:

Click "New" → "Database" → "Add PostgreSQL"
Railway provisions a database automatically
Connection string appears in the service variables

Add Redis:

Click "New" → "Database" → "Add Redis"
Railway provisions Redis automatically

Add Backend Service:

Click "New" → "GitHub Repo" → Select your repo
Configure:
- Root Directory: backend
- Build Command: pip install -r requirements.txt && python manage.py collectstatic --noinput && python manage.py migrate
- Start Command: gunicorn core.wsgi:application --bind 0.0.0.0:$PORT

Add Frontend Service:

Click "New" → "GitHub Repo" → Select same repo
Configure:
- Root Directory: frontend
- Build Command: npm install && npm run build
- Start Command: npm start

Step 3: Configure Environment Variables

Backend Service:
Click on backend → Variables → Add Reference Variables:

Railway automatically creates DATABASE_URL and REDIS_URL when you add those services.

Add these manually:

DJANGO_ENV=production
SECRET_KEY=<your-secret-key>
DEBUG=False
ALLOWED_HOSTS=${{RAILWAY_PUBLIC_DOMAIN}}
CSRF_TRUSTED_ORIGINS=https://${{RAILWAY_PUBLIC_DOMAIN}}
CLOUDINARY_CLOUD_NAME=<your-value>
CLOUDINARY_API_KEY=<your-value>
CLOUDINARY_API_SECRET=<your-value>
CORS_ALLOWED_ORIGINS=https://${{frontend.RAILWAY_PUBLIC_DOMAIN}}

Frontend Service:

NEXT_PUBLIC_API_URL=https://${{backend.RAILWAY_PUBLIC_DOMAIN}}

The ${{}} syntax is Railway's variable reference system. It automatically resolves to the actual domain of each service.

Step 4: Generate Public Domains

Each service gets a Railway domain automatically:

Click backend service → Settings → Public Networking → Generate Domain
Click frontend service → Settings → Public Networking → Generate Domain

You'll get:

Backend: https://housing-backend-production-abc.up.railway.app
Frontend: https://housing-frontend-production-xyz.up.railway.app

Step 5: Deploy

Railway automatically deploys when you:

Push to GitHub
Change environment variables
Update settings

Watch the deployment logs in each service.

Test:
Open the frontend URL. Everything should work!

Step 6: Custom Domain (Optional)

Add custom domain to frontend:

Frontend service → Settings → Custom Domains
Add housing.yourdomain.com
Update DNS (CNAME to Railway domain)
Railway provisions SSL automatically

Add custom domain to backend:

Backend service → Settings → Custom Domains
Add api.yourdomain.com
Update DNS
Update CORS_ALLOWED_ORIGINS to include new domain

Step 7: Railway Pricing

Railway charges $5/month per service that uses resources.

Free tier:

$5 free credit/month
Shared resources
Good for one service

Hobby plan:

$5/service/month
Dedicated resources
No sleep
Automatic scaling

For our 4 services:

PostgreSQL: $5/month
Redis: $5/month
Backend: $5/month
Frontend: $5/month Total: $20/month

Part E: Path 2 — Vercel + Railway (RECOMMENDED FOR PRODUCTION)

This is the optimal architecture for production apps. Frontend on Vercel's edge network (global CDN), backend on Railway (simple PaaS).

What we're deploying:

Frontend: Vercel (edge network, 100+ locations globally)
Backend: Railway
Database: Railway PostgreSQL
Redis: Railway Redis or Upstash

Cost: $15-35/month

Vercel: Free (hobby) or $20/month (pro)
Railway: $15/month (3 services: backend, DB, Redis)

Why this is optimal:

✅ Frontend served from edge (lowest latency globally)
✅ Backend centralized (simpler to manage)
✅ Best performance-to-cost ratio
✅ Each component runs on its optimal platform

Step 1: Deploy Backend to Railway

Follow the same steps as Path 1 for:

PostgreSQL service
Redis service
Backend service

Environment variables:

DJANGO_ENV=production
SECRET_KEY=<your-secret>
ALLOWED_HOSTS=${{RAILWAY_PUBLIC_DOMAIN}}
CSRF_TRUSTED_ORIGINS=https://${{RAILWAY_PUBLIC_DOMAIN}}
DATABASE_URL=${{Postgres.DATABASE_URL}}
REDIS_URL=${{Redis.REDIS_URL}}
CLOUDINARY_CLOUD_NAME=<your-value>
CLOUDINARY_API_KEY=<your-value>
CLOUDINARY_API_SECRET=<your-value>
CORS_ALLOWED_ORIGINS=https://your-app.vercel.app

Generate domain for backend.

You'll get: https://housing-backend-production.up.railway.app

Step 2: Deploy Frontend to Vercel

Go to vercel.com
Import project from GitHub

Configure:

Root Directory: frontend
Framework: Next.js
Environment Variable:

 NEXT_PUBLIC_API_URL=https://housing-backend-production.up.railway.app

Deploy

You'll get: https://housing-portal.vercel.app

Step 3: Update CORS

Go back to Railway → Backend → Environment Variables → Update:

CORS_ALLOWED_ORIGINS=https://housing-portal.vercel.app

Save and redeploy.

Step 4: Test the Integration

Open https://housing-portal.vercel.app:

First load: Served from Vercel edge (fast globally)
API calls: Go to Railway backend
Images: Served from Cloudinary CDN
Static files: Served from Vercel edge

Check DevTools:

Network tab → API calls to housing-backend-production.up.railway.app
Images from res.cloudinary.com
HTML/CSS/JS from Vercel edge

Step 5: Add Custom Domains

Frontend:

Vercel → Project → Settings → Domains
Add housing.yourdomain.com and www.housing.yourdomain.com
Update DNS:

   CNAME  housing  cname.vercel-dns.com
   CNAME  www      cname.vercel-dns.com

Vercel provisions SSL (automatic)

Backend:

Railway → Backend → Settings → Custom Domains
Add api.yourdomain.com
Update DNS:

   CNAME  api  housing-backend-production.up.railway.app

Railway provisions SSL (automatic)

Update environment variables:

Railway: ALLOWED_HOSTS=api.yourdomain.com
Railway: CORS_ALLOWED_ORIGINS=https://housing.yourdomain.com,https://www.housing.yourdomain.com
Vercel: NEXT_PUBLIC_API_URL=https://api.yourdomain.com

Step 6: Performance Optimization

Vercel Edge Caching:
Next.js automatically caches static assets on Vercel's edge network. No configuration needed.

SWR Client Caching:
Already implemented in Part 5. Works automatically.

Railway Connection Pooling:
Railway handles this automatically.

Result:

US user: ~50ms TTFB
EU user: ~80ms TTFB
Asia user: ~150ms TTFB

Much better than a single-region deployment!

📊 [Exercise: Vercel Deployment Dashboard]

Instructions: Capture screenshot of Vercel deployment success page showing build logs and deployment URL.

Part F: Path 3 — AWS DIY (Advanced)

This is for readers who want full control or need to learn infrastructure.

What we're deploying:

Frontend: Vercel (or AWS Amplify if you want 100% AWS)
Backend: EC2 instance with Nginx + Gunicorn
Database: RDS PostgreSQL
Redis: ElastiCache
Load Balancer: Application Load Balancer
SSL: AWS Certificate Manager
Static files: S3 + CloudFront (optional, or use WhiteNoise)

Cost: ~$90/month

EC2 t3.small: $15/month
RDS db.t3.micro: $15/month
ElastiCache cache.t3.micro: $12/month
ALB: $16/month
S3/CloudFront: $5/month
Data transfer: $10-20/month

Complexity: High

This section provides the architecture and key steps. Full AWS deployment would be a separate multi-part series.

AWS Architecture Overview

Internet
  ↓
Route 53 (DNS)
  ↓
CloudFront (optional, for static files)
  ↓
Application Load Balancer
  ↓
EC2 Instance (Django + Gunicorn + Nginx)
  ↓
├─ RDS PostgreSQL (VPC, private subnet)
└─ ElastiCache Redis (VPC, private subnet)

High-Level Steps (Not Complete Tutorial)

Due to complexity, this is an overview. For full AWS deployment, refer to the Django Deployment Guide by SaaS Pegasus.

1. Provision Infrastructure:

Create VPC with public/private subnets
Launch RDS PostgreSQL in private subnet
Launch ElastiCache Redis in private subnet
Launch EC2 instance in public subnet
Configure security groups (allow port 80, 443, 22)

2. Set Up EC2 Instance:

SSH into instance:

ssh -i your-key.pem ubuntu@ec2-instance-ip

Install dependencies:

sudo apt update
sudo apt install python3-pip python3-venv nginx postgresql-client redis-tools

Clone your repository:

git clone https://github.com/yourusername/housing-caching-demo.git
cd housing-caching-demo/backend

Create virtual environment:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Configure Gunicorn:

Create /etc/systemd/system/gunicorn.service:

4. Configure Nginx:

Create /etc/nginx/sites-available/housing-portal:

📄 [GIST 1: nginx.conf]

Gist URL: https://gist.github.com/urwithajit9/11b25980f75649279fec917f05cb5565

5. SSL with Let's Encrypt:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d api.yourdomain.com

Certbot automatically updates Nginx config for HTTPS.

6. Set Up Application:

Set environment variables in /home/ubuntu/.env:

DJANGO_ENV=production
SECRET_KEY=...
DATABASE_URL=postgres://user:pass@rds-endpoint:5432/housing_db
REDIS_URL=redis://elasticache-endpoint:6379
...

Run migrations:

python manage.py migrate
python manage.py collectstatic --noinput

Start services:

sudo systemctl start gunicorn
sudo systemctl enable gunicorn
sudo systemctl restart nginx

7. Deploy Frontend:

Either:

Deploy to Vercel (same as Path 2)
Or deploy to AWS Amplify
Or serve from same EC2 with Nginx

8. Set Up Load Balancer (Optional but Recommended):

Create Application Load Balancer
Point to EC2 instance(s)
Configure health checks
Update Route 53 to point to ALB instead of EC2 directly

9. Auto Scaling (Optional):

Create Auto Scaling Group
Define scaling policies
ALB distributes traffic across instances

When to Use AWS DIY

Use AWS when:

You need full control (compliance, security requirements)
You're learning DevOps
You have high traffic (100k+ requests/day)
You're already on AWS ecosystem
You need custom infrastructure (VPN, peering, etc.)

Don't use AWS when:

You're building a side project (overkill)
You don't have DevOps experience
You want to focus on features, not infrastructure
You're a solo developer (too much to manage)

AWS Cost Optimization Tips

Use Reserved Instances: Save 30-40% by committing to 1 year
Right-size instances: Start with t3.micro, scale up only when needed
Use S3 for static files: Cheaper than serving from EC2
Enable CloudWatch alarms: Catch runaway costs early
Use AWS Free Tier: RDS and EC2 have free tiers for 12 months

Part G: Domain and SSL Configuration

Regardless of which path you chose, you'll want custom domains and HTTPS.

Step 1: Register a Domain

Popular registrars:

Namecheap: ~$10/year for .com
Google Domains: ~$12/year
Cloudflare Registrar: At-cost pricing (~$9/year)
Porkbun: Budget option (~$8/year)

For this tutorial, we'll use housing-demo.com as an example.

Step 2: DNS Configuration

You need to point your domain to your deployed services.

For Vercel (Frontend):

Type: CNAME
Name: housing (or www or @)
Value: cname.vercel-dns.com
TTL: 3600

For Railway (Backend):

Type: CNAME
Name: api
Value: your-service.up.railway.app
TTL: 3600

For Render (Backend):

Type: CNAME
Name: api
Value: your-service.onrender.com
TTL: 3600

For AWS (Backend with ALB):

Type: A (Alias)
Name: api
Value: your-alb.us-east-1.elb.amazonaws.com
TTL: 3600

Step 3: SSL Certificates

Vercel: Automatic. Provisions SSL within minutes of adding domain.

Railway: Automatic. Provisions SSL via Let's Encrypt.

Render: Automatic. Provisions SSL via Let's Encrypt.

AWS: Use AWS Certificate Manager:

Request certificate for api.yourdomain.com
Verify via email or DNS
Attach certificate to ALB

Self-managed (VPS): Use Certbot:

sudo certbot --nginx -d api.yourdomain.com

Step 4: Force HTTPS

Django (already configured in production.py):

SECURE_SSL_REDIRECT = True

Nginx (if using):

server {
    listen 80;
    server_name api.yourdomain.com;
    return 301 https://$server_name$request_uri;
}

Next.js: Handled by platform (Vercel/Railway).

Step 5: DNS Propagation

After updating DNS, changes take time to propagate:

Minimum: 5 minutes
Typical: 1-2 hours
Maximum: 48 hours (rare)

Check propagation:

dig housing-demo.com
dig api.housing-demo.com

Or use online tool: dnschecker.org

Step 6: Test HTTPS

curl -I https://housing-demo.com
# Should show: HTTP/2 200
# Should NOT redirect to HTTP

curl -I https://api.housing-demo.com/api/properties/cached/
# Should show: HTTP/2 200
# Should return JSON

Part H: CI/CD Pipeline with GitHub Actions

Automate deployments so every push to main deploys automatically.

Deployment Flow

Developer pushes to GitHub
  ↓
GitHub Actions triggers
  ↓
Run tests
  ↓
Deploy to Vercel (frontend)
  ↓
Deploy to Railway/Render (backend)
  ↓
Run migrations
  ↓
Send notification (Slack/email)

📊 [ CI/CD Pipeline Flow]

GitHub Actions for Frontend (Vercel)

Vercel automatically deploys on push if you connected via GitHub. No action needed!

But if you want explicit control, create .github/workflows/deploy-frontend.yml:

📄 [GIST 2: deploy-frontend.yml]

GitHub Actions workflow for deploying frontend to Vercel.
Gist URL: hhttps://gist.github.com/urwithajit9/a68c22276c30501a6b4d14a06d86df3b

GitHub Actions for Backend (Railway)

Railway also auto-deploys on push. But for explicit control:

.github/workflows/deploy-backend.yml:

📄 [GIST 3: deploy-backend.yml]

GitHub Actions workflow for deploying backend to Railway.
Gist URL: https://gist.github.com/urwithajit9/cc0026c991cf7669a0083f396b64c9a6

GitHub Actions for Backend (AWS)

For AWS, you need to SSH into the server and pull latest code:

.github/workflows/deploy-aws.yml:

📄 [GIST 4: deploy-aws.yml]

GitHub Actions workflow for deploying to AWS EC2.
Gist URL: https://gist.github.com/urwithajit9/880d644d7026a2cc1cb4bc4d03c32282

Environment-Specific Deployments

Create separate workflows for staging and production:

.github/workflows/
├── deploy-staging.yml    # Triggers on push to 'staging' branch
└── deploy-production.yml # Triggers on push to 'main' branch

This lets you test changes in staging before production.

Part I: Monitoring and Maintenance

Production apps need monitoring to catch errors, track performance, and ensure uptime.

Step 1: Error Tracking with Sentry

Sentry captures errors from both frontend and backend.

1. Create Sentry account:

Go to sentry.io
Sign up (free tier: 5k errors/month)
Create a project for Django
Create a project for Next.js

2. Install Sentry in Django:

pip install sentry-sdk

Add to backend/core/settings/production.py:

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

sentry_sdk.init(
    dsn=os.environ.get('SENTRY_DSN'),
    integrations=[DjangoIntegration()],
    traces_sample_rate=0.1,  # 10% of transactions
    send_default_pii=False,
)

Add SENTRY_DSN to environment variables.

3. Install Sentry in Next.js:

npm install @sentry/nextjs

Run configuration wizard:

npx @sentry/wizard@latest -i nextjs

This creates sentry.client.config.js, sentry.server.config.js, etc.

Add NEXT_PUBLIC_SENTRY_DSN to Vercel environment variables.

4. Test error tracking:

Trigger an error in Django:

# In a view
raise Exception("Test error from Django")

Trigger an error in Next.js:

// In a component
throw new Error("Test error from Next.js")

Check Sentry dashboard — errors should appear.

📊 [Exercise: Sentry Error Dashboard]

Instructions: Capture screenshot of Sentry dashboard showing captured errors from both Django and Next.js.

Step 2: Uptime Monitoring

Use UptimeRobot (free tier) to monitor if your site is down.

1. Create account: uptimerobot.com

2. Add monitors:

Frontend: https://housing-demo.com (HTTP monitor, check every 5 min)
Backend: https://api.housing-demo.com/api/properties/cached/ (HTTP monitor)

3. Configure alerts:

Email when down
Email when back up
Optional: Slack, Discord, SMS

4. Public status page (optional):
UptimeRobot can generate a public status page: https://stats.uptimerobot.com/ABC123

You can embed this on your site or share with users.

📊 [Exercise: UptimeRobot Dashboard]

Instructions: Capture screenshot showing uptime monitors for frontend and backend with 100% uptime.

Step 3: Performance Monitoring

Option 1: Vercel Analytics (Frontend)

Vercel → Project → Analytics
Automatically tracks Web Vitals (LCP, FID, CLS)
Free on all plans

Option 2: Railway Metrics (Backend)

Railway → Service → Metrics
Shows CPU, memory, request latency
Free on all plans

Option 3: New Relic or Datadog (Advanced)

Full APM (Application Performance Monitoring)
Tracks database queries, external API calls, etc.
Expensive ($100+/month)

For this tutorial, Vercel and Railway built-in metrics are sufficient.

Step 4: Log Aggregation

Railway/Render:

Built-in logs in dashboard
Can export to external services

AWS:

CloudWatch Logs (built-in)
Or use Papertrail, LogDNA, Datadog

Centralized logging pattern:

All services → Log aggregator (Papertrail) → Search/analyze

This lets you search logs across all services in one place.

Step 5: Database Backups

Neon.tech:

Automatic daily backups (free tier)
Point-in-time recovery (paid tiers)

Railway:

Automatic daily backups
Can restore from dashboard

RDS:

Automatic daily backups (configurable)
Can restore to any point in time (within retention period)

Manual backup:

# Dump database
pg_dump $DATABASE_URL > backup.sql

# Restore
psql $DATABASE_URL < backup.sql

Schedule this with cron or GitHub Actions.

Part J: Cost and Performance Comparison

Let's compare all four deployment paths side by side.

Cost Comparison

Component	Path 0 (Free)	Path 1 (Railway)	Path 2 (Vercel+Railway)	Path 3 (AWS)
Frontend	Free (Vercel)	$5 (Railway)	Free (Vercel Hobby)	$30 (EC2 or Vercel Pro)
Backend	Free* (Render)	$5 (Railway)	$5 (Railway)	$30 (EC2)
Database	Free** (Neon)	$5 (Railway)	$5 (Railway)	$15 (RDS)
Redis	Free*** (Upstash)	$5 (Railway)	$5 (Railway or Upstash)	$15 (ElastiCache)
Monitoring	Free (UptimeRobot)	Free (Railway)	Free (Vercel Analytics)	$20 (CloudWatch)
SSL	Free (auto)	Free (auto)	Free (auto)	Free (ACM)
Domain	Optional $10/year	Optional $10/year	Optional $10/year	Optional $10/year
TOTAL/month	$0	$20	$15-20	$110

* Sleeps after 15min inactivity

** 0.5GB storage limit

*** 10k requests/day limit

Performance Comparison

Measured from different global locations:

Location	Path 0 (Free)	Path 1 (Railway)	Path 2 (Vercel+Railway)	Path 3 (AWS)
US East	150ms (cold start) / 80ms (warm)	50ms	30ms	40ms
US West	180ms / 100ms	80ms	40ms	120ms
Europe	250ms / 180ms	150ms	60ms	200ms
Asia	400ms / 300ms	280ms	100ms	350ms

Winner: Path 2 (Vercel + Railway) — Vercel's global edge network gives best latency worldwide.

📊 [Exercise: Actual Performance Metrics]

Instructions: Use tools like Pingdom or WebPageTest to measure actual TTFB from different locations. Record your results here.

Uptime Comparison

Deployment Path	Expected Uptime	SLA
Path 0 (Free)	95-98%	None
Path 1 (Railway)	99%+	None (hobby) / 99.9% (pro)
Path 2 (Vercel+Railway)	99.9%+	99.9% (Vercel Pro)
Path 3 (AWS)	99.95%+	99.99% (with multi-AZ)

Scalability Comparison

Deployment Path	Max Concurrent Users	Bottleneck
Path 0 (Free)	~50	Redis rate limit (10k/day)
Path 1 (Railway)	~5,000	Database connections
Path 2 (Vercel+Railway)	~50,000	Backend capacity
Path 3 (AWS)	~500,000+	Cost 💰

Decision Matrix

Choose based on:

Stage → Deployment Path

Idea/Demo → Path 0 (Free)
MVP/Beta → Path 1 (Railway)
Launched/Growth → Path 2 (Vercel+Railway)
Enterprise/Scale → Path 3 (AWS)

Part K: Troubleshooting Deployment

Common issues and solutions.

Issue 1: Static Files Not Loading

Symptom: CSS/JS broken, images don't load (404 errors).

Cause: Static files not collected or not served properly.

Fix:

Django:

python manage.py collectstatic --noinput

Make sure in production settings:

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATIC_URL = '/static/'

With WhiteNoise:

MIDDLEWARE.insert(1, 'whitenoise.middleware.WhiteNoiseMiddleware')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

Issue 2: CORS Errors

Symptom: Browser console shows:

Access to fetch at 'https://api.yourdomain.com' has been blocked by CORS policy

Cause: Backend doesn't allow frontend domain in CORS settings.

Fix:

Check Django settings:

CORS_ALLOWED_ORIGINS = [
    'https://yourdomain.com',
    'https://www.yourdomain.com',
]

Restart backend after changing.

Issue 3: 502 Bad Gateway

Symptom: Nginx shows 502 error.

Cause: Gunicorn not running or crashed.

Fix:

Check Gunicorn status:

sudo systemctl status gunicorn

Check logs:

sudo journalctl -u gunicorn -n 50

Restart:

sudo systemctl restart gunicorn

Issue 4: Database Connection Errors

Symptom:

django.db.utils.OperationalError: could not connect to server

Cause: Wrong DATABASE_URL or database not accessible.

Fix:

Test connection:

psql $DATABASE_URL

Check:

Is DATABASE_URL correct?
Is database accepting connections from your server IP?
For RDS/Railway: Check security groups/firewall rules
For Neon: Check connection pooler settings

Issue 5: Environment Variables Not Loading

Symptom: Settings reference missing environment variables.

Cause: .env file not loaded or variables not set in platform.

Fix:

Railway/Render/Vercel:

Check dashboard → Service → Environment Variables
Redeploy after adding variables

AWS EC2:

Check /home/ubuntu/.env file exists
Source it in systemd service file

Local testing:

export DJANGO_ENV=production
python manage.py check --deploy

Issue 6: SSL Certificate Errors

Symptom:

NET::ERR_CERT_AUTHORITY_INVALID

Cause: SSL not provisioned yet or DNS not propagated.

Fix:

Wait for DNS propagation (up to 48 hours, usually < 2 hours).

Check DNS:

dig yourdomain.com

Force SSL renewal (Certbot):

sudo certbot renew --force-renewal

Issue 7: Next.js Build Fails

Symptom: Deployment fails with TypeScript errors.

Cause: Type errors not caught in development.

Fix:

Test build locally:

cd frontend
npm run build

Fix all TypeScript errors before deploying.

Temporary bypass (not recommended):

// next.config.js
module.exports = {
  typescript: {
    ignoreBuildErrors: true,  // DON'T use in production
  },
}

Part L: The Complete Deployment Checklist

Use this checklist for every deployment.

Pre-Deployment Checklist

Code Preparation:

[ ] All tests passing locally
[ ] No TypeScript errors (npm run build)
[ ] No Django warnings (python manage.py check --deploy)
[ ] Environment variables documented
[ ] .gitignore includes .env* files
[ ] requirements.txt / package.json up to date

Settings Verification:

[ ] DEBUG = False in production
[ ] SECRET_KEY is strong and from environment
[ ] ALLOWED_HOSTS configured
[ ] CSRF_TRUSTED_ORIGINS configured
[ ] CORS_ALLOWED_ORIGINS configured
[ ] Database connection pooling enabled
[ ] Static files configuration (WhiteNoise or S3)

Security Checklist:

[ ] SSL redirect enabled (SECURE_SSL_REDIRECT = True)
[ ] Secure cookies (SESSION_COOKIE_SECURE = True)
[ ] HSTS enabled
[ ] No secrets in code (all in environment variables)
[ ] CORS properly configured (not CORS_ALLOW_ALL_ORIGINS = True)

Infrastructure Ready:

[ ] Domain registered (if using custom domain)
[ ] DNS records planned
[ ] Hosting accounts created
[ ] Database provisioned
[ ] Redis provisioned

Deployment Checklist

Backend Deployment:

[ ] Code deployed to server/platform
[ ] Environment variables set
[ ] Database migrations run
[ ] Static files collected
[ ] Gunicorn/service started
[ ] Health check endpoint responding

Frontend Deployment:

[ ] Code deployed to Vercel/platform
[ ] Environment variables set (NEXT_PUBLIC_API_URL)
[ ] Build successful
[ ] Preview URL accessible

Database Setup:

[ ] Migrations applied
[ ] Superuser created
[ ] Test data loaded (if needed)
[ ] Backups configured

Domain and SSL:

[ ] DNS records added
[ ] DNS propagated (check with dig)
[ ] SSL certificates provisioned
[ ] HTTPS enforced
[ ] HTTP redirects to HTTPS

Integration Testing:

[ ] Frontend loads without errors
[ ] API calls work (check DevTools Network tab)
[ ] No CORS errors
[ ] Images load from Cloudinary
[ ] Authentication works (if implemented)
[ ] Database queries succeed

Post-Deployment Checklist

Monitoring Setup:

[ ] Error tracking configured (Sentry)
[ ] Uptime monitoring configured (UptimeRobot)
[ ] Performance monitoring enabled
[ ] Log aggregation set up
[ ] Alerts configured (email/Slack)

Backups Verified:

[ ] Database backups enabled
[ ] Backup restoration tested
[ ] Backup schedule confirmed

Performance Verification:

[ ] Lighthouse score > 90
[ ] TTFB < 200ms
[ ] No console errors
[ ] API response times < 500ms
[ ] Images loading fast

Documentation:

[ ] Deployment process documented
[ ] Environment variables documented
[ ] Rollback procedure documented
[ ] Incident response plan created

Load Testing:

[ ] Test with expected traffic volume
[ ] Verify database connection pooling works
[ ] Verify Redis cache hit rate
[ ] No memory leaks under sustained load

User Acceptance:

[ ] Staging environment tested
[ ] Stakeholders approve
[ ] Users can access the site
[ ] All features work in production

What We Built — The Complete System

We entered Part 7 with a housing portal running on localhost. We exit with four complete deployment paths to choose from.

Path 0 (Free Tier):

Perfect for this tutorial
$0/month, ideal for demos and MVPs
Accept cold starts as a tradeoff
Easy migration path to paid tiers when needed

Path 1 (Railway):

Simplest paid deployment
$20/month for everything in one place
No infrastructure management
Great for side projects and small apps

Path 2 (Vercel + Railway):

Optimal for production
$15-35/month
Best performance globally
This is what you choose for a real product

Path 3 (AWS DIY):

Full control
$90+/month
For enterprise and compliance needs
Requires DevOps knowledge

📊 [Complete Production Request Flow]

The Production Architecture (Path 2)

User in Tokyo
  ↓
Vercel Edge (Tokyo) — serves Next.js SSR, cached static files
  ↓
Railway Backend (US-East) — Django API
  ↓
├─ PostgreSQL (Railway) — persistent data
├─ Redis (Railway) — cache layer
└─ Cloudinary CDN (Tokyo edge) — images

Result: < 100ms TTFB globally

Every layer is optimized. Every cache works together. The user experience is instant.

What's Different from Development

Development (localhost):

DEBUG = True (detailed errors)
SQLite or PostgreSQL on same machine
Redis on same machine
No SSL
No monitoring
Single server
Unlimited resources

Production (live site):

DEBUG = False (generic errors)
Managed PostgreSQL with backups
Managed Redis or ElastiCache
SSL enforced
Error tracking, uptime monitoring
Distributed across multiple servers/regions
Resource limits, cost constraints
Autoscaling
Security hardening

Beyond Deployment — What's Next

The series ends here, but your learning continues:

Part 8 (Bonus): Advanced Topics

Celery background tasks
WebSockets for real-time updates
Full-text search with Elasticsearch
Payment integration (Stripe)
Email notifications (SendGrid)
Admin customization

Part 9 (Bonus): Scaling Beyond

Kubernetes deployment
Microservices architecture
Multi-region deployment
Database sharding
Read replicas

But you don't need any of that yet. What you have now — this complete, optimized, deployed housing portal — is production-ready and can handle thousands of users.

Checkpoint: Push to GitHub

git add .
git commit -m "feat: add production settings, deployment configs, CI/CD pipeline"
git checkout -b part-7-deployment
git push origin part-7-deployment

The repo now has seven branches:

part-1-setup — infrastructure
part-2-data — database and seed
part-3-caching — API and Redis
part-4-optimization — query optimization and signals
part-5-frontend — Next.js UI and SWR
part-6-images — Cloudinary and lazy loading
part-7-deployment — production deployment

View the live site:

Free tier: https://your-app.vercel.app
With domain: https://housing-demo.com

Acknowledgments and References

This deployment guide was informed by:

Django Deployment Guide by SaaS Pegasus — Comprehensive Django deployment resource
Vercel Documentation — Next.js deployment best practices
Railway Documentation — PaaS deployment patterns
AWS Well-Architected Framework — Cloud architecture principles
Django Official Deployment Checklist — Security and settings verification

Special thanks to the Django, Next.js, and Railway communities for excellent documentation.

The series is complete. You've built a production-grade housing portal from zero to deployed. You understand caching at every layer. You know how to optimize databases. You can deploy to production. Most importantly, you understand the why behind every decision.

Now go build something amazing. 🚀

Series Complete:

When `venv` Lies: Debugging a Ghost Python 3.4 on Ubuntu

Ajit Kumar — Wed, 11 Feb 2026 05:54:11 +0000

Virtual environments are supposed to isolate your project. But what happens when your venv quietly binds itself to a decade-old Python interpreter?

Here’s a real-world debugging story of how Python 3.4 hijacked a modern Ubuntu system running Python 3.12 — and how to diagnose and fix it correctly.

The Symptom

Everything looked normal:

python3 -m venv venv
source venv/bin/activate

But then:

pip didn’t exist inside the venv
ensurepip failed with SSL errors
get-pip.py refused to run
pip freeze showed nothing
SSL module failed to import

Critical clue:

python3 --version
Python 3.4.10

But Ubuntu reported:

sudo apt install python3
python3 is already the newest version (3.12.3)

How can both be true?

Why This Happens

Linux resolves executables using $PATH.

Most systems have:

/usr/local/bin
/usr/bin

If an old Python was manually compiled and installed into:

/usr/local/bin/python3
/usr/local/lib/python3.4

It takes precedence over Ubuntu’s managed Python at:

/usr/bin/python3

So when you run:

python3 -m venv venv

You unknowingly create a virtual environment using Python 3.4 — not 3.12.

And since Python 3.4:

Is end-of-life (2019)
Lacks modern SSL support
Cannot run modern pip
Is incompatible with current tooling

Your venv becomes broken from the start.

How to Detect the Problem

1️⃣ Check Which Python Is Being Used

which python3

If it shows:

/usr/local/bin/python3

That’s the problem.

Correct system Python should be:

/usr/bin/python3

2️⃣ Verify Version Explicitly

python3 --version
/usr/bin/python3 --version

If they differ, you have multiple Python installations.

3️⃣ Check SSL Module

python3 -c "import ssl"

If it throws:

ImportError: No module named '_ssl'

Your interpreter is broken or outdated.

4️⃣ Inspect Virtual Environment

Inside venv:

python --version
which python
which pip

If version shows 3.4.x, your venv was created from the wrong interpreter.

Why `venv` Doesn’t Protect You

venv is not magic.

It simply clones the interpreter used to create it:

pythonX -m venv venv

If pythonX is wrong, your venv is wrong.

Garbage in → garbage out.

The Correct Fix

Step 1 — Delete the Broken Virtual Environment

deactivate
rm -rf venv

Step 2 — Use Explicit System Python

Never rely on ambiguous python3.

/usr/bin/python3 -m venv venv
source venv/bin/activate

Step 3 — Verify Immediately

python --version
which python
which pip

Expected:

Python 3.12.3
.../venv/bin/python
.../venv/bin/pip

Step 4 — Remove Old Python (If Safe)

If you confirm /usr/local/bin/python3 is obsolete:

sudo rm /usr/local/bin/python3
sudo rm /usr/local/bin/python
sudo rm -rf /usr/local/lib/python3.4
hash -r

Now verify:

which python3
python3 --version

Should resolve to:

/usr/bin/python3
Python 3.12.3

Key Lessons

1. `venv` uses whatever interpreter you give it

It does not automatically select the newest Python.

2. Always verify interpreter path

Before creating a venv:

which python3
python3 --version

3. Prefer explicit paths in production

/usr/bin/python3 -m venv venv

4. Old `/usr/local` installations often override system tools

Manual builds are invisible to apt.

Final State (Healthy Setup)

System Python:

/usr/bin/python3 → 3.12

Project virtual environment:

venv → created from 3.12
pip → inside venv

Now:

SSL works
pip works
Modern packages install
Environment is reproducible

Conclusion

This issue is subtle because nothing crashes immediately. The system appears to work — until you hit SSL, pip, or dependency errors.

When debugging Python environments on Linux:

Always verify the interpreter path before blaming pip, venv, or Ubuntu.

Most environment problems are PATH problems in disguise.

From Zero to Cached: Building a High-Performance Housing Portal with Django, Next.js, and Redis- Part 6: Image Optimization

Ajit Kumar — Tue, 10 Feb 2026 05:52:13 +0000

Part 6: The Image Layer — From Blue Gradients to Production CDN

In Part 1, we containerized the stack. In Part 2, we built the database. In Part 3, we cached the API. In Part 4, we optimized queries. In Part 5, we built the UI. Today, we make it look real.

If you're jumping in here, you need the full context. Part 1 through Part 5 built the entire system — infrastructure, database, API, caching, and UI. If you're continuing from Part 5, you have a working housing portal with property cards displaying blue gradient placeholders where images should be.

Today we replace those placeholders with real property photos. By the end of this post, you'll have images hosted on a CDN, automatic thumbnails, lazy loading, WebP conversion, and responsive images that adapt to device size. The page will load 85% faster. The images will be 90% smaller. Everything will feel production-ready.

The Image Problem

A single unoptimized property photo is 3-5MB. Twenty of them on a listing page is 60-100MB. Over a 3G connection, that's 90 seconds of loading — or a timeout. Even over WiFi, it's 5-10 seconds of white screen.

The naive approach — serving images directly from Django — makes this worse. Django reads the file from disk, holds it in memory, and streams it over HTTP. Every image request ties up a Django worker. Twenty simultaneous users requesting images = 400 concurrent requests to Django. Your API stops responding. The database queries start timing out. The entire system chokes.

We need a different architecture:

Offload image hosting to a CDN — images are served from edge servers geographically close to users, not from your single Django server
Generate thumbnails on-demand — a 400x300px card thumbnail is 50KB, not 5MB
Convert to modern formats — WebP is 30% smaller than JPEG for the same visual quality
Lazy load — only load images when they're about to enter the viewport
Optimize with Next.js — automatic responsive images, format selection, and caching

By the end of this part, the system will handle images the way Netflix handles video: distributed, optimized, and invisible to the user.

Part A: The CDN Decision — Choosing Cloudinary

We need an image hosting service that can:

Store our images (thousands of them)
Generate thumbnails and crops on-demand
Serve images from a global CDN
Convert formats automatically (JPEG → WebP → AVIF)
Integrate with Django (for uploads) and Next.js (for display)

The Options

Cloudinary (recommended for this series)

Free tier: 25GB storage + 25GB bandwidth/month (enough for ~50,000 images)
Automatic transformations via URL parameters
Excellent Django SDK (django-cloudinary-storage)
Works seamlessly with Next.js <Image> component
Well-documented, widely used

ImageKit

Similar to Cloudinary, slightly better pricing at scale
Less Django integration (requires custom code)
Good Next.js support

AWS S3 + CloudFront

Most control, cheapest at massive scale
Requires Lambda@Edge for transformations
More setup complexity
Best for teams already on AWS

Vercel Blob Storage

Native Next.js integration
Simple API
Expensive (no free tier, $0.15/GB storage + $0.40/GB bandwidth)
Upload from Next.js API routes, not Django

We're using Cloudinary for this series. It has the best free tier, the simplest setup, and works perfectly with both Django and Next.js. If you're already using AWS or Vercel, the concepts transfer — the URL structure and transformations are similar.

Step 1: Sign Up for Cloudinary

Go to cloudinary.com and click "Sign Up Free"
Complete the registration (email, password, company name)
You'll land on the dashboard

The dashboard shows three critical values:

Cloud name — this appears in all your image URLs (e.g., res.cloudinary.com/YOUR_CLOUD_NAME/...)
API Key — used for uploads
API Secret — used to sign upload requests (keep this secret)

Copy these three values. We'll store them in environment variables.

Step 2: Store Credentials Securely

Never commit API credentials to git. They go in environment variables.

Create backend/.env (if it doesn't exist):

# backend/.env

# Database (already exists from Part 1)
DATABASE_URL=postgres://user:password@db:5432/housing_db

# Redis (already exists from Part 3)
REDIS_URL=redis://redis:6379/1

# Cloudinary (NEW - add these)
CLOUDINARY_CLOUD_NAME=your_cloud_name_here
CLOUDINARY_API_KEY=your_api_key_here
CLOUDINARY_API_SECRET=your_api_secret_here

Replace your_cloud_name_here, etc. with your actual values from the Cloudinary dashboard.

Verify .env is in .gitignore:

# Check if .env is already ignored
cd backend
cat .gitignore | grep ".env"

If you don't see .env in the output, add it:

echo ".env" >> .gitignore

This prevents you from accidentally committing secrets.

Step 3: Pass Environment Variables to Docker

Docker containers don't automatically see .env files. We need to pass the variables explicitly.

Update docker-compose.yml:

  backend:
    build: ./backend
    volumes:
      - ./backend:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgres://user:password@db:5432/housing_db
      - REDIS_URL=redis://redis:6379/1
      # Add these three lines ↓
      - CLOUDINARY_CLOUD_NAME=${CLOUDINARY_CLOUD_NAME}
      - CLOUDINARY_API_KEY=${CLOUDINARY_API_KEY}
      - CLOUDINARY_API_SECRET=${CLOUDINARY_API_SECRET}
    depends_on:
      - db
      - redis
    # Add this block ↓ to load .env file from backend directory
    env_file:
      - ./backend/.env

What this does:

env_file: ./backend/.env tells Docker Compose to load environment variables from that file
${CLOUDINARY_CLOUD_NAME} reads the value from the file and passes it to the container

Restart the backend to pick up the new environment:

docker compose restart backend

Step 4: Install Cloudinary SDK

cd backend
source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install cloudinary django-cloudinary-storage
pip freeze > requirements.txt

What we just installed:

cloudinary — the Python SDK for uploading, transforming, and managing images
django-cloudinary-storage — Django integration that makes FileFields automatically upload to Cloudinary instead of local disk

Step 5: Configure Django to Use Cloudinary

Update backend/core/settings.py:

# At the top, with other imports
import os
import cloudinary
import cloudinary.uploader
import cloudinary.api

# ... (keep existing settings) ...

# In INSTALLED_APPS, add cloudinary_storage BEFORE django.contrib.staticfiles
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',

    'cloudinary_storage',  # ← Add this BEFORE staticfiles
    'django.contrib.staticfiles',

    'rest_framework',
    'corsheaders',
    'debug_toolbar',
    'django_filters',

    'housing',
]

# ... (keep existing settings) ...

# Add this at the bottom of settings.py
CLOUDINARY_STORAGE = {
    'CLOUD_NAME': os.environ.get('CLOUDINARY_CLOUD_NAME'),
    'API_KEY': os.environ.get('CLOUDINARY_API_KEY'),
    'API_SECRET': os.environ.get('CLOUDINARY_API_SECRET'),
}

# Configure Cloudinary
cloudinary.config(
    cloud_name=CLOUDINARY_STORAGE['CLOUD_NAME'],
    api_key=CLOUDINARY_STORAGE['API_KEY'],
    api_secret=CLOUDINARY_STORAGE['API_SECRET'],
    secure=True  # Use HTTPS
)

# Set default file storage (for FileFields and ImageFields)
DEFAULT_FILE_STORAGE = 'cloudinary_storage.storage.MediaCloudinaryStorage'

What this does:

Reads the Cloudinary credentials from environment variables
Configures the SDK with those credentials
Sets DEFAULT_FILE_STORAGE so any Django ImageField or FileField automatically uploads to Cloudinary instead of saving to local disk

Step 6: Verify the Connection

Before we modify models, let's prove Cloudinary is configured correctly.

Create a test script: backend/scripts/test_cloudinary.py

"""
scripts/test_cloudinary.py

Test script to verify Cloudinary connection.
Uploads a dummy image and prints the URL.

Run with:
    docker compose exec backend python scripts/test_cloudinary.py
"""

import os
import sys
import django

# Add the parent directory to the Python path so we can import Django settings
sys.path.append('/app')
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core.settings')
django.setup()

import cloudinary
import cloudinary.uploader

def test_upload():
    print("Testing Cloudinary connection...")
    print(f"Cloud name: {cloudinary.config().cloud_name}")

    # Create a tiny test image in memory
    from PIL import Image
    import io

    img = Image.new('RGB', (100, 100), color='blue')
    buffer = io.BytesIO()
    img.save(buffer, format='PNG')
    buffer.seek(0)

    # Upload to Cloudinary
    try:
        result = cloudinary.uploader.upload(
            buffer,
            folder="housing_test",  # Optional: organize in folders
            public_id="test_image"
        )

        print("✅ Upload successful!")
        print(f"Image URL: {result['secure_url']}")
        print(f"Public ID: {result['public_id']}")
        return True
    except Exception as e:
        print(f"❌ Upload failed: {e}")
        return False

if __name__ == '__main__':
    success = test_upload()
    sys.exit(0 if success else 1)

Install Pillow (required for the test script):

pip install Pillow
pip freeze > requirements.txt

Rebuild the backend container:

docker compose build backend
docker compose restart backend

Run the test:

docker compose exec backend python scripts/test_cloudinary.py

Expected output:

Testing Cloudinary connection...
Cloud name: your_cloud_name
✅ Upload successful!
Image URL: https://res.cloudinary.com/your_cloud_name/image/upload/v1234567890/housing_test/test_image.png
Public ID: housing_test/test_image

If you see ✅, the connection works. If you see ❌, check:

The environment variables are set correctly in backend/.env
The docker-compose.yml has env_file: ./backend/.env
You restarted the backend after adding the env vars

Go to your Cloudinary dashboard → Media Library. You should see the test image there.

Part B: The Upload Flow — Django Admin to CDN

Now that Cloudinary is connected, we need to update the Django admin so admins can upload property images.

Step 7: Update the PropertyImage Model

Our current model has original_url as a URLField — a text field that stores a URL string. We need to change it to an ImageField — a special field that handles file uploads.

Update backend/housing/models.py:

from django.db import models
from cloudinary.models import CloudinaryField  # ← Add this import


# ... (keep Location, Office, Agent, Property models unchanged) ...


class PropertyImage(models.Model):
    """
    Images for a property listing.

    The image field uses CloudinaryField instead of Django's ImageField.
    This automatically uploads to Cloudinary and stores the Cloudinary URL.
    """
    listing = models.ForeignKey(
        Property,
        on_delete=models.CASCADE,
        related_name="images",
        db_index=False,
    )

    # Replace the old URLField with CloudinaryField
    image = CloudinaryField(
        'image',
        folder='housing/properties',  # Organize in Cloudinary folders
        transformation={
            'quality': 'auto',  # Automatic quality optimization
            'fetch_format': 'auto',  # Automatic format selection (WebP, AVIF, etc.)
        }
    )

    display_order = models.IntegerField(default=0)
    alt_text = models.CharField(max_length=255, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        ordering = ['display_order']
        verbose_name_plural = "property images"

    def __str__(self):
        return f"Image {self.display_order} for {self.listing.title}"

    def get_original_url(self):
        """
        Returns the original uploaded image URL.
        """
        if self.image:
            return self.image.url
        return None

    def get_thumbnail_url(self, width=400, height=300):
        """
        Returns a thumbnail URL with Cloudinary transformations.
        Cloudinary generates this on-demand — no separate file is stored.
        """
        if not self.image:
            return None

        # Build transformation string
        # w_400,h_300,c_fill = width 400px, height 300px, fill mode (crop to fit)
        return self.image.build_url(transformation=[
            {'width': width, 'height': height, 'crop': 'fill'},
            {'quality': 'auto'},
            {'fetch_format': 'auto'},
        ])

    def get_webp_url(self):
        """
        Returns a WebP version of the image.
        WebP is 25-35% smaller than JPEG for the same visual quality.
        """
        if not self.image:
            return None

        return self.image.build_url(transformation=[
            {'fetch_format': 'webp'},
            {'quality': 'auto'},
        ])

What changed:

original_url → image (CloudinaryField instead of URLField)
Added get_original_url(), get_thumbnail_url(), get_webp_url() methods
Cloudinary transformations are applied via URL parameters, not separate files

Important: This is a schema change. We need a migration.

Step 8: Create and Run the Migration

Before we migrate, we have existing data. The old PropertyImage rows have URLs in the original_url field (picsum.photos links from the seed command). When we change the field to image (CloudinaryField), those rows will have a null image field.

We have two options:

Option A: Keep old data, manually upload images later via admin

Run the migration
The old original_url field is removed
Existing images disappear from the frontend (until you upload new ones)

Option B: Migrate existing picsum.photos images to Cloudinary

Write a data migration that downloads each picsum image and uploads to Cloudinary (Check the troubleshoot section with new seed command)
Takes time (5000+ images) but preserves existing data

For this tutorial, we'll use Option A (simpler). If you want to preserve the seed data, I'll provide a migration script in the troubleshooting section.

Run the migration:

# Generate the migration file
docker compose exec backend python manage.py makemigrations housing

# Django will detect the field change and ask:
# "You are trying to change the nullable field 'original_url' on propertyimage to non-nullable field 'image' without a default"
# Select option 1: Provide a one-off default now
# Enter: None

# Apply the migration
docker compose exec backend python manage.py migrate housing

If you get errors about the migration, use the --fake option to skip applying it (we're dropping and recreating anyway):

docker compose exec backend python manage.py migrate housing --fake

Step 9: Update the Django Admin for Image Uploads

Now that the model accepts uploads, we need an admin interface.

Update backend/housing/admin.py:

from django.contrib import admin
from .models import Location, Office, Agent, Property, PropertyImage


# ... (keep Location, Office, Agent admin classes unchanged) ...


class PropertyImageInline(admin.TabularInline):
    """
    Inline admin for uploading multiple images per property.
    Shows as a table below the property form.
    """
    model = PropertyImage
    extra = 3  # Show 3 empty upload slots by default
    fields = ['image', 'display_order', 'alt_text']

    # Optional: show thumbnail preview in admin
    readonly_fields = ['image_preview']

    def image_preview(self, obj):
        """Display a small thumbnail in the admin list"""
        if obj.image:
            return f'<img src="{obj.get_thumbnail_url(100, 75)}" />'
        return "No image"
    image_preview.allow_tags = True


@admin.register(Property)
class PropertyAdmin(admin.ModelAdmin):
    list_display = [
        'title', 'property_type', 'price', 'bedrooms',
        'bathrooms', 'status', 'location', 'agent',
        'is_published', 'view_count', 'created_at',
    ]
    list_filter = ['property_type', 'status', 'is_published']
    search_fields = ['title', 'description']
    ordering = ['-created_at']
    list_select_related = ['location', 'agent']

    # Add the inline ↓
    inlines = [PropertyImageInline]


# The PropertyImage admin is now handled by the inline above
# But we can still register it separately for direct access
@admin.register(PropertyImage)
class PropertyImageAdmin(admin.ModelAdmin):
    list_display = ['listing', 'display_order', 'alt_text', 'created_at']
    list_select_related = ['listing']
    ordering = ['listing', 'display_order']

What this does:

PropertyImageInline adds an image upload section directly on the Property edit page
Admins can upload multiple images at once
extra = 3 means 3 empty upload slots appear by default
image_preview shows a thumbnail of each uploaded image

Restart the backend:

docker compose restart backend

Step 10: Test the Upload Flow

Open http://localhost:8000/admin/
Log in with your superuser credentials (from Part 2)
Go to Housing → Properties
Click any property to edit it
Scroll down — you should see "Property images" section with 3 upload slots
Click "Choose File" and select an image from your computer
Set Display order to 0 (this will be the main image)
Set Alt text to something descriptive (e.g., "Front view of property")
Click "Save and continue editing"

After saving, Django uploads the image to Cloudinary and stores the URL in the database. Refresh the page — you should see a small thumbnail preview of the image you just uploaded.

📊 [Screenshot: Django Admin Image Upload Interface]

Go to your Cloudinary dashboard → Media Library → housing/properties folder. You should see the image there.

The URL pattern looks like:

https://res.cloudinary.com/YOUR_CLOUD_NAME/image/upload/v1234567890/housing/properties/abc123.jpg

This is the URL Django stores in the database. Every transformation (thumbnail, WebP, etc.) is generated from this base URL by adding parameters.

Part C: The Frontend Integration — Next.js Image Component

The backend is uploading images to Cloudinary. Now we need the frontend to display them.

Step 11: Update the Django Serializer

The API needs to return image URLs. Currently, the PropertySerializer doesn't include images.

Update backend/housing/serializers.py:

from rest_framework import serializers
from .models import Office, Agent, Location, Property, PropertyImage


# ... (keep existing serializers unchanged) ...


class PropertyImageSerializer(serializers.ModelSerializer):
    """
    Serializer for property images.
    Returns multiple URL variants for different use cases.
    """
    original_url = serializers.SerializerMethodField()
    thumbnail_url = serializers.SerializerMethodField()
    webp_url = serializers.SerializerMethodField()

    class Meta:
        model = PropertyImage
        fields = ['id', 'original_url', 'thumbnail_url', 'webp_url', 'display_order', 'alt_text']

    def get_original_url(self, obj):
        """Full-size image URL"""
        return obj.get_original_url()

    def get_thumbnail_url(self, obj):
        """400x300 thumbnail URL"""
        return obj.get_thumbnail_url(width=400, height=300)

    def get_webp_url(self, obj):
        """WebP version URL (smaller file size)"""
        return obj.get_webp_url()


class PropertySerializer(serializers.ModelSerializer):
    location = LocationSerializer(read_only=True)
    agent = AgentSerializer(read_only=True)
    images = PropertyImageSerializer(many=True, read_only=True)  # ← Add this

    class Meta:
        model = Property
        fields = [
            'id', 'title', 'description', 'property_type', 'price',
            'bedrooms', 'bathrooms', 'location', 'agent', 'status',
            'view_count', 'created_at', 'images',  # ← Add to fields list
        ]

What this does:

Adds PropertyImageSerializer that returns original_url, thumbnail_url, and webp_url
Includes images in PropertySerializer so the API returns an array of images per property
Uses SerializerMethodField to call our model methods

Step 12: Update the View to Prefetch Images

Without prefetch_related, Django will fire N+1 queries again — one query to get properties, then one query per property to get its images.

Update backend/housing/views.py:

class OptimizedPropertyListView(generics.ListAPIView):
    serializer_class = PropertySerializer
    filter_backends = [DjangoFilterBackend, SearchFilter]
    filterset_fields = ['property_type', 'status', 'location__city']
    search_fields = ['title', 'description']

    def get_queryset(self):
        return Property.objects.select_related(
            'location',
            'agent__office'
        ).prefetch_related('images').all().order_by('-created_at')  # ← Add prefetch_related

Restart the backend:

docker compose restart backend

Test the API directly:

curl -s http://localhost:8000/api/properties/cached/ | jq '.results[0].images'

You should see an array of image objects (or an empty array if you haven't uploaded images yet):

[
  {
    "id": 1,
    "original_url": "https://res.cloudinary.com/.../housing/properties/abc123.jpg",
    "thumbnail_url": "https://res.cloudinary.com/.../w_400,h_300,c_fill/.../abc123.jpg",
    "webp_url": "https://res.cloudinary.com/.../f_webp/.../abc123.jpg",
    "display_order": 0,
    "alt_text": "Front view of property"
  }
]

Step 13: Configure Next.js for External Images

Next.js doesn't allow external image URLs by default (security feature). We need to whitelist Cloudinary.

Update frontend/next.config.js:

/** @type {import('next').NextConfig} */
const nextConfig = {
  images: {
    remotePatterns: [
      {
        protocol: 'https',
        hostname: 'res.cloudinary.com',
        pathname: '/**',
      },
      // Keep picsum.photos for any old seed data that might still exist
      {
        protocol: 'https',
        hostname: 'picsum.photos',
        pathname: '/**',
      },
    ],
    // Optional: configure device sizes for responsive images
    deviceSizes: [640, 750, 828, 1080, 1200, 1920, 2048, 3840],
    imageSizes: [16, 32, 48, 64, 96, 128, 256, 384],
  },
}

module.exports = nextConfig

What this does:

remotePatterns tells Next.js which external domains are allowed
deviceSizes defines the widths Next.js should generate (for responsive images)
imageSizes defines smaller icon sizes

Restart the frontend:

docker compose restart frontend

Step 14: Update PropertyCard to Use Real Images

Replace the blue gradient placeholder with the Next.js <Image> component.

Update frontend/components/PropertyCard.tsx:

import { Property } from '@/types/property';
import { formatPrice, capitalize } from '@/lib/formatters';
import Image from 'next/image';  // ← Add this import

interface PropertyCardProps {
  property: Property;
}

export function PropertyCard({ property }: PropertyCardProps) {
  const statusColors = {
    available: 'bg-green-100 text-green-800',
    pending: 'bg-yellow-100 text-yellow-800',
    sold: 'bg-red-100 text-red-800',
  };

  // Get the first image (display_order = 0) or fallback to placeholder
  const mainImage = property.images && property.images.length > 0
    ? property.images.find(img => img.display_order === 0) || property.images[0]
    : null;

  return (
    <div className="border border-gray-200 rounded-lg overflow-hidden hover:shadow-lg transition-shadow duration-200">
      {/* Image section - UPDATED */}
      <div className="relative h-48 w-full bg-gray-100">
        {mainImage ? (
          <Image
            src={mainImage.thumbnail_url}
            alt={mainImage.alt_text || property.title}
            fill
            className="object-cover"
            sizes="(max-width: 768px) 100vw, (max-width: 1200px) 50vw, 33vw"
            loading="lazy"
          />
        ) : (
          // Fallback gradient if no image uploaded
          <div className="h-full w-full bg-gradient-to-br from-blue-400 to-blue-600 flex items-center justify-center text-white text-6xl font-bold">
            {property.bedrooms}BR
          </div>
        )}
      </div>

      {/* Content - keep unchanged */}
      <div className="p-4">
        <div className="mb-2">
          <span className={`inline-block px-2 py-1 text-xs font-semibold rounded ${statusColors[property.status]}`}>
            {capitalize(property.status)}
          </span>
        </div>

        <h3 className="text-lg font-bold text-gray-900 mb-2 line-clamp-2">
          {property.title}
        </h3>

        <p className="text-2xl font-bold text-blue-600 mb-3">
          {formatPrice(property.price)}
        </p>

        <div className="grid grid-cols-2 gap-2 text-sm text-gray-600 mb-3">
          <div>
            <span className="font-semibold">{property.bedrooms}</span> Beds
          </div>
          <div>
            <span className="font-semibold">{property.bathrooms}</span> Baths
          </div>
          <div className="col-span-2">
            <span className="font-semibold">{capitalize(property.property_type)}</span>
          </div>
        </div>

        <div className="border-t pt-3">
          <p className="text-sm text-gray-600">
            📍 {property.location.city}, {property.location.state}
          </p>
        </div>

        {property.agent && (
          <div className="mt-2 text-xs text-gray-500">
            Listed by {property.agent.name}
          </div>
        )}
      </div>
    </div>
  );
}

What changed:

Added Image import from next/image
Added logic to find the main image (display_order === 0 or first image)
Replaced the <div> with <Image> component
Used mainImage.thumbnail_url (the 400x300 version, not the full-size original)
Added fill prop (makes image fill the parent container)
Added sizes prop (tells the browser what size image to request at different viewport widths)
Added loading="lazy" (browser only loads images as they enter viewport)
Kept the gradient fallback for properties without images

Understanding the props:

fill — Image expands to fill the parent div. Requires position: relative on parent (which we have via Tailwind's relative class).
sizes — Tells the browser "on mobile (≤768px), this image takes 100% width. On tablet (≤1200px), 50% width. On desktop, 33% width." The browser uses this to request the appropriately-sized image.
loading="lazy" — Defers loading images that are off-screen. Standard HTML attribute, works in all modern browsers.

Step 15: Update the TypeScript Types

The Property interface needs to include the images field.

Update frontend/types/property.ts:

// ... (keep existing interfaces) ...

export interface PropertyImage {
  id: number;
  original_url: string;
  thumbnail_url: string;
  webp_url: string;
  display_order: number;
  alt_text: string;
}

export interface Property {
  id: number;
  title: string;
  description: string;
  property_type: 'apartment' | 'house' | 'villa' | 'studio' | 'condo';
  price: string;
  bedrooms: number;
  bathrooms: number;
  location: Location;
  agent: Agent;
  status: 'available' | 'pending' | 'sold';
  view_count: number;
  created_at: string;
  images: PropertyImage[];  // ← Add this line
}

// ... (keep PaginatedResponse unchanged) ...

Step 16: Test the Real Images

Make sure you uploaded at least one image via Django admin (Step 10)
Open http://localhost:3000/ in your browser
You should see real property photos instead of blue gradients

If you see blue gradients still:

Check the API response: curl http://localhost:8000/api/properties/cached/ | jq '.results[0].images'
If the images array is empty, no images are uploaded yet
If the images array has data but the frontend shows gradients, check the browser console for errors

📊 [Exercise - : Property Cards with Real Images]

Instructions: Open the listing page in your browser. Capture a screenshot showing the property grid with real images loaded from Cloudinary instead of blue gradients. Highlight one card to show the image quality.

📊 [Example Screenshot - : Property Cards with Real Images]

Part D: Lazy Loading and Performance

Right now, every image on the page loads immediately — even images that are off-screen. On a page with 20 properties, that's 20 network requests happening at once. We need lazy loading.

How Lazy Loading Works

The browser's Intersection Observer API watches elements as they scroll into view. When an image is about to become visible (say, 200px before it enters the viewport), the browser starts loading it. By the time the user scrolls to it, it's ready.

Next.js <Image> component does this automatically if you set loading="lazy" (which we did in Step 14). But let's verify it's working and understand what's happening.

Step 17: Verify Lazy Loading

Open http://localhost:3000/ in your browser with DevTools open (F12 → Network tab).

Clear the network log — Click the 🚫 icon in DevTools to clear existing requests
Hard refresh — Press Ctrl+Shift+R (Windows/Linux) or Cmd+Shift+R (Mac)
Watch the Network tab — You should see ~6-8 image requests initially (only images above the fold)
Scroll down slowly — As you scroll, more image requests appear in the Network tab
Count total requests — By the time you reach the bottom, you should see 20 image requests total

What this proves:

Initial load: only visible images (6-8 requests)
On scroll: remaining images (12-14 requests)
Total transfer size is lower initially, page loads faster

📊 [Exercise : DevTools Network Tab Showing Lazy Loading]

Instructions: With DevTools Network tab open, load the page and immediately take a screenshot showing only 6-8 image requests. Then scroll to the bottom and take another screenshot showing all 20 requests. Highlight the difference in request count.

Step 18: Custom Lazy Loading with Intersection Observer (Advanced)

The native loading="lazy" attribute is great, but it has limitations:

You can't control when the loading starts (browsers decide)
You can't show a custom loading state (skeleton, blur placeholder, etc.)

For more control, we can use the react-intersection-observer library.

Install it:

cd frontend
npm install react-intersection-observer

Create frontend/components/LazyImage.tsx:

/**
 * components/LazyImage.tsx
 * 
 * Custom lazy loading component with Intersection Observer.
 * Shows a skeleton loader until the image enters the viewport.
 */

'use client';

import { useInView } from 'react-intersection-observer';
import Image from 'next/image';
import { useState } from 'react';

interface LazyImageProps {
  src: string;
  alt: string;
  fill?: boolean;
  width?: number;
  height?: number;
  className?: string;
  sizes?: string;
}

export function LazyImage({ src, alt, fill, width, height, className, sizes }: LazyImageProps) {
  const { ref, inView } = useInView({
    triggerOnce: true,  // Load once, don't unload when scrolling back up
    rootMargin: '200px',  // Start loading 200px before entering viewport
  });

  const [isLoading, setIsLoading] = useState(true);

  return (
    <div ref={ref} className="relative w-full h-full">
      {inView ? (
        <>
          {/* Skeleton loader - shows while image is loading */}
          {isLoading && (
            <div className="absolute inset-0 bg-gray-200 animate-pulse" />
          )}

          {/* Actual image */}
          <Image
            src={src}
            alt={alt}
            fill={fill}
            width={width}
            height={height}
            className={className}
            sizes={sizes}
            onLoadingComplete={() => setIsLoading(false)}
          />
        </>
      ) : (
        /* Placeholder before image enters viewport */
        <div className="w-full h-full bg-gray-200" />
      )}
    </div>
  );
}

What this does:

useInView hook detects when the component is about to enter the viewport
rootMargin: '200px' means start loading 200px before it's visible
Shows a gray placeholder before loading
Shows an animated skeleton while loading
Shows the image after loading completes

To use it, replace <Image> with <LazyImage> in PropertyCard.tsx:

import { LazyImage } from './LazyImage';

// Inside PropertyCard:
{mainImage ? (
  <LazyImage
    src={mainImage.thumbnail_url}
    alt={mainImage.alt_text || property.title}
    fill
    className="object-cover"
    sizes="(max-width: 768px) 100vw, (max-width: 1200px) 50vw, 33vw"
  />
) : (
  <div className="h-full w-full bg-gradient-to-br from-blue-400 to-blue-600" />
)}

This is optional — the native loading="lazy" works fine for most cases. Use LazyImage if you want custom loading states.

Step 19: Blur Placeholder (Premium UX)

Next.js supports blur placeholders — a tiny, blurred version of the image shows while the full image loads. This is the pattern used by Medium, Unsplash, and other high-end sites.

To use it, you need a blurDataURL — a tiny base64-encoded version of the image.

Two approaches:

Approach A: Generate blur hash at build time (requires plaiceholder library)

Best for static images
Doesn't work well for dynamic user-uploaded content
Skip this for now

Approach B: Use Cloudinary's blur transformation (recommended)

Cloudinary can generate a tiny, blurred version on-demand. We create a helper function that transforms any Cloudinary URL into a blur placeholder.

Create frontend/lib/cloudinary.ts:

/**
 * lib/cloudinary.ts
 * 
 * Helper functions for Cloudinary URL transformations.
 */

/**
 * Generate a tiny blurred version of a Cloudinary image for use as a placeholder.
 * Returns a 20px-wide, heavily blurred version.
 */
export function getCloudinaryBlurURL(url: string): string {
  if (!url || !url.includes('res.cloudinary.com')) {
    // Not a Cloudinary URL, return a solid color data URL
    return 'data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDAwIiBoZWlnaHQ9IjMwMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48cmVjdCB3aWR0aD0iNDAwIiBoZWlnaHQ9IjMwMCIgZmlsbD0iI2UwZTBlMCIvPjwvc3ZnPg==';
  }

  // Parse the URL and inject blur transformation
  // Cloudinary URL format: https://res.cloudinary.com/{cloud_name}/image/upload/{transformations}/{path}
  const parts = url.split('/upload/');
  if (parts.length !== 2) return url;  // Invalid format, return as-is

  const [base, path] = parts;

  // Build blur transformation: width=20, quality=1, blur=1000
  const blurTransformation = 'w_20,q_1,e_blur:1000';

  return `${base}/upload/${blurTransformation}/${path}`;
}

What this does:

Splits the Cloudinary URL at /upload/
Injects transformation parameters (w_20,q_1,e_blur:1000)
Returns a URL that generates a 20px-wide, heavily blurred image
Falls back to a gray SVG if the URL isn't from Cloudinary

Use it in PropertyCard:

import { getCloudinaryBlurURL } from '@/lib/cloudinary';

// Inside PropertyCard, in the Image component:
{mainImage ? (
  <Image
    src={mainImage.thumbnail_url}
    alt={mainImage.alt_text || property.title}
    fill
    className="object-cover"
    sizes="(max-width: 768px) 100vw, (max-width: 1200px) 50vw, 33vw"
    placeholder="blur"
    blurDataURL={getCloudinaryBlurURL(mainImage.thumbnail_url)}
  />
) : (
  <div className="h-full w-full bg-gradient-to-br from-blue-400 to-blue-600" />
)}

What changed:

Added placeholder="blur" prop
Added blurDataURL={getCloudinaryBlurURL(...)} prop

Now when images load, you'll see a blurred preview first, then the sharp image fades in. This is a subtle effect but makes the page feel much more polished.

Part E: The Property Detail Page with Image Gallery

So far, we only show the first image on each property card. We need a detail page that shows all images.

Step 20: Create the Detail Page API Endpoint

We need a Django endpoint that returns a single property by ID.

Update backend/housing/urls.py:

from django.urls import path
from .views import PropertyListView, CachedPropertyListView, OptimizedPropertyListView, PropertyDetailView  # ← Add PropertyDetailView

urlpatterns = [
    path('properties/live/naive/', PropertyListView.as_view(), name='property-naive'),
    path('properties/cached/', CachedPropertyListView.as_view(), name='property-cached'),
    path('properties/live/optimized/', OptimizedPropertyListView.as_view(), name='property-optimized'),
    path('properties/<int:pk>/', PropertyDetailView.as_view(), name='property-detail'),  # ← Add this
]

Add the view to backend/housing/views.py:

from rest_framework import generics
# ... (keep existing imports) ...


# ... (keep existing views) ...


class PropertyDetailView(generics.RetrieveAPIView):
    """
    Retrieve a single property by ID.
    Returns all images, not just the first one.
    """
    queryset = Property.objects.select_related(
        'location',
        'agent__office'
    ).prefetch_related('images').all()
    serializer_class = PropertySerializer

Restart the backend:

docker compose restart backend

Test it:

curl http://localhost:8000/api/properties/1/ | jq

You should see a single property object with all its images.

Step 21: Create the Image Gallery Component

Create frontend/components/ImageGallery.tsx:

/**
 * components/ImageGallery.tsx
 * 
 * Image gallery with main image and thumbnail strip.
 * Click thumbnails to change the main image.
 */

'use client';

import { useState } from 'react';
import Image from 'next/image';
import type { PropertyImage } from '@/types/property';

interface ImageGalleryProps {
  images: PropertyImage[];
}

export function ImageGallery({ images }: ImageGalleryProps) {
  const [currentIndex, setCurrentIndex] = useState(0);

  if (!images || images.length === 0) {
    return (
      <div className="w-full h-96 bg-gradient-to-br from-blue-400 to-blue-600 flex items-center justify-center text-white text-4xl font-bold">
        No Images
      </div>
    );
  }

  const currentImage = images[currentIndex];

  return (
    <div className="space-y-4">
      {/* Main image */}
      <div className="relative w-full h-96 bg-gray-100 rounded-lg overflow-hidden">
        <Image
          src={currentImage.original_url}
          alt={currentImage.alt_text || 'Property image'}
          fill
          className="object-contain"
          priority  // Load first image immediately
          sizes="100vw"
        />

        {/* Navigation arrows (if more than 1 image) */}
        {images.length > 1 && (
          <>
            <button
              onClick={() => setCurrentIndex((currentIndex - 1 + images.length) % images.length)}
              className="absolute left-4 top-1/2 -translate-y-1/2 bg-white/80 hover:bg-white p-2 rounded-full shadow-lg"
              aria-label="Previous image"
            >
              <svg className="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M15 19l-7-7 7-7" />
              </svg>
            </button>
            <button
              onClick={() => setCurrentIndex((currentIndex + 1) % images.length)}
              className="absolute right-4 top-1/2 -translate-y-1/2 bg-white/80 hover:bg-white p-2 rounded-full shadow-lg"
              aria-label="Next image"
            >
              <svg className="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M9 5l7 7-7 7" />
              </svg>
            </button>
          </>
        )}

        {/* Image counter */}
        <div className="absolute bottom-4 right-4 bg-black/70 text-white px-3 py-1 rounded-full text-sm">
          {currentIndex + 1} / {images.length}
        </div>
      </div>

      {/* Thumbnail strip */}
      {images.length > 1 && (
        <div className="flex gap-2 overflow-x-auto pb-2">
          {images.map((image, index) => (
            <button
              key={image.id}
              onClick={() => setCurrentIndex(index)}
              className={`relative flex-shrink-0 w-24 h-18 rounded overflow-hidden ${
                index === currentIndex ? 'ring-2 ring-blue-600' : 'opacity-60 hover:opacity-100'
              }`}
            >
              <Image
                src={image.thumbnail_url}
                alt={image.alt_text || `Thumbnail ${index + 1}`}
                fill
                className="object-cover"
                sizes="96px"
              />
            </button>
          ))}
        </div>
      )}
    </div>
  );
}

What this does:

Shows a large main image
Shows a thumbnail strip below (if more than 1 image)
Click a thumbnail to change the main image
Left/right arrow buttons to navigate
Image counter (1 / 5, 2 / 5, etc.)
Uses object-contain for main image (shows full image without cropping)
Uses object-cover for thumbnails (crops to fill the square)

Step 22: Create the Property Detail Page

Create frontend/app/properties/[id]/page.tsx:

/**
 * app/properties/[id]/page.tsx
 * 
 * Property detail page showing full information and all images.
 */

import { fetchAPI } from '@/lib/api';
import { Property } from '@/types/property';
import { ImageGallery } from '@/components/ImageGallery';
import { formatPrice, capitalize, formatDate } from '@/lib/formatters';
import Link from 'next/link';

interface PropertyDetailPageProps {
  params: {
    id: string;
  };
}

export default async function PropertyDetailPage({ params }: PropertyDetailPageProps) {
  const property: Property = await fetchAPI(`/api/properties/${params.id}/`);

  return (
    <div className="min-h-screen bg-gray-50">
      {/* Header */}
      <header className="bg-white border-b border-gray-200">
        <div className="max-w-7xl mx-auto px-4 py-6">
          <Link href="/" className="text-blue-600 hover:text-blue-800 mb-4 inline-block">
            ← Back to listings
          </Link>
          <h1 className="text-3xl font-bold text-gray-900">{property.title}</h1>
          <p className="text-gray-600 mt-1">
            📍 {property.location.city}, {property.location.state} {property.location.zip_code}
          </p>
        </div>
      </header>

      {/* Main content */}
      <main className="max-w-7xl mx-auto px-4 py-8">
        <div className="grid grid-cols-1 lg:grid-cols-3 gap-8">
          {/* Left column - Images and details */}
          <div className="lg:col-span-2 space-y-6">
            {/* Image gallery */}
            <ImageGallery images={property.images} />

            {/* Description */}
            <div className="bg-white rounded-lg p-6 shadow">
              <h2 className="text-xl font-bold mb-4">About this property</h2>
              <p className="text-gray-700 whitespace-pre-line">{property.description}</p>
            </div>

            {/* Details */}
            <div className="bg-white rounded-lg p-6 shadow">
              <h2 className="text-xl font-bold mb-4">Property Details</h2>
              <dl className="grid grid-cols-2 gap-4">
                <div>
                  <dt className="text-sm text-gray-600">Type</dt>
                  <dd className="text-lg font-semibold">{capitalize(property.property_type)}</dd>
                </div>
                <div>
                  <dt className="text-sm text-gray-600">Status</dt>
                  <dd className="text-lg font-semibold">{capitalize(property.status)}</dd>
                </div>
                <div>
                  <dt className="text-sm text-gray-600">Bedrooms</dt>
                  <dd className="text-lg font-semibold">{property.bedrooms}</dd>
                </div>
                <div>
                  <dt className="text-sm text-gray-600">Bathrooms</dt>
                  <dd className="text-lg font-semibold">{property.bathrooms}</dd>
                </div>
                <div>
                  <dt className="text-sm text-gray-600">Listed</dt>
                  <dd className="text-lg font-semibold">{formatDate(property.created_at)}</dd>
                </div>
                <div>
                  <dt className="text-sm text-gray-600">Views</dt>
                  <dd className="text-lg font-semibold">{property.view_count.toLocaleString()}</dd>
                </div>
              </dl>
            </div>
          </div>

          {/* Right column - Price and agent */}
          <div className="space-y-6">
            {/* Price card */}
            <div className="bg-white rounded-lg p-6 shadow sticky top-4">
              <div className="text-4xl font-bold text-blue-600 mb-6">
                {formatPrice(property.price)}
              </div>

              {/* Agent info */}
              {property.agent && (
                <div className="border-t pt-6">
                  <h3 className="text-sm font-semibold text-gray-600 mb-2">LISTED BY</h3>
                  <p className="text-lg font-semibold">{property.agent.name}</p>
                  {property.agent.email && (
                    <p className="text-sm text-gray-600 mt-1">{property.agent.email}</p>
                  )}
                  {property.agent.phone && (
                    <p className="text-sm text-gray-600">{property.agent.phone}</p>
                  )}
                  {property.agent.office && (
                    <p className="text-sm text-gray-500 mt-2">{property.agent.office.name}</p>
                  )}
                </div>
              )}

              {/* Contact button */}
              <button className="w-full mt-6 bg-blue-600 text-white py-3 px-4 rounded-lg font-semibold hover:bg-blue-700 transition">
                Contact Agent
              </button>
            </div>
          </div>
        </div>
      </main>
    </div>
  );
}

What this page shows:

Image gallery at the top
Property description
Property details (type, bedrooms, bathrooms, etc.)
Price and agent info in a sticky sidebar
Back button to return to listings

Step 23: Make Property Cards Clickable

Update frontend/components/PropertyCard.tsx to link to the detail page:

import Link from 'next/link';  // ← Add this import

// Wrap the entire card in a Link:
export function PropertyCard({ property }: PropertyCardProps) {
  // ... (keep all existing code) ...

  return (
    <Link href={`/properties/${property.id}`} className="block">
      <div className="border border-gray-200 rounded-lg overflow-hidden hover:shadow-lg transition-shadow duration-200">
        {/* ... (keep all existing content) ... */}
      </div>
    </Link>
  );
}

Now when users click a property card, they navigate to the detail page.

Test it:

Open http://localhost:3000/
Click any property card
You should see the detail page with the image gallery

📊 [Screenshot : Property Detail Page with Image Gallery]

Part F: Performance Measurement — The Before and After

Let's prove the optimizations work by measuring actual performance.

Step 24: Create a Test Page Without Optimization

To show the difference, we need a "before" page that doesn't use any of our optimizations.

Create frontend/app/test-unoptimized/page.tsx:

/**
 * app/test-unoptimized/page.tsx
 * 
 * Test page WITHOUT Next.js Image optimization.
 * Uses regular <img> tags to show the performance difference.
 */

import { fetchAPI } from '@/lib/api';
import { PaginatedResponse, Property } from '@/types/property';

export default async function UnoptimizedTestPage() {
  const data: PaginatedResponse<Property> = await fetchAPI('/api/properties/cached/');

  return (
    <div className="min-h-screen bg-gray-50 p-8">
      <div className="max-w-7xl mx-auto">
        <h1 className="text-3xl font-bold mb-4">Unoptimized Images Test</h1>
        <p className="text-red-600 mb-8">
          ⚠️ This page uses regular &lt;img&gt; tags without optimization. Do NOT use this in production.
        </p>

        <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
          {data.results.map((property) => {
            const mainImage = property.images?.[0];

            return (
              <div key={property.id} className="border rounded-lg overflow-hidden">
                <div className="h-48 bg-gray-200">
                  {mainImage && (
                    /* Regular img tag - NO optimization */
                    <img
                      src={mainImage.original_url}  // Full-size original (5MB+)
                      alt={property.title}
                      className="w-full h-full object-cover"
                    />
                  )}
                </div>
                <div className="p-4">
                  <h3 className="font-bold">{property.title}</h3>
                  <p className="text-xl font-bold text-blue-600">${property.price}</p>
                </div>
              </div>
            );
          })}
        </div>
      </div>
    </div>
  );
}

What this page does:

Uses regular <img> tags (not Next.js <Image>)
Loads original_url (full-size images, 3-5MB each)
No lazy loading
No format conversion (JPEG only, no WebP)
No responsive sizing

This is the "worst case" — how the page would perform without any optimization.

Step 25: Measure the Unoptimized Page

Open http://localhost:3000/test-unoptimized with DevTools open (F12 → Network tab).

What to measure:

Total transfer size — Look at the bottom of the Network tab. It shows "XXX MB transferred"
Total load time — The time until all images finish loading
Number of requests — Count the image requests
Largest Contentful Paint (LCP) — Go to Lighthouse tab, run audit, check LCP metric

Record these numbers.

📊 [Exercise - Data: Unoptimized Performance Metrics]

Instructions: Load the test-unoptimized page with DevTools Network tab open. Record:

Metric Value

Total transfer size [YOUR MB]

Total load time [YOUR SECONDS]

Number of image requests [YOUR COUNT]

LCP (Lighthouse) [YOUR MS]

Metric	Value
Total transfer size	[YOUR MB]
Total load time	[YOUR SECONDS]
Number of image requests	[YOUR COUNT]
LCP (Lighthouse)	[YOUR MS]

Step 26: Measure the Optimized Page

Now measure the main listing page (http://localhost:3000/) which uses all our optimizations.

Clear the browser cache first (important):

DevTools → Network tab → Right-click → Clear browser cache

Then hard refresh (Ctrl+Shift+R or Cmd+Shift+R).

Measure the same metrics:

Total transfer size
Total load time
Number of requests (initially vs after scrolling)
LCP

📊 [Exercise - Data : Optimized Performance Metrics]

Instructions: Load the main listing page with DevTools Network tab open. Record:

Metric Unoptimized Optimized Improvement

Total transfer size [YOUR MB] [YOUR MB] [CALCULATE %]

Total load time [YOUR S] [YOUR S] [CALCULATE %]

LCP [YOUR MS] [YOUR MS] [CALCULATE %]

Initial image requests 20 [YOUR COUNT] -

Metric	Unoptimized	Optimized	Improvement
Total transfer size	[YOUR MB]	[YOUR MB]	[CALCULATE %]
Total load time	[YOUR S]	[YOUR S]	[CALCULATE %]
LCP	[YOUR MS]	[YOUR MS]	[CALCULATE %]
Initial image requests	20	[YOUR COUNT]	-

Expected improvements:

Transfer size: 85-90% reduction (100MB → 10-15MB)
Load time: 80-90% faster (10s → 1-2s)
LCP: 70-80% faster (3-5s → 0.5-1s)
Initial requests: 70% reduction (20 → 6-8)

The exact numbers depend on your internet speed and how many properties have images uploaded.

Step 27: Lighthouse Performance Audit

Lighthouse is Chrome's built-in performance auditing tool. It measures real-world metrics.

Open http://localhost:3000/ in Chrome
Open DevTools (F12)
Go to Lighthouse tab
Select:
- Categories: Performance
- Device: Mobile (more realistic than desktop)
Click Analyze page load

Wait 30-60 seconds for the audit to complete.

What to look for:

Performance score — should be 90-100 (green)
LCP (Largest Contentful Paint) — should be under 2.5s (green)
CLS (Cumulative Layout Shift) — should be under 0.1 (green)
FID (First Input Delay) — should be under 100ms (green)

The <Image> component with fill prop and lazy loading should give you excellent scores.

📊 [Exercise : Lighthouse Performance Score]

Instructions: Run a Lighthouse audit on the main listing page. Capture a screenshot showing the Performance score (should be 90+) and the Core Web Vitals metrics (LCP, CLS, FID).

Part G: Troubleshooting

Issue 1: "Invalid src prop" Error

Symptom:

Error: Invalid src prop (https://res.cloudinary.com/...) on `next/image`, hostname "res.cloudinary.com" is not configured under images in your `next.config.js`

Cause: Next.js doesn't allow external images by default.

Fix: Add Cloudinary to remotePatterns in next.config.js (Step 13). Restart the frontend container.

Issue 2: Images Upload to Cloudinary but Don't Display

Symptom: Image appears in Cloudinary dashboard but frontend shows gradient placeholder.

Possible causes:

The serializer doesn't include images — Check PropertySerializer has images = PropertyImageSerializer(many=True, read_only=True)
The view doesn't prefetch images — Check the view has .prefetch_related('images')
The TypeScript interface is wrong — Check Property interface has images: PropertyImage[]

Debug:

# Check the API response directly
curl http://localhost:8000/api/properties/cached/ | jq '.results[0].images'

If this returns an empty array, the problem is in the backend. If it returns data, the problem is in the frontend.

Issue 3: "Failed to fetch" When Uploading in Admin

Symptom: Image upload form submits but shows an error. No image appears in Cloudinary.

Cause: Cloudinary credentials are wrong or not loaded.

Fix:

Verify credentials in backend/.env
Verify docker-compose.yml loads the .env file
Restart backend: docker compose restart backend
Run the test script from Step 6

Issue 4: Images Are Slow Even with Optimization

Symptom: Images still take 5-10 seconds to load.

Possible causes:

Using original_url instead of thumbnail_url — Check PropertyCard uses mainImage.thumbnail_url, not mainImage.original_url
Cloudinary transformations not applied — Check the model methods include quality: 'auto' and fetch_format: 'auto'
Network is genuinely slow — Test on a different network. Use Lighthouse's "throttling" feature to simulate 3G and see if Cloudinary is actually helping.

Issue 5: Layout Shift / CLS Issues

Symptom: Images "pop in" and push content down when they load. Lighthouse shows high CLS score.

Cause: The <Image> component doesn't know the aspect ratio before the image loads.

Fix: Always use fill prop with a container that has a fixed height:

<div className="relative h-48 w-full">  {/* Fixed height */}
  <Image src="..." alt="..." fill className="object-cover" />
</div>

Or if you know the image dimensions, use width and height props:

<Image src="..." alt="..." width={400} height={300} />

Issue 6: Next.js Build Fails (Type Errors)

Symptom:

Type error: Property 'images' does not exist on type 'Property'

Cause: TypeScript interface doesn't match the actual data structure.

Fix: Make sure frontend/types/property.ts has the PropertyImage interface and Property has images: PropertyImage[]. Restart the dev server.

Here’s a short, dev.to–ready troubleshooting section you can drop straight into your post.
It’s concise, practical, and aligned with the exact failure modes you hit.

🧹 Troubleshooting : Cleaning the Housing Models & Reseeding Data

If you run into errors like:

relation "housing_location" already exists
relation "housing_location" does not exist
Cloudinary uploads failing during seeding
Migrations applied but seed scripts crashing

you’re almost certainly dealing with schema drift: your Django models, migration history, and database schema are out of sync.

This commonly happens after:

Refactoring models (e.g. moving from URL fields → CloudinaryField)
Deleting or rewriting migration files
Reusing an old Docker database volume

✅ The Clean Reset (recommended)

In development, the safest fix is a full reset:

docker compose down -v

This removes the PostgreSQL volume and guarantees a clean schema.
You can use code from previous GIST for seed_data_with_cloudinary:

Then rebuild everything in the correct order:

docker compose up -d
docker compose exec backend python manage.py makemigrations
docker compose exec backend python manage.py migrate
docker compose exec backend python manage.py seed_data_with_cloudinary --flush

At this point:

The database schema exactly matches the current models
All housing data is regenerated
Property images are uploaded to Cloudinary using the new seed script

🔑 Important rule

Always run migrations before seeding, and always run both inside Docker.

Seeding assumes tables already exist. If migrations haven’t been applied—or were applied to a different database—you’ll see table-related errors immediately.

🧠 Why this works

The updated seed script:

Matches the new PropertyImage model
Uploads images explicitly via the Cloudinary SDK
Stores only deterministic public_ids in the database
Is safe to re-run after a flush

If something feels “haunted,” it’s almost always stale schema state. Resetting once saves hours of debugging later.

⚠️ When `--fake` Migrations Make Things Worse

Django’s --fake and --fake-initial options can be useful in very specific scenarios, but they come with a sharp edge.

If you’ve ever done something like:

python manage.py migrate --fake
python manage.py migrate app_name 0001 --fake

and later started seeing errors such as:

relation already exists
relation does not exist
migrations applying out of order
Django insisting a migration is applied while PostgreSQL disagrees

then your database is in an unstable migration state.

What went wrong

Using --fake tells Django:

“Trust me, the database schema already matches this migration.”

But Django does not verify the actual tables, columns, or indexes.
If the schema doesn’t exactly match the models:

Django’s migration history lies
PostgreSQL’s schema tells a different story
Every future migration becomes unpredictable

At that point, you’re debugging migration metadata, not your application.

The correct recovery path

Once fake migrations start cascading into different errors, stop patching.
The fastest and safest fix in development is a clean reset:

docker compose down -v
docker compose up -d
docker compose exec backend python manage.py makemigrations
docker compose exec backend python manage.py migrate
docker compose exec backend python manage.py seed_data_with_cloudinary --flush

This guarantees:

Migration history matches the actual schema
Models, tables, and constraints are aligned
Seed scripts run without hidden side effects

Rule of thumb

If you used --fake and things start breaking later, don’t fight it.
Drop the dev database and rebuild once — it’s cheaper than chasing ghosts.

In production, --fake requires surgical care.
In development, a clean rebuild is almost always the right answer.

Part H: The Complete Image Pipeline

Let's map the full journey of an image from admin upload to user viewing.

The Flow (End to End)

Step 1: Admin uploads image

Admin goes to http://localhost:8000/admin/housing/property/1/change/
Scrolls to "Property images" inline section
Clicks "Choose File" and selects a photo
Sets display_order = 0, alt_text = "Front view"
Clicks "Save"

Step 2: Django processes the upload

Django receives the file in memory
django-cloudinary-storage intercepts the save operation
Calls cloudinary.uploader.upload(file, folder='housing/properties')
Cloudinary uploads the file to their CDN
Cloudinary returns a response: { "secure_url": "https://res.cloudinary.com/...", "public_id": "housing/properties/abc123" }
Django saves the URL to PropertyImage.image field

Step 3: User requests the listing page

User navigates to http://localhost:3000/
Next.js (server-side) fetches from http://backend:8000/api/properties/cached/
Django queries the database with prefetch_related('images')
PropertyImageSerializer calls obj.get_thumbnail_url() for each image
The method generates a Cloudinary URL with transformations: https://res.cloudinary.com/.../w_400,h_300,c_fill,q_auto,f_auto/.../abc123.jpg
Django returns JSON to Next.js

Step 4: Next.js renders the page

Next.js Server Component receives the property data
Renders the page HTML with <Image> components
Browser receives the HTML
React hydrates (makes the page interactive)

Step 5: Browser loads images

Browser sees <Image src="https://res.cloudinary.com/.../w_400,h_300..." loading="lazy" />
For images above the fold (visible immediately), browser sends request to Cloudinary
Cloudinary checks its cache: does a 400x300 WebP version of this image exist?
If yes: Cloudinary serves it from edge cache (~10ms response time)
If no: Cloudinary generates it on-demand (resize original to 400x300, convert to WebP), caches it, serves it (~100ms first time)
Next.js caches the image in the browser (memory + disk cache)
For images below the fold (not visible), browser waits until they're about to scroll into view, then repeats steps 23-27

Step 6: User navigates away and comes back

User visits another page, then clicks Back button
Browser checks its cache: does it have the HTML for http://localhost:3000/?
Yes → renders immediately (0ms)
Browser checks image cache: does it have https://res.cloudinary.com/.../w_400...?
Yes → displays images immediately from disk cache (0ms)
No network requests needed

Where Caching Happens

Layer	Technology	What's cached	Invalidation
User's browser	Browser HTTP cache	HTML, images, CSS, JS	Time-based (headers)
Next.js CDN	Vercel edge network	Rendered HTML (on deploy)	On deploy
SWR	Browser memory	API responses	Focus, interval, manual
Cloudinary CDN	Cloudinary edge cache	Transformed images	Never expires (immutable URLs)
Redis	Backend cache	API responses (JSON)	Django signals
PostgreSQL	Database	Source data	Never (source of truth)

Six layers of caching. Each one serves a different purpose. Remove any one, and the system still works — but it's slower.

What We Built — And What's Left

We started Part 6 with blue gradients. We ended with production-grade image delivery:

Images are hosted on a CDN. Cloudinary serves from edge servers globally. A user in Tokyo and a user in New York both get 50ms response times.

Images are optimized automatically. Cloudinary converts JPEG to WebP for Chrome users (30% smaller). It adjusts quality based on content (higher for photos, lower for graphics). It generates responsive sizes on-demand (400px for mobile, 800px for desktop).

Images lazy load. Only visible images load initially (6-8 images). The rest load as you scroll. This cuts initial transfer from 100MB to 10MB.

Images have blur placeholders. A tiny, blurred preview shows while the full image loads. This is the pattern used by Medium and Unsplash. It makes the page feel faster even when it's loading at the same speed.

The system is measurable. We went from 10s load time to 2s. From 100MB transferred to 10MB. From LCP of 5s to 1s. The Lighthouse score went from ~40 to ~95. We didn't guess. We measured.

But we're not done. The system works. It's fast. But it's not deployed. It runs on localhost. No one outside your machine can access it. Part 7 fixes that. We deploy to production. Vercel for the frontend. Railway or Fly.io for the backend. Cloudinary stays as-is (it's already a CDN). We add custom domains. We add SSL. We add monitoring. We make it real.

Checkpoint: Push to GitHub

git add .
git commit -m "feat: add Cloudinary CDN integration, image optimization, lazy loading, and image gallery"
git checkout -b part-6-images
git push origin part-6-images

The repo now has six branches:

part-1-setup — infrastructure
part-2-data — database and seed
part-3-caching — API and Redis
part-4-optimization — query optimization and signals
part-5-frontend — Next.js UI and SWR
part-6-images — Cloudinary, lazy loading, gallery

Diff any two branches:

git diff part-5-frontend..part-6-images

Next: Part 7 — Deployment: From Localhost to Production. Custom domains, SSL, monitoring, and CI/CD. Stay tuned.

From Zero to Cached: Building a High-Performance Housing Portal with Django, Next.js, and Redis - Part -5 : Frontend

Ajit Kumar — Fri, 06 Feb 2026 04:41:00 +0000

Part 5: The Frontend Layer — Bringing the API to Life

In Part 1, we built the infrastructure. In Part 2, we created the database. In Part 3, we cached the API. In Part 4, we optimized the database and made the cache smart. Today, we make it visible.

If you're jumping in here, start from the beginning. Part 1 sets up Docker. Part 2 seeds the data. Part 3 introduces Redis. Part 4 fixes the queries. If you're continuing from Part 4, you have a Django API that responds in 4ms with fresh data and 1 optimized query on cache misses.

Today we build the actual housing portal — the UI users see, the filters they interact with, and the third caching layer: the browser. By the end of this post, you'll have a working real estate listing page powered by three layers of caching working together.

What We're Building Today

We've spent four parts building the backend. Fast API. Smart cache. Optimized database. But if you open http://localhost:3000 right now, you see the default Next.js welcome page. No properties. No filters. No housing portal.

Today we change that. We're building:

A property listing page — grid of cards showing title, price, location, agent
Filters — search by type, city, price range
Pagination — browse through all 5,000 properties
Client-side caching — instant repeat visits via SWR
The third caching layer — Redis (backend) + Next.js SSR + SWR (browser)

By the end, the system will be complete. A user can search for apartments in Seattle under $500k, see 20 results instantly, paginate to see more, and when they come back tomorrow, the page loads from the browser cache in 0ms.

Part A: The Foundation — Connecting Next.js to Django

The frontend container is already running from Part 1. But it's isolated. It doesn't know the Django API exists. We need to connect them — and handle the fact that "localhost" means different things depending on where the code runs.

The URL Problem

Here's the issue. When Next.js runs on the server (inside the Docker container), it needs to call the API at http://backend:8000 — that's the Docker service name from docker-compose.yml. But when Next.js runs in the browser (on your machine), it needs to call the API at http://localhost:8000 — because the browser has no access to Docker's internal network.

We need code that works in both contexts.

Step 1: Create the API Configuration

Create frontend/lib/api.ts:

/**
 * lib/api.ts
 * 
 * Centralized API configuration.
 * Handles the URL difference between server-side (Docker) and client-side (browser).
 */

// Determine the base URL based on where the code is running
const getBaseURL = () => {
  // Check if we're running on the server (Node.js) or in the browser
  if (typeof window === 'undefined') {
    // Server-side: use the Docker service name
    return 'http://backend:8000';
  }
  // Client-side: use localhost
  return 'http://localhost:8000';
};

export const API_BASE_URL = getBaseURL();

/**
 * Centralized fetch wrapper.
 * All API calls go through this function.
 */
export async function fetchAPI(endpoint: string, options?: RequestInit) {
  const url = `${API_BASE_URL}${endpoint}`;

  const response = await fetch(url, {
    ...options,
    headers: {
      'Content-Type': 'application/json',
      ...options?.headers,
    },
  });

  if (!response.ok) {
    throw new Error(`API call failed: ${response.status} ${response.statusText}`);
  }

  return response.json();
}

What this does:

typeof window === 'undefined' is the standard way to detect server-side rendering in Next.js. In Node.js (the server), window doesn't exist. In the browser, it does.
getBaseURL() returns the correct URL for the context.
fetchAPI() is a wrapper around fetch that adds error handling and the correct base URL.

If you're new to TypeScript, don't worry about the type annotations (endpoint: string, options?: RequestInit) — they're just documentation that tells TypeScript what types of data to expect. The code works the same as JavaScript.

Step 2: Create TypeScript Types for the API Response

Next.js uses TypeScript by default (we chose "Yes" to TypeScript in Part 1 when running create-next-app). TypeScript needs to know what shape the API response has so it can catch errors at build time.

Create frontend/types/property.ts:

/**
 * types/property.ts
 * 
 * TypeScript interfaces that mirror the Django serializers.
 * These define the shape of the data we get from the API.
 */

export interface Location {
  id: number;
  city: string;
  state: string;
  zip_code: string;
}

export interface Office {
  id: number;
  name: string;
  city: string;
  phone: string;
}

export interface Agent {
  id: number;
  name: string;
  email: string;
  phone: string;
  office: Office;
}

export interface Property {
  id: number;
  title: string;
  description: string;
  property_type: 'apartment' | 'house' | 'villa' | 'studio' | 'condo';
  price: string;  // Django returns this as a string: "450000.00"
  bedrooms: number;
  bathrooms: number;
  location: Location;
  agent: Agent;
  status: 'available' | 'pending' | 'sold';
  view_count: number;
  created_at: string;  // ISO 8601 date string
}

export interface PaginatedResponse<T> {
  count: number;
  next: string | null;
  previous: string | null;
  results: T[];
}

What this does:

Each interface defines the structure of an object. If you try to access property.invalidField, TypeScript will error at build time.
property_type: 'apartment' | 'house' | ... means the field can only be one of those exact strings. This is a union type.
PaginatedResponse<T> is a generic interface. It works with any type. PaginatedResponse<Property> means a paginated response where results is an array of Property objects.

These interfaces match the serializers we wrote in Part 3. If you change a serializer in Django, change the corresponding interface here. TypeScript will then tell you everywhere in the frontend that needs to update.

Step 3: Verify the Connection

Before we build the UI, let's prove the connection works. Create a simple test page.

Create frontend/app/test/page.tsx:

/**
 * app/test/page.tsx
 * 
 * A simple test page to verify the API connection works.
 * This is a Server Component — it fetches on the server before rendering.
 */

import { fetchAPI } from '@/lib/api';
import { PaginatedResponse, Property } from '@/types/property';

export default async function TestPage() {
  try {
    const data: PaginatedResponse<Property> = await fetchAPI('/api/properties/cached/');

    return (
      <div className="p-8">
        <h1 className="text-2xl font-bold mb-4">API Connection Test</h1>
        <p className="mb-4">
          Successfully fetched {data.results.length} properties out of {data.count} total.
        </p>
        <div className="space-y-2">
          {data.results.slice(0, 3).map((property) => (
            <div key={property.id} className="border p-4 rounded">
              <h2 className="font-bold">{property.title}</h2>
              <p>Price: ${property.price}</p>
              <p>Location: {property.location.city}, {property.location.state}</p>
            </div>
          ))}
        </div>
      </div>
    );
  } catch (error) {
    return (
      <div className="p-8">
        <h1 className="text-2xl font-bold text-red-600">Error</h1>
        <p>Failed to fetch data from API.</p>
        <pre className="mt-4 p-4 bg-gray-100 rounded">
          {error instanceof Error ? error.message : 'Unknown error'}
        </pre>
      </div>
    );
  }
}

What this does:

export default async function — this is an async Server Component. It runs on the server, fetches data, and renders HTML before sending it to the browser.
await fetchAPI('/api/properties/cached/') — calls our API wrapper. On the server, this uses http://backend:8000. The await syntax is standard JavaScript for handling promises.
try/catch — if the fetch fails (API is down, wrong URL, etc.), we show an error page instead of crashing.
The JSX (the HTML-like syntax) renders the properties. className is React's version of class (because class is a reserved word in JavaScript).

A note on the @/ import: Next.js configures TypeScript to treat @/ as an alias for the project root. @/lib/api means frontend/lib/api.ts. This makes imports cleaner than ../../lib/api.

Step 4: Test It

Make sure the backend is running:

docker compose ps

You should see backend, frontend, db, and redis all in the "Up" state.

Open your browser and navigate to:

http://localhost:3000/test

If everything is configured correctly, you should see:

API Connection Test

Successfully fetched 20 properties out of 5000 total.

[Three property cards with title, price, and location]

Here is a screenshot:
📊 [Screenshot : Filters in Action]

If you see an error instead, check:

Is the backend running? docker compose logs backend should show no errors.
Is the API endpoint correct? The URL should be /api/properties/cached/ (with trailing slash).
Is CORS configured? Check settings.py — CORS_ALLOWED_ORIGINS should include http://localhost:3000.

If you see "Failed to fetch" in the browser console, open DevTools (F12), go to the Console tab, and check the error message. It will tell you exactly what went wrong.

Part B: The UI Layer — Building the Property Card

The test page proves the connection works. Now we build the actual UI. We start with the smallest piece: a single property card.

Step 5: Create Helper Functions for Formatting

Before we build components, we need helpers to format the data. Django returns prices as "450000.00" — a string with two decimal places. Users expect $450,000. We need a formatter.

Create frontend/lib/formatters.ts:

/**
 * lib/formatters.ts
 * 
 * Utility functions for formatting data for display.
 */

/**
 * Format a price string from the API into a readable currency format.
 * Input: "450000.00"
 * Output: "$450,000"
 */
export function formatPrice(price: string): string {
  const numericPrice = parseFloat(price);
  return new Intl.NumberFormat('en-US', {
    style: 'currency',
    currency: 'USD',
    minimumFractionDigits: 0,  // No cents
    maximumFractionDigits: 0,
  }).format(numericPrice);
}

/**
 * Format a date string into a readable format.
 * Input: "2024-01-15T10:30:00Z"
 * Output: "Jan 15, 2024"
 */
export function formatDate(dateString: string): string {
  const date = new Date(dateString);
  return new Intl.DateTimeFormat('en-US', {
    month: 'short',
    day: 'numeric',
    year: 'numeric',
  }).format(date);
}

/**
 * Capitalize the first letter of each word.
 * Input: "apartment"
 * Output: "Apartment"
 */
export function capitalize(str: string): string {
  return str.charAt(0).toUpperCase() + str.slice(1);
}

What this does:

Intl.NumberFormat and Intl.DateTimeFormat are built-in JavaScript APIs for internationalized formatting. They handle things like currency symbols, thousands separators, and date formats automatically.
parseFloat(price) converts the string "450000.00" into the number 450000.
These are pure functions — same input always produces the same output. No side effects. Easy to test.

Step 6: Create the PropertyCard Component

This is the visual representation of a single property. It's reusable — we'll map over an array of properties and render one card per property.

Create frontend/components/PropertyCard.tsx:

/**
 * components/PropertyCard.tsx
 * 
 * A card component that displays a single property listing.
 * Used in the grid on the listing page.
 */

import { Property } from '@/types/property';
import { formatPrice, capitalize } from '@/lib/formatters';

interface PropertyCardProps {
  property: Property;
}

export function PropertyCard({ property }: PropertyCardProps) {
  // Determine badge color based on status
  const statusColors = {
    available: 'bg-green-100 text-green-800',
    pending: 'bg-yellow-100 text-yellow-800',
    sold: 'bg-red-100 text-red-800',
  };

  return (
    <div className="border border-gray-200 rounded-lg overflow-hidden hover:shadow-lg transition-shadow duration-200">
      {/* Image placeholder - we'll add real images in Part 6 */}
      <div className="h-48 bg-gradient-to-br from-blue-400 to-blue-600 flex items-center justify-center text-white text-6xl font-bold">
        {property.bedrooms}BR
      </div>

      {/* Content */}
      <div className="p-4">
        {/* Status badge */}
        <div className="mb-2">
          <span className={`inline-block px-2 py-1 text-xs font-semibold rounded ${statusColors[property.status]}`}>
            {capitalize(property.status)}
          </span>
        </div>

        {/* Title */}
        <h3 className="text-lg font-bold text-gray-900 mb-2 line-clamp-2">
          {property.title}
        </h3>

        {/* Price */}
        <p className="text-2xl font-bold text-blue-600 mb-3">
          {formatPrice(property.price)}
        </p>

        {/* Details grid */}
        <div className="grid grid-cols-2 gap-2 text-sm text-gray-600 mb-3">
          <div>
            <span className="font-semibold">{property.bedrooms}</span> Beds
          </div>
          <div>
            <span className="font-semibold">{property.bathrooms}</span> Baths
          </div>
          <div className="col-span-2">
            <span className="font-semibold">{capitalize(property.property_type)}</span>
          </div>
        </div>

        {/* Location */}
        <div className="border-t pt-3">
          <p className="text-sm text-gray-600">
            📍 {property.location.city}, {property.location.state}
          </p>
        </div>

        {/* Agent info */}
        {property.agent && (
          <div className="mt-2 text-xs text-gray-500">
            Listed by {property.agent.name}
          </div>
        )}
      </div>
    </div>
  );
}

What this does:

PropertyCardProps defines what data this component expects. It requires one prop: property, which must be a Property object.
The { property }: PropertyCardProps syntax is destructuring. It extracts the property field from the props object.
Tailwind CSS classes handle all the styling. border border-gray-200 creates a light gray border. rounded-lg rounds the corners. hover:shadow-lg adds a shadow when you hover over the card.
line-clamp-2 truncates the title to 2 lines and adds ... if it's longer. This keeps cards uniform height even if titles vary.
The image placeholder uses a blue gradient with the number of bedrooms displayed. We'll replace this with real images in Part 6.

If you're unfamiliar with Tailwind: Each className is a utility class that applies a specific CSS property. text-2xl sets font size. font-bold makes it bold. text-blue-600 makes it blue. You don't write CSS files — you compose classes directly in the JSX. It feels weird at first, then it becomes addictive.

Step 7: Create the Listing Page

Now we use the card component to build a full page.

Replace frontend/app/page.tsx (the homepage) with this:

/**
 * app/page.tsx
 * 
 * The main listing page.
 * Server Component that fetches properties and renders a grid.
 */

import { fetchAPI } from '@/lib/api';
import { PaginatedResponse, Property } from '@/types/property';
import { PropertyCard } from '@/components/PropertyCard';

export default async function HomePage() {
  const data: PaginatedResponse<Property> = await fetchAPI('/api/properties/cached/');

  return (
    <div className="min-h-screen bg-gray-50">
      {/* Header */}
      <header className="bg-white border-b border-gray-200">
        <div className="max-w-7xl mx-auto px-4 py-6">
          <h1 className="text-3xl font-bold text-gray-900">
            Housing Portal
          </h1>
          <p className="text-gray-600 mt-1">
            {data.count.toLocaleString()} properties available
          </p>
        </div>
      </header>

      {/* Main content */}
      <main className="max-w-7xl mx-auto px-4 py-8">
        {/* Grid of property cards */}
        <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
          {data.results.map((property) => (
            <PropertyCard key={property.id} property={property} />
          ))}
        </div>

        {/* Pagination info */}
        <div className="mt-8 text-center text-gray-600">
          Showing {data.results.length} of {data.count} properties
        </div>
      </main>
    </div>
  );
}

What this does:

This replaces the default Next.js welcome page with our actual housing portal.
grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 creates a responsive grid: 1 column on mobile, 2 on tablets, 3 on desktop. Tailwind's responsive prefixes (md:, lg:) handle this automatically.
data.results.map((property) => ...) is the standard React pattern for rendering lists. It loops over the array and returns a component for each item.
key={property.id} is required by React. It helps React track which items changed when the list updates.

Step 8: Verify the Listing Page

Open your browser and navigate to:

http://localhost:3000/

You should see:

A header showing "Housing Portal" and the total count (5,000 properties)
A grid of 20 property cards
Each card shows a blue gradient placeholder, the title, price, beds/baths, location, and agent name

📊 [Screenshot: The Listing Page]

If you don't see cards, open the browser console (F12 → Console tab). Look for errors. Common issues:

"Cannot find module '@/components/PropertyCard'" — You created the file in the wrong location. It should be frontend/components/PropertyCard.tsx (no app/ prefix for components).
"property.location is undefined" — The API response doesn't match the TypeScript interface. Check the Django serializer — make sure it includes nested location and agent fields.
The page is blank but no errors — The server-side fetch might have failed silently. Check docker compose logs frontend for errors.

Part C: The Filter Bar — Interactivity

The listing page works, but it shows the same 20 properties every time. We need filters so users can search for specific types of properties in specific cities.

Filters are interactive. They use React state. That means they must be Client Components, not Server Components.

A Quick Primer on Server vs Client Components

Next.js 14+ uses a new model where components are Server Components by default. Server Components:

Run only on the server
Can await data fetches directly
Cannot use React state or event handlers (onClick, onChange, etc.)
Result in smaller JavaScript bundles (they don't ship to the browser)

Client Components:

Run on both server (for initial HTML) and browser (for interactivity)
Can use React state and event handlers
Must be marked with 'use client' directive at the top of the file

The pattern: Server Components fetch data. Client Components handle interactivity. Server Components can import and render Client Components. Client Components can receive data from Server Components as props.

Step 9: Create the FilterBar Component

Create frontend/components/FilterBar.tsx:

/**
 * components/FilterBar.tsx
 * 
 * A client component that renders filter inputs.
 * Updates URL query parameters when filters change.
 */

'use client';

import { useRouter, useSearchParams } from 'next/navigation';
import { useState, useEffect } from 'react';

export function FilterBar() {
  const router = useRouter();
  const searchParams = useSearchParams();

  // Initialize state from URL parameters
  const [propertyType, setPropertyType] = useState(searchParams.get('type') || '');
  const [city, setCity] = useState(searchParams.get('city') || '');
  const [minPrice, setMinPrice] = useState(searchParams.get('min_price') || '');
  const [maxPrice, setMaxPrice] = useState(searchParams.get('max_price') || '');

  // Update URL when filters change
  useEffect(() => {
    const params = new URLSearchParams();

    if (propertyType) params.set('type', propertyType);
    if (city) params.set('city', city);
    if (minPrice) params.set('min_price', minPrice);
    if (maxPrice) params.set('max_price', maxPrice);

    // Push to URL without reloading the page
    const queryString = params.toString();
    router.push(queryString ? `/?${queryString}` : '/');
  }, [propertyType, city, minPrice, maxPrice, router]);

  return (
    <div className="bg-white border border-gray-200 rounded-lg p-4 mb-6">
      <div className="grid grid-cols-1 md:grid-cols-4 gap-4">
        {/* Property Type Filter */}
        <div>
          <label htmlFor="type" className="block text-sm font-medium text-gray-700 mb-1">
            Property Type
          </label>
          <select
            id="type"
            value={propertyType}
            onChange={(e) => setPropertyType(e.target.value)}
            className="w-full border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:ring-2 focus:ring-blue-500"
          >
            <option value="">All Types</option>
            <option value="apartment">Apartment</option>
            <option value="house">House</option>
            <option value="villa">Villa</option>
            <option value="studio">Studio</option>
            <option value="condo">Condo</option>
          </select>
        </div>

        {/* City Filter */}
        <div>
          <label htmlFor="city" className="block text-sm font-medium text-gray-700 mb-1">
            City
          </label>
          <select
            id="city"
            value={city}
            onChange={(e) => setCity(e.target.value)}
            className="w-full border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:ring-2 focus:ring-blue-500"
          >
            <option value="">All Cities</option>
            <option value="Seattle">Seattle</option>
            <option value="Portland">Portland</option>
            <option value="San Francisco">San Francisco</option>
            <option value="Los Angeles">Los Angeles</option>
            <option value="Austin">Austin</option>
            <option value="Denver">Denver</option>
            <option value="Chicago">Chicago</option>
            <option value="Miami">Miami</option>
            <option value="New York">New York</option>
            <option value="Boston">Boston</option>
          </select>
        </div>

        {/* Min Price Filter */}
        <div>
          <label htmlFor="min-price" className="block text-sm font-medium text-gray-700 mb-1">
            Min Price
          </label>
          <input
            id="min-price"
            type="number"
            value={minPrice}
            onChange={(e) => setMinPrice(e.target.value)}
            placeholder="No minimum"
            className="w-full border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:ring-2 focus:ring-blue-500"
          />
        </div>

        {/* Max Price Filter */}
        <div>
          <label htmlFor="max-price" className="block text-sm font-medium text-gray-700 mb-1">
            Max Price
          </label>
          <input
            id="max-price"
            type="number"
            value={maxPrice}
            onChange={(e) => setMaxPrice(e.target.value)}
            placeholder="No maximum"
            className="w-full border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:ring-2 focus:ring-blue-500"
          />
        </div>
      </div>

      {/* Clear filters button */}
      {(propertyType || city || minPrice || maxPrice) && (
        <button
          onClick={() => {
            setPropertyType('');
            setCity('');
            setMinPrice('');
            setMaxPrice('');
          }}
          className="mt-4 text-sm text-blue-600 hover:text-blue-800 font-medium"
        >
          Clear all filters
        </button>
      )}
    </div>
  );
}

What this does:

'use client' at the top marks this as a Client Component.
useRouter and useSearchParams are Next.js hooks for reading and updating the URL.
useState is the React hook for component state. When you call setPropertyType('apartment'), React re-renders the component with the new value.
useEffect runs after every render. The dependency array [propertyType, city, ...] means "run this effect whenever any of these values change."
router.push() updates the URL without a full page reload. This is client-side navigation.
The conditional render {(propertyType || city || ...) && <button>} means "only show the clear button if at least one filter is active."

If you're new to React: useState and useEffect are the two most important hooks. useState holds data that can change. useEffect runs side effects (like updating the URL) when that data changes.

Step 10: Update the Listing Page to Use Filters

Now we need to make the Server Component read the URL params and pass them to the API.

Update frontend/app/page.tsx:

/**
 * app/page.tsx
 * 
 * Updated to support filtering via URL query parameters.
 */

import { fetchAPI } from '@/lib/api';
import { PaginatedResponse, Property } from '@/types/property';
import { PropertyCard } from '@/components/PropertyCard';
import { FilterBar } from '@/components/FilterBar';

interface HomePageProps {
  searchParams: {
    type?: string;
    city?: string;
    min_price?: string;
    max_price?: string;
    page?: string;
  };
}

export default async function HomePage({ searchParams }: HomePageProps) {
  // Build the API URL with query parameters
  const params = new URLSearchParams();

  if (searchParams.type) params.set('property_type', searchParams.type);
  if (searchParams.city) params.set('location__city', searchParams.city);
  if (searchParams.min_price) params.set('price__gte', searchParams.min_price);
  if (searchParams.max_price) params.set('price__lte', searchParams.max_price);
  if (searchParams.page) params.set('page', searchParams.page);

  const queryString = params.toString();
  const endpoint = `/api/properties/cached/${queryString ? `?${queryString}` : ''}`;

  const data: PaginatedResponse<Property> = await fetchAPI(endpoint);

  return (
    <div className="min-h-screen bg-gray-50">
      {/* Header */}
      <header className="bg-white border-b border-gray-200">
        <div className="max-w-7xl mx-auto px-4 py-6">
          <h1 className="text-3xl font-bold text-gray-900">
            Housing Portal
          </h1>
          <p className="text-gray-600 mt-1">
            {data.count.toLocaleString()} properties available
          </p>
        </div>
      </header>

      {/* Main content */}
      <main className="max-w-7xl mx-auto px-4 py-8">
        {/* Filter bar */}
        <FilterBar />

        {/* Results count */}
        {data.count === 0 ? (
          <div className="text-center py-12">
            <p className="text-gray-600 text-lg">No properties match your filters.</p>
            <p className="text-gray-500 text-sm mt-2">Try adjusting your search criteria.</p>
          </div>
        ) : (
          <>
            {/* Grid of property cards */}
            <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
              {data.results.map((property) => (
                <PropertyCard key={property.id} property={property} />
              ))}
            </div>

            {/* Pagination info */}
            <div className="mt-8 text-center text-gray-600">
              Showing {data.results.length} of {data.count} properties
            </div>
          </>
        )}
      </main>
    </div>
  );
}

What changed:

Added searchParams prop. Next.js automatically passes URL query parameters to page components as searchParams.
location__city uses Django's field lookup syntax. location__city=Seattle filters on the city field of the related location object.
price__gte means "price greater than or equal to" (gte = greater than or equal). price__lte is "less than or equal to."
Added a "no results" message when data.count === 0.

Important: Django REST Framework needs to be configured to support these filters. We need to add a filter backend.

Step 11: Add Filtering to the Django API

The frontend is sending filter parameters. The backend needs to understand them.

Update housing/views.py:

"""
housing/views.py

Updated to support filtering.
"""

from django.utils.decorators import method_decorator
from django.views.decorators.cache import cache_page
from rest_framework import generics
from rest_framework.filters import SearchFilter
from django_filters.rest_framework import DjangoFilterBackend  # ← New import
from .models import Property
from .serializers import PropertySerializer


class PropertyListView(generics.ListAPIView):
    """
    Naive baseline. No caching. No optimization.
    """
    queryset = Property.objects.all().order_by('-created_at')
    serializer_class = PropertySerializer
    filter_backends = [DjangoFilterBackend, SearchFilter]  # ← Add this
    filterset_fields = ['property_type', 'status', 'location__city']  # ← Add this
    search_fields = ['title', 'description']  # ← Add this


class CachedPropertyListView(PropertyListView):
    """
    Cached version.
    """
    @method_decorator(cache_page(60))
    def dispatch(self, *args, **kwargs):
        return super().dispatch(*args, **kwargs)


class OptimizedPropertyListView(generics.ListAPIView):
    """
    Optimized queryset.
    """
    serializer_class = PropertySerializer
    filter_backends = [DjangoFilterBackend, SearchFilter]  # ← Add this
    filterset_fields = ['property_type', 'status', 'location__city']  # ← Add this
    search_fields = ['title', 'description']  # ← Add this

    def get_queryset(self):
        return Property.objects.select_related(
            'location',
            'agent__office'
        ).all().order_by('-created_at')

Install the django-filter package:

cd backend
source venv/bin/activate
pip install django-filter
pip freeze > requirements.txt

Add it to INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    # ...existing apps...
    'django_filters',  # ← Add this
    'housing',
]

Restart the backend:

docker compose restart backend

Step 12: Test the Filters

Open http://localhost:3000/ in your browser.

Select "Apartment" from the Property Type dropdown. The URL changes to /?type=apartment. The grid updates to show only apartments.
Select "Seattle" from the City dropdown. The URL changes to /?type=apartment&city=Seattle. The grid updates again.
Enter 200000 in Min Price and 500000 in Max Price. The URL updates. The grid shows only apartments in Seattle between $200k and $500k.
Click "Clear all filters". The URL resets to /. The grid shows all properties again.

📊 [Screenshot: Filters in Action]

If filters don't work:

URL changes but results don't update — The backend isn't reading the filter parameters. Check that django-filter is installed and added to INSTALLED_APPS.
"Unknown field: location__city" — The field lookup syntax is wrong. Check the Django model — location must be a ForeignKey, and the Location model must have a city field.
Filters work but cache doesn't — The cache key includes the full URL. Each filter combination creates a different cache key. That's correct behavior.

Part D: Client-Side Caching with SWR

The listing page works. Filters work. But every navigation re-fetches from the server. If a user loads the homepage, clicks a property (we haven't built that yet, but imagine), then hits the browser back button, the homepage re-fetches. That's slow.

We need client-side caching. The browser should remember data it already fetched and show it instantly on repeat visits.

What is SWR?

SWR (stale-while-revalidate) is a React hook library by Vercel (the company behind Next.js). The name describes its strategy:

Stale: Show cached data immediately (even if it might be slightly outdated)
While: While showing the cached data, fetch fresh data in the background
Revalidate: When the fresh data arrives, update the UI if anything changed

The user sees instant results. The app stays fresh. It's the best of both worlds.

Step 13: Install SWR

cd frontend
npm install swr

Wait for the install to finish. You'll see added 1 package in the output.

Step 14: Create the SWR Configuration

SWR needs a fetcher function — a function that knows how to fetch data. We already have one: fetchAPI from lib/api.ts. But SWR expects a slightly different signature.

Create frontend/lib/fetcher.ts:

/**
 * lib/fetcher.ts
 * 
 * Fetcher function for SWR.
 * SWR calls this function with a URL and expects a promise that resolves to data.
 */

import { API_BASE_URL } from './api';

export async function fetcher(endpoint: string) {
  const url = `${API_BASE_URL}${endpoint}`;
  const response = await fetch(url);

  if (!response.ok) {
    const error = new Error('An error occurred while fetching the data.');
    // Attach extra info to the error object
    throw error;
  }

  return response.json();
}

What this does:

Takes an endpoint (e.g., /api/properties/cached/) and returns the JSON response.
Throws an error if the request fails. SWR catches this and passes it to your component as the error value.

Step 15: Create the SWR Provider

SWR has global configuration options. We set them in a provider component that wraps the entire app.

Create frontend/app/providers.tsx:

/**
 * app/providers.tsx
 * 
 * Client component that wraps the app with SWR configuration.
 */

'use client';

import { SWRConfig } from 'swr';
import { fetcher } from '@/lib/fetcher';
import { ReactNode } from 'react';

interface ProvidersProps {
  children: ReactNode;
}

export function Providers({ children }: ProvidersProps) {
  return (
    <SWRConfig
      value={{
        fetcher,  // Default fetcher for all useSWR calls
        revalidateOnFocus: false,  // Don't refetch when user focuses the tab
        revalidateOnReconnect: true,  // Refetch when internet reconnects
        dedupingInterval: 2000,  // Don't refetch same endpoint within 2 seconds
      }}
    >
      {children}
    </SWRConfig>
  );
}

What this does:

SWRConfig sets default options for all useSWR hooks in the app.
revalidateOnFocus: false — by default, SWR refetches data whenever the user focuses the browser tab. That's aggressive. We turn it off.
dedupingInterval: 2000 — if you call useSWR('/api/properties/') twice within 2 seconds, SWR only makes one network request and shares the result.

Step 16: Wrap the App in the Provider

Update frontend/app/layout.tsx:

/**
 * app/layout.tsx
 * 
 * The root layout that wraps all pages.
 */

import type { Metadata } from 'next';
import { Inter } from 'next/font/google';
import './globals.css';
import { Providers } from './providers';

const inter = Inter({ subsets: ['latin'] });

export const metadata: Metadata = {
  title: 'Housing Portal',
  description: 'Find your perfect home',
};

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body className={inter.className}>
        <Providers>
          {children}
        </Providers>
      </body>
    </html>
  );
}

What changed:

Wrapped {children} in <Providers>. Now every page in the app has access to SWR configuration.

Step 17: Convert the Listing Page to Use SWR

This is the big change. We're converting from a Server Component (fetches on every request) to a Client Component (fetches once, caches in browser).

Replace frontend/app/page.tsx with this:

/**
 * app/page.tsx
 * 
 * Client Component using SWR for data fetching and caching.
 */

'use client';

import useSWR from 'swr';
import { useSearchParams } from 'next/navigation';
import { PaginatedResponse, Property } from '@/types/property';
import { PropertyCard } from '@/components/PropertyCard';
import { FilterBar } from '@/components/FilterBar';

export default function HomePage() {
  const searchParams = useSearchParams();

  // Build the endpoint URL with current filters
  const params = new URLSearchParams();
  const type = searchParams.get('type');
  const city = searchParams.get('city');
  const minPrice = searchParams.get('min_price');
  const maxPrice = searchParams.get('max_price');
  const page = searchParams.get('page');

  if (type) params.set('property_type', type);
  if (city) params.set('location__city', city);
  if (minPrice) params.set('price__gte', minPrice);
  if (maxPrice) params.set('price__lte', maxPrice);
  if (page) params.set('page', page);

  const queryString = params.toString();
  const endpoint = `/api/properties/cached/${queryString ? `?${queryString}` : ''}`;

  // SWR hook - fetches data and handles caching
  const { data, error, isLoading } = useSWR<PaginatedResponse<Property>>(endpoint);

  // Loading state
  if (isLoading) {
    return (
      <div className="min-h-screen bg-gray-50 flex items-center justify-center">
        <div className="text-center">
          <div className="inline-block animate-spin rounded-full h-12 w-12 border-b-2 border-blue-600"></div>
          <p className="mt-4 text-gray-600">Loading properties...</p>
        </div>
      </div>
    );
  }

  // Error state
  if (error) {
    return (
      <div className="min-h-screen bg-gray-50 flex items-center justify-center">
        <div className="text-center">
          <p className="text-red-600 text-lg font-semibold">Failed to load properties</p>
          <p className="text-gray-600 mt-2">Please try again later.</p>
        </div>
      </div>
    );
  }

  // Data loaded successfully
  if (!data) return null;

  return (
    <div className="min-h-screen bg-gray-50">
      {/* Header */}
      <header className="bg-white border-b border-gray-200">
        <div className="max-w-7xl mx-auto px-4 py-6">
          <h1 className="text-3xl font-bold text-gray-900">
            Housing Portal
          </h1>
          <p className="text-gray-600 mt-1">
            {data.count.toLocaleString()} properties available
          </p>
        </div>
      </header>

      {/* Main content */}
      <main className="max-w-7xl mx-auto px-4 py-8">
        {/* Filter bar */}
        <FilterBar />

        {/* Results */}
        {data.count === 0 ? (
          <div className="text-center py-12">
            <p className="text-gray-600 text-lg">No properties match your filters.</p>
            <p className="text-gray-500 text-sm mt-2">Try adjusting your search criteria.</p>
          </div>
        ) : (
          <>
            {/* Grid of property cards */}
            <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
              {data.results.map((property) => (
                <PropertyCard key={property.id} property={property} />
              ))}
            </div>

            {/* Pagination info */}
            <div className="mt-8 text-center text-gray-600">
              Showing {data.results.length} of {data.count} properties
            </div>
          </>
        )}
      </main>
    </div>
  );
}

What changed:

Added 'use client' at the top. This is now a Client Component.
Replaced await fetchAPI() with const { data, error, isLoading } = useSWR(endpoint).
Added explicit loading and error states. Server Components handle this automatically. Client Components need to handle it manually.
The rest of the JSX is identical.

How SWR works:

First visit: isLoading is true. Shows spinner. Fetches data. When data arrives, isLoading becomes false, data has the response. Renders the grid.
Second visit (same URL): SWR checks its cache. Data is there. isLoading is false immediately. Shows grid instantly. No network request.
Revalidation: After showing cached data, SWR refetches in the background (based on revalidateOnFocus, refreshInterval, etc.). If the data changed, it updates.

Step 18: Test the Client-Side Cache

Open http://localhost:3000/ in your browser.

First load: You'll see the loading spinner for a moment, then the property grid appears.
Open DevTools: Press F12. Go to the Network tab.
Navigate away and back: Click the URL bar, type http://localhost:3000/test, press Enter. Then click Back.
Watch the Network tab: When you go back to the homepage, you'll see no new request to /api/properties/cached/. The data came from SWR's cache.
The grid appears instantly. No spinner. No delay. The browser remembered.

📊 [practice : Network Tab - SWR Cache Hit]

Instructions: With DevTools Network tab open, load the homepage, navigate to /test, then click the browser Back button. Check the Network tab showing no new API request when returning to the homepage. This proves SWR is caching.

To force SWR to revalidate:

Wait 60 seconds (the cache_page TTL on the backend). SWR will eventually refetch.
Focus a different browser tab, then focus this one again (if revalidateOnFocus was true).
Call mutate() manually (we'll add this later for the refresh button).

Part E: Pagination

Right now, the page shows 20 properties. There are 5,000 in the database. We need pagination.

Django REST Framework already paginates the response. The next and previous fields in the API response tell us the URLs for the next/previous pages. We just need to read them and create buttons.

Step 19: Create the Pagination Component

Create frontend/components/Pagination.tsx:

/**
 * components/Pagination.tsx
 * 
 * Pagination controls for browsing through paginated results.
 */

'use client';

import { useRouter, useSearchParams } from 'next/navigation';

interface PaginationProps {
  count: number;
  next: string | null;
  previous: string | null;
}

export function Pagination({ count, next, previous }: PaginationProps) {
  const router = useRouter();
  const searchParams = useSearchParams();
  const currentPage = parseInt(searchParams.get('page') || '1');
  const pageSize = 20;  // DRF default
  const totalPages = Math.ceil(count / pageSize);

  const goToPage = (page: number) => {
    const params = new URLSearchParams(searchParams.toString());
    params.set('page', page.toString());
    router.push(`/?${params.toString()}`);
  };

  if (totalPages <= 1) {
    // Don't show pagination if there's only one page
    return null;
  }

  return (
    <div className="mt-8 flex items-center justify-center gap-2">
      {/* Previous button */}
      <button
        onClick={() => goToPage(currentPage - 1)}
        disabled={!previous}
        className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
      >
        Previous
      </button>

      {/* Page numbers */}
      <div className="flex items-center gap-2">
        {/* First page */}
        {currentPage > 2 && (
          <>
            <button
              onClick={() => goToPage(1)}
              className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50"
            >
              1
            </button>
            {currentPage > 3 && <span className="text-gray-500">...</span>}
          </>
        )}

        {/* Previous page number */}
        {currentPage > 1 && (
          <button
            onClick={() => goToPage(currentPage - 1)}
            className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50"
          >
            {currentPage - 1}
          </button>
        )}

        {/* Current page (highlighted) */}
        <button
          disabled
          className="px-4 py-2 border-2 border-blue-600 rounded-md text-blue-600 bg-blue-50 font-semibold"
        >
          {currentPage}
        </button>

        {/* Next page number */}
        {currentPage < totalPages && (
          <button
            onClick={() => goToPage(currentPage + 1)}
            className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50"
          >
            {currentPage + 1}
          </button>
        )}

        {/* Last page */}
        {currentPage < totalPages - 1 && (
          <>
            {currentPage < totalPages - 2 && <span className="text-gray-500">...</span>}
            <button
              onClick={() => goToPage(totalPages)}
              className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50"
            >
              {totalPages}
            </button>
          </>
        )}
      </div>

      {/* Next button */}
      <button
        onClick={() => goToPage(currentPage + 1)}
        disabled={!next}
        className="px-4 py-2 border border-gray-300 rounded-md text-gray-700 bg-white hover:bg-gray-50 disabled:opacity-50 disabled:cursor-not-allowed"
      >
        Next
      </button>
    </div>
  );
}

What this does:

Shows Previous/Next buttons (disabled if you're on the first/last page).
Shows the current page number, one page before, one page after.
Shows page 1 and the last page if you're far from them (with ... in between).
Updates the URL with ?page=2 when you click.

Step 20: Add Pagination to the Listing Page

Update frontend/app/page.tsx to include the Pagination component:

// ... (keep all the existing imports and code)
import { Pagination } from '@/components/Pagination';  // ← Add this import

export default function HomePage() {
  // ... (keep all the existing useSWR and loading/error state code)

  return (
    <div className="min-h-screen bg-gray-50">
      <header>...</header>

      <main className="max-w-7xl mx-auto px-4 py-8">
        <FilterBar />

        {data.count === 0 ? (
          <div>...</div>
        ) : (
          <>
            <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
              {data.results.map((property) => (
                <PropertyCard key={property.id} property={property} />
              ))}
            </div>

            {/* Pagination - ADD THIS */}
            <Pagination count={data.count} next={data.next} previous={data.previous} />

            <div className="mt-8 text-center text-gray-600">
              Showing {data.results.length} of {data.count} properties
            </div>
          </>
        )}
      </main>
    </div>
  );
}

Step 21: Test Pagination

Open http://localhost:3000/ in your browser.

Scroll to the bottom of the page. You should see pagination controls.
Click "Next" or "2". The URL changes to /?page=2. The grid updates with properties 21-40.
Click "3". URL becomes /?page=3. Properties 41-60 appear.

4. Click "Previous". You go back to page 2.

Each page is a different URL. Each URL is a different SWR cache key. Once you visit page 2, going back to page 2 later will be instant — SWR remembers it.

Part F: The Complete Caching Stack

We now have three caching layers working together. Let's map the complete request flow.

The Three Layers

Layer 1: Redis (Backend)

Technology: Redis with Django's cache_page decorator
Scope: Caches the full HTTP response (JSON string)
Invalidation: Signal-based (immediate when data changes)
Speed: 4ms

Layer 2: Next.js SSR (Server)

Technology: React Server Components
Scope: Renders HTML on the server before sending to browser
Invalidation: On every deploy (build time) or revalidation trigger
Speed: 20ms (if Redis is warm) or 40ms (if Redis is cold but DB is optimized)

Layer 3: SWR (Browser)

Technology: SWR client-side cache
Scope: Caches API responses in browser memory
Invalidation: Manual (mutate), on focus, on interval, or on mount
Speed: 0ms (instant from memory)

Request Flow Examples

Scenario 1: Brand new visitor, cold cache everywhere

User visits http://localhost:3000/
Next.js Client Component mounts
useSWR('/api/properties/cached/') fires
SWR checks cache: miss
Fetches from http://localhost:8000/api/properties/cached/
Django checks Redis: miss
Django queries PostgreSQL with select_related (1 query, ~15ms)
Django serializes 20 properties to JSON
Django saves to Redis with 60s TTL
Django returns JSON to browser
SWR caches the response in browser memory
React renders the property grid
Total time: ~50ms (network + query + serialization)

Scenario 2: Second visitor, Redis is warm

User visits http://localhost:3000/
Next.js Client Component mounts
useSWR('/api/properties/cached/') fires
SWR checks cache: miss (different user, different browser)
Fetches from Django
Django checks Redis: hit (previous user warmed it)
Django returns cached JSON (4ms)
SWR caches in this user's browser
React renders
Total time: ~20ms (network + Redis)

Scenario 3: Same user returns (SWR cache is warm)

User visits http://localhost:3000/ (they were here 5 minutes ago)
useSWR('/api/properties/cached/') fires
SWR checks cache: hit
React renders immediately with cached data
Total time: 0ms (instant)
In the background, SWR revalidates (fetches fresh data)
If data changed, React updates the UI
User sees instant results, data stays fresh

Scenario 4: Admin updates a property

Admin changes Property #42's price in Django admin
Django's post_save signal fires
Signal handler calls cache.clear()
Redis cache is now empty
Next SWR request: cache miss in SWR, cache miss in Redis, queries DB
Redis cache repopulates
SWR cache repopulates (on next revalidation)
Fresh data propagates across all three layers

The Caching Matrix

Layer	Hit Speed	Miss Fallback	Data Freshness	Scope
SWR (Browser)	0ms	Fetch from Django (20ms)	Revalidates in background	Per user, per browser
Redis (Backend)	4ms	Query PostgreSQL (15ms)	Immediate via signals	Global, all users
PostgreSQL (Source)	15ms	N/A (source of truth)	Always fresh	Global, all users

📊 [Practice: Your Cache Performance Matrix]

Instructions: Measure each scenario using Browser DevTools Network tab. Record the actual timings from your system. Hard refresh (Ctrl+Shift+R) to bypass SWR cache. Use Incognito mode to simulate new users. You can use following table template to fill the data to understand the achivement. Often, it goes in performance measuring of developer

Scenario Total Time Redis State SWR State DB Queries

Cold everywhere [YOUR MS] miss miss 1

Warm Redis, cold SWR [YOUR MS] hit miss 0

Warm SWR [YOUR MS] N/A hit 0

Scenario	Total Time	Redis State	SWR State	DB Queries
Cold everywhere	[YOUR MS]	miss	miss	1
Warm Redis, cold SWR	[YOUR MS]	hit	miss	0
Warm SWR	[YOUR MS]	N/A	hit	0

Part G: Performance Verification

Let's prove the caching works by measuring it.

Step 22: Measure First Load vs Repeat Load

Open your browser with DevTools (F12 → Network tab).

Test 1: Cold cache (hard refresh)

Make sure you're on http://localhost:3000/
Press Ctrl+Shift+R (Windows/Linux) or Cmd+Shift+R (Mac) to hard refresh
This bypasses all browser caches
In the Network tab, find the request to /api/properties/cached/
Look at the "Time" column. Record this number.

Expected: 20-50ms on the first load (depends on whether Redis cache was warm).

Test 2: Warm cache (soft refresh)

Press F5 to soft refresh (or click the refresh button)
SWR cache might be cleared, but Redis cache is still warm
Find the /api/properties/cached/ request in Network tab
Record the time.

Expected: 10-25ms (slightly faster because Redis is warm).

Test 3: Navigation (SWR cache hit)

Navigate to /test by typing in the URL bar
Click the browser Back button
The homepage loads instantly
Check the Network tab — there should be no request to /api/properties/cached/

Expected: 0ms network time. The data came from SWR's cache.

📊 [Practice: Performance Comparison - DevTools Network Tab]

Instructions: Run all three tests above. You can capture three screenshots of the Network tab showing: (1) Hard refresh with ~50ms, (2) Soft refresh with ~20ms, (3) Back button with no API request.

Step 23: Measure Cache Hit Rate

Add a cache status indicator to the page so you can see when SWR is serving from cache vs fetching fresh data.

Update frontend/app/page.tsx to show the cache state:

'use client';

import useSWR from 'swr';
import { useSearchParams } from 'next/navigation';
import { PaginatedResponse, Property } from '@/types/property';
import { PropertyCard } from '@/components/PropertyCard';
import { FilterBar } from '@/components/FilterBar';
import { Pagination } from '@/components/Pagination';

export default function HomePage() {
  const searchParams = useSearchParams();

  // ... (keep all the existing endpoint building code)

  const { data, error, isLoading, isValidating } = useSWR<PaginatedResponse<Property>>(endpoint);

  // ... (keep all the existing loading/error state code)

  if (!data) return null;

  return (
    <div className="min-h-screen bg-gray-50">
      <header className="bg-white border-b border-gray-200">
        <div className="max-w-7xl mx-auto px-4 py-6">
          <div className="flex items-center justify-between">
            <div>
              <h1 className="text-3xl font-bold text-gray-900">Housing Portal</h1>
              <p className="text-gray-600 mt-1">
                {data.count.toLocaleString()} properties available
              </p>
            </div>

            {/* Cache status indicator - ADD THIS */}
            {isValidating && (
              <div className="flex items-center gap-2 text-sm text-gray-600">
                <div className="animate-spin rounded-full h-4 w-4 border-b-2 border-blue-600"></div>
                <span>Refreshing...</span>
              </div>
            )}
          </div>
        </div>
      </header>

      <main className="max-w-7xl mx-auto px-4 py-8">
        {/* ... (keep existing FilterBar, grid, Pagination) */}
      </main>
    </div>
  );
}

What this does:

isValidating is true when SWR is fetching (either initial load or revalidation in the background).
When SWR shows cached data and revalidates in the background, you'll see "Refreshing..." briefly in the header.
This proves SWR is working: instant data display (from cache) + background refresh (to keep data fresh).

Part H: Troubleshooting

Issue 1: "Cannot find module '@/lib/api'"

Symptom: Build fails or Next.js can't import the file.

Cause: The file is in the wrong location or has a typo in the filename.

Fix:

Verify the file exists: frontend/lib/api.ts (NOT frontend/app/lib/api.ts)
Check the import path: import { fetchAPI } from '@/lib/api' (no .ts extension)
Restart the dev server: Ctrl+C in the terminal running docker compose up, then docker compose up again.

Issue 2: "useSearchParams() should be wrapped in a suspense boundary"

Symptom: Warning in console when using useSearchParams().

Cause: Next.js wants you to handle the loading state while search params are being read (for edge cases where the params aren't available immediately).

Fix: Wrap the component in a Suspense boundary. Update frontend/app/page.tsx:

import { Suspense } from 'react';

function HomePageContent() {
  // ... (move all the existing page code here)
}

export default function HomePage() {
  return (
    <Suspense fallback={<div>Loading...</div>}>
      <HomePageContent />
    </Suspense>
  );
}

This is optional — the warning doesn't break functionality, but it's good practice.

Issue 3: SWR Shows Stale Data Forever

Symptom: You change data in Django admin. The frontend never updates, even after waiting several minutes.

Cause: SWR's cache is persistent (survives page refreshes in some browsers), and revalidation isn't happening.

Fix: Add a manual refresh button. Update the header in page.tsx:

import { mutate } from 'swr';

// ... inside the component, in the header section:

<button
  onClick={() => mutate(endpoint)}
  className="px-4 py-2 bg-blue-600 text-white rounded-md hover:bg-blue-700"
>
  Refresh Data
</button>

mutate(endpoint) tells SWR: "the data for this endpoint is stale, refetch it immediately."

Issue 4: Filters Don't Work

Symptom: You select a filter, the URL changes, but the results don't update.

Cause: The Django backend isn't reading the filter parameters.

Fix:

Verify django-filter is installed: pip list | grep django-filter
Verify it's in INSTALLED_APPS in settings.py
Verify the view has filter_backends and filterset_fields configured
Restart the backend: docker compose restart backend

Test the API directly:

curl "http://localhost:8000/api/properties/cached/?property_type=apartment" | jq '.results[0].property_type'

All results should be "apartment". If not, the backend filtering is broken.

Issue 5: Images Don't Load (Broken Icon)

Symptom: Property cards show broken image icons.

Cause: We're using a blue gradient placeholder. There are no real images yet.

Fix: This is expected. We'll add real images in Part 6. For now, the placeholder is correct.

If you want to test with real images temporarily, replace the image div in PropertyCard.tsx:

<div className="h-48 bg-gray-300 flex items-center justify-center">
  <img
    src="https://via.placeholder.com/400x300"
    alt={property.title}
    className="w-full h-full object-cover"
  />
</div>

🛠 Troubleshooting - 2

1. Next.js 15 Synchronous Dynamic APIs

The Issue: The app crashed with an error stating searchParams is a Promise and must be unwrapped.
The Cause: In Next.js 15, searchParams and params props in Server Components are now asynchronous. Accessing them directly (e.g., searchParams.type) throws a TypeError.
The Solution: Use await to unwrap the props at the start of the component function.

export default async function Page({ searchParams }: { searchParams: Promise<any> }) {
  const filters = await searchParams;
  // Now use filters.type
}

2. Docker Networking: `ECONNREFUSED`

The Issue: The Frontend container could not connect to the Backend API, even though the backend appeared to be running.
The Cause: Containers don't share localhost. On the server side (SSR), the frontend must use the Docker service name (e.g., http://backend:8000) instead of localhost.
The Solution: Create a dynamic base URL utility that checks if the code is running in the browser or on the server.

const API_URL = typeof window === "undefined" ? "http://backend:8000" : "http://localhost:8000";

3. Missing Python Dependencies in Docker

The Issue: Backend logs showed ModuleNotFoundError: No module named 'django_filters'.
The Cause: The Django app required a package that wasn't included in the initial Docker image build or requirements.txt.
The Solution: Add the package to requirements.txt and rebuild the container using docker-compose up --build.

4. Permission Denied (`EACCES`) on `npm install`

The Issue: Running npm install locally failed with permission errors in the node_modules folder.
The Cause: Because Docker was run with root privileges, it created the node_modules folder on the host machine as root, preventing the local user from modifying it.
The Solution: Reclaim ownership of the project files using chown.

sudo chown -R $USER:$USER .

5. Module Not Found inside Docker Volumes

The Issue: swr was installed locally, but the frontend container still claimed it couldn't find the module.
The Cause: An Anonymous Volume in docker-compose.yml (- /app/node_modules) was masking the local folder. The container was looking at an isolated, outdated version of the modules.
The Solution: Use docker-compose exec to install the package directly inside the running container volume, or perform a clean rebuild with docker-compose down -v.

6. Missing Database Relations

The Issue: The frontend finally loaded, but the backend threw an error: relation "housing_property" does not exist.
The Cause: The Postgres database was running, but the Django migrations hadn't been applied to create the table schema.
The Solution: Run the migrations through the Docker container.

docker-compose exec backend python manage.py migrate

What We Built — And What's Next

We started Part 5 with a Django API and a blank Next.js page. We ended with a complete housing portal:

The UI is functional. Users can browse 5,000 properties, filter by type/city/price, and paginate through results. The cards display all the essential information: title, price, location, agent.

The caching is triple-layered. Redis caches at the backend (4ms). SWR caches in the browser (0ms on repeat visits). The system is both fast and fresh.

The performance is measurable. First load: ~50ms. Repeat load: instant. We can prove it in DevTools.

The filters work. URL state drives the filters. The URL is shareable — send someone http://localhost:3000/?type=house&city=Seattle and they see exactly what you see.

But there's still something missing. The blue gradient placeholders. No real images. No property detail pages. No agent contact info. No map view. Those are polish, not core functionality — but they matter. Part 6 tackles images. Part 7 tackles deployment.

Here's what we'll do in Part 6 - The Image Layer:

Image hosting — Integrate Cloudinary or a similar CDN service
Upload flow — Let admins upload images in the Django admin
Thumbnail generation — Automatically create thumbnails on upload
Fill the null fields — Populate cdn_url and thumbnail_url that have been sitting empty since Part 2
Lazy loading — Load images only when they scroll into view
Next.js Image optimization — Automatic WebP conversion, responsive srcsets
Measure the difference — Compare serving images from Django vs a CDN

The image layer is the final performance bottleneck. A high-res property photo can be 2-5MB. Twenty of them on a page is 40-100MB. Over a slow connection, that's 30 seconds of loading. The CDN and Next.js Image component solve this. We'll measure the before and after.

Checkpoint: Push to GitHub

git add .
git commit -m "feat: add Next.js frontend with SWR caching, filters, and pagination"
git checkout -b part-5-frontend
git push origin part-5-frontend

The repo now has five branches:

part-1-setup — infrastructure
part-2-data — database and seed
part-3-problem — API and Redis
part-4-optimization — query optimization and signals
part-5-frontend — Next.js UI with SWR

Diff any two branches to see what changed:

git diff part-4-optimization..part-5-frontend

Next: Part 6 — The Image Layer: CDN Integration and Lazy Loading. Stay tuned.

From Zero to Cached: Building a High-Performance Housing Portal with Django, Next.js, and Redis - Part 4 : Optimization Pillars

Ajit Kumar — Wed, 04 Feb 2026 06:59:57 +0000

Part 4: The Two Pillars — Fixing the Foundation Before It Breaks

In Part 1, we built the infrastructure. In Part 2, we planted performance problems into the database. In Part 3, we achieved 95% speed improvements with Redis — and discovered that speed without correctness is worse than no speed at all.

If you're jumping in here, you need the context. Part 1 gives you the containerized stack. Part 2 gives you the database. Part 3 gives you the cache. If you're continuing from Part 3, you already have an API that responds in 4ms when the cache is warm — and 80ms with 61 queries when the cache is cold.

Today we fix both problems. We turn 61 queries into 1. We make cache misses disappear by invalidating stale data the moment it changes. By the end of this post, your system will be both fast and correct — which is the only combination that matters in production.

The Two Problems We're Solving

Part 3 left us with a system that works beautifully under ideal conditions and breaks under real ones. Let's be precise about what "breaks" means.

Problem 1: The N+1 Query Trap

Every time the cache expires or is empty, the API fires 61 separate database queries to render 20 properties. This happens because our serializers are nested — Property → Agent → Office, Property → Location — and Django fetches each related object separately instead of using a JOIN.

Why this matters: A cache miss isn't rare. It happens every time the cache expires (every 60 seconds in Part 3's config). It happens when a user hits a URL for the first time. It happens when you deploy a new version and the cache resets. Under high traffic, cache misses are constant. If each miss costs 61 queries and 80ms, your database chokes.

The symptom: Response times spike from 4ms to 80ms during a cache miss. Under load, PostgreSQL's connection pool fills up. The 51st concurrent request waits. The 101st request times out.

The fix: SQL JOINs via select_related. One query. No matter how nested the serializers are.

Problem 2: Stale Data

When a property's price changes in the admin, the cached API continues serving the old price for up to 60 seconds. That's the cache TTL we set in Part 3. The cache has no awareness of the underlying data. It only knows how to count.

Why this matters: In real estate, a 60-second delay might be tolerable. In e-commerce during a flash sale, it's catastrophic. In stock trading, it's illegal. But even in real estate, there's a trust problem. A user sees a price, calls the agent, and the agent quotes a different number. The user blames the portal. The agent blames the portal. Both are right. The cache is wrong.

The symptom: You change data in the admin. The API shows the old data. You wait. Eventually it updates. You have no control over when.

The fix: Signal-based invalidation. When the data changes, Django tells Redis: delete this key. Immediately. Not in 60 seconds. Now.

Part A: The Diagnostic Tool — Django Debug Toolbar

Before we fix anything, we need to see the problem in a way that's impossible to ignore. Terminal logs show you SQL queries. Django Debug Toolbar shows you SQL queries and makes you feel the weight of them.

Step 1: Install Django Debug Toolbar

cd backend
source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install django-debug-toolbar
pip freeze > requirements.txt

Step 2: Configure It for Docker

The toolbar only appears when Django thinks you're making requests from a trusted IP. Inside Docker, the request comes from the Docker network, not 127.0.0.1. We need to tell Django that Docker IPs are safe.

Add this to core/settings.py:

# At the top, with other imports
import socket

# In INSTALLED_APPS (add near the top, before your apps)
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    # Third-party
    'rest_framework',
    'corsheaders',
    'debug_toolbar',        # ← Add this

    # Our apps
    'housing',
]

# In MIDDLEWARE (add at the very top of the list)
MIDDLEWARE = [
    'debug_toolbar.middleware.DebugToolbarMiddleware',  # ← Add this FIRST
    'django.middleware.security.SecurityMiddleware',
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

# At the bottom of settings.py, add this block
# This allows the toolbar to show up when running inside Docker
INTERNAL_IPS = [
    '127.0.0.1',
]

# Docker-specific: detect the host machine's IP from inside the container
try:
    hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
    INTERNAL_IPS += [ip[:-1] + '1' for ip in ips]
except:
    pass

Step 3: Add the Toolbar URLs

Update core/urls.py:

from django.contrib import admin
from django.urls import path, include
from django.conf import settings  # ← Add this import

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('housing.urls')),
]

# Only add the debug toolbar URLs if DEBUG is True
if settings.DEBUG:
    urlpatterns += [
        path('__debug__/', include('debug_toolbar.urls')),
    ]

Step 4: Rebuild and Restart

docker compose build backend
docker compose restart backend

Step 5: Verify It Works

Open your browser and navigate to:

http://localhost:8000/api/properties/live/naive/

You should see a sidebar on the right side of the browser with tabs labeled "History", "Versions", "Time", "SQL", "Cache", etc. If you don't see it, the INTERNAL_IPS detection didn't work. Hard-code your Docker gateway IP instead:

# Find your Docker gateway IP
docker network inspect housing-caching-demo_default | grep Gateway

Add that IP to INTERNAL_IPS manually:

INTERNAL_IPS = [
    '127.0.0.1',
    '172.18.0.1',  # ← Your gateway IP here (yours will be different)
]

Restart again. The toolbar should appear.

What the Toolbar Shows You

Click the SQL tab in the toolbar. You'll see a list of every query Django fired to render the page. Scroll through it. Count them. You should see 61 queries for the naive endpoint.

Click any query. The toolbar shows you:

The raw SQL
The execution time
The stack trace (where in your code this query was triggered)
Whether it's a duplicate (same query fired multiple times)

This is the smoking gun. This is what we're about to fix.

📊 [Screenshot : Django Debug Toolbar - Naive Endpoint SQL Tab]

Part B: Pillar 1 — Fixing the N+1 Queries with `select_related`

The N+1 problem isn't a Django bug. It's the default behavior when you serialize nested relationships without telling Django to optimize. We fix it with select_related — a method that tells Django to perform SQL JOINs upfront instead of lazy-loading related objects one at a time.

How `select_related` Works

Here's what Django does without it:

-- Query 1: Get 20 properties
SELECT * FROM housing_property ORDER BY created_at DESC LIMIT 20;

-- Query 2-21: Get each property's location (20 separate queries)
SELECT * FROM housing_location WHERE id = 1;
SELECT * FROM housing_location WHERE id = 2;
...
SELECT * FROM housing_location WHERE id = 20;

-- Query 22-41: Get each property's agent (20 separate queries)
SELECT * FROM housing_agent WHERE id = 1;
SELECT * FROM housing_agent WHERE id = 2;
...

-- Query 42-61: Get each agent's office (20 separate queries)
SELECT * FROM housing_office WHERE id = 1;
SELECT * FROM housing_office WHERE id = 2;
...

Total: 61 queries. Each one is a round-trip to PostgreSQL. Even if each query is fast, the cumulative network latency kills you.

Here's what Django does with select_related:

-- Query 1: Get 20 properties WITH their related data via JOINs
SELECT 
    property.*, 
    location.*, 
    agent.*, 
    office.*
FROM housing_property AS property
LEFT OUTER JOIN housing_location AS location ON property.location_id = location.id
LEFT OUTER JOIN housing_agent AS agent ON property.agent_id = agent.id
LEFT OUTER JOIN housing_office AS office ON agent.office_id = office.id
ORDER BY property.created_at DESC
LIMIT 20;

Total: 1 query. Everything comes back in a single round-trip. The database does the work of combining the tables. Python just deserializes the result.

The Code

We already teased this in Part 3's OptimizedPropertyListView. Now we're making it real.

Update housing/views.py. The OptimizedPropertyListView class should look like this:

class OptimizedPropertyListView(generics.ListAPIView):
    """
    The database-optimized version. Uses select_related to fetch
    Property + Location + Agent + Office in a single query with JOINs.

    This is what "fast without cache" looks like.
    61 queries → 1 query.
    80ms → 15-20ms.
    """
    serializer_class = PropertySerializer

    def get_queryset(self):
        return Property.objects.select_related(
            'location',         # JOIN on property.location_id = location.id
            'agent__office'     # Double underscore: JOIN agent, then JOIN office
        ).all().order_by('-created_at')

The double underscore (agent__office) is the key. It tells Django: "Follow the agent ForeignKey from Property to Agent, then follow the office ForeignKey from Agent to Office." Both JOINs happen in the same query.

Test It

Restart the backend:

docker compose restart backend

Open your browser and navigate to:

http://localhost:8000/api/properties/live/optimized/

Click the SQL tab in the Debug Toolbar. You should see 1 query. It's a long query — the SELECT statement spans multiple lines because it's pulling columns from four tables — but it's one query.

Click the Time tab. Check the total time. It should be 15-20ms. That's without a cache. Just fast SQL.

📊 [Screenshot: Django Debug Toolbar - Optimized Endpoint SQL Tab]

The Before-and-After Table

Metric	Naive (No Optimization)	Optimized (`select_related`)	Improvement
Query count	61	1	98% reduction
Query time (PostgreSQL)	~40ms	~8ms	5x faster
Total response time (no cache)	80ms	15-20ms	4x faster
Database connections used	61	1	98% reduction

The database is now fast. But the cache is still dumb. Let's fix that.

Part C: Pillar 2 — Smart Invalidation with Django Signals

The cache from Part 3 is time-based. It expires after 60 seconds whether the data changed or not. This creates two problems:

Stale data for up to 60 seconds after an update
Unnecessary cache misses every 60 seconds even if the data hasn't changed in days

We're replacing time-based expiry with event-based invalidation. When the data changes, Django sends a signal. We listen for that signal and delete the cache key immediately.

How Django Signals Work

Django's ORM fires signals at specific moments in a model's lifecycle:

pre_save — fired before a model is saved to the database
post_save — fired after a model is saved
pre_delete — fired before a model is deleted
post_delete — fired after a model is deleted

We register a function to listen for these signals. When the signal fires, our function runs. That function can do anything — log the event, send an email, update a search index, or in our case, delete a Redis cache key.

The Implementation

Create a new file: housing/signals.py

"""
housing/signals.py

Signal handlers for cache invalidation.

Whenever a Property is saved or deleted, we clear the cached API responses
immediately instead of waiting for the TTL to expire.
"""

from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver
from django.core.cache import cache
from .models import Property


@receiver([post_save, post_delete], sender=Property)
def invalidate_property_cache(sender, instance, **kwargs):
    """
    Automatic cache invalidation.

    Fires whenever a Property is created, updated, or deleted.
    Clears the entire cache to ensure fresh data on the next request.

    In a production system, you'd invalidate specific cache keys
    (e.g., only the listing page, only pages that include this property).
    For this demo, we use cache.clear() for simplicity and guaranteed freshness.
    """
    cache.clear()
    print(f"[SIGNAL] Property {instance.id} changed. Redis cache purged.")

The @receiver decorator registers this function as a signal handler. The [post_save, post_delete] list means "fire this function after a save OR after a delete". The sender=Property parameter means "only fire this for the Property model, not all models".

Register the Signals

Signals don't auto-register. Django needs to import signals.py at startup. The correct place to do this is in the app's AppConfig class.

Update housing/apps.py:

from django.apps import AppConfig


class HousingConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'housing'

    def ready(self):
        """
        Called when Django starts up.
        This is where we import signals to register them.
        """
        import housing.signals  # noqa

The # noqa comment tells linters "this import has a side effect (registration), it's not unused."

Restart the Backend

docker compose restart backend

Watch the logs as it restarts:

docker compose logs -f backend

You should see Django start up. No errors. The signal is registered.

Part D: Testing the Smart Invalidation

This is the moment where the system behavior changes in a way you can see with your own eyes.

The Test

Prime the cache. Hit the cached endpoint once:

curl -s http://localhost:8000/api/properties/cached/ | jq '.results[0] | {id, title, price}'

You'll see output like:

{
  "id": 1,
  "title": "Modern Apartment in Seattle",
  "price": "450000.00"
}

Note the price.

Verify the cache is serving it. Open the cached endpoint in your browser:

http://localhost:8000/api/properties/cached/

Click the SQL tab in the Debug Toolbar. You should see 0 queries. The cache is serving it.

Change the data. Open the Django admin:

http://localhost:8000/admin/housing/property/

Find Property #1. Change the price from 450000 to 395000. Save.

Watch the terminal where docker compose logs -f backend is running. You should see:

[SIGNAL] Property 1 changed. Redis cache purged.

Verify the cache was purged. Refresh the browser on the cached endpoint.

Click the SQL tab. You should see 1 query (the optimized query with JOINs). The cache was empty. Django queried the database with the optimized queryset.

Check the JSON response. The price should be 395000.00 — the new price. Immediately. No 60-second wait.

Verify the cache is repopulated. Refresh again.

Click the SQL tab. You should see 0 queries. The fresh data is now cached. The next request will serve from Redis at 4ms.

What Just Happened

The cache is no longer time-based. It's event-based. The data in Redis is always fresh because stale data is deleted the moment the source of truth (PostgreSQL) changes. There's no 60-second window where the API lies to users.

This is the pattern production systems use. E-commerce sites invalidate product caches when prices change. News sites invalidate article caches when headlines are edited. Real estate portals invalidate listing caches when prices or availability change. The TTL still exists as a safety net (in case a signal fails to fire, the cache expires eventually), but invalidation happens immediately, not eventually.

Part E: The Trade-Off — When `cache.clear()` Isn't Enough

Our signal uses cache.clear() — it deletes every key in Redis. This is simple and guarantees freshness, but it's blunt. If you have 100 different cached endpoints (different search filters, different pages, featured listings, agent profiles), changing one property blows away all 100 caches.

In a high-traffic system, that creates a new problem: the cache stampede returns. Everyone hits empty caches at the same moment. The database gets hammered.

The Better Approach: Targeted Invalidation

Instead of cache.clear(), delete specific keys:

@receiver([post_save, post_delete], sender=Property)
def invalidate_property_cache(sender, instance, **kwargs):
    """
    Invalidate only the cache keys related to this property.
    """
    # The cache key pattern that Django's cache_page decorator creates
    # This is a simplified example — the real key includes query params, headers, etc.
    cache_key_pattern = f"views.decorators.cache.cache_page.*.properties.*"

    # In production, you'd use a pattern-matching delete or maintain a registry
    # of cache keys per property. For now, we clear everything.
    cache.clear()
    print(f"[SIGNAL] Property {instance.id} changed. Cache purged.")

The challenge: Django's cache_page decorator auto-generates keys based on the URL, query parameters, request headers, and more. You can't easily predict the exact key unless you control the caching logic yourself (which we'll do in Part 5 with low-level caching).

For this post, cache.clear() is correct. It's simple, it works, and it guarantees correctness. The performance cost (everyone refills their cache simultaneously) is acceptable at this scale. When you hit 10,000 requests per second, you move to more sophisticated patterns like cache warming, versioned keys, or probabilistic invalidation. But you don't need those yet. Solve the problem you have, not the problem you imagine.

Part F: The Complete System — Fast and Correct

Let's map what we built across all four parts and see the final state.

The Request Lifecycle (Cached Endpoint)

Request 1 — Cache Miss:

User requests /api/properties/cached/
Django checks Redis: "Do you have this URL?"
Redis: "No."
Django queries PostgreSQL with select_related (1 query, 15ms)
Django serializes the data to JSON
Django saves the response to Redis with a 60-second TTL
Django returns the JSON to the user
Total time: ~20ms (query + serialization + network)

Request 2-N — Cache Hit:

User requests /api/properties/cached/
Django checks Redis: "Do you have this URL?"
Redis: "Yes, here it is."
Django returns the cached JSON directly
Total time: ~4ms (Redis read + network)

When Data Changes:

Admin updates Property #42 in the admin panel
Django saves the change to PostgreSQL
Django fires the post_save signal
The signal handler calls cache.clear()
Redis deletes all cached responses
Next request is a cache miss (goes through Request 1 flow above)
Data is fresh immediately, not in 60 seconds

The Performance Table — All Four Parts Combined

Metric	Part 2 (Naive DB)	Part 3 (Basic Cache)	Part 4 (Optimized + Smart Invalidation)
Query count (cache miss)	61	61	1
Query count (cache hit)	61	0	0
Response time (cache miss)	80ms	80ms	15-20ms
Response time (cache hit)	80ms	4ms	4ms
Data freshness	Instant	Up to 60s stale	Instant
Cache invalidation	N/A	Time-based (dumb)	Event-based (smart)
Under load (50 concurrent users)	500-1500ms	4-10ms (warm cache)	4-10ms (warm), 20-40ms (cold)

Part G: Tools and Patterns — The Professional Toolkit

Here are the tools and patterns we used in Part 4, documented for reference.

Django Debug Toolbar

What it does: Shows you every SQL query, cache hit/miss, template render, and signal fired during a request. Lives in the browser as a sidebar.

When to use it: During development, whenever you're optimizing a view or debugging unexpected behavior.

Key panels:

SQL — query count, query time, duplicate detection
Cache — hits/misses, key names, sizes
Time — breakdown of where time is spent (DB, template, Python, middleware)
Signals — which signals fired and how long their handlers took

Installation:

pip install django-debug-toolbar

Configuration: See Part A above. The INTERNAL_IPS trick is critical for Docker.

`select_related` vs `prefetch_related`

select_related — for ForeignKey and OneToOne relationships. Uses SQL JOINs. One query.

Property.objects.select_related('location', 'agent__office')

Produces:

SELECT property.*, location.*, agent.*, office.*
FROM housing_property
LEFT JOIN housing_location ON ...
LEFT JOIN housing_agent ON ...
LEFT JOIN housing_office ON ...

prefetch_related — for ManyToMany and reverse ForeignKey relationships. Uses separate queries but fetches related objects in bulk instead of one at a time.

Property.objects.prefetch_related('images')

Produces:

-- Query 1
SELECT * FROM housing_property LIMIT 20;

-- Query 2 (one query for ALL images, not 20 separate ones)
SELECT * FROM housing_propertyimage WHERE property_id IN (1,2,3,...,20);

When to use which:

select_related — when you're traversing ForeignKeys forward (Property → Agent)
prefetch_related — when you're traversing reverse ForeignKeys or ManyToMany (Property → all its images)

Django Signals

Common signals:

pre_save / post_save — before/after a model is saved
pre_delete / post_delete — before/after a model is deleted
m2m_changed — when a ManyToMany relationship changes

Registration pattern:

from django.db.models.signals import post_save
from django.dispatch import receiver

@receiver(post_save, sender=MyModel)
def my_handler(sender, instance, created, **kwargs):
    if created:
        # This is a new object
        pass
    else:
        # This is an update
        pass

Where to put signal handlers: In a signals.py file in your app. Import it in apps.py in the ready() method.

When to use signals: Cache invalidation, search index updates, audit logging, triggering async tasks. Don't use them for business logic that belongs in model methods or views.

Cache Invalidation Strategies

Time-based (Part 3):

@cache_page(60) — cache expires after 60 seconds
Simple, predictable, safe
Guaranteed stale data for up to TTL duration
Thundering herd on expiry under high traffic

Event-based (Part 4):

Invalidate on post_save / post_delete signals
Data is always fresh
More complex, requires testing
Avoids stale data but creates cold cache moments

Hybrid (Production):

Event-based invalidation + long TTL (e.g., 3600 seconds) as a safety net
If invalidation fails for any reason, cache still expires eventually
Best of both worlds

Advanced patterns (Part 5+):

Cache warming — background task refreshes cache before expiry
Versioned keys — property:123:v2 instead of deleting, increment version
Probabilistic early expiry — randomly refresh 5% of requests when cache is 90% expired
Cache locking — only one process refreshes an expired key, others wait or serve stale

Part H: Troubleshooting

Debug Toolbar Doesn't Appear

Symptom: You open the API endpoint in the browser. No sidebar. No toolbar.

Possible causes:

INTERNAL_IPS is wrong. The toolbar only shows if Django thinks the request is coming from a trusted IP. Inside Docker, the request comes from the Docker bridge network, not 127.0.0.1.

Fix: Add your Docker gateway IP to INTERNAL_IPS. Find it with:

docker network inspect housing-caching-demo_default | grep Gateway

Add that IP (e.g., 172.18.0.1) to the INTERNAL_IPS list in settings.py.

Middleware is in the wrong order. DebugToolbarMiddleware must be at the top of the MIDDLEWARE list.

Fix: Check settings.py. The toolbar middleware should be the first item in the list.

DEBUG is False. The toolbar is disabled if DEBUG = False.

Fix: Check settings.py. DEBUG should be True in development.

You're hitting the API via curl or Locust. The toolbar only works in a browser. It injects JavaScript and HTML into the response.

Fix: Open the URL in Chrome/Firefox.

Signal Fires But Cache Doesn't Clear

Symptom: You update a property in the admin. The terminal shows the "[SIGNAL] Property X changed" message. But the cached endpoint still shows old data.

Possible causes:

You're hitting a different URL. Django's cache_page decorator creates different keys for different URLs. /api/properties/cached/ and /api/properties/cached (no trailing slash) are different keys.

Fix: Check the exact URL you're testing. Match it exactly, including trailing slash.

The cache backend isn't Redis. If CACHES in settings.py is misconfigured, Django might be using LocMemCache (local memory) instead of Redis. cache.clear() would clear local memory, not Redis.

Fix: Verify the cache backend:

docker compose exec backend python manage.py shell

from django.core.cache import cache
print(cache.__class__)  # Should be RedisCache or django_redis
cache.set('test', 'value')

Then check Redis:

docker compose exec redis redis-cli keys "*"

You should see a key. If not, the cache isn't going to Redis.

The signal isn't registered. If you didn't add the ready() method to apps.py, the signal handler never gets imported.

Fix: Verify the signal is registered. Add a print statement to signals.py at the module level (outside any function):

print("[DEBUG] signals.py imported")

Restart the backend. If you don't see that print statement in the logs, the file isn't being imported.

`select_related` Doesn't Reduce Query Count

Symptom: You added select_related to the view. The Debug Toolbar still shows 61 queries.

Possible causes:

You're testing the wrong endpoint. select_related is only on the OptimizedPropertyListView, not the PropertyListView.

Fix: Make sure you're hitting /api/properties/live/optimized/, not /api/properties/live/naive/.

The serializer is accessing related objects Django didn't fetch. If your serializer accesses a field you didn't include in select_related, Django makes an extra query.

Fix: Check what fields the serializer uses. Make sure every ForeignKey accessed in the serializer is in the select_related() call.

You forgot to restart. Code changes don't apply until the container restarts.

Fix: docker compose restart backend

What We Built — And What's Next

We entered Part 4 with a system that was fast when the cache was warm and broken when the cache was cold. We leave Part 4 with a system that's fast all the time and correct all the time.

The database is optimized. 61 queries became 1. Cache misses that used to take 80ms now take 15ms. The database can handle 10x more traffic before it chokes.

The cache is smart. It no longer serves stale data. When a property changes, the cache updates immediately. Users see fresh data without sacrificing speed.

The system is observable. Django Debug Toolbar shows us exactly where time is spent. We can see the difference between a cache hit and a cache miss. We can see which queries are slow. We can prove the optimizations work.

But we're still serving data from Django to... nobody. The frontend is a blank Next.js welcome page. Part 5 changes that. We build the actual housing portal UI — the listing cards, the filters, the search. And we introduce a third caching layer: the browser. Client-side caching with Next.js revalidation. That's when the system becomes whole.

Checkpoint: Push to GitHub

git add .
git commit -m "feat: add select_related optimization, signal-based cache invalidation, and Django Debug Toolbar"
git checkout -b part-4-full-cache
git push origin part-4-full-cache

The repo now has four branches:

part-1-setup — infrastructure
part-2-data — database schema and seed
part-3-problem — API layer and basic Redis cache
part-4-full-cache — query optimization and smart invalidation

Diff any two branches to see what changed:

git diff part-3-problem..part-4-full-cache

Next: Part 5 — The Frontend Layer: Next.js, SWR, and Client-Side Caching. Stay tuned.

Forem: Ajit Kumar

From AWS Key Leak to evnx: The Origin Story of a Developer's Safety Net

Part 1: The Accident — An AWS Key Leak That Changed Everything

Part 2: Years of .env Pain Across Projects

Part 3: The Poorly Written Prompt That Started It All

Part 4: From dotenv-cli to evnx — A Name Born From Distribution

Part 5: Where Things Stand Now

Get Started with evnx

🌐 Official Website

📦 Main Repository

🧪 Test Repository

⚙️ GitHub Actions Integration

🍺 Homebrew Tap (macOS / Linux)

🪟 Scoop Bucket (Windows)

Python 3 & Security: A Deep-Dive Tutorial

Table of Contents

1. Python 3.13 — What Changed and Why It Matters

1.1 The New Interactive Interpreter (REPL)

1.2 input() — The Security-Critical Change ⚠️

1.3 Free-Threaded Mode (Experimental in 3.13)

1.4 Improved Error Messages

1.5 Type Parameter Syntax (PEP 695, stabilised in 3.13)

✏️ Quick Check 1

2. Input & Safe Data Entry

2.1 The Full input() Contract

2.2 Robust Input Validation Patterns

2.3 Security-First Input Rules

3. Core Iteration Built-ins — zip, enumerate, map, filter

3.1 zip — Parallel Iteration

3.2 enumerate — Index + Value

3.3 map — Apply a Function to Every Element

3.4 filter — Keep Elements Matching a Predicate

✏️ Practice Problem 3

4. Aggregation Built-ins — sum, any, all, min, max

4.1 sum — Flexible Accumulation

4.2 any and all — Logical Reductions

4.3 min and max — With Custom Keys

5. Sorting — sorted, sort, key functions

5.1 sorted vs .sort()

5.2 The key Parameter — Transformation Before Comparison

5.3 Security Relevance of Sorting

6. Lambda Functions & Functional Style

6.1 Lambda Syntax and Scope

6.2 Lambda with Built-ins

6.3 When NOT to Use Lambda

7. Comprehensions — list, dict, set, generator

7.1 List Comprehensions

7.2 Dict and Set Comprehensions

7.3 Generator Expressions — Lazy Evaluation

8. Exception Handling — try, except, else, finally

8.1 The Full try/except/else/finally Anatomy

8.2 Exception Hierarchy — Know What You're Catching

8.3 Exception Patterns for Secure Code

8.4 The else Clause — Often Forgotten

9. Custom Exceptions & Exception Chaining

9.1 Designing a Custom Exception Hierarchy

9.2 Exception Chaining — raise ... from ...

9.3 Exception Groups (Python 3.11+)

10. Context Managers & Resource Safety

10.1 The with Statement Contract

10.2 contextlib — Building Context Managers

10.3 Security-Critical Resource Patterns

11. String Security — Formatting, Sanitisation, Validation

11.1 f-strings and Format Security

11.2 String Methods for Validation

11.3 Encoding and Unicode Security

12. Injection Attacks — SQL, Command, Path Traversal

12.1 SQL Injection — The Most Common Critical Vulnerability

12.2 Command Injection

12.3 Path Traversal

13. Password & Secret Handling

13.1 Why You Must NEVER Use SHA-256 for Passwords

13.2 bcrypt — Industry Standard

13.3 Python's hashlib.scrypt (Built-in)

13.4 Secrets Module — Cryptographically Secure Randomness

13.5 Environment Variables & Secret Management

14. Cryptography Fundamentals in Python

14.1 HMAC — Message Authentication Code

14.2 Symmetric Encryption — Fernet (AES-128-CBC)

14.3 Hashing for Integrity (Non-Password Use)

Part 2: Years of `.env` Pain Across Projects

Part 4: From `dotenv-cli` to `evnx` — A Name Born From Distribution