Forem: Nyson Markus

PySpark to Pandas/scikit-learn: A Practical Migration Guide for Data Engineers Learning ML

Nyson Markus — Fri, 10 Apr 2026 11:09:34 +0000

If you've spent years writing PySpark pipelines, the first time you open a Jupyter notebook full of pd.DataFrame and sklearn.fit() calls, it can feel like you've switched languages entirely.

You haven't. The concepts are the same: transformations, aggregations, pipelines, model evaluation. But the execution model, API design, and idioms are different enough to cause real friction when you're trying to learn ML fast.

This guide is not a beginner tutorial. It's a translation layer, a direct mapping from what you already know in PySpark to its equivalent in Pandas and scikit-learn, with side-by-side code, gotchas, and practical advice for anyone making the shift from data engineer to machine learning engineer.

The Single Biggest Mental Model Shift

Before any code, understand this: PySpark uses lazy evaluation. Pandas and scikit-learn do not.

In PySpark, transformations like .filter(), .select(), and .groupBy() build a logical execution plan. Nothing runs until you call an action like .collect() or .show(). This is what enables distributed optimization across a cluster.

In Pandas, every operation executes immediately on the data in memory. When you write df['col'].mean(), it computes right then.

# PySpark: lazy, nothing computed yet
df_filtered = spark_df.filter(spark_df['age'] > 30)

# Pandas: eager, executes immediately
df_filtered = pandas_df[pandas_df['age'] > 30]

This difference has downstream consequences. Debugging is easier in Pandas because errors surface instantly. But Pandas cannot handle datasets that exceed your machine's RAM. PySpark adds cluster overhead that makes it slower than Pandas for datasets under roughly 5 to 10 GB.

Part 1: Core DataFrame Operations Side-by-Side

Most of your day-to-day DE work maps cleanly. Here's the translation table for the operations you use most.

Filtering Rows

# PySpark
df.filter(df['salary'] > 100000)
df.filter("salary > 100000")           # SQL-style string also works

# Pandas
df[df['salary'] > 100000]
df.query("salary > 100000")            # equivalent SQL-style

Selecting Columns

# PySpark
df.select('name', 'salary', 'department')

# Pandas
df[['name', 'salary', 'department']]

GroupBy and Aggregation

# PySpark
df.groupBy('department').agg(
    F.mean('salary').alias('avg_salary'),
    F.count('*').alias('headcount')
)

# Pandas
df.groupby('department').agg(
    avg_salary=('salary', 'mean'),
    headcount=('salary', 'count')
).reset_index()

Note the small but important differences. PySpark uses groupBy (camelCase) while Pandas uses groupby (lowercase). PySpark's .agg() requires explicit column references via F.mean(), while Pandas uses tuple notation (column, function).

Joins

# PySpark
df1.join(df2, on='user_id', how='left')

# Pandas
pd.merge(df1, df2, on='user_id', how='left')

Adding and Transforming Columns

# PySpark
df.withColumn('salary_k', df['salary'] / 1000)

# Pandas
df['salary_k'] = df['salary'] / 1000

# or the non-mutating version
df = df.assign(salary_k=df['salary'] / 1000)

The .withColumn() pattern in PySpark has no single Pandas equivalent. You can use direct assignment or .assign(). Prefer .assign() in chained operations because it returns a new DataFrame without modifying the original.

Handling Nulls

# PySpark
df.dropna(subset=['age', 'salary'])
df.fillna({'age': 0, 'salary': df.agg(F.mean('salary')).collect()})

# Pandas
df.dropna(subset=['age', 'salary'])
df.fillna({'age': 0, 'salary': df['salary'].mean()})

This is one of the cleanest translations. The APIs are nearly identical, except Pandas lets you call .mean() directly on a Series without a .collect() call.

Part 2: What `.toPandas()` Actually Costs

As a data engineer, you've probably used .toPandas() to pull Spark results into a notebook. It's tempting to treat this as a bridge that makes the two worlds equivalent. It is not, or at least not for free.

# This works, but has important implications
pandas_df = spark_df.toPandas()

When you call .toPandas():

All data is collected from every executor back to the driver node
It must fit entirely in driver memory
Arrow-based transfer (enabled by spark.sql.execution.arrow.pyspark.enabled = true) makes this significantly faster but does not change the memory constraint

For ML work on training datasets, your feature table often fits comfortably in memory. Feature stores, for example, typically serve pre-aggregated feature vectors that are far smaller than raw event data. But for very large training sets, this is where tools like Ray, Dask, or Spark's native MLlib become relevant.

Part 3: Feature Engineering: MLlib vs. scikit-learn Transformers

This is where the API divergence is most significant, and where most data engineers hit the steepest learning curve.

The Fundamental Difference: Vectors vs. Native Columns

PySpark MLlib requires all numerical features to be assembled into a single dense or sparse vector column before training. scikit-learn works on native NumPy arrays and Pandas DataFrames directly.

# PySpark MLlib: you MUST assemble features into a vector first
from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(
    inputCols=['age', 'salary', 'tenure'],
    outputCol='features'
)
df_assembled = assembler.transform(df)
# df now has a 'features' column of DenseVector([age, salary, tenure])

# scikit-learn: no assembly needed, just pass the DataFrame directly
from sklearn.linear_model import LogisticRegression

X = df[['age', 'salary', 'tenure']]
y = df['label']
model = LogisticRegression()
model.fit(X, y)

The VectorAssembler requirement is a common source of confusion when coming from PySpark. In scikit-learn, you skip it entirely and work with 2D arrays directly.

Encoding Categorical Variables

# PySpark MLlib: two-step process
from pyspark.ml.feature import StringIndexer, OneHotEncoder

indexer = StringIndexer(inputCol='city', outputCol='city_index')
encoder = OneHotEncoder(inputCols=['city_index'], outputCols=['city_ohe'])

# scikit-learn: single step
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
city_encoded = encoder.fit_transform(df[['city']])

PySpark requires a two-step process (StringIndexer then OneHotEncoder) because it operates in a distributed setting where integer indices must be assigned consistently across partitions. scikit-learn handles this in a single step.

Scaling Features

# PySpark MLlib
from pyspark.ml.feature import StandardScaler

scaler = StandardScaler(inputCol='features', outputCol='scaled_features')
scaler_model = scaler.fit(df_assembled)
df_scaled = scaler_model.transform(df_assembled)

# scikit-learn
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Part 4: Building Pipelines: The Most Transferable Concept

This is the good news for data engineers: both PySpark and scikit-learn use the Pipeline abstraction, and the mental model translates almost directly.

A pipeline chains transformers and a final estimator into a single reusable object that can be fit, serialized, and applied to new data.

# PySpark MLlib Pipeline
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier

pipeline = Pipeline(stages=[
    StringIndexer(inputCol='city', outputCol='city_idx'),
    OneHotEncoder(inputCols=['city_idx'], outputCols=['city_ohe']),
    VectorAssembler(inputCols=['age', 'salary', 'city_ohe'], outputCol='features'),
    StandardScaler(inputCol='features', outputCol='scaled_features'),
    RandomForestClassifier(featuresCol='scaled_features', labelCol='label')
])

model = pipeline.fit(train_df)
predictions = model.transform(test_df)

# scikit-learn Pipeline
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier

preprocessor = ColumnTransformer([
    ('num', StandardScaler(), ['age', 'salary']),
    ('cat', OneHotEncoder(handle_unknown='ignore'), ['city'])
])

pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier())
])

model = pipeline.fit(X_train, y_train)
predictions = model.predict(X_test)

The key structural difference: scikit-learn's ColumnTransformer lets you apply different transformations to different columns in parallel, while PySpark's Pipeline applies stages sequentially. You must explicitly order every step in PySpark. The scikit-learn approach is more concise for heterogeneous feature sets.

Part 5: Model Evaluation: A Familiar Concept, Cleaner API

PySpark's evaluation API requires instantiating a separate evaluator object. scikit-learn bundles everything into sklearn.metrics.

# PySpark
from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator

evaluator = BinaryClassificationEvaluator(labelCol='label', metricName='areaUnderROC')
auc = evaluator.evaluate(predictions)

acc_evaluator = MulticlassClassificationEvaluator(metricName='accuracy')
accuracy = acc_evaluator.evaluate(predictions)

# scikit-learn
from sklearn.metrics import roc_auc_score, accuracy_score, classification_report

auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
accuracy = accuracy_score(y_test, predictions)
print(classification_report(y_test, predictions))  # precision, recall, F1 in one call

scikit-learn's classification_report() is significantly more informative than PySpark's single-metric evaluators. It gives precision, recall, F1, and support for every class in a single call.

Part 6: When PySpark Still Wins

Making the shift to Pandas/scikit-learn for ML does not mean PySpark becomes irrelevant. As an ML engineer, you'll continue using PySpark for:

Feature computation at scale: computing rolling aggregations, user behavior features, or entity embeddings over billions of rows before they land in your feature store
Training data generation pipelines: joining raw event tables, filtering time windows, and assembling labeled datasets
Batch inference at scale: applying trained sklearn or PyTorch models inside a Spark UDF to score millions of records
Large-scale data validation: running data quality checks (Great Expectations, Deequ) on incoming training data

The common production pattern is: PySpark for data preparation, then Pandas/scikit-learn/PyTorch for model training, then PySpark again for batch scoring. Your DE background gives you a direct advantage in both the first and last mile of this pipeline.

Part 7: Practical Learning Path

If you're working through this transition, here's a sequenced approach based on the operations data engineers use most.

Week 1: Core Pandas fluency

Rebuild 3 of your existing PySpark ETL jobs in Pandas
Master: filtering, groupby/agg, merge, apply, pivot_table
Read: Pandas User Guide: Comparison with Spark

Week 2: NumPy foundations

Understand array broadcasting, which is the engine under Pandas
Practice vectorized operations instead of loops
This directly maps to understanding tensor operations in PyTorch later

Week 3: scikit-learn core API

The estimator interface: .fit(), .transform(), .predict()
Build your first Pipeline with at least one ColumnTransformer
Implement cross-validation with cross_val_score()

Week 4: End-to-end project

Pick a Kaggle tabular dataset, preferably structured like a transaction dataset that resembles your DE work
Build a full pipeline: EDA, feature engineering, model training, evaluation, and serialization with joblib
This becomes your first ML portfolio project, exactly the kind of hands-on work that signals readiness for an ML engineering role

Quick Reference: PySpark to Pandas/scikit-learn Cheat Sheet

Operation	PySpark	Pandas / scikit-learn
Filter rows	`df.filter(condition)`	`df[condition]` or `df.query()`
Select columns	`df.select('a', 'b')`	`df[['a', 'b']]`
Add column	`df.withColumn('c', expr)`	`df.assign(c=expr)`
GroupBy agg	`df.groupBy('x').agg(F.mean('y'))`	`df.groupby('x').agg(avg_y=('y','mean'))`
Join	`df1.join(df2, on='id', how='left')`	`pd.merge(df1, df2, on='id', how='left')`
Drop nulls	`df.dropna(subset=['col'])`	`df.dropna(subset=['col'])`
Fill nulls	`df.fillna({'col': val})`	`df.fillna({'col': val})`
Encode categoricals	`StringIndexer` + `OneHotEncoder`	`OneHotEncoder` (single step)
Scale features	`StandardScaler(inputCol='features')`	`StandardScaler()` on array directly
Build pipeline	`Pipeline(stages=[...])`	`Pipeline([('step', transformer)])`
Evaluate model	`BinaryClassificationEvaluator`	`roc_auc_score()`, `classification_report()`
Lazy evaluation	Yes, actions trigger execution	No, eager by default

Key Takeaways

The shift from PySpark to Pandas/scikit-learn is a surface-level API change over a familiar conceptual foundation. Every concept you already know, including transformations, aggregations, joins, pipelines, and train/test splits, exists in both worlds. What changes is:

Execution model: eager vs. lazy; in-memory vs. distributed
Feature representation: native columns vs. assembled vectors
Pipeline structure: sequential stages vs. parallel ColumnTransformer
Evaluation API: single-metric evaluators vs. a comprehensive metrics module

Your data engineering instincts around schema validation, null handling, data quality, and pipeline reproducibility are directly applicable to ML work. The transition is about building on top of what you have, not starting over.

8 Machine Learning Projects for Software Engineers to Build in 2026

Nyson Markus — Mon, 06 Apr 2026 01:32:53 +0000

Most ML project lists are built for data science students. This one is built for software engineers who already know how to ship production code and want to demonstrate ML competence to hiring teams, not just familiarity with Scikit-learn.

Every project here is chosen for one reason: it forces you to solve problems that show up in real ML engineering roles, not just in Kaggle notebooks. The stack choices are opinionated and current. The "what it actually demonstrates" notes are written from the perspective of what a hiring manager at a product company looks for, not what makes a clean tutorial.

Projects are ordered from foundational to advanced. Each builds on patterns from the one before it.

1. Text Classification Pipeline With Drift Monitoring

What you build: A sentiment or topic classifier trained on a public dataset (Amazon reviews, AG News), wrapped in a FastAPI endpoint, with a basic drift detection layer that flags when incoming text starts diverging from the training distribution.

Stack: Python, Scikit-learn or HuggingFace, FastAPI, Evidently AI, Docker

The production element most people skip: The drift monitor. Most engineers build the classifier and stop. Adding Evidently to track feature drift over time and log alerts when distribution shifts exceed a threshold is what turns this from a tutorial into an ML system.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
import pandas as pd

def check_drift(reference_data: pd.DataFrame, current_data: pd.DataFrame) -> dict:
    report = Report(metrics=[DataDriftPreset()])
    report.run(reference_data=reference_data, current_data=current_data)
    result = report.as_dict()
    drift_detected = result["metrics"][0]["result"]["dataset_drift"]
    return {"drift_detected": drift_detected, "report": result}

What it demonstrates: Model serving, containerization, and the monitoring mindset that separates MLEs from notebook practitioners.

2. Feature Store From Scratch

What you build: A lightweight feature store that computes, stores, and serves features for a tabular ML problem (churn prediction, loan default). Features are computed offline, stored in a database, and retrieved at inference time via a point-in-time correct query that prevents future leakage.

Stack: Python, PostgreSQL or Redis, Feast (or hand-rolled), FastAPI

The production element most people skip: Point-in-time correctness. Most engineers join features naively on entity ID, which leaks future data into training. A real feature store retrieves the feature value that existed at the time of the label, not the latest value.

def get_features_at_timestamp(
    entity_id: str,
    timestamp: datetime,
    feature_names: list[str],
    conn
) -> dict:
    query = """
        SELECT feature_name, feature_value
        FROM feature_store
        WHERE entity_id = %s
          AND feature_name = ANY(%s)
          AND computed_at <= %s
        ORDER BY computed_at DESC
    """
    rows = conn.execute(query, (entity_id, feature_names, timestamp)).fetchall()
    seen = {}
    for name, value in rows:
        if name not in seen:
            seen[name] = value
    return seen

What it demonstrates: Understanding of training-serving skew, data leakage, and production feature pipelines — one of the most commonly tested concepts in MLE system design interviews.

3. Fine-Tuned LLM With Evaluation Harness

What you build: A domain-specific fine-tuned model using LoRA/QLoRA on a task like legal clause classification, medical note summarization, or code review comment generation. The evaluation harness runs the model against a golden test set on every training run and logs results to an experiment tracker.

Stack: Python, HuggingFace PEFT, QLoRA, Weights & Biases, PyTorch

The production element most people skip: The evaluation harness. Most engineers fine-tune, check loss curves, and call it done. Building a golden set of 50-100 human-labeled examples and writing automated evaluation that runs on every checkpoint is what makes this a system.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

def apply_lora(model_name: str, r: int = 8, lora_alpha: int = 16) -> object:
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        load_in_4bit=True,
        device_map="auto"
    )
    config = LoraConfig(
        r=r,
        lora_alpha=lora_alpha,
        target_modules=["q_proj", "v_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM"
    )
    return get_peft_model(model, config)

What it demonstrates: Modern LLM adaptation techniques, experiment tracking discipline, and evaluation methodology — all directly relevant to applied ML roles in 2026.

4. Real-Time Fraud Detection System

What you build: A streaming fraud detection pipeline that consumes transaction events from Kafka, computes real-time features (time since last transaction, rolling spend deviation), runs a trained classifier, and logs decisions with confidence scores for auditing.

Stack: Python, Apache Kafka, Redis (for real-time feature retrieval), XGBoost or LightGBM, FastAPI

The production element most people skip: Handling class imbalance correctly in both training and threshold selection. Fraud datasets are typically 0.1-1% positive. Training without addressing this produces a model that predicts "not fraud" for everything and achieves 99% accuracy. The threshold for flagging fraud should be tuned on business cost, not F1.

from sklearn.utils.class_weight import compute_sample_weight
import lightgbm as lgb

def train_fraud_model(X_train, y_train):
    weights = compute_sample_weight(class_weight='balanced', y=y_train)
    model = lgb.LGBMClassifier(
        n_estimators=500,
        learning_rate=0.05,
        num_leaves=31,
        class_weight='balanced'
    )
    model.fit(X_train, y_train, sample_weight=weights)
    return model

def select_threshold_by_cost(
    y_true, y_proba,
    cost_fn: float = 10,
    cost_fp: float = 1
) -> float:
    best_threshold, best_cost = 0.5, float('inf')
    for t in [i / 100 for i in range(1, 100)]:
        preds = (y_proba >= t).astype(int)
        fn = ((preds == 0) & (y_true == 1)).sum()
        fp = ((preds == 1) & (y_true == 0)).sum()
        total_cost = fn * cost_fn + fp * cost_fp
        if total_cost < best_cost:
            best_cost, best_threshold = total_cost, t
    return best_threshold

What it demonstrates: Streaming data pipelines, imbalanced classification, business-aware threshold tuning, and real-time serving — a complete production ML system.

5. RAG System With Retrieval Evaluation

What you build: A retrieval-augmented generation system over a document corpus (company docs, research papers, a Wikipedia subset). The system chunks documents, generates embeddings, stores them in a vector database, retrieves context at query time, and passes it to an LLM. Critically, it includes retrieval evaluation that measures whether the right chunks are being retrieved.

Stack: Python, LangChain or LlamaIndex, Pinecone or ChromaDB, OpenAI or open-source LLM, RAGAS

The production element most people skip: Retrieval evaluation. Most engineers build the RAG pipeline and eyeball a few outputs. RAGAS gives you automated metrics for context precision, context recall, and answer faithfulness. Without these, you have no way to know if chunking strategy or embedding model changes actually improved the system.

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy
)
from datasets import Dataset

def evaluate_rag_pipeline(
    questions: list[str],
    answers: list[str],
    contexts: list[list[str]],
    ground_truths: list[str]
) -> dict:
    data = {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths
    }
    dataset = Dataset.from_dict(data)
    result = evaluate(
        dataset,
        metrics=[context_precision, context_recall, faithfulness, answer_relevancy]
    )
    return result

What it demonstrates: The full LLM application stack, embedding and retrieval systems, and evaluation discipline for generative systems — directly aligned with what most AI product teams are hiring for in 2026.

6. ML Pipeline With Full CI/CD

What you build: A complete ML pipeline where every commit triggers automated tests, the model is retrained on new data if tests pass, evaluation metrics are compared against the currently deployed model, and deployment only proceeds if the new model wins on a held-out test set. No manual steps.

Stack: Python, GitHub Actions, DVC (data version control), MLflow, Docker, any cloud (AWS/GCP/Azure)

The production element most people skip: The model promotion gate. Most CI/CD tutorials for ML cover training automation but stop before the comparison step. A real pipeline only deploys if the challenger model outperforms the champion on the evaluation set.

# .github/workflows/ml_pipeline.yml
name: ML Pipeline

on:
  push:
    branches: [main]
    paths:
      - 'data/**'
      - 'src/**'
      - 'params.yaml'

jobs:
  train-and-evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run data validation tests
        run: pytest tests/test_data.py

      - name: Train model
        run: python src/train.py

      - name: Evaluate and compare vs champion
        run: python src/evaluate.py --compare-champion

      - name: Deploy if challenger wins
        if: ${{ steps.evaluate.outputs.challenger_wins == 'true' }}
        run: python src/deploy.py

What it demonstrates: MLOps maturity, reproducible training pipelines, and automated model governance — the skills that hiring managers consistently say separate production-ready candidates from notebook-only practitioners.

7. Computer Vision Inference Service With Batching

What you build: An object detection or image classification model (fine-tuned on a custom dataset using a YOLO or EfficientNet backbone) served behind an API that supports dynamic request batching — grouping individual inference requests together and processing them as a batch to maximize GPU throughput.

Stack: Python, PyTorch, Ultralytics YOLO or timm, FastAPI, NVIDIA Triton Inference Server or custom batching logic

The production element most people skip: Dynamic batching. Most engineers serve one image per request, which leaves GPU utilization at 10-20% under real load. A batching layer collects requests over a short time window and processes them together, dramatically improving throughput without increasing per-request latency at moderate traffic.

import asyncio
from collections import defaultdict

class DynamicBatcher:
    def __init__(self, model, max_batch_size: int = 32, max_wait_ms: float = 10.0):
        self.model = model
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms
        self.queue = asyncio.Queue()

    async def infer(self, image_tensor):
        future = asyncio.get_event_loop().create_future()
        await self.queue.put((image_tensor, future))
        return await future

    async def process_batches(self):
        while True:
            batch, futures = [], []
            deadline = asyncio.get_event_loop().time() + self.max_wait_ms / 1000

            while len(batch) < self.max_batch_size:
                timeout = deadline - asyncio.get_event_loop().time()
                if timeout <= 0:
                    break
                try:
                    tensor, future = await asyncio.wait_for(
                        self.queue.get(), timeout=timeout
                    )
                    batch.append(tensor)
                    futures.append(future)
                except asyncio.TimeoutError:
                    break

            if batch:
                import torch
                results = self.model(torch.stack(batch))
                for future, result in zip(futures, results):
                    future.set_result(result)

What it demonstrates: GPU-aware serving, latency vs throughput tradeoffs, and inference optimization — all of which appear in MLE system design interviews and on-the-job performance reviews.

8. End-to-End Recommendation System

What you build: A two-tower retrieval and ranking system. The retrieval tower generates user and item embeddings and uses approximate nearest neighbor search to retrieve candidates. A separate ranking model scores the candidates using additional features. Both stages are served via API and the full system logs impressions and clicks for future retraining.

Stack: Python, PyTorch, Faiss (ANN search), FastAPI, PostgreSQL (interaction logging), Airflow (retraining schedule)

The production element most people skip: The two-stage architecture itself. Most engineers build a single model that scores all items, which doesn't scale past a few thousand items. The retrieval-then-ranking split is how Netflix, Spotify, YouTube, and every serious recommendation system at scale actually works.

import torch
import torch.nn as nn
import faiss
import numpy as np

class TwoTowerModel(nn.Module):
    def __init__(self, user_dim: int, item_dim: int, embedding_dim: int = 64):
        super().__init__()
        self.user_tower = nn.Sequential(
            nn.Linear(user_dim, 128),
            nn.ReLU(),
            nn.Linear(128, embedding_dim)
        )
        self.item_tower = nn.Sequential(
            nn.Linear(item_dim, 128),
            nn.ReLU(),
            nn.Linear(128, embedding_dim)
        )

    def forward(self, user_features, item_features):
        user_emb = self.user_tower(user_features)
        item_emb = self.item_tower(item_features)
        return torch.sum(user_emb * item_emb, dim=1)

def build_faiss_index(item_embeddings: np.ndarray) -> faiss.Index:
    dim = item_embeddings.shape[1]
    index = faiss.IndexFlatIP(dim)
    faiss.normalize_L2(item_embeddings)
    index.add(item_embeddings)
    return index

def retrieve_candidates(
    user_embedding: np.ndarray,
    index: faiss.Index,
    k: int = 100
) -> np.ndarray:
    faiss.normalize_L2(user_embedding)
    distances, indices = index.search(user_embedding, k)
    return indices[0]

What it demonstrates: The most commonly asked ML system design question in interviews ("design a recommendation system"), implemented end-to-end with the architecture that actually scales — retrieval, ranking, logging, and retraining loop included.

What Separates These Projects From Tutorial Clones

Every project above has one thing in common: it includes the part that tutorials skip. Drift monitoring, point-in-time correct features, retrieval evaluation, model promotion gates, dynamic batching, two-stage retrieval. These are the elements that show up in production ML systems and almost never in beginner resources.

Building these projects also changes how you talk about your work in interviews. The difference between a candidate who says "I built a RAG system" and one who says "I built a RAG system and measured context recall and faithfulness using RAGAS across three chunking strategies" is a gap in how they approached the project.

The other thing worth knowing is that project selection matters as much as project execution. Engineers who are making the transition from software engineer to machine learning engineer often overbuild in one area and neglect others.

A portfolio with three NLP projects and nothing on serving or monitoring reads differently to a hiring team than one that covers the full ML lifecycle. For a detailed breakdown of how to structure and present these projects, this ML engineer portfolio guide covers what hiring managers actually look for beyond the GitHub link.