Forem: Zafer Dace

When Your AI Wiki Outgrows the Context Window — A Practical Guide to RAG

Zafer Dace — Wed, 08 Apr 2026 13:24:33 +0000

Karpathy showed us how to build LLM-powered knowledge bases. But what happens when your wiki gets too big for the context window? Here's the missing piece.

In a recent post, Andrej Karpathy described a workflow that resonated with thousands of developers: use LLMs to build and maintain personal knowledge bases as markdown wikis. Raw documents go in, the LLM compiles them into structured articles, and you query the wiki like a research assistant.

He also noted something important:

"I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries... at this ~small scale."

The key phrase is "at this small scale." His wiki is ~100 articles and ~400K words. That fits in a large context window. But what happens when you hit 500 articles? 1,000? 2 million words?

The context window runs out. Your LLM can't read everything anymore. This is where RAG comes in — and it's simpler than you think.

What is RAG?

RAG (Retrieval Augmented Generation) is a three-step pattern:

Retrieve — Find the most relevant documents for a given question
Augment — Attach those documents to the prompt
Generate — LLM answers using only the relevant context

Think of it as an open-book exam. The LLM doesn't memorize your entire wiki — it looks up the right pages before answering.

You: "How does attention differ from convolution?"
          ↓
    1. Search vector DB → top 5 relevant articles found
    2. Attach articles to prompt
    3. LLM reads 5 articles (not 500) → generates answer
          ↓
LLM: "Based on your wiki articles on attention mechanisms
      and CNN architectures, the key differences are..."

Without RAG, you'd need to feed all 500 articles into the context window. With RAG, you feed only 5. Same quality, 100x less tokens.

How It Works Under the Hood

RAG relies on vector embeddings — turning text into numbers that capture meaning.

Step 1: Index your wiki

Every article gets converted into a vector (a list of numbers) by an embedding model:

"Attention mechanism" → [0.42, 0.68, 0.35, -0.12, ...]
"CNN architecture"   → [0.39, 0.71, 0.30, -0.15, ...]  ← similar topic, close vectors
"Cooking recipes"    → [0.85, 0.10, 0.92, 0.44, ...]   ← different topic, far apart

These vectors are stored in a vector database — a specialized database that finds similar vectors fast.

Step 2: Query

When you ask a question, the same embedding model converts your question to a vector, then the vector DB finds the closest matches:

"How does self-attention work?"
    → vector → search → top 5 closest articles
    → attention-mechanism.md, transformer-architecture.md, ...

Step 3: Generate

Those articles are injected into the LLM prompt:

System: Answer based on the following context:
[article 1 content]
[article 2 content]
[article 3 content]

User: How does self-attention work?

The LLM now has the right context and generates an accurate, grounded answer.

The Landscape: Existing Tools

Since Karpathy's post, several tools have emerged. Here's a comparison of the most notable ones:

Tool	Stack	Best For
ObsidianRAG	ChromaDB + Ollama + GraphRAG	Full-featured local RAG with wikilink-aware search
obsidian-notes-rag	SQLite-vec + MCP server	Claude Code / AI agent integration
llmwiki	Web UI + Claude	Non-technical users who want a GUI
obsidian-note-taking-assistant	DuckDB + Web app	Combined note-taking + RAG
obsidianRAGsody	CLI + URL clipper	CLI-first workflow with web scraping

Which one should you use?

Want everything local + privacy? → ObsidianRAG (Ollama + ChromaDB)
Using Claude Code as your agent? → obsidian-notes-rag (MCP server)
Just want to try RAG quickly? → obsidianRAGsody (simple CLI)

What Makes a Good RAG Pipeline?

A naive RAG (embed → search → generate) works, but production-quality tools like ObsidianRAG go further:

1. Hybrid Search (Vector + Keyword)
Vector search finds semantically similar content ("How do transformers work?" → finds articles about attention). But it can miss exact terms. BM25 keyword search catches those. The best systems combine both — ObsidianRAG uses a 60/40 vector/keyword split.

2. Reranking
Initial retrieval returns ~20 candidates. A CrossEncoder reranker (like bge-reranker-v2-m3) then scores each candidate against the original query more carefully, keeping only the top 5. This dramatically improves precision.

3. Graph-Aware Expansion
If article A is retrieved and it contains [[article B]] wikilinks, a smart system also pulls in article B. This follows the knowledge graph your LLM already built — exactly how Obsidian's backlinks work.

4. Multilingual Embeddings
If your wiki has mixed-language content, use paraphrase-multilingual-mpnet-base-v2 instead of English-only models. It covers 50+ languages.

Simple RAG:    Query → Vector Search → Top 5 → LLM
Better RAG:    Query → Hybrid Search → Top 20 → Rerank → Top 5 → Expand Links → LLM

Build It Yourself: Minimal RAG in 50 Lines

If you want to understand the core concept, here's a minimal implementation. For production use, consider the tools listed above.

Prerequisites

pip install chromadb sentence-transformers ollama

The Code

import os
import glob
import chromadb
from sentence_transformers import SentenceTransformer

# 1. Setup
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")  # local, no API. Use paraphrase-multilingual-mpnet-base-v2 for multilingual wikis
chroma = chromadb.PersistentClient(path="./wiki_vectors")
collection = chroma.get_or_create_collection("wiki")

# 2. Index your wiki
def index_wiki(wiki_path):
    md_files = glob.glob(os.path.join(wiki_path, "**/*.md"), recursive=True)

    for filepath in md_files:
        with open(filepath) as f:
            content = f.read()

        doc_id = os.path.relpath(filepath, wiki_path)

        # Chunk long articles (simple split by sections)
        chunks = content.split("\n## ")
        for i, chunk in enumerate(chunks):
            chunk_id = f"{doc_id}::chunk_{i}"
            collection.upsert(
                ids=[chunk_id],
                documents=[chunk],
                metadatas=[{"source": doc_id, "chunk": i}]
            )

    print(f"Indexed {len(md_files)} files")

# 3. Search
def search(query, n_results=5):
    results = collection.query(
        query_texts=[query],
        n_results=n_results
    )
    return results["documents"][0], results["metadatas"][0]

# 4. Ask with RAG
def ask(question, wiki_path=None):
    if wiki_path:
        index_wiki(wiki_path)

    docs, metas = search(question)

    context = "\n\n---\n\n".join(
        f"[Source: {m['source']}]\n{doc}"
        for doc, m in zip(docs, metas)
    )

    prompt = f"""Answer the question based on the following context from my wiki.
Cite your sources.

Context:
{context}

Question: {question}"""

    # Use Ollama for local LLM
    import ollama
    response = ollama.chat(model="llama3.2", messages=[
        {"role": "user", "content": prompt}
    ])

    return response["message"]["content"]

# Usage
index_wiki("~/knowledge-base/wiki")
answer = ask("What are the key differences between GPT and BERT?")
print(answer)

That's it. ~50 lines. Fully local. No API keys. No cloud.

When to Use RAG vs. Direct Context

Not everything needs RAG. Here's a simple decision guide:

Wiki Size	Approach	Why
< 50 articles	Direct context	Fits in most context windows
50-200 articles	Index file + direct	Karpathy's approach — LLM reads index, then relevant files
200-1000 articles	RAG	Too big for context, but RAG handles it easily
1000+ articles	RAG + hybrid search	Add keyword search alongside vector search for precision

The sweet spot for adding RAG is when you notice your LLM starting to miss information that's definitely in your wiki, or when token costs become significant.

Tips for Better RAG

1. Chunk wisely

Don't index entire articles as single vectors. Split by sections (## headings). A 5,000-word article as one chunk loses precision — the vector becomes a blur of all topics in that article. Smaller chunks = more precise retrieval.

2. Keep metadata

Store the source file path, section title, and date with each chunk. This lets you filter results ("only search articles from the last month") and cite sources in answers.

3. Use hybrid search

Vector search finds semantically similar content. Keyword search finds exact matches. Combine both:

Vector: "How do transformers handle long sequences?" → finds articles about attention, context windows
Keyword: "RoPE" → finds the exact article mentioning Rotary Position Embeddings

4. Re-index incrementally

Don't rebuild the entire index when you add one article. Use upsert to add/update only the changed files. Most vector DBs support this natively.

5. Let the LLM maintain the wiki, RAG maintains the retrieval

Keep Karpathy's workflow intact — the LLM still writes and organizes the wiki. RAG is just the lookup layer. Don't let RAG complexity infect your clean wiki structure.

What's Next: The Compounding Knowledge Loop

The real power emerges when you combine Karpathy's wiki pattern with RAG in a feedback loop:

Raw Sources → LLM compiles wiki → RAG indexes wiki
                    ↑                      ↓
                    └──── You ask questions ─┘
                          Answers filed back
                          into the wiki

Every question you ask, every answer you file back — they compound. The wiki grows smarter. The RAG index gets richer. Six months in, you have a personal research assistant that knows your domain better than any general-purpose LLM ever could.

And the best part? It all runs on your laptop.

Credit: The LLM knowledge base concept was originally described by Andrej Karpathy. This post explores the RAG extension for scaling beyond context window limits.

If you're new to Karpathy's approach, check out my previous post on building the wiki itself.

Further Reading:

Karpathy's original LLM Wiki gist
ObsidianRAG — Full-featured local Obsidian RAG
obsidian-notes-rag — MCP server for AI agents
ChromaDB docs — Getting started with vector databases

Build Your Own AI-Powered Knowledge Base with LLMs and Obsidian

Zafer Dace — Tue, 07 Apr 2026 16:12:40 +0000

A practical guide to Andrej Karpathy's approach for turning raw research into a living, LLM-maintained wiki.

Last week, Andrej Karpathy shared a fascinating workflow on X: instead of using LLMs primarily for code, he's been using them to build and maintain personal knowledge bases. Raw documents go in, and the LLM compiles them into a structured markdown wiki — complete with summaries, backlinks, concept articles, and cross-references.

The idea is simple but powerful: you rarely touch the wiki yourself. The LLM writes it, maintains it, and answers questions from it.

I loved this concept and decided to build my own version. In this post, I'll walk you through exactly how to set it up using Obsidian as your viewer and Claude Code (or any LLM coding agent) as the engine that manages everything.

The Architecture

The system has four layers:

There's no fancy integration or plugin needed. Obsidian and Claude Code simply share the same directory. Obsidian watches the files and renders them beautifully. Claude Code reads and writes them. That's it.

Step 1: Set Up the Vault

Create a folder structure for your knowledge base:

mkdir -p ~/knowledge-base/{raw,wiki/concepts,wiki/topics,output}
cd ~/knowledge-base

Create a CLAUDE.md file at the root — this tells Claude Code how to behave in this project:

# Knowledge Base Instructions

## Structure
- `raw/` — Source documents (articles, papers, notes). Never modify these.
- `wiki/` — LLM-maintained wiki. All articles are markdown with YAML frontmatter.
- `wiki/concepts/` — Individual concept articles.
- `wiki/topics/` — Broader topic overviews.
- `output/` — Generated outputs (comparisons, slides, charts).
- `_index.md` — Master index of all wiki articles with one-line summaries.

## Article Format
Every wiki article must have:
- YAML frontmatter with: title, tags, sources (list of raw/ files), last_updated
- A brief summary (2-3 sentences) at the top
- Backlinks to related concepts using [[wiki links]]
- Sources section at the bottom linking to raw/ documents

## Rules
- Always update `_index.md` when creating or modifying articles.
- Use [[double bracket]] links for cross-references.
- Never delete or modify files in `raw/`.
- When adding new information, cite the source file from `raw/`.

Now open this folder as an Obsidian vault:

Open Obsidian
"Open folder as vault" → select ~/knowledge-base
Done — Obsidian is now your viewer

Step 2: Collect Raw Data

This is the "data ingest" phase. You have several options:

Obsidian Web Clipper (Recommended)

Install the Obsidian Web Clipper browser extension. Configure it to save clipped articles into your raw/ folder. One click saves any web article as clean markdown.

Manual Copy

For PDFs, papers, or notes — just drop markdown files into raw/:

---
title: "Attention Is All You Need"
source: https://arxiv.org/abs/1706.03762
type: paper
date_added: 2025-04-07
---

# Attention Is All You Need

The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks...

Images

Save related images into raw/images/ and reference them in your markdown. Obsidian renders them inline, and Claude Code can analyze them too.

Step 3: Compile the Wiki

This is where the magic happens. Open Claude Code in your knowledge base directory and ask it to compile:

Read all files in raw/ and compile a wiki:
- Create concept articles in wiki/concepts/ for each key concept
- Create topic overviews in wiki/topics/ for broader themes
- Add backlinks between related articles
- Update _index.md with all articles and one-line summaries

Claude Code will:

Read every document in raw/
Identify key concepts and themes
Create structured markdown articles with frontmatter
Cross-link everything with [[wiki links]]
Build a master index

The result looks something like this in Obsidian's graph view — a connected web of knowledge that you never had to organize manually.

Incremental Updates

When you add new documents to raw/, you don't need to rebuild everything:

I added 3 new articles to raw/. Read them and integrate into the existing wiki.
Update existing articles if there's new info, create new ones if needed,
and update _index.md.

The LLM reads the new sources, figures out what's new vs. what's already covered, and surgically updates the wiki.

Step 4: Ask Questions

Once your wiki reaches a decent size, you can query it like a research assistant:

Based on the wiki, compare the training approaches of GPT-4 and Llama 3.
Write the comparison as output/gpt4-vs-llama3.md with a summary table.

What are the main unsolved problems in RLHF according to our sources?
Write a brief report to output/rlhf-challenges.md

Create a Marp slide deck summarizing the key concepts in wiki/topics/
Save as output/overview-slides.md

The LLM reads the relevant wiki articles, synthesizes an answer, and writes it as a markdown file — which you immediately see in Obsidian.

Pro tip: File the best outputs back into the wiki. Your explorations compound over time.

Step 5: Lint and Maintain

As Karpathy mentioned, you can run "health checks" on your wiki:

Scan the entire wiki for:
- Inconsistent information between articles
- Missing backlinks (concepts mentioned but not linked)
- Articles that reference deleted or missing sources
- Stub articles that need expansion
Report findings in output/health-check.md

Look at the wiki and suggest 5 new article topics that would
fill gaps in our coverage. Explain why each would be valuable.

This is surprisingly useful — the LLM often finds connections and gaps you wouldn't notice yourself.

Tips and Tricks

Use CLAUDE.md Wisely

The CLAUDE.md file is your control plane. As your wiki grows, refine the instructions. Add domain-specific terminology, preferred article structure, or naming conventions.

Keep _index.md Updated

This is the LLM's "table of contents." When the wiki gets large (100+ articles), the LLM reads _index.md first to understand what exists before diving into specific files. Keep it clean and current.

Obsidian Graph View

Enable Obsidian's graph view to visualize connections. The [[wiki links]] that the LLM creates show up as edges in the graph. It's a great way to spot isolated articles or missing connections.

Marp for Presentations

Install the Marp plugin for Obsidian to render slide decks. Ask Claude Code to generate presentations in Marp format — instant slides from your knowledge base.

Scale Considerations

Karpathy reports his wiki works well at ~100 articles and ~400K words without needing RAG. The key is the _index.md with brief summaries — the LLM reads this first, then dives into relevant articles. At much larger scales, you might need a search tool or embeddings-based retrieval.

Why This Works

The insight behind this approach is subtle: LLMs are better at maintaining structured knowledge than we are. They don't forget to add backlinks. They don't leave articles half-finished (unless you tell them to). They can read 50 articles and produce a consistent summary faster than we can read 5.

You bring the judgment — which sources to add, which questions to ask, which outputs to keep. The LLM handles the grunt work of organizing, linking, summarizing, and maintaining.

As Karpathy put it:

"You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts."

Until that product exists, Obsidian + Claude Code gets you 90% of the way there — today, for free, with tools you might already have.

Getting Started

Create a folder, add CLAUDE.md with your wiki rules
Open it as an Obsidian vault
Clip or drop 5-10 articles into raw/
Run claude in the folder and ask it to compile
Explore the result in Obsidian
Start asking questions

The beauty of this system is that it compounds. Every article you add, every question you ask, every health check you run — they all make the knowledge base richer and more connected. After a few weeks, you'll have a personal research assistant that actually knows your domain.

Credit: This approach was originally described by Andrej Karpathy. This post is a practical implementation guide based on his concept.

Choosing the Right Local LLM for Your Mac: A Developer's Real-World Guide to Parameters, Quantization, and Model Architecture

Zafer Dace — Sat, 04 Apr 2026 11:37:52 +0000

I tested four local LLMs on my 36GB Apple Silicon Mac with the same Unity/C# prompt, and the results were not what the model names suggested. The fastest model was roughly 10x faster than the slowest. The "code" model refused to write the code. The best answer came from a distilled model that felt smarter in practice than a larger alternative.

That is why choosing a local model is harder than sorting by parameter count. Architecture, quantization, active parameters, context window, and actual behavior under your prompt matter more than the headline number.

Why Run LLMs Locally?

I do not think local models replace Claude, GPT, or other frontier cloud systems. I use them as supplements, not substitutes. But they are already useful enough that every Mac developer should understand where they fit.

The biggest benefit is cost. If I want to iterate on the same task ten times, local inference turns that into a zero-API-cost workflow. Then there is offline capability, IP protection, and freedom from rate limits or daily quotas.

The tradeoff is also obvious: local models still trail the best cloud systems on reasoning and large-scale architecture work. I use them as part of a stack, not as replacements.

Understanding the Jargon

The local LLM ecosystem is full of terms that make simple tradeoffs sound more mysterious than they are. Here is the practical version.

Parameters (7B, 14B, 31B)

When you see 7B, 14B, or 31B, the B means billion parameters. You can think of parameters as the model's learned internal connections.

My rough mental model:

7B = a capable high school student
14B = a university graduate
31B = a specialist
400B+ = frontier cloud territory

That analogy is crude but useful. More parameters usually mean better outputs. The cost is more RAM and slower inference.

Dense vs MoE (Mixture of Experts)

A dense model means the full network participates in every token. I think of it as a 14-person team where everybody works on every question together.

An MoE model is different. A 30B-A3B model might have 30 billion total parameters, but only 3 billion are active for a given token. That is more like a 30-person office where only three people handle the current task.

The practical implication is simple: total parameters are not the same as active reasoning depth.

Real example from my test:

Qwen3 Coder 30B-A3B (MoE, 3B active): 51.67 tok/s, but basic architecture output
Qwen3.5 27B (dense): 8.53 tok/s, but much stronger modular design and implementation detail

That is why I do not assume 30B beats 14B or 27B. Active parameters matter.

Quantization (Q4, Q6, Q8)

Quantization is compression for model weights. The easiest analogy is image compression.

FP16 = the original full-quality photo
Q8 = high-quality JPEG, much smaller with minimal visible loss
Q4 = medium-quality JPEG, smaller again with more noticeable degradation
Q2 = thumbnail-level compression, technically visible but not something you want to rely on

For a 14B model, the memory picture looks roughly like this:

FP16: about 28GB
Q8: about 14GB
Q4: about 8GB

The exact numbers vary by format and runtime, but the rule is stable. If your RAM allows it, use Q8. If memory is tight, use Q4. I avoid Q2 for serious work.

KV Cache

Every generated token depends on the tokens that came before it. KV cache stores that attention state so the model does not have to recompute everything from scratch.

The catch is memory use. Bigger context means more RAM pressure. Roughly speaking:

8K context can cost around 2GB extra
32K can push toward 8GB

Exact usage depends on the model and backend, but the tradeoff is real. In my setup, TurboQuant+ helped Gemma by compressing KV cache so I could get more practical use out of limited memory.

Context Window

Context window is how much text the model can see at one time.

8K = around 500 lines of code
32K = around 2,000 lines
128K = around 8,000 lines
262K = large multi-file chunks
1M = cloud-model territory

For developers, this matters immediately. An 8K model may be fine for one short file, but it becomes restrictive fast once you include package structure, interfaces, or multiple files.

My Test Setup

I wanted a realistic prompt, not a benchmark toy. So I used a Unity/C# request that checks more than raw syntax:

"Write a Firebase Analytics tool for Unity using VContainer, UniTask, and MessagePipe. Make it modular for reuse across games. Package it as UPM."

My machine was a 36GB Apple Silicon Mac using unified memory. I ran Qwen models through LM Studio with the MLX backend, and Gemma through a llama.cpp TurboQuant+ fork because that runtime gave me better memory behavior for that particular model.

This was not a scientific benchmark shootout. It was a practical developer test: same machine, same task, same expectation of usable output.

The Results

Model 1: Qwen3 Coder 30B-A3B (MoE)

This was the speed monster.

It is a 30B MoE model with only 3B active parameters per token, and it showed. I measured 51.67 tok/s, and it felt genuinely responsive. It generated 1682 tokens in roughly half a minute.

The output was decent: solid explanations and a usable class outline, but not production-ready architecture. It left important initialization details to me and stayed at the "good draft" level.

My conclusion: excellent for quick questions, boilerplate, and fast ideation. Not enough for deep architecture work.

Model 2: Qwen3.5 27B Claude Distilled (Dense)

This was the clear winner on quality.

It is a dense 27B model, reportedly distilled from Claude 4.6 Opus behavior, and the output quality difference was obvious. It ran at 8.53 tok/s, much slower than the MoE model, but the answer was in a different class.

It produced 5138 tokens over about three to four minutes, and most of them were useful. The naming was cleaner. The module boundaries made sense. It handled service registration, dependency injection, and reusable package structure with much more confidence.

This is the model that felt most like a serious coding partner.

My conclusion: if the task involves architecture, modular design, or reusable package-level code, this is the one worth waiting for.

Model 3: Qwen 2.5 Coder 14B (Dense, code-specialized)

This was the biggest disappointment.

On paper, it should have been a strong fit: dense 14B, code-specialized, manageable size. In practice, it refused to do the work. Instead of writing the package, it explained how I could do it. When I pushed further, it said the task was too complex.

That matters more to me than benchmark scores. A coding model that declines to code on a realistic prompt is not a reliable tool for my workflow.

My conclusion: probably fine for completions and short snippets, not dependable for larger scoped generation.

Model 4: Gemma 4 31B (Dense, TurboQuant+)

Gemma 4 31B was interesting because it felt strong in theory and limited in practice.

It is a dense 31B model, but the 8K context window was the major bottleneck. Even with TurboQuant+ helping on memory through KV cache compression, I still felt boxed in by the context limit. It ran at 5.83 tok/s and produced 2454 tokens in about seven minutes.

The output quality was decent. I would place it closer to Qwen3 Coder than to Qwen3.5 distilled. It gave useful guidance, but not the modular, production-grade design I wanted.

My conclusion: capable, but constrained. TurboQuant+ helps it fit and run, but it cannot fix the small context window.

Results Table

Model	Architecture	Context	Speed	Output	Quality Summary	Verdict
Qwen3 Coder 30B-A3B	MoE, `30B` total / `3B` active	`262K`	`51.67 tok/s`	`1682` tokens in ~30s	Good explanations, basic structure, shallow architecture	Best for speed, boilerplate, quick questions
Qwen3.5 27B Claude Distilled	Dense `27B`	`262K`	`8.53 tok/s`	`5138` tokens in 3-4 min	Best modularity, DI patterns, naming, package structure	Best overall code quality
Qwen 2.5 Coder 14B	Dense `14B`	`32K`	N/A	Refused full implementation	Explained approach instead of coding; failed on complexity	Disappointing for complex prompts
Gemma 4 31B	Dense `31B`, TurboQuant+ runtime	`8K`	`5.83 tok/s`	`2454` tokens in ~7 min	Useful guidance, but not detailed enough for the speed	Limited by context, hard to justify

RAM Guide: What to Download for Your Mac

16GB RAM

At 16GB, I would stay modest and optimize for responsiveness.

Qwen 2.5 7B Q8
Llama 3.1 8B Q8

These are good for completions, simple questions, and small utility generation. I would not expect serious architecture work from them.

32GB RAM

Qwen3.5 27B Claude Distilled Q4 for the best quality
Qwen 2.5 Coder 14B Q8 for fast code-oriented tasks
Gemma 4 31B Q4 via TurboQuant+ if you want to experiment with larger dense models

This is where local LLMs start becoming genuinely useful. For me, the distilled 27B is the most compelling choice in this tier.

64GB+ RAM

Qwen 2.5 Coder 32B Q8
Llama 3.1 70B Q4
Multiple models loaded simultaneously

This is the tier where local work becomes much more flexible. You can keep a fast model and a smart model loaded at the same time.

Tools I Actually Found Useful

The tooling matters almost as much as the model choice.

LM Studio: the easiest place to start. Drag-and-drop workflow, clean interface, and MLX optimization make it especially friendly on Apple Silicon.
llama.cpp / TurboQuant+: the better choice if you want more control, server mode, and memory optimization tricks like improved KV cache handling.
Ollama: good for quick CLI testing and simple local serving.
llmfit (github.com/AlexsJones/llmfit): useful for estimating what model and quantization level will actually fit on your hardware before you waste time downloading huge files.

If you are new to local LLMs on Mac, I would start with LM Studio. Once you care about squeezing more performance or memory efficiency out of your machine, llama.cpp-style runtimes are worth the extra complexity.

My Recommendation

For me, the best setup is a multi-model workflow:

Cloud models like Claude or Codex for architecture decisions, complex reasoning, and bigger refactors
Local Qwen3.5 distilled for offline code generation, iterative package drafting, and zero-cost repetition
Local Qwen3 Coder MoE for quick questions, boilerplate, and fast turnaround

If I had to recommend one local model from this test for a 32GB-class Mac developer who wants the best coding output, I would choose Qwen3.5 27B Claude Distilled. If I had to recommend one for speed, I would choose Qwen3 Coder 30B-A3B.

Those are different winners, and that is exactly the point.

Conclusion

Local LLMs in 2026 are genuinely useful for developers, but only if you understand what the labels do and do not mean. Parameters alone are not enough. Architecture, quantization, context window, runtime, and training all matter.

The surprising result from my test was how differently the models failed and succeeded on the same prompt. The fastest model was useful but shallow. The code-specialized model failed the assignment. The biggest model was constrained by context. The best answer came from a distilled dense model that balanced capability and usability.

If your goal is to write better code faster on a Mac, the winning strategy is not "download the largest model." It is to build a local stack that matches your hardware and your actual development loop.

Multi-Model AI Orchestration for Software Development: How I Ship 10x Faster with Claude, Codex, and Gemini

Zafer Dace — Thu, 02 Apr 2026 22:05:40 +0000

I shipped 19 tools across 2 npm packages, got them reviewed, fixed 10 bugs, and published, all in one evening. I did not do it by typing faster. I did it by orchestrating multiple AI models the same way I would coordinate a small development team.

That shift changed how I use AI for software work. Instead of asking one model to do everything, I assign roles: one model plans, another researches, another writes code, another reviews, and another handles large-scale analysis when the codebase is too broad for everyone else.

The Problem

Most developers start with a simple pattern: open one chat, paste some code, and keep asking the same model to help with everything. That works for small tasks. It breaks down on real projects.

The first problem is context pressure. As the conversation grows, the model’s context window fills with stale details, exploratory dead ends, copied logs, and half-finished code. Even when the window is technically large enough, quality often degrades because the model is trying to juggle too many concerns at once.

The second problem is that modern codebases are not tidy, single-language systems. The projects I work on often span TypeScript, Python, C#, shell scripts, README docs, test suites, CI config, and package metadata. The mental model required to review a TypeScript AST transform is not the same as the one required to inspect Unity C# editor code or write reliable Python tests.

The third problem is that software development is not one task. It is a bundle of different tasks:

writing implementation code
researching project conventions
reviewing for defects
running builds and tests
comparing architectures
doing large-scale cross-file analysis
answering quick lookup questions

Using one model for all of that is like asking one engineer to do product design, coding, testing, documentation, DevOps, and code review at the same time.

The Architecture: Each Model Has a Role

I now use a multi-model setup where each model has a clear job.

Model	Role	Why This Model
Claude Opus (Orchestrator)	Decision-making, planning, user communication, coordination	Strongest reasoning, sees the big picture
Claude Sonnet (Subagent)	Codebase research, file reading, build/test, pattern finding	Fast, cheap, parallelizable
Codex MCP	Code writing in sandbox, counter-analysis, code review	Independent context, can debate with Opus
Gemini 2.5 Pro	Large-scale analysis (10+ files), cross-cutting research	1M token context for massive codebases

This is the important constraint: Opus almost never reads more than three files directly, and it never writes code spanning more than two files.

Opus is my scarce resource. I want its context window reserved for decisions, tradeoffs, and coordination. If I let it spend tokens reading ten implementation files, parsing test fixtures, or editing code across half the repo, I am wasting the most valuable reasoning surface in the system.

So I deliberately make Opus act more like a tech lead than a hands-on individual contributor:

It decides what needs to be built.
It asks subagents to gather evidence.
It synthesizes findings into an implementation spec.
It asks Codex to challenge that spec.
It resolves disagreements.
It sends implementation to the right execution agent.

The Core Principle: Preserve the Orchestrator

The best model should not be your file reader, log parser, or bulk code generator.

If I need to answer questions like these:

What conventions does this repo use for new tools?
Which helper utilities are already available?
How do existing tests structure edge cases?
Where does platform-specific formatting happen?

I do not spend Opus on that. I send Sonnet agents to inspect the codebase and return structured findings. If the question spans a huge number of files, I use Gemini for the broad scan and have it summarize patterns, architectural seams, and constraints.

Then Opus makes the decision with clean inputs instead of raw noise.

Real-World Example 1: Building 4 Platform Mappers in One Session

One of the clearest examples was figma-spec-mcp, an open source MCP server that bridges Figma designs to code platforms. The package already had a React mapper, and I wanted to expand it with React Native, Flutter, and SwiftUI support while preserving shared conventions and reusing the normalized UI AST.

Instead, I split the work.

Workflow

A Sonnet subagent researched the codebase: tool conventions, type patterns, existing React mapper design, shared helpers, and how the normalized AST flowed through the system.
Opus synthesized those findings into a detailed implementation spec.
I sent a single Codex prompt: create all three new mappers by reusing the normalized UI AST and following the discovered conventions.
Codex wrote more than 2,000 lines across the new mapper surfaces.
In a separate Codex review session, I asked it to review the output like a skeptical senior engineer, not like the original author.
That review found ten platform-specific bugs.
Three Sonnet subagents fixed those bugs in parallel.
The full toolset passed TypeScript, ESLint, Prettier, and publint.

What the review caught

The review surfaced bugs that were not obvious from a green-looking implementation:

Flutter color output used the wrong byte ordering.
React Native had shadowOffset represented as a string instead of an object.
SwiftUI output relied on a missing color initializer.
A few generated platform props matched one framework’s conventions but not the actual target platform’s API.

Result

I ended that session with four platform mappers, reviewed, fixed, lint-clean, and production-ready in about two hours. The speed came from specialization and parallelism, not from asking one model to “be smarter.”

Real-World Example 2: Contributing to `CoplayDev/unity-mcp`

The second example was a series of open source contributions to CoplayDev/unity-mcp, a Unity MCP server with over 1,000 stars. The most significant was adding an execute_code tool that lets AI agents run arbitrary C# code directly inside the Unity Editor, with in-memory compilation via Roslyn, safety checks, execution history, and replay support.

The interesting part is how the feature gap was identified. I was already using a different Unity MCP server (AnkleBreaker) for my own projects, and I noticed it had capabilities that CoplayDev lacked. Rather than manually comparing 78 tools against 34, I had AI agents do the comparison systematically.

Workflow

I identified the gap myself by working with both MCP servers daily, then used a Sonnet exploration agent to systematically map all tools from AnkleBreaker’s 78-tool set against CoplayDev’s 34 tools. The agent returned a structured comparison table showing exactly which features were missing.
From that gap analysis, I picked execute_code as the highest-impact contribution: it unlocks an entire class of workflows where AI agents can inspect live Unity state, run editor automation, and validate assumptions without requiring manual steps.
A Sonnet agent deep-dived CoplayDev’s dual-codebase conventions (Python MCP server + C# Unity plugin), studying the tool registration pattern, parameter handling, response envelope format, and test structure.
Opus synthesized the research into a detailed implementation spec covering four actions (execute, get_history, replay, clear_history), safety checks for dangerous patterns, Roslyn/CSharpCodeProvider fallback, and execution history management.
Codex wrote the full implementation: ExecuteCode.cs (C# Unity handler with in-memory compilation), execute_code.py (Python MCP tool), and test_execute_code.py (unit tests). Over 1,600 lines of additions.
Opus reviewed the output and caught issues before the PR went out.
The PR was merged after reviewer feedback was addressed.

What the review caught

Safety check patterns needed tightening for edge cases around System.IO and Process usage
Error line number normalization had to account for the wrapper class offset
Compiler selection logic needed a cleaner fallback path

Result

The execute_code tool became one of the more significant contributions to the project, enabling AI agents to do things like inspect scene hierarchies at runtime, validate component references programmatically, and run editor automation scripts. The contribution was grounded in a real gap analysis rather than guesswork, and the multi-model workflow ensured the implementation matched the project’s conventions across two languages.

Real-World Example 3: `roblox-shipcheck` Shooter Audit Expansion

The third example was roblox-shipcheck, an open source Roblox game audit tool. I wanted to add six shooter-genre-specific tools and expand the package around them with tests, documentation, examples, and release notes.

Workflow

Background Sonnet agents worked in parallel on the README rewrite, CHANGELOG, usage examples, and unit tests.
Codex wrote all six shooter tools: weapon config audit, hitbox audit, scope UI audit, mobile HUD audit, team infrastructure audit, and anti-cheat surface audit.
In a separate review session, Codex reviewed the generated implementation and found eight issues.
A Sonnet agent fixed those issues and got 124 tests passing.
Sourcery AI, acting as an automated reviewer, found three additional issues.
Another Sonnet agent addressed the review feedback and tightened the remaining edge cases.

What the review caught

The first review wave found:

ESLint violations
heuristics that were too strict for real-world projects
false positives for free-for-all game modes

The automated reviewer then found:

opportunities to consolidate shared test helpers
missing edge cases in the audit suite
rough spots in the implementation details around reuse and consistency

Result

The package ended with 49 tools total, 124 passing tests, a cleaner README, updated examples, release notes, and green CI across TypeScript, ESLint, Prettier, and SonarCloud. That is the difference between “I added some code” and “I shipped a maintainable release.”

Token Budget Rules: The Key Insight

The most important lesson in all of this is simple: your orchestrator’s context window is the scarcest resource in the system.

These are the rules I follow now:

Opus reads three files or fewer per task. If I need more than that, I delegate the reading to Sonnet or Gemini and ask for a structured summary.
Opus writes code in two files or fewer. If the task spans more than two files, I send it to Codex with a detailed spec.
Before starting any task, I ask: “Can a subagent do this?” If the answer is yes, I stop and delegate.
Codex reviews everything. Even code Codex wrote itself. The review happens in a separate session so it can challenge its own assumptions.
Independent work gets parallel agents. If docs, tests, examples, and changelog updates do not depend on each other, they should run at the same time.

Here is the mental model I use:

Opus = scarce strategic bandwidth
Sonnet = cheap parallel investigation
Codex = isolated implementation and review
Gemini = massive-context research pass

Once I started treating context like a budget instead of an infinite buffer, my sessions became dramatically more reliable.

The Debate Pattern

One of the most effective techniques in this setup is what I call the debate pattern.

Instead of asking one model for a solution and immediately implementing it, I force a disagreement phase.

The process

Opus analyzes the problem and proposes a solution.
Codex receives that analysis and produces counter-analysis: where it agrees, where it disagrees, and what it would change.
If there are conflicts, I do one follow-up round to resolve them.
Once there is consensus, I convert that into an implementation plan.
Codex implements.
A separate Codex session reviews the result.

This works because disagreement exposes hidden assumptions.

In one session, that debate caught:

Flutter Color formatting confusion between 0xRRGGBBAA and 0xAARRGGBB
React Native Paper prop mismatch using mode where variant was correct
a non-existent SwiftUI Color(hex:) initializer

None of those issues were broad architectural failures. They were the kind of platform-specific correctness bugs that burn time after merge if you do not catch them early.

The debate pattern turns AI assistance from “fast autocomplete” into “adversarial design review plus implementation.”

Results

The performance difference is large enough that I now think in terms of orchestration by default.

Metric	Single Model	Multi-Model Orchestration
Tools shipped per session	2-3	10-15
Bugs caught before publish	~60%	~95% (Codex review)
Parallel workstreams	1	6+ simultaneous
Context preservation	Degrades after 3-4 files	Stays sharp (delegated)
Convention compliance	Often drifts	Exact match (research first)

Getting Started

If you want to try this workflow, start simple. You do not need a huge automation stack on day one. You just need role separation and a few clear rules.

My practical setup

Claude Code CLI with Opus as orchestrator for planning, decisions, and user-facing coordination
Codex MCP server (npm: codex) for implementation, sandboxed code changes, and review
Gemini MCP (npm: gemini-mcp-tool) for large-scale repo analysis and broad research across many files
Sonnet subagents via Claude Code’s Agent tool for codebase research, builds, tests, pattern extraction, docs, and support work

The most important operational detail is to write your rules down in CLAUDE.md. If the orchestrator has to rediscover your preferences every session, you lose consistency and waste tokens.

My CLAUDE.md contains rules like:

- Opus reads <= 3 files directly
- Opus writes <= 2 files directly
- Delegate codebase exploration to Sonnet
- Use Codex for implementation spanning multiple files
- Always run a separate review pass before publish
- Prefer parallel subagents for independent tasks

That single file turns ad hoc prompting into a repeatable operating model.

A good first workflow

If you want a low-friction way to start, try this:

Use Sonnet to inspect the repo and summarize conventions.
Use Opus to write a short implementation spec.
Use Codex to implement across the affected files.
Use a fresh Codex session to review for defects.
Use Sonnet to fix issues and run tests.

Practical Lessons

Three habits made the biggest difference for me.

First, I stopped treating AI output as a finished artifact and started treating it as a managed workstream. Every meaningful code change has research, implementation, review, and verification phases. Different models are better at different phases.

Second, I learned that independent context is a feature, not a limitation. When Codex reviews code from a separate session, it does not inherit all the assumptions of the implementation pass. That distance is exactly why it catches bugs.

Third, I stopped optimizing for “best prompt” and started optimizing for “best routing.” The better question is: which model should spend tokens on this specific task?

Conclusion

The future of AI-assisted development is not a single omniscient model sitting in one giant chat. It is orchestration: using the right model for the right task, preserving your strongest model’s context for decisions, and letting specialized agents handle research, implementation, review, and verification.

If you are already using AI in development, my practical advice is simple: stop asking one model to do everything. Give each model a role, protect your orchestrator’s context window, and add a real review pass. That is where the 10x improvement comes from.

Forem: Zafer Dace

When Your AI Wiki Outgrows the Context Window — A Practical Guide to RAG

What is RAG?

How It Works Under the Hood

Step 1: Index your wiki

Step 2: Query

Step 3: Generate

The Landscape: Existing Tools

Which one should you use?

What Makes a Good RAG Pipeline?

Build It Yourself: Minimal RAG in 50 Lines

Prerequisites

The Code

When to Use RAG vs. Direct Context

Tips for Better RAG

1. Chunk wisely

2. Keep metadata

3. Use hybrid search

4. Re-index incrementally

5. Let the LLM maintain the wiki, RAG maintains the retrieval

What's Next: The Compounding Knowledge Loop

Build Your Own AI-Powered Knowledge Base with LLMs and Obsidian

A practical guide to Andrej Karpathy's approach for turning raw research into a living, LLM-maintained wiki.

The Architecture

Step 1: Set Up the Vault

Step 2: Collect Raw Data

Obsidian Web Clipper (Recommended)

Manual Copy

Images

Step 3: Compile the Wiki

Incremental Updates

Step 4: Ask Questions

Step 5: Lint and Maintain

Tips and Tricks

Use CLAUDE.md Wisely

Keep _index.md Updated

Obsidian Graph View

Marp for Presentations

Scale Considerations

Why This Works

Getting Started

Choosing the Right Local LLM for Your Mac: A Developer's Real-World Guide to Parameters, Quantization, and Model Architecture

Why Run LLMs Locally?

Understanding the Jargon

Parameters (7B, 14B, 31B)

Dense vs MoE (Mixture of Experts)

Quantization (Q4, Q6, Q8)

KV Cache

Context Window

My Test Setup

The Results

Model 1: Qwen3 Coder 30B-A3B (MoE)

Model 2: Qwen3.5 27B Claude Distilled (Dense)

Model 3: Qwen 2.5 Coder 14B (Dense, code-specialized)

Model 4: Gemma 4 31B (Dense, TurboQuant+)

Results Table

RAM Guide: What to Download for Your Mac

16GB RAM

32GB RAM

64GB+ RAM

Tools I Actually Found Useful

My Recommendation

Conclusion

Multi-Model AI Orchestration for Software Development: How I Ship 10x Faster with Claude, Codex, and Gemini

The Problem

The Architecture: Each Model Has a Role

The Core Principle: Preserve the Orchestrator

Real-World Example 1: Building 4 Platform Mappers in One Session

Workflow

What the review caught

Result

Real-World Example 2: Contributing to CoplayDev/unity-mcp

Workflow

What the review caught

Result

Real-World Example 3: roblox-shipcheck Shooter Audit Expansion

Workflow

What the review caught

Result

Token Budget Rules: The Key Insight

Real-World Example 2: Contributing to `CoplayDev/unity-mcp`

Real-World Example 3: `roblox-shipcheck` Shooter Audit Expansion