Forem: Linghua Jin

Your AI Coding Agent is Blind. Here's the Fix.

Linghua Jin — Wed, 18 Mar 2026 04:25:45 +0000

I've been using Claude Code, Cursor, and Codex daily. And I kept hitting the same wall.

The agent would hallucinate functions. Suggest code that almost worked. Miss obvious patterns that were right there in the codebase.

I thought it was a model problem. It wasn't.

It was a context problem.

The Real Reason Your AI Agent Keeps Getting It Wrong

When your coding agent tries to understand your codebase, it does something naive by default: it reads files. Sometimes whole files. Sometimes random chunks.

The problem? Most codebases are too large to fit in a context window. So the agent gets a sliced, incomplete, often misleading view of your code.

Imagine asking a surgeon to operate while only being able to see through a 2-inch hole. That's your AI agent right now.

The agent isn't dumb. It's just blind.

The Fix: AST-Based Semantic Search

Here's what changes everything: instead of feeding your agent raw file contents or naive text chunks, you give it semantically meaningful code units — extracted using the Abstract Syntax Tree (AST).

AST-based chunking understands code structure. It knows where functions start and end. It won't split a class in half. It keeps imports with their context.

The result? Your agent gets exactly the code it needs — no noise, no half-functions, no hallucination-inducing garbage.

This is what cocoindex-code does.

What is cocoindex-code?

It's a lightweight, open-source CLI tool that builds a semantic search index over your codebase using AST-based chunking + local embedding models.

Key facts:

⚡ 70% fewer tokens consumed by your agent
🚀 1-minute setup — zero config, zero API keys required
🌳 AST-based chunking for 28+ languages (Python, TypeScript, Rust, Go, Java, C/C++ and more)
🔄 Incremental indexing — only re-indexes changed files
🔌 Works with Claude Code, Cursor, Codex, OpenCode via Skills or MCP

The default embedding model runs locally (sentence-transformers/all-MiniLM-L6-v2) — completely free, no API key needed.

Wait, Isn't That What LSP Does?

Great question. LSP (Language Server Protocol) is incredible for editors — it gives you go-to-definition, find references, real-time type errors, rename refactoring.

But LSP and cocoindex-code solve different problems:

	LSP	cocoindex-code
Purpose	Real-time editing assistance	Semantic search across entire codebase
Search type	Exact symbol matching	Natural language / fuzzy semantic search
Best for	Jump to definition, rename	"Where is the auth logic?"
Target user	You, in your editor	Your AI coding agent

LSP answers: "Where is getUserById defined?"

cocoindex-code answers: "Find all code related to user authentication and session management."

They're complementary, not competing. Use both.

How to Get Started (Seriously, 1 Minute)

Install:

pipx install cocoindex-code

Add to your Claude Code agent (Skill integration):

npx skills add cocoindex-io/cocoindex-code

That's literally it. The skill teaches your agent to automatically initialize, index, and search your codebase whenever it's helpful. No ccc init, no ccc index required manually.

Or use it directly from CLI:

ccc index                          # build the index
ccc search "authentication logic"  # semantic search
ccc search "database connection"   # finds related code even without exact names

MCP Server support is also available for Cursor, Codex, OpenCode, and any MCP-compatible agent.

Real-World Impact

A 10K+ file codebase. An agent that previously hallucinated constantly. After adding cocoindex-code:

The agent finds relevant code in seconds instead of scanning entire directories
Token usage dropped by 70% — meaning faster responses AND lower costs
Hallucinations on codebase-specific logic went down dramatically

One community user shared: they were scaling up a 10K+ file codebase with Codex and said it "just worked."

Why This Matters for the Future of AI Coding

We're entering an era where AI agents don't just autocomplete — they plan, refactor, and ship features. For that to work reliably, agents need to understand codebases, not just guess at them.

AST-based semantic search is one of the most important missing primitives in the current AI coding stack. cocoindex-code is one of the first open-source tools to make it trivially easy to use.

The CLI is having a comeback. Not because terminals are trendy — but because small, composable, offline-first tools are exactly what AI agents need.

Try It

🔗 GitHub: https://github.com/cocoindex-io/cocoindex-code

Apache-2.0 license. Fully open source. Drop a ⭐ if you find it useful — it genuinely helps the project grow.

Have you tried semantic code search with your AI agent? I'd love to hear what workflows people are building in the comments 👇

I Built a Code AST MCP That Saves 70% Tokens and Went Viral (54K+ Views)

Linghua Jin — Wed, 11 Mar 2026 01:54:54 +0000

Last week, I open-sourced a lightweight Code MCP server that uses AST (Abstract Syntax Tree) parsing to give coding agents semantic understanding of your codebase. It went viral on X with 54K+ views.

Here's the tweet that started it all:

// Detect dark theme var iframe = document.getElementById('tweet-2031366453153157139-479'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2031366453153157139&theme=dark" }

The Problem: Coding Agents Are Burning Tokens

If you've used Claude Code, Codex, Cursor, or any AI coding agent on a real codebase, you've probably noticed: they dump entire files into context just to understand your code structure. That's expensive, slow, and wasteful.

The agent doesn't need your whole file. It needs to know what functions exist, what classes are defined, and how they relate to each other.

The Solution: AST-Based Semantic Code Search

I built cocoindex-code - a super lightweight embedded MCP that:

Parses your code into ASTs using tree-sitter, extracting meaningful chunks (functions, classes, methods)
Creates semantic embeddings of those chunks
Lets your coding agent search by meaning, not just text matching
Only re-indexes changed files - built on a Rust-based incremental indexing engine

The result? 70% token savings and noticeably faster coding agent responses.

1-Minute Setup - No Config Needed

For Claude Code:

pipx install cocoindex-code
claude mcp add cocoindex-code -- cocoindex-code

For Codex:

codex mcp add cocoindex-code -- cocoindex-code

That's it. No database, no API keys, no config files. It just works.

How It Works Under the Hood

Tree-sitter parsing breaks your code into semantic chunks (functions, classes, etc.) across 20+ languages
Local embedding model (SentenceTransformers) creates vector representations - completely free, no API key needed
SQLite + vector search stores everything locally and portably
Incremental indexing via CocoIndex (Rust engine) means only changed files get re-processed

When your agent needs to find code, it calls the search MCP tool with a natural language query and gets back exactly the relevant code chunks with file paths and line numbers.

Why It Went Viral

I think people resonated with a few things:

Real pain point - everyone using coding agents feels the token burn
Zero friction - one pip install and one MCP add command
No vendor lock-in - works with Claude, Codex, Cursor, or any MCP-compatible agent
Open source (Apache 2.0) - you can inspect every line of code
No API keys required - the default embedding model runs locally for free

Supported Languages

Python, JavaScript/TypeScript, Rust, Go, Java, C/C++, C#, Ruby, Kotlin, Swift, SQL, Shell, and more. It uses tree-sitter grammars so adding new languages is straightforward.

What's Next

We're actively working on:

Better embedding models optimized for code (try nomic-ai/CodeRankEmbed with a GPU)
Enterprise features for large codebases and shared indexing across teams
More MCP tools beyond search

The repo is at github.com/cocoindex-io/cocoindex-code - 520+ stars and growing fast.

Built with CocoIndex, our open-source Rust-based data indexing framework.

Would love to hear your experience if you try it out. Drop a comment or open an issue on GitHub!

I Built a Tiny MCP That Understands Your Code and Saves 70% Tokens

Linghua Jin — Sun, 22 Feb 2026 04:11:28 +0000

Every coding agent demo looks magical... until you point it at a real codebase. Then it either:

Chokes on context windows
Hallucinates around stale code
Or becomes so slow you might as well just grep

I hit this wall building AI workflows with large Rust/Python/TS repos, so I built something I actually wanted for my own stack: a super light-weight, AST-based embedded MCP that just works on your codebase. It's called cocoindex-code and it's already saving me ~70% tokens and a lot of waiting time.

If you're using Claude, Codex, Cursor, or any MCP-friendly coding agent, this post is for you.

The Core Idea: AST + Incremental Indexing

Most "code RAG" setups feel like infra projects: spin up a vector DB, write ETL, fight schema drift, tune chunking, maintain workers. Then you pray it all stays in sync.

cocoindex-code takes the opposite approach:

Embedded MCP: It runs locally as an MCP server, no separate DB to run or maintain.
AST-based indexing: It understands code structure via Tree-sitter, so you get meaningful chunks (functions, classes, blocks) instead of random 200-line windows.
Incremental updates: Built on top of the Rust-based CocoIndex engine, it only re-indexes changed files.
Real multi-language support: Python, JS/TS, Rust, Go, Java, C/C++, C#, SQL, Shell, and more.

The goal: you ask an agent a question, it pulls precisely the code it needs, without blowing up your context window.

What You Get Out of the Box

Here's what you get by just adding the MCP:

Semantic code search tool: search(query, limit, offset, refresh_index) as an MCP tool.
Instant token savings: Because only relevant code chunks go into prompts, not entire files or folders.
Speed: Incremental indexing + Rust engine means updates feel near-instant on typical dev repos.
No-key local embeddings by default: Uses sentence-transformers/all-MiniLM-L6-v2 locally via SentenceTransformers.
Optional power-ups: Swap in any LiteLLM-supported embedding model (OpenAI, Gemini, Mistral, Voyage for code, Ollama, etc.).

This means you can go from "plain coding agent" to "coding agent that actually knows your codebase" in about a minute.

1-Minute Setup for Claude, Codex, and OpenCode

First, install uv if you don't have it yet:

curl -LsSf https://astral.sh/uv/install.sh | sh

Claude

claude mcp add cocoindex-code \
  -- uvx --prerelease=explicit --with \
  "cocoindex>=1.0.0a16" \
  cocoindex-code@latest

Codex

codex mcp add cocoindex-code \
  -- uvx --prerelease=explicit --with \
  "cocoindex>=1.0.0a16" \
  cocoindex-code@latest

OpenCode

You can do it interactively:

opencode mcp add
# MCP server name: cocoindex-code
# type: local
# command:
# uvx --prerelease=explicit --with cocoindex>=1.0.0a16 cocoindex-code@latest

That's it. Point your agent at your repo, and you now have semantic search over your codebase as an MCP tool.

How the `search` MCP Tool Works

Once connected, the MCP exposes a search tool:

search(
  query: str,        # natural language or code snippet
  limit: int = 10,   # 1-100
  offset: int = 0,   # pagination
  refresh_index: bool = True  # re-index before querying
)

Each result comes back with:

File path
Language
Code content
Start/end line numbers
Similarity score

I've found three killer use cases:

"Where is the actual implementation of X?" - when the repo has 5 similarly named functions.
"Show me all the auth-related logic touching JWT refresh."
"Find the code that matches this stack trace snippet."

Because the index is kept up to date incrementally, you can refactor, run tests, and immediately use the agent against the new code layout without re-running some giant offline job.

Supported Languages and Smart Defaults

cocoindex-code ships with a very practical language matrix:

C, C++, C#, CSS/SCSS, Go, HTML, Java, JavaScript/TypeScript/TSX, JSON/YAML/TOML, Kotlin, Markdown/MDX, Pascal, PHP, Python, R, Ruby, Rust, Scala, Solidity, SQL, Swift, XML

It also auto-excludes noisy directories like __pycache__, node_modules, target, dist, and vendored dependencies.

Root path is auto-discovered via .cocoindex_code/, .git/, or falling back to current working directory. In practice, you usually don't set any env vars at all - it just finds your repo root.

Embeddings: Start Free, Scale Later

Out of the box, the project uses a local SentenceTransformers model:

Default: sbert/sentence-transformers/all-MiniLM-L6-v2
No API key, no billing surprises, completely local.

If you want stronger semantic understanding for code-heavy repos, you can point COCOINDEX_CODE_EMBEDDING_MODEL to any LiteLLM-supported embedding model:

Ollama (local)
OpenAI / Azure OpenAI
Gemini
Mistral
Voyage (code-optimized)
Cohere
AWS Bedrock
Nebius

Basically: start with free local, upgrade only if/when you actually need it.

What About Huge / Enterprise Codebases?

Under the hood, cocoindex-code uses CocoIndex, a Rust-based indexing engine built for large-scale, incremental data workflows.

For big org setups, you can:

Share indexes across teammates instead of re-indexing on every machine.
Take advantage of features like branch dedupe to avoid duplicate work.
Run it as part of a larger data/indexing platform on top of CocoIndex.

If You Want to Try It, Here's the Ask

If this sounds useful, here's a small but meaningful way you can help:

Star the repo: cocoindex-code and the underlying cocoindex.
Try it on your main project (the messy one, not the toy one).
Drop feedback, issues, or ideas in the GitHub repo.

I'm especially interested in:

Repos where existing "code RAG" tools failed you
Languages or frameworks you want better support for
Workflows where you want your coding agent to feel 10x more context-aware

If you do try it, let me know in the comments what stack you used it on - I'd love to feature a few real-world examples in a follow-up post.

I Stopped Manually Documenting My Repos: Now They Generate Their Own Wikis

Linghua Jin — Thu, 05 Feb 2026 06:25:34 +0000

Every few months I'd open an old repo and ask the same question:

"Who wrote this, and why were they so angry?"

Function names that lie, classes with no explanation, a main.py that secretly runs the whole company. The code worked, but the documentation never kept up. It was always "I'll document this later," and later never came.

So I did the only reasonable thing: I stopped trying.

Instead, I wired up a pipeline that reads my projects, calls an LLM, and spits out a one-pager wiki for each codebase—automatically, every time the code changes.

In this post, I'll walk through how to build that: a multi-codebase summarization pipeline using CocoIndex that:

Scans multiple projects in one go.
Uses structured LLM outputs (Pydantic + Instructor) to extract functions, classes, and relationships.
Aggregates everything into a project-level summary.
Generates Markdown docs with Mermaid diagrams so you actually understand the architecture.
Only re-runs the minimum work when files change, thanks to incremental processing and memoization.

If you've ever wished your monorepo came with a live-updating architecture wiki, this is for you.

The Idea: A Self-Updating Wiki For Every Project

Let's say you have a projects/ folder like this:

projects/
  ├── my_project_1/
  │   ├── main.py
  │   └── utils.py
  ├── my_project_2/
  │   └── app.py
  └── ...

What we want:

For each subdirectory, generate a Markdown file like output/my_project_1.md.
That Markdown includes:
- An overview of the project's purpose.
- Key public classes and functions, with human-readable summaries.
- Mermaid diagrams showing how components connect.
- Optional per-file details if the project spans multiple files.

And whenever you:

Add a new project,
Modify a file, or
Change the LLM extraction logic,

you just run:

cocoindex update main.py

CocoIndex figures out what changed and recomputes only what's necessary.

Step 1: Setup – Point the Pipeline at Your Code

First, install CocoIndex and the supporting libraries:

pip install --pre "cocoindex>=1.0.0a6" instructor litellm pydantic

Create a project folder and enter it:

mkdir multi-codebase-summarization
cd multi-codebase-summarization

Set up your LLM configuration:

export GEMINI_API_KEY="your-api-key"
export LLM_MODEL="gemini/gemini-2.5-flash"

Tell CocoIndex where to keep its state:

echo "COCOINDEX_DB=./cocoindex.db" > .env

Then create your projects/ directory and drop in any Python projects you want summarized:

mkdir projects

Step 2: The App – Treat Each Directory as a Project

CocoIndex has the notion of an App: a top-level unit that defines how data flows from sources to outputs.

from __future__ import annotations
import os
import asyncio
import pathlib
from typing import Collection

import instructor
from litellm import acompletion
from pydantic import BaseModel, Field

import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

LLM_MODEL = os.environ.get("LLM_MODEL", "gemini/gemini-2.5-flash")

app = coco.App(
    "MultiCodebaseSummarization",
    app_main,
    root_dir=pathlib.Path("./projects"),
    output_dir=pathlib.Path("./output"),
)

The app_main function scans subdirectories and mounts a processing component per project:

@coco.function
def app_main(root_dir: pathlib.Path, output_dir: pathlib.Path) -> None:
    for entry in root_dir.resolve().iterdir():
        if not entry.is_dir() or entry.name.startswith("."):
            continue
        project_name = entry.name
        files = list(
            localfs.walk_dir(
                entry,
                recursive=True,
                path_matcher=PatternFilePathMatcher(
                    included_patterns=["*.py"],
                    excluded_patterns=[".*", "__pycache__"],
                ),
            )
        )
        if files:
            coco.mount(
                coco.component_subpath("project", project_name),
                process_project,
                project_name,
                files,
                output_dir,
            )

Step 3: Extract Structured File Info with an LLM

Define Pydantic models that describe exactly what we want:

class FunctionInfo(BaseModel):
    name: str = Field(description="Function name")
    signature: str = Field(description="Function signature")
    is_coco_function: bool = Field(description="Whether decorated with @coco.function")
    summary: str = Field(description="Brief summary")

class ClassInfo(BaseModel):
    name: str = Field(description="Class name")
    summary: str = Field(description="Brief summary")

class CodebaseInfo(BaseModel):
    name: str = Field(description="File path or project name")
    summary: str = Field(description="Brief summary")
    public_classes: list[ClassInfo] = Field(default_factory=list)
    public_functions: list[FunctionInfo] = Field(default_factory=list)
    mermaid_graphs: list[str] = Field(default_factory=list)

With Instructor wrapping LiteLLM, we tell the model to fill in this schema:

_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)

@coco.function(memo=True)
async def extract_file_info(file: FileLike) -> CodebaseInfo:
    content = file.read_text()
    prompt = f"Analyze the following Python file...\n{content}"
    result = await _instructor_client.chat.completions.create(
        model=LLM_MODEL,
        response_model=CodebaseInfo,
        messages=[{"role": "user", "content": prompt}],
    )
    return CodebaseInfo.model_validate(result.model_dump())

Key point: memo=True caches results by file content—unchanged files skip the LLM entirely.

Step 4: Aggregate Files into a Project-Level Summary

@coco.function
async def aggregate_project_info(
    project_name: str,
    file_infos: list[CodebaseInfo],
) -> CodebaseInfo:
    if len(file_infos) == 1:
        return file_infos[0]  # Single file, reuse

    # Multi-file: LLM synthesizes
    files_text = "\n".join(f"{i.name}: {i.summary}" for i in file_infos)
    prompt = f"Aggregate these files into one summary:\n{files_text}"
    result = await _instructor_client.chat.completions.create(
        model=LLM_MODEL,
        response_model=CodebaseInfo,
        messages=[{"role": "user", "content": prompt}],
    )
    return result

Step 5: Generate Markdown (With Mermaid Diagrams)

@coco.function
def generate_markdown(project_name: str, info: CodebaseInfo) -> str:
    lines = [f"# {project_name}", "", "## Overview", info.summary, ""]

    if info.public_classes:
        lines.append("**Classes:**")
        for cls in info.public_classes:
            lines.append(f"- `{cls.name}`: {cls.summary}")

    if info.public_functions:
        lines.append("**Functions:**")
        for fn in info.public_functions:
            lines.append(f"- `{fn.signature}`: {fn.summary}")

    if info.mermaid_graphs:
        lines.extend(["## Pipeline", "```

mermaid"])
        lines.extend(info.mermaid_graphs)
        lines.append("

```")

    return "\n".join(lines)

Step 6: Wire It All Together

@coco.function(memo=True)
async def process_project(
    project_name: str,
    files: Collection[localfs.File],
    output_dir: pathlib.Path,
) -> None:
    # Extract info from each file concurrently
    file_infos = await asyncio.gather(
        *[extract_file_info(f) for f in files]
    )

    # Aggregate into project-level summary
    project_info = await aggregate_project_info(project_name, file_infos)

    # Generate and output markdown
    markdown = generate_markdown(project_name, project_info)

    localfs.declare_file(
        output_dir / f"{project_name}.md",
        markdown,
        create_parent_dirs=True,
    )

Run it:

cocoindex update main.py
ls output/
# my_project_1.md  my_project_2.md  ...

Why This Pattern Is Powerful

This pipeline demonstrates a reusable pattern:

Structured LLM outputs: LLMs become typed, predictable components via Pydantic + Instructor
Memoized LLM calls: You stop paying for the same prompt multiple times
Async concurrency: LLM becomes a parallel compute resource
Hierarchical aggregation: File → project, page → document, message → conversation
Incremental processing: "Live" documentation without nightly rebuilds

Anywhere you have "items that need LLM enrichment, plus a rolled-up view," this pattern applies.

Try It Yourself

Full example: Multi-Codebase Summarization
CocoIndex repo: github.com/cocoindex/cocoindex
Step by Step Tutorial

If you end up generating auto-wikis for your own monorepo, drop a link in the comments—I'd love to see what your "self-documenting" codebase looks like.

Found this useful? Give CocoIndex a ⭐ on GitHub!

I Built a Real-Time HackerNews Trend Radar With AI (And It Runs Itself)

Linghua Jin — Fri, 16 Jan 2026 03:11:14 +0000

Every day, HackerNews quietly decides what the dev world will care about next.
But unless you're doom-scrolling it all day, you're missing the real signal: which topics are actually taking off right now, across threads and deep comment chains.

So instead of manually refreshing HN, I built a real-time "trend radar" on top of it:

Continuously ingests fresh HN stories and comments
Uses an LLM to extract structured topics (companies, tools, models, tech terms)
Streams everything into Postgres for instant querying like:
- "What's trending on HN right now?"
- "Which threads are driving the most hype for Claude / LangChain / Rust today?"

All of this runs as a declarative CocoIndex flow with incremental syncs, LLM-powered extraction, and simple query handlers.

In this post, you'll see how it works end-to-end and how you can fork it to track any community (Reddit, X, Discord, internal Slack, etc.).

Why HN Is a Goldmine (If You Can Structure It)

HackerNews is one of the strongest early signals for:

New tools and frameworks devs actually try
Which AI models/products are gaining mindshare
Real sentiment and feedback in the comments
Emerging startups and obscure libraries that might be big in 6-12 months

But raw HN has three problems:

Threads are noisy; comments are nested and messy
There's no notion of "topics" beyond free text
There's no built-in way to ask: "What's trending across the whole firehose?"

The HackerNews Trending Topics example in CocoIndex is essentially: "turn HN into a structured, continuously updating topics index that AI agents and dashboards can query in milliseconds."

Architecture: From HN Firehose to Queryable Topics

At a high level, the pipeline looks like this:

HackerNews API
    ↓
HackerNewsConnector (Custom Source)
    ├─ list() → thread IDs + updated_at
    ├─ get_value() → full threads + comments
    └─ provides_ordinal() → enables incremental sync
    ↓
CocoIndex Flow
    ├─ LLM topic extraction on threads + comments
    ├─ message_index collector (content)
    └─ topic_index collector (topics)
    ↓
Postgres
    ├─ hn_messages
    └─ hn_topics
    ↓
Query Handlers
    ├─ search_by_topic("Claude")
    ├─ get_trending_topics(limit=20)
    └─ get_threads_for_topic("Rust")

Key idea: separate discovery from fetching.

list() hits the HN Algolia search API to get lightweight metadata: thread IDs + updated_at timestamps.
get_value() only runs for threads whose updated_at changed, fetching full content + comments from the items API.
Ordinals (timestamps) let CocoIndex skip everything that hasn't changed, cutting API calls by >90% on subsequent syncs.

This is what enables "live mode" with a 30-second polling interval without melting APIs or your wallet.

Step 1: Turning HackerNews Into a First-Class Incremental Source

First, you define the data model for threads and comments.

class _HackerNewsThreadKey(NamedTuple):
    thread_id: str

@dataclasses.dataclass
class _HackerNewsComment:
    id: str
    author: str | None
    text: str | None
    created_at: datetime | None

@dataclasses.dataclass
class _HackerNewsThread:
    author: str | None
    text: str
    url: str | None
    created_at: datetime | None
    comments: list[_HackerNewsComment]

Then you declare a SourceSpec that configures how to query HN:

class HackerNewsSource(SourceSpec):
    """Source spec for HackerNews API."""
    tag: str | None = None      # e.g. "story"
    max_results: int = 100      # hits per poll

The custom source connector wires this spec into actual HTTP calls:

list() → calls https://hn.algolia.com/api/v1/search_by_date with hitsPerPage=max_results, yields PartialSourceRow objects keyed by thread ID, with ordinals based on updated_at.
get_value() → calls https://hn.algolia.com/api/v1/items/{thread_id} and parses the full thread + nested comments into _HackerNewsThread and _HackerNewsComment.
provides_ordinal() → returns True so CocoIndex can do incremental sync.

CocoIndex handles the hard part: tracking ordinals and only re-pulling changed rows on each sync.

Step 2: Using an LLM to Extract Topics From Every Thread and Comment

Once the source is in the flow, the fun part starts: semantic enrichment.

You define a minimal Topic type that the LLM will fill:

@dataclasses.dataclass
class Topic:
    """
    A single topic extracted from text:
    - products, tools, frameworks
    - people, companies
    - domains (e.g. "vector search", "fintech")
    """
    topic: str

Inside the flow, every thread gets its topics extracted with a single declarative transform:

with data_scope["threads"].row() as thread:
    thread["topics"] = thread["text"].transform(
        cocoindex.functions.ExtractByLlm(
            llm_spec=cocoindex.LlmSpec(
                api_type=cocoindex.LlmApiType.OPENAI,
                model="gpt-4o-mini",
            ),
            output_type=list[Topic],
        )
    )

Same for comments:

with thread["comments"].row() as comment:
    comment["topics"] = comment["text"].transform(
        cocoindex.functions.ExtractByLlm(
            llm_spec=cocoindex.LlmSpec(
                api_type=cocoindex.LlmApiType.OPENAI,
                model="gpt-4o-mini",
            ),
            output_type=list[Topic],
        )
    )

Under the hood, CocoIndex:

Calls the LLM with a structured prompt and enforces output_type=list[Topic]
Normalizes messy free text into consistent topic strings
Makes this just another column in your flow instead of a separate glue script

This is what turns HN from "some text" into something an AI agent or SQL query can reason about.

Step 3: Indexing Into Postgres for Fast Topic Queries

All structured data is collected into two logical indexes:

message_index: threads + comments with their raw text and metadata
topic_index: individual topics linked back to messages

Collectors are declared once and then exported to Postgres:

message_index = data_scope.add_collector()
topic_index = data_scope.add_collector()

message_index.export(
    "hn_messages",
    cocoindex.targets.Postgres(),
    primary_key_fields=["id"],
)

topic_index.export(
    "hn_topics",
    cocoindex.targets.Postgres(),
    primary_key_fields=["topic", "message_id"],
)

Now you have two tables you can poke with SQL or via CocoIndex query handlers:

hn_messages: full-text search, content analytics, author stats
hn_topics: topic-level analytics, trend tracking, per-topic thread ranking

Step 4: Query Handlers - From "Cool Pipeline" to Real Product

Here's where it stops being just a nice ETL project and becomes something you can actually ship.

4.1 `search_by_topic(topic)`: "Show Me All Claude Mentions"

This query handler lets you search HN content by topic across threads and comments:

@hackernews_trending_topics_flow.query_handler()
def search_by_topic(topic: str) -> cocoindex.QueryOutput:
    topic_table = cocoindex.utils.get_target_default_name(
        hackernews_trending_topics_flow, "hn_topics"
    )
    message_table = cocoindex.utils.get_target_default_name(
        hackernews_trending_topics_flow, "hn_messages"
    )

    with connection_pool().connection() as conn:
        with conn.cursor() as cur:
            cur.execute(
                f"""
                SELECT m.id, m.thread_id, m.author, m.content_type,
                       m.text, m.created_at, t.topic
                FROM {topic_table} t
                JOIN {message_table} m ON t.message_id = m.id
                WHERE LOWER(t.topic) LIKE LOWER(%s)
                ORDER BY m.created_at DESC
                """,
                (f"%{topic}%",),
            )

            results = [
                {
                    "id": row[0],
                    "url": f"https://news.ycombinator.com/item?id={row[1]}",
                    "author": row[2],
                    "type": row[3],
                    "text": row[4],
                    "created_at": row[5].isoformat(),
                    "topic": row[6],
                }
                for row in cur.fetchall()
            ]

    return cocoindex.QueryOutput(results=results)

You can literally run:

cocoindex query main.py search_by_topic --topic "Claude"

...and get a clean JSON response with URLs, authors, timestamps, and which piece of content the topic appeared in.

4.2 `get_threads_for_topic(topic)`: Rank Threads by Topic Score

Not all mentions are equal.

If "Rust" is in the thread title, that's a primary discussion
If it's buried in a comment, that's more of a side mention

get_threads_for_topic uses a weighted scoring model to prioritize threads where the topic is central.

4.3 `get_trending_topics(limit=20)`: The Actual Trend Radar

Finally, the endpoint that powers dashboards and agents - this surfaces a list like:

["Claude 3.7 Sonnet", "OpenAI o4-mini", "LangChain", "Modal", ...] with scores and latest mention times
Each topic includes the top threads where it's being discussed right now

You can wire this into:

A live dashboard showing "top 20 topics in the last N hours"
A Slack bot posting a daily "what's trending on HN" summary
An internal research agent that watches for signals relevant to your stack

Running It in Real Time

Once the flow is defined, keeping it live is a one-liner:

# On-demand refresh
cocoindex update main

# Live mode: keeps polling HN and updating indexes
cocoindex update -L main

CocoIndex handles:

Polling HN every 30 seconds (configurable)
Incrementally syncing only changed threads
Re-running LLM extraction only where needed
Exporting into Postgres and making query handlers available

For debugging, CocoInsight lets you explore the flow, see lineage, and play with queries from a UI:

cocoindex server -ci main
# Then open: https://cocoindex.io/cocoinsight

What You Can Build on Top of This (Beyond "Just HN")

Once you have this pattern, you're not limited to HackerNews.

Some obvious extensions:

Cross-community trend tracking
- Add Reddit subs, X lists, Discord channels, internal Slack, etc. as additional sources
- Normalize topics across them to see which ideas propagate where and when
Sentiment-aware trend analysis
- Plug in an LLM-based sentiment extraction step alongside topics
- Track not just what is trending, but whether devs love or hate it
Influencer and key-contributor maps
- Use the author field to see who starts important discussions and whose comments move the conversation
Continuous knowledge graphs
- Treat topics as nodes, threads as edges, and build a graph of tools, companies, and people linked by real discussions
Real-time AI research agents
- Point an agent at the Postgres-backed index and let it answer questions like
- "What are the top new vector DBs people are experimenting with this week?"
- "Which AI eval frameworks are getting traction?"

If you live in data, infra, or AI-land, this is basically a self-updating signal layer over HN that your tools and agents can query.

Want to Try It Yourself?

You can find the fully working example (including flow definition, custom source, query handlers, and Postgres export) in the official HackerNews Trending Topics example on the CocoIndex docs and GitHub.

If you end up:

Pointing this at a different community
Layering in embeddings, RAG, or sentiment
Wiring it into a real product or agent

...definitely share it back. The coolest part of this pattern is how little code you need to go from "raw community noise" to a live, queryable trend radar.

Build an AI-Powered Competitive Intelligence Monitor

Linghua Jin — Thu, 15 Jan 2026 07:21:09 +0000

Staying ahead of competitors requires constant vigilance—tracking product launches, funding rounds, partnerships, and strategic moves across the web. The open-source Competitive Intelligence Monitor project demonstrates how to automate this process using CocoIndex, Tavily Search, and LLM extraction to continuously track and structure competitor news into a queryable PostgreSQL database.

What the Project Does

The system automates web monitoring by using Tavily's AI-native search to pull full-text articles, then feeding them through a GPT-4o-mini–based extraction layer to detect structured "competitive events" such as:

Product launches and feature releases
Partnerships and collaborations
Funding rounds and financial news
Key executive hires/departures
Acquisitions and mergers

These events and their source articles are stored in PostgreSQL so teams can ask natural questions like "What has Anthropic been doing recently?" or "Which competitors are making the most news this week?"

Core Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Tavily AI   │────▶│  CocoIndex   │────▶│  PostgreSQL  │
│    Search    │     │   Pipeline   │     │   Database   │
└──────────────┘     └──────────────┘     └──────────────┘
       │                    │                    │
       ▼                    ▼                    ▼
   Articles           Extraction           Intelligence
  (web data)        (GPT-4o-mini)         (structured)

Data flows from Tavily search results into an LLM extraction step that produces CompetitiveEvent objects, then into dual indexes—one table for raw articles and another for normalized events.

Data Model: The CompetitiveEvent Class

At the heart of the extraction is the CompetitiveEvent dataclass that defines what the LLM should extract from each article:

@dataclasses.dataclass
class CompetitiveEvent:
    """A competitive intelligence event extracted from text.

    Examples:
    - Product Launch: "OpenAI released GPT-5 with multimodal capabilities"
    - Partnership: "Anthropic partnered with Google Cloud for enterprise AI"
    - Funding: "Mistral AI raised $400M Series B led by Andreessen Horowitz"
    - Key Hire: "Former Meta AI director joined Cohere as Chief Scientist"
    - Strategic Move: "Microsoft acquired AI startup Inflection for $650M"
    """
    event_type: str      # "product_launch", "partnership", "funding", "key_hire", "acquisition", "other"
    competitor: str      # Company name (e.g., "OpenAI", "Anthropic", "Google AI")
    description: str     # Brief description of the event
    significance: str    # "high", "medium", "low" - based on market impact
    related_companies: list[str]  # Other companies mentioned

Custom Tavily Source Connector

The project implements a custom CocoIndex source connector that interfaces with Tavily's AI-native search API:

class TavilySearchSource(SourceSpec):
    """Fetches competitive intelligence using Tavily AI Search API."""
    competitor: str
    days_back: int = 7
    max_results: int = 10

@source_connector(
    spec_cls=TavilySearchSource,
    key_type=_ArticleKey,
    value_type=_Article,
)
class TavilySearchConnector:
    async def list(self) -> AsyncIterator[PartialSourceRow[_ArticleKey, _Article]]:
        """List articles from Tavily search."""
        search_query = (
            f"{self._spec.competitor} AND "
            f"(funding OR partnership OR product launch OR acquisition OR executive hire)"
        )

        client = TavilyClient(api_key=self._api_key)
        response = client.search(
            query=search_query,
            search_depth="advanced",
            max_results=self._spec.max_results,
            include_raw_content=True,
        )

        for result in response.get("results", []):
            url = result["url"]
            yield PartialSourceRow(
                key=_ArticleKey(url=url),
                data=PartialSourceRowData(ordinal=ordinal),
            )

The CocoIndex Pipeline Definition

The main pipeline uses CocoIndex's flow builder to orchestrate data collection and LLM extraction:

@cocoindex.flow_def(name="CompetitiveIntelligence")
def competitive_intelligence_flow(
    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
    """Main pipeline for competitive intelligence monitoring."""

    competitors = os.getenv("COMPETITORS", "OpenAI,Anthropic").split(",")
    refresh_interval = int(os.getenv("REFRESH_INTERVAL_SECONDS", "3600"))

    # Add Tavily search source for each competitor
    for competitor in competitors:
        data_scope[f"articles_{competitor.strip()}"] = flow_builder.add_source(
            TavilySearchSource(
                competitor=competitor.strip(),
                days_back=search_days_back,
                max_results=10,
            ),
            refresh_interval=timedelta(seconds=refresh_interval),
        )

    articles_index = data_scope.add_collector()
    events_index = data_scope.add_collector()

    # Process each competitor's articles
    for competitor in competitors:
        articles = data_scope[f"articles_{competitor.strip()}"]

        with articles.row() as article:
            # Extract competitive events using GPT-4o-mini via OpenRouter
            article["events"] = article["content"].transform(
                cocoindex.functions.ExtractByLlm(
                    llm_spec=cocoindex.LlmSpec(
                        api_type=cocoindex.LlmApiType.OPENAI,
                        model="openai/gpt-4o-mini",
                        address="https://openrouter.ai/api/v1",
                    ),
                    output_type=list[CompetitiveEvent],
                    instruction=(
                        "Extract competitive intelligence events from this article. "
                        "Focus on: product launches, partnerships, funding rounds, key hires, "
                        "acquisitions, and other strategic moves."
                    ),
                )
            )

Query Handlers for Analysis

The project includes built-in query handlers that enable SQL-powered intelligence retrieval:

@competitive_intelligence_flow.query_handler()
def search_by_competitor(
    competitor: str, 
    event_type: str | None = None, 
    limit: int = 20
) -> cocoindex.QueryOutput:
    """Find recent competitive intelligence about a specific competitor."""

    with connection_pool().connection() as conn:
        with conn.cursor() as cur:
            sql = f"""
                SELECT e.competitor, e.event_type, e.description, e.significance,
                       e.related_companies, a.title, a.url, a.source, a.published_at
                FROM {events_table} e
                JOIN {articles_table} a ON e.article_id = a.id
                WHERE LOWER(e.competitor) LIKE LOWER(%s)
            """
            params = [f"%{competitor}%"]

            if event_type:
                sql += " AND e.event_type = %s"
                params.append(event_type)

            sql += " ORDER BY a.published_at DESC LIMIT %s"
            cur.execute(sql, params)

            return cocoindex.QueryOutput(results=[...])

Getting Started

Configuration is controlled through environment variables:

DATABASE_URL=postgresql://user:password@localhost:5432/competitive_intel
COCOINDEX_DATABASE_URL=postgresql://user:password@localhost:5432/competitive_intel
OPENAI_API_KEY=sk-or-v1-...
TAVILY_API_KEY=tvly-...
COMPETITORS=OpenAI,Anthropic,Google AI,Meta AI,Mistral AI
REFRESH_INTERVAL_SECONDS=3600
SEARCH_DAYS_BACK=7

Run the interactive CLI for first-time setup:

python3 run_interactive.py

Or use CocoIndex directly for automated deployments:

cocoindex update main -f          # Initial sync
cocoindex update -L main.py       # Continuous monitoring

Why This Approach Matters

By combining AI-native search with structured LLM extraction, the monitor:

Avoids brittle scraping - Tavily handles content extraction
De-duplicates work - CocoIndex tracks processed articles via incremental processing
Turns noise into signal - Structured events with significance scoring
Enables flexible analysis - Dual indexing (raw + extracted) for maximum flexibility

The project supports multiple query types:

Search by competitor name
Filter by event type (funding, partnerships, acquisitions, etc.)
Rank by significance (high=3, medium=2, low=1 weighted scoring)
Trend analysis across time periods

Get the Code

The project is MIT-licensed and available on GitHub:

Laksh-star / competitive-intelligence

AI-powered competitive intelligence monitor using CocoIndex, Tavily Search, and LLM extraction

Competitive Intelligence Monitor

Track competitor mentions across the web using AI-powered search and LLM extraction. Automatically monitors competitors, extracts competitive intelligence events, and stores structured data in PostgreSQL for analysis.

What This Does

This pipeline automatically:

Searches the web using Tavily AI (AI-native search engine optimized for agents)
Extracts competitive intelligence events using DeepSeek LLM analysis
- Product launches and feature releases
- Partnerships and collaborations
- Funding rounds and financial news
- Key executive hires/departures
- Acquisitions and mergers
Indexes both raw articles and extracted events in PostgreSQL
Enables queries like:
- "What has OpenAI been doing recently?"
- "Which competitors are making the most news?"
- "Find all partnership announcements"
- "What are the most significant competitive moves this week?"

Prerequisites

PostgreSQL Database - Choose one option
- Local PostgreSQL installation
- Cloud PostgreSQL (AWS RDS, Google Cloud SQL, Azure Database, etc.)
Python 3.11+ - Required for CocoIndex
API Keys (required)
- Tavily API key from tavily.com (free tier: 1,000…

View on GitHub

Built with:

CocoIndex - Modern data pipeline framework
Tavily AI Search - AI-native search engine
OpenRouter - Multi-model API gateway

Have questions or want to contribute? Drop a comment below or open an issue on GitHub!

Stop Writing Fragile Prompts: Extract Structured Data from PDFs with DSPy + CocoIndex

Linghua Jin — Tue, 13 Jan 2026 19:13:35 +0000

TL;DR: Traditional prompt engineering is fragile—small changes break everything. This tutorial shows how to extract structured patient data from PDF intake forms using DSPy's typed Signatures + CocoIndex's incremental processing. No OCR preprocessing, no regex, just declarative code.

The Problem with Prompt Engineering

If you've built LLM applications, you know the pain:

You write a carefully crafted prompt with instructions and few-shot examples
It works... until the model changes, or the data shifts slightly
Your output format breaks, and you're back to debugging strings

Logic buried in strings is hard to test, compose, or version.

What if there was a better way?

Enter DSPy + CocoIndex

DSPy (from Stanford) replaces prompt engineering with a programming model. You define what each LLM step should do (inputs, outputs, constraints), and the framework figures out how to prompt the model.

CocoIndex is an ultra-performant data processing engine (Rust-powered) that handles:

File ingestion from any source
Incremental processing (only reprocess changed documents)
Caching and lineage tracking

Together, they form a powerful combo: DSPy owns "how the model thinks," CocoIndex owns "how data moves and stays fresh."

What We're Building

A pipeline that:

Reads PDF patient intake forms
Converts pages to images
Extracts structured Patient data using vision models
Exports to PostgreSQL with automatic updates

Step 1: Define Your Schema with Pydantic

Instead of parsing unstructured text, we define exactly what we want:

from pydantic import BaseModel, Field
from datetime import date

class Address(BaseModel):
    street: str
    city: str
    state: str
    zip_code: str

class Insurance(BaseModel):
    provider: str
    policy_number: str
    group_number: str | None = None
    policyholder_name: str

class Patient(BaseModel):
    name: str
    dob: date
    gender: str
    address: Address
    phone: str
    email: str
    insurance: Insurance | None = None
    reason_for_visit: str
    allergies: list[str] = Field(default_factory=list)
    current_medications: list[str] = Field(default_factory=list)
    consent_given: bool

This FHIR-inspired schema gives us validation, nested models, and type safety out of the box.

Step 2: Create the DSPy Signature

A Signature is DSPy's way of declaring the task contract:

import dspy

class PatientExtractionSignature(dspy.Signature):
    """Extract structured patient information from a medical intake form image."""

    form_images: list[dspy.Image] = dspy.InputField(
        desc="Images of the patient intake form pages"
    )
    patient: Patient = dspy.OutputField(
        desc="Extracted patient information with all available fields filled"
    )

Notice: No prompts, no examples, no parsing logic. Just a typed contract.

Step 3: Build the Extractor Module

class PatientExtractor(dspy.Module):
    def __init__(self):
        super().__init__()
        self.extract = dspy.ChainOfThought(PatientExtractionSignature)

    def forward(self, form_images: list[dspy.Image]) -> Patient:
        result = self.extract(form_images=form_images)
        return result.patient

ChainOfThought handles the reasoning—DSPy translates your signature into an effective prompt automatically.

Step 4: Wire It Into CocoIndex

Here's where incremental processing magic happens:

import cocoindex
import pymupdf

@cocoindex.op.function(cache=True, behavior_version=1)
def extract_patient(pdf_content: bytes) -> Patient:
    # Convert PDF to images
    pdf_doc = pymupdf.open(stream=pdf_content, filetype="pdf")
    form_images = []

    for page in pdf_doc:
        pix = page.get_pixmap(matrix=pymupdf.Matrix(2, 2))  # 2x resolution
        form_images.append(dspy.Image(pix.tobytes("png")))

    pdf_doc.close()

    # Extract with DSPy
    extractor = PatientExtractor()
    return extractor(form_images=form_images)

The cache=True decorator means repeated calls with the same PDF reuse results—no wasted API calls.

Step 5: Define the Flow

@cocoindex.flow_def(name="PatientIntakeExtraction")
def patient_intake_flow(flow_builder, data_scope):
    # Source: local PDF files
    data_scope["documents"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(path="data/patient_forms", binary=True)
    )

    patients_index = data_scope.add_collector()

    with data_scope["documents"].row() as doc:
        doc["patient_info"] = doc["content"].transform(extract_patient)
        patients_index.collect(
            filename=doc["filename"],
            patient_info=doc["patient_info"]
        )

    # Export to Postgres
    patients_index.export(
        "patients",
        cocoindex.storages.Postgres(table_name="patients_info"),
        primary_key_fields=["filename"]
    )

Why This Approach Wins

Traditional Approach	DSPy + CocoIndex
Fragile prompts	Typed Signatures
Manual parsing	Automatic structured output
Full reprocessing	Incremental updates
No audit trail	Built-in lineage tracking
String debugging	Testable modules

Run It

pip install cocoindex dspy-ai pydantic pymupdf
cocoindex update main

That's it. Your PDFs are now structured data in Postgres, automatically kept in sync.

Key Takeaways

DSPy replaces prompt engineering with a programming model—define the contract, not the implementation
Vision models eliminate OCR complexity—just pass images directly
CocoIndex handles the plumbing—caching, incremental updates, lineage
Pydantic gives you validation and nested structures for free
Neither tool tries to be the whole stack—they compose beautifully

Resources

Have you tried DSPy or CocoIndex? Drop a comment with your experience—I'd love to hear how others are solving structured extraction problems!

We Built an Open-Source Pipeline That Turned Meeting Notes Into a Live Knowledge Graph — And It Went Viral (200K Impressions)

Linghua Jin — Mon, 05 Jan 2026 03:22:44 +0000

🚀 The Result: 200K Social Impressions and Viral Engagement

Our latest project just exploded on LinkedIn with 200K+ impressions, and for good reason. We built something that solves a real problem most companies face: their meeting notes are a goldmine of untapped knowledge, but nobody has time to manually organize them.

💡 The Problem

Most companies sit on an ocean of meeting notes scattered across Google Drive. Inside those documents are:

Critical decisions that shape product direction
Action items and task assignments
Key relationships between people, projects, and initiatives
Institutional knowledge that disappears when people leave

But here's the catch: these documents are constantly changing. Traditional data pipelines would reprocess everything from scratch every time, wasting compute and money on unchanged files.

⚡ The Solution: Incremental Processing with CocoIndex

We built an open-source pipeline that:

Connects directly to Google Drive - no manual exports needed
Only processes what changed - incremental LLM extraction means zero reprocessing of unchanged docs
Builds a live knowledge graph - automatically extracts entities, relationships, and updates Neo4j in real-time
Production-ready - fully open-sourced under Apache 2.0 license

🔧 How It Works

The pipeline continuously:

Monitors your Google Drive for changes
Uses LLM to extract structured data (people, decisions, tasks, relationships)
Incrementally updates the knowledge graph - only changed documents get reprocessed
Serves fresh insights through Neo4j queries

Tech Stack:

CocoIndex - for incremental data processing
Neo4j - for the knowledge graph
LLM - for entity and relationship extraction
Google Drive API - for document access

📚 Full Tutorial Available

We've published a complete step-by-step tutorial with:

Full source code (Apache 2.0)
Architecture explanations
Setup instructions
Real examples

🔗 Links:

GitHub Repo: https://github.com/cocoindex-io/cocoindex (⭐ star it if you find it useful!)
Tutorial: https://cocoindex.io/blogs/meeting-notes-graph
Live Example: https://cocoindex.io/examples/meeting_notes_graph

🎯 Why This Matters

Incremental processing is the key differentiator here. Most pipelines are "dumb" - they reprocess everything even if only one document changed. That's:

❌ Expensive (LLM costs add up fast)
❌ Slow (unnecessary compute time)
❌ Wasteful (environmental impact)

With CocoIndex's incremental approach:

✅ Only pay for what changed
✅ Real-time updates
✅ Scales with your document library

🌟 Built With Open Source

This entire project is open source and production-ready. Whether you're a startup drowning in meeting notes or an enterprise looking to unlock institutional knowledge, you can deploy this today.

What could you build with a live knowledge graph of your company's meeting notes?

Drop your thoughts in the comments! And if you're working on similar problems, I'd love to hear about your approach. 👇

Why Your AI Agent is Living in the Past (And How to Fix It) 🚀

Linghua Jin — Wed, 24 Dec 2025 00:22:35 +0000

The Stale Context Problem

Imagine this: You've built a beautiful AI agent that can answer questions about your codebase. You spent weeks setting up the perfect data pipeline, carefully chunked your documents, and embedded everything into a vector database.

Then someone pushes a new feature to main.

Your AI agent? Still answering questions based on yesterday's code. 😬

This is the dirty secret of production AI systems: maintaining fresh, structured context is harder than building the AI itself.

Why Fresh Context Matters for AI Agents

Here's the reality: AI agents in 2025 aren't just answering static FAQs. They're:

Monitoring live codebases that change dozens of times per day
Processing incoming emails and turning them into structured data
Analyzing meeting notes to build dynamic knowledge graphs
Watching PDF documents that get updated in real-time
Tracking customer data that evolves every second

Every time your source data changes, you face a painful choice:

Re-index everything (slow, expensive, wastes compute)
Let your context go stale (fast way to lose user trust)
Build complex change tracking (hello technical debt!)

There has to be a better way. Spoiler: there is.

Enter CocoIndex: Context Engineering Made Simple

CocoIndex just hit #1 on GitHub Trending for Rust, and for good reason. It's a data transformation framework built specifically for keeping AI context fresh.

Here's what makes it different:

🚀 Incremental Processing by Default

No more re-processing your entire dataset when one file changes. CocoIndex tracks dependencies and only recomputes what's necessary.

# This automatically handles incremental updates
data["documents"] = flow_builder.add_source(
    cocoindex.sources.LocalFile(path="docs")
)

When you update a single document, CocoIndex:

Detects exactly what changed
Re-processes only affected chunks
Updates your vector store with minimal operations
Preserves everything else

No index swaps. No downtime. No stale data.

🧱 Dataflow Programming Model

Define your transformations once, and CocoIndex handles the orchestration:

@cocoindex.flow_def(name="SmartContext")
def smart_context_flow(flow_builder, data_scope):
    # Source: Read from anywhere
    data_scope["docs"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(path="markdown_files")
    )

    collector = data_scope.add_collector()

    # Transform: Process each document
    with data_scope["docs"].row() as doc:
        # Split into chunks
        doc["chunks"] = doc["content"].transform(
            cocoindex.functions.SplitRecursively(),
            chunk_size=2000
        )

        # Embed each chunk
        with doc["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].transform(
                cocoindex.functions.SentenceTransformerEmbed(
                    model="all-MiniLM-L6-v2"
                )
            )

            collector.collect(
                filename=doc["filename"],
                text=chunk["text"],
                embedding=chunk["embedding"]
            )

    # Export: Send to your vector store
    collector.export(
        "docs",
        cocoindex.targets.Postgres(),
        vector_indexes=[...]
    )

Notice what you DON'T see:

No explicit update logic
No manual cache invalidation
No index swap coordination
No "when to re-embed" decisions

Just pure transformation logic. CocoIndex handles the rest.

🔧 Built for Production, Not Demos

Ultra-performant Rust core: The heavy lifting happens in Rust, giving you C-level performance with Python ergonomics.

Data lineage out of the box: Track exactly where each piece of context came from. Debug your AI's reasoning, not just its output.

Plug-and-play components: Switch between embedding models, vector stores, or data sources with single-line changes.

Real-World Use Cases

Here's what developers are building:

Live Code Search: Index your entire monorepo, keep embeddings fresh as PRs merge. No more "this was refactored last week" moments.

Meeting Notes → Knowledge Graph: Extract entities and relationships from Google Drive meeting notes, automatically update your knowledge base.

Smart PDF Processing: Parse complex PDFs (text + images), embed both modalities, and serve multimodal search that stays current.

Customer Context for Support AI: Keep your support agent's context synchronized with live customer data, product updates, and recent tickets.

The Context Engineering Paradigm Shift

Traditional RAG: "Let's embed everything and query it"
Context Engineering: "Let's define transformations and keep everything synchronized"

The difference? Production AI systems that actually work at scale.

Try It Yourself

CocoIndex is open source (Apache 2.0) and dead simple to get started:

pip install -U cocoindex

Check out the examples:

Text embedding with auto-updates
PDF processing with live refresh
Knowledge graph extraction
Custom transformations
Multi-format indexing

👉 GitHub: github.com/cocoindex-io/cocoindex
📖 Docs: cocoindex.io/docs

The Bottom Line

2026 is the year autonomous agents go mainstream. But they won't succeed with stale context.

If you're building AI systems that need to stay synchronized with reality — not just answer questions about the past — context engineering is your unlock.

And CocoIndex? It's the framework that makes it actually feasible.

Give it a star if you're tired of rebuilding indexes manually ⭐

What's your biggest challenge keeping AI context fresh? Drop a comment below! 👇

Build a Real-Time Codebase Index in 5 Minutes with CocoIndex (Rust + Tree-sitter)

Linghua Jin — Mon, 15 Dec 2025 01:41:56 +0000

Why Another Codebase Indexing Tool?

Let's be honest: managing code context for AI agents is a nightmare.

Your AI coding assistant needs to understand your entire codebase—not just one file at a time. Whether you're building RAG systems for Claude, context for Cursor, or semantic code search, you need:

✅ Fast, incremental updates (not rebuilding everything)
✅ Proper code parsing (not just text chunking)
✅ Vector embeddings for semantic search
✅ Real-time sync when your code changes

That's exactly what CocoIndex delivers.

What Makes CocoIndex Special?

Built-in Tree-sitter Support: Unlike generic text splitters, CocoIndex uses Tree-sitter to parse your code semantically. It understands functions, classes, and code structure—not just lines of text.

Incremental Processing: Only reprocess what changed. No more waiting 10 minutes every time you update a single file.

Native Vector Search: Built-in support for embedding generation and vector search with PostgreSQL + pgvector.

MCP Compatible: Works seamlessly with AI editors like Cursor, Windsurf, and Claude.

Real-World Use Cases

🤖 AI Coding Agents: Give Claude, Codex, or Gemini the right code context

🔍 Semantic Code Search: Find code by meaning, not keywords

📝 Auto Documentation: Keep design docs synced with actual code

🔧 Code Review Automation: AI-powered PR analysis

🚨 SRE Workflows: Index infrastructure-as-code for incident response

Tutorial: Build Your Codebase Index

Let me show you how ridiculously simple this is.

Step 1: Install

pip install -U cocoindex

You'll also need PostgreSQL with pgvector extension. Installation guide here.

Step 2: Define Your Flow

Create a flow that reads your codebase, chunks it with Tree-sitter, and generates embeddings:

import os
import cocoindex

@cocoindex.flow_def(name="CodeEmbedding")
def code_embedding_flow(
    flow_builder: cocoindex.FlowBuilder,
    data_scope: cocoindex.DataScope
):
    # Load your codebase
    data_scope["files"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(
            path=os.path.join('..', '..'),
            included_patterns=["*.py", "*.rs", "*.toml"],
            excluded_patterns=[".*", "target", "**/node_modules"]
        )
    )

    code_embeddings = data_scope.add_collector()

Step 3: Extract Language & Chunk Code

@cocoindex.op.function()
def extract_extension(filename: str) -> str:
    return os.path.splitext(filename)[1]

with data_scope["files"].row() as file:
    # Extract extension for Tree-sitter
    file["extension"] = file["filename"].transform(extract_extension)

    # Chunk code semantically
    file["chunks"] = file["content"].transform(
        cocoindex.functions.SplitRecursively(),
        language=file["extension"],
        chunk_size=1000,
        chunk_overlap=300
    )

Step 4: Embed & Index

@cocoindex.transform_flow()
def code_to_embedding(
    text: cocoindex.DataSlice[str]
) -> cocoindex.DataSlice[list[float]]:
    return text.transform(
        cocoindex.functions.SentenceTransformerEmbed(
            model="sentence-transformers/all-MiniLM-L6-v2"
        )
    )

with file["chunks"].row() as chunk:
    chunk["embedding"] = chunk["text"].call(code_to_embedding)

    code_embeddings.collect(
        filename=file["filename"],
        location=chunk["location"],
        code=chunk["text"],
        embedding=chunk["embedding"]
    )

# Export to PostgreSQL with vector index
code_embeddings.export(
    "code_embeddings",
    cocoindex.storages.Postgres(),
    primary_key_fields=["filename", "location"],
    vector_indexes=[
        cocoindex.VectorIndex(
            "embedding",
            cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY
        )
    ]
)

Step 5: Run It

cocoindex update main

Boom. Your codebase is now indexed with semantic embeddings.

Step 6: Query Your Index

def search(pool: ConnectionPool, query: str, top_k: int = 5):
    table_name = cocoindex.utils.get_target_storage_default_name(
        code_embedding_flow, "code_embeddings"
    )

    query_vector = code_to_embedding.eval(query)

    with pool.connection() as conn:
        with conn.cursor() as cur:
            cur.execute(f"""
                SELECT filename, code, embedding <=> %s::vector AS distance
                FROM {table_name}
                ORDER BY distance LIMIT %s
            """, (query_vector, top_k))

            return [{
                "filename": row[0],
                "code": row[1],
                "score": 1.0 - row[2]
            } for row in cur.fetchall()]

Now you can search your codebase semantically:

python main.py
# Enter: "authentication middleware"
# Returns relevant auth code across your entire codebase

Language Support

CocoIndex supports all major languages via Tree-sitter:

Python, JavaScript, TypeScript
Rust, Go, C, C++, Java
Ruby, PHP, Swift, Kotlin
And 30+ more

Full language list here.

Visualize with CocoInsight

Want to debug your indexing flow visually?

cocoindex server -ci main

This spins up CocoInsight at https://cocoindex.io/cocoinsight where you can inspect your data flow step-by-step.

Why You Should Care

AI coding tools are only as good as the context you give them. If you're building:

AI agents that need code awareness
Semantic code search engines
Automated documentation generators
Code review automation

...you need a proper codebase index.

CocoIndex makes it stupidly simple.

Try It Yourself

⭐ Star the repo: github.com/cocoindex-io/cocoindex

📖 Read the docs: cocoindex.io/docs

🎥 Watch the tutorial: YouTube guide

💬 Join Discord: discord.com/invite/zpA9S2DR7s

What are you building with AI code tools? Drop a comment below—I'd love to hear your use case!

If you found this useful, give CocoIndex a star on GitHub. It's open source and built by developers who actually understand the pain of managing code context for AI. 🚀

From Repo to Movement: Building an Open Source Project and Its Community

Linghua Jin — Sat, 13 Dec 2025 21:59:26 +0000

Publishing a public GitHub repository is easy; turning it into a healthy open source project with an active community is hard work and very intentional. The projects that endure treat both their code and their community as core product surfaces, with the README acting as the front door to everything.

Start with a sharp problem and vision

Every great open source project starts with a real, painful problem and a specific audience, not just an experiment pushed to GitHub. Define who you are building for, what they are trying to do, and why existing tools are insufficient.

Write this down as a concise vision: what success looks like, what is in scope, and what is explicitly out of scope. This vision will later anchor your roadmap, help you say "no" to distracting feature requests, and give contributors something clear to align with.

Set the foundations: license, structure, and tooling

Before you ask anyone to contribute, make the project safe and predictable to use. Choose an explicit open source license that matches your goals, add it to the repository, and structure your code in a way that is easy to navigate.

Set up basic automation and hygiene early: tests, linters, continuous integration, and clear directory conventions. These guardrails reduce friction for both you and future contributors, and signal that the project is maintained professionally.

The README: your project's front door

A README is the welcome mat and elevator pitch of your project: it explains what the project is, why it exists, and how to get value from it in the first five minutes. Visitors skim the README to decide whether to try the project, whether it seems maintained, and whether it looks friendly to contributions.

A great README typically answers, in order:

What problem does this solve, and for whom?
How do I install it and run a minimal example?
How does it fit into my stack or workflow?
Where do I go for docs, questions, and contributions?

When this information is clear, users reach "first success" much faster, which directly increases adoption and the pool of potential contributors. Well-structured READMEs are strongly associated with higher contribution rates because they reduce ambiguity, confusion, and support noise for new people.

How a great README supercharges your community

The README does more than explain the code; it actively shapes your community's growth curve. First, it acts as always-on marketing and documentation: search engines index it, blog posts link to it, and people share screenshots of it in chats and social feeds.

Second, it acts as a community filter and funnel. When you include links to "good first issues," contributor guides, chat channels, and governance docs, the README quietly converts curious users into engaged participants. Clear expectations about scope, stability, and contribution norms reduce low-quality issues and misaligned requests, which keeps the environment healthier for everyone.

Design for contributor experience from day one

Once the foundations and README are in place, design the project around contributors, not just maintainers. Add a CONTRIBUTING file that explains how to set up the environment, run tests, and submit issues or pull requests, and pair it with issue and PR templates.

Label beginner-friendly tasks, keep a small set of "good first issues" fresh, and ensure that at least one maintainer watches and responds quickly. Early, fast feedback often matters more than the actual decision, because it tells new contributors that their time is respected.

Governance and roles: make power explicit

As your project grows, unclear decision-making is one of the fastest ways to frustrate contributors. Choose a governance model—strong maintainer, core team, or more formal steering committee—and write down how decisions are made, who has which permissions, and how new people can grow into those roles.

Document this in a lightweight governance or project charter file and reference it from the README so it is easy to find. Pair governance with a Code of Conduct that defines expected behavior, reporting channels, and enforcement to keep the space safe and inclusive.

Treat communication as part of the product

Healthy communities are built on predictable, transparent communication. Use your README and docs to advertise where discussions happen (GitHub Discussions, Discord, mailing list), and keep most technical decisions and design conversations in public channels by default.

Maintain a cadence of release notes, changelogs, and occasional blog posts or newsletters that explain not just what changed, but why. Publicly thanking contributors in these updates reinforces good behavior and encourages more people to step up.

Balance sustainability and growth

Maintaining an open source project is a marathon, not a sprint, and burnout is real. Use automation—bots for formatting, dependency updates, stale issue triage—to free maintainers from repetitive tasks, and be comfortable sunsetting features or saying "no" when requests drift from your stated vision.

Over time, consider sustainability levers such as sponsorships, grants, or institutional backing if your project becomes critical infrastructure. Your goal is to build a project where code, community, and documentation—anchored by a great README—reinforce each other, so no single maintainer has to carry the entire weight alone.

When building CocoIndex, the goal was to solve a very specific pain: keeping AI indexes fresh without drowning in custom data pipelines, so the core engine ships in Rust with a Python-first developer experience and opinionated building blocks for incremental data processing. From day one, the README doubled as a narrative and a playbook—showing real pipelines, quickstarts, and examples—so that people could go from "what is this?" to "I have it running on my data" as fast as possible, which turned early adopters into collaborators. To keep the Discord community warm, the focus has been on fast, friendly responses, sharing work-in-progress features, tagging good first issues from GitHub, and using the server as an open design room rather than a support ticket queue, so that contributors feel like co-builders of the framework, not just users asking for help.

This Week in Rust Trending: Storage, AI Agents, and Real‑World Infra

Linghua Jin — Sat, 13 Dec 2025 02:13:37 +0000

This week's Rust trending list isn't about toy crates or hobby side‑projects. It's about storage engines, AI agents that actually ship code, blockchains, local‑first AI, and serious infra you can run in production (or soon enough).

Here's a quick tour of the top 10 trending Rust repos, what they do, and why developers are starring them so hard right now.

RustFS: Rust Takes on S3 Object Storage

RustFS is a high‑performance, S3‑compatible distributed object storage system written in Rust. It targets data lakes, AI pipelines, and big‑data workloads with support for migration and coexistence with other S3 platforms like MinIO and Ceph. RustFS leans heavily on Rust's concurrency and memory safety story, aiming for very small 4KB object latencies and advertises up to 2.3x performance vs MinIO in its own benchmarks.

Why it's trending:

It hits a real, expensive problem: on‑prem and cloud‑agnostic S3‑compatible storage with an Apache‑2.0 license.
The combination of "S3‑compatible", "Rust‑based", and "faster than MinIO" is catnip for infra engineers.

Goose: AI Agent That Actually Touches Your Codebase

Goose is an open‑source, extensible AI agent that goes far beyond autocomplete. It installs dependencies, edits your files, runs tests, and integrates with multiple LLMs while evolving toward a Rust core and a rich tool/extension ecosystem. Goose is designed to be the glue between your repo, your terminal, and your models.

Why it's trending:

It embodies the "agents that do, not just chat" movement: people want assistants that can refactor, fix, and test real projects.
A clear roadmap, active community, and focus on reproducible, scriptable workflows make it appealing for serious teams, not just tinkerers.

Polkadot SDK: Full‑Stack Blockchain Toolkit in Rust

The Polkadot SDK is the official Rust toolkit for building on the Polkadot ecosystem. It bundles networking, consensus, Substrate primitives, and tooling so teams can build full blockchains and parachains. The project ships opinionated versioned releases and tooling like a CLI to manage SDK versions.

Why it's trending:

It's one of the most mature "build your own chain in Rust" stacks, backing production networks and serious research.
With a consolidated SDK instead of a dozen scattered crates, teams get a more approachable entry point into Polkadot development.

Turso: SQLite‑Compatible, Rust‑Powered Database

Turso is an in‑process SQL database written in Rust that is intentionally compatible with SQLite at the file, C API, and SQL levels. On top of that, it layers features like async I/O, change data capture, multi‑language bindings, and vector operations to support modern, edge‑heavy and AI workloads.

Why it's trending:

Developers get the familiarity of SQLite with a Rust engine designed for edge, serverless, or embedded scenarios.
Built‑in support for CDC and vectors lines up perfectly with the "AI plus event‑driven plus edge" stack a lot of teams are chasing right now.

CocoIndex: Data Transformation for AI Context

CocoIndex is a data transformation framework focused on AI and context‑heavy workloads. It uses a Rust core to deliver ultra‑fast, incremental processing between sources and targets like object storage, databases, and vector stores. The design emphasizes reproducible, composable pipelines for building and maintaining AI‑ready indexes and knowledge graphs.

Why it's trending:

It speaks directly to teams building RAG, search, and analytics systems who need low‑latency indexing instead of generic batch ETL.
Its "building blocks" model for sources, transforms, and sinks makes it feel like Lego for AI data infrastructure.

Rustlings: The Perennial Rust On‑Ramp

Rustlings is the canonical "little exercises" repo that teaches Rust by making you fix compiler errors and complete tiny challenges. It ships with an installer, rust‑analyzer integration, quizzes, and a curated curriculum that tracks the language's evolution.

Why it's trending (again):

Every new wave of devs trying Rust ends up here; it's often the first repo people star when starting their Rust journey.
The exercises mirror real compiler messages and tooling, which makes the learning path feel tightly aligned with day‑to‑day Rust work.

Magisk: Android Modding Meets Modern Systems Code

Magisk is the legendary Android "magic mask": a systemless rooting and module platform that lets power users and developers customize their devices without touching the system partition. Around this ecosystem, Rust is increasingly used for performance‑ and security‑sensitive components, reflecting a broader trend in Android tooling.

Why it's trending:

Magisk is widely used in the real world; when its ecosystem touches Rust, it showcases Rust in practical, high‑impact scenarios.
It sits at the intersection of low‑level systems work, security, and hacker culture—exactly where Rust enthusiasts love to play.

Yew: Build Web Apps in Rust + WASM

Yew is a Rust/WASM framework for building client‑side web apps with a component‑based model, inspired by the ergonomics of modern JavaScript frameworks. It gives Rust developers a way to write frontends without switching to TypeScript, while still targeting the browser with WebAssembly.

Why it's trending:

Yew is the go‑to answer when someone asks "Can I build a SPA in Rust?" and continues to polish its DX and performance story.
It offers a unified language stack for teams heavily invested in Rust, especially in full‑stack or embedded‑plus‑web contexts.

Vibe Kanban: A Kanban Board for AI Coding Agents

Vibe Kanban is a Kanban board designed specifically to orchestrate AI coding agents. Instead of traditional tickets, you manage tasks that are executed by agents, with support for running them in parallel or sequence, spinning up dev servers, and managing configuration via MCP‑style tooling.

Why it's trending:

It captures a new workflow: engineers supervising fleets of agents rather than writing every line of code themselves.
The "just run it with npx and connect GitHub" story makes it easy to try on a real repo in minutes.

Meeting Minutes (Meetily): Local‑First AI Meeting Assistant

meeting-minutes (Meetily) is an open‑source AI meeting assistant that runs locally, giving you recording, live transcription (Parakeet/Whisper), speaker diarization, and summarization via Ollama or cloud LLMs. It's built in Rust and targets macOS, Windows, and Linux with a strong privacy‑first, self‑hosted posture.

Why it's trending:

It solves a real, universal problem—meeting notes—without sending your audio to someone else's cloud by default.
Combining Rust for performance with GPU‑accelerated speech models and local LLMs hits the sweet spot of "AI you actually own."

Fresh: A New TUI Editor Written in Rust

fresh is a terminal text editor that aims to be easy to use, powerful, and fast, all written in Rust. It targets developers who want a modern alternative to Vim/Emacs/Helix but with a gentler learning curve and sensible defaults.

Why it's trending:

"New Rust TUI editor" is practically a genre, and performance‑obsessed devs love trying editors that promise speed and simplicity.
It offers a familiar terminal workflow with a more approachable UX than the classic modal or Lisp‑heavy editors.

The Big Picture: What This Week Says About Rust

Across these trending projects, a few clear themes emerge:

Theme	Examples	Why it matters
AI‑native tooling	Goose, CocoIndex, Vibe Kanban, Meeting Minutes	Rust is becoming a backbone for serious, stateful AI infra and agents.
Infra you can run today	RustFS, Turso, Magisk ecosystem	These are real workloads—storage, databases, Android systems—built in Rust.
Developer experience focus	Rustlings, Yew, Fresh	Better learning paths and tools keep lowering the barrier into Rust.

Rust is no longer just "the safe systems language with cool borrow‑checker memes." This week's trending list shows it powering object storage, blockchains, databases, AI agents, and local‑first productivity tools—all projects that developers can run today, not just read about.

If you're Rust‑curious, this list is a great set of repos to star, clone, and learn from next.

Forem: Linghua Jin

Your AI Coding Agent is Blind. Here's the Fix.

The Real Reason Your AI Agent Keeps Getting It Wrong

The Fix: AST-Based Semantic Search

What is cocoindex-code?

Wait, Isn't That What LSP Does?

How to Get Started (Seriously, 1 Minute)

Real-World Impact

Why This Matters for the Future of AI Coding

Try It

I Built a Code AST MCP That Saves 70% Tokens and Went Viral (54K+ Views)

The Problem: Coding Agents Are Burning Tokens

The Solution: AST-Based Semantic Code Search

1-Minute Setup - No Config Needed

How It Works Under the Hood

Why It Went Viral

Supported Languages

What's Next

I Built a Tiny MCP That Understands Your Code and Saves 70% Tokens

The Core Idea: AST + Incremental Indexing

What You Get Out of the Box

1-Minute Setup for Claude, Codex, and OpenCode

Claude

Codex

OpenCode

How the search MCP Tool Works

Supported Languages and Smart Defaults

Embeddings: Start Free, Scale Later

What About Huge / Enterprise Codebases?

If You Want to Try It, Here's the Ask

I Stopped Manually Documenting My Repos: Now They Generate Their Own Wikis

The Idea: A Self-Updating Wiki For Every Project

Step 1: Setup – Point the Pipeline at Your Code

Step 2: The App – Treat Each Directory as a Project

Step 3: Extract Structured File Info with an LLM

Step 4: Aggregate Files into a Project-Level Summary

Step 5: Generate Markdown (With Mermaid Diagrams)

Step 6: Wire It All Together

Why This Pattern Is Powerful

Try It Yourself

I Built a Real-Time HackerNews Trend Radar With AI (And It Runs Itself)

Why HN Is a Goldmine (If You Can Structure It)

Architecture: From HN Firehose to Queryable Topics

Step 1: Turning HackerNews Into a First-Class Incremental Source

Step 2: Using an LLM to Extract Topics From Every Thread and Comment

Step 3: Indexing Into Postgres for Fast Topic Queries

Step 4: Query Handlers - From "Cool Pipeline" to Real Product

4.1 search_by_topic(topic): "Show Me All Claude Mentions"

4.2 get_threads_for_topic(topic): Rank Threads by Topic Score

4.3 get_trending_topics(limit=20): The Actual Trend Radar

Running It in Real Time

What You Can Build on Top of This (Beyond "Just HN")

Want to Try It Yourself?

Build an AI-Powered Competitive Intelligence Monitor

What the Project Does

Core Architecture

Data Model: The CompetitiveEvent Class

Custom Tavily Source Connector

The CocoIndex Pipeline Definition

Query Handlers for Analysis

Getting Started

Why This Approach Matters

Get the Code

Laksh-star / competitive-intelligence

AI-powered competitive intelligence monitor using CocoIndex, Tavily Search, and LLM extraction

Competitive Intelligence Monitor

What This Does

Prerequisites

Stop Writing Fragile Prompts: Extract Structured Data from PDFs with DSPy + CocoIndex

The Problem with Prompt Engineering

Enter DSPy + CocoIndex

What We're Building

Step 1: Define Your Schema with Pydantic

Step 2: Create the DSPy Signature

Step 3: Build the Extractor Module

Step 4: Wire It Into CocoIndex

Step 5: Define the Flow

Why This Approach Wins

Run It

Key Takeaways

How the `search` MCP Tool Works

4.1 `search_by_topic(topic)`: "Show Me All Claude Mentions"

4.2 `get_threads_for_topic(topic)`: Rank Threads by Topic Score

4.3 `get_trending_topics(limit=20)`: The Actual Trend Radar