Forem: eyanpen

Why Does Semantic Chunking Need an Embedding API?

eyanpen — Mon, 04 May 2026 05:54:39 +0000

Fixed-length chunking requires no external services, yet semantic chunking absolutely needs an Embedding API — why?

The Short Answer

The core idea of semantic chunking is to split text at semantic boundaries. Determining whether "two pieces of text belong to the same topic" requires converting text into vectors and computing similarity — that's exactly what the Embedding API does.

Traditional Chunking vs Semantic Chunking

Dimension	Fixed-Length / Recursive	Semantic Chunking
Split criteria	Character count, token count, delimiters	Semantic similarity between adjacent sentences
Requires Embedding	❌ No	✅ Yes
Split quality	May break in the middle of a topic	Splits at topic transitions, preserving semantic coherence

Fixed-length chunking is like measuring paper with a ruler — regardless of content, it cuts every 500 characters. Semantic chunking is like a reader who, after finishing a paragraph, asks "is the next part still about the same thing?" If not, that's where the cut goes.

Two Mainstream Semantic Chunking Strategies

Strategy 1: Adjacent Similarity (Kamradt Method)

Core idea: Compute semantic distances between adjacent sentences and split where distances spike.

Process:
1. Split text into small sentences
2. For each sentence, concatenate buffer_size adjacent sentences as context
3. Call Embedding API to get vectors for each combined sentence
4. Compute cosine distances between adjacent combined sentences
5. Use binary search to find a threshold, split where distance exceeds it

Pseudocode:

# Step 1: Build context windows
for i, sentence in enumerate(sentences):
    combined = ""
    for j in range(max(0, i - buffer_size), i):
        combined += sentences[j] + " "
    combined += sentence
    for j in range(i + 1, min(n, i + 1 + buffer_size)):
        combined += " " + sentences[j]
    combined_texts.append(combined)

# Step 2: Get embeddings for all combined sentences (one batch call)
embeddings = embedding_client.embed_texts(combined_texts)
embedding_matrix = np.array(embeddings)

# Step 3: Compute cosine distances only between adjacent sentences
distances = []
for i in range(len(sentences) - 1):
    similarity = dot(embedding_matrix[i], embedding_matrix[i + 1])
    distances.append(1 - similarity)  # Higher distance = greater topic difference

# Step 4: Binary search for threshold targeting total_size / avg_chunk_size cuts
threshold = binary_search_threshold(distances, target_cuts)

# Step 5: Split where distance exceeds threshold
breakpoints = [i for i, d in enumerate(distances) if d > threshold]

Intuition: Imagine reading an article sentence by sentence, asking yourself after each one: "Is the next sentence still about the same thing?" When you feel the topic has jumped, you cut there.

Key characteristic: Only looks at adjacent relationships. It only computes the distance between sentence[i] and sentence[i+1] — a local greedy strategy.

Strategy 2: Cluster Optimal Segmentation (Dynamic Programming Method)

Core idea: Build a similarity matrix between all sentence pairs and use dynamic programming to find the segmentation that maximizes intra-cluster similarity.

Process:
1. Split text into small sentences
2. Call Embedding API to get vectors for all sentences
3. Build an N×N similarity matrix
4. Normalize the matrix by subtracting the mean (prevents degeneration into one giant cluster)
5. Use dynamic programming to find the optimal segmentation

Pseudocode:

# Step 1: Get embeddings for all sentences (note: no buffer concatenation)
embeddings = embedding_client.embed_texts(sentences)
embedding_matrix = np.array(embeddings)

# Step 2: Build N×N similarity matrix
similarity_matrix = dot(embedding_matrix, embedding_matrix.T)

# Step 3: Mean normalization to prevent DP from putting everything in one cluster
mean_sim = mean(upper_triangle(similarity_matrix))
similarity_matrix -= mean_sim
fill_diagonal(similarity_matrix, 0)

# Step 4: Dynamic programming for optimal segmentation
# dp[i] = maximum intra-cluster similarity sum for the first i+1 sentences
for i in range(n):
    for size in range(1, i + 2):
        start = i - size + 1
        if cluster_size(start, i) > max_chunk_size and size > 1:
            break
        reward = sum(similarity_matrix[start:i+1, start:i+1])
        if start > 0:
            reward += dp[start - 1]
        dp[i] = max(dp[i], reward)

# Step 5: Backtrack to get optimal segmentation
clusters = backtrack(segmentation)

Key characteristic: Globally optimal. It considers relationships between all sentence pairs and uses DP to find the overall best segmentation.

Deep Comparison of the Two Strategies

Fundamental Algorithmic Differences

Dimension	Kamradt (Adjacent Similarity)	Cluster (Dynamic Programming)
Scope	Local — only adjacent sentences	Global — all sentence pairs
Decision method	Greedy: cut when distance exceeds threshold	Optimization: maximize intra-cluster similarity
Threshold	Binary search for target cut count	No threshold needed, DP decides automatically
Context enhancement	✅ buffer_size concatenation	❌ Uses raw sentences directly
Size constraints	avg_chunk_size + max_chunk_size dual constraint	max_chunk_size hard constraint

The core difference in one sentence:

Kamradt asks: "Is there a topic transition between these two adjacent sentences?"
Cluster asks: "Which grouping makes sentences within each group most similar to each other?"

An Intuitive Example

Consider 6 sentences with the following topic distribution:

Sentence 1: Discussing Apple's earnings report
Sentence 2: Discussing Apple's new products
Sentence 3: Discussing the weather forecast
Sentence 4: Discussing tomorrow's temperature
Sentence 5: Discussing Apple's stock price
Sentence 6: Discussing Apple's competitors

Kamradt's approach: Compare adjacent pairs

Sentence 2→3: Topic jump (Apple → weather), cut!
Sentence 4→5: Topic jump (weather → Apple), cut!
Result: [1,2] [3,4] [5,6]

Cluster's approach: The global similarity matrix shows sentences 1,2,5,6 are highly similar to each other

But since DP requires contiguous segmentation (can't skip around), it can only cut contiguous spans
Result is likely also [1,2] [3,4] [5,6], but the reasoning is different

The key difference emerges when boundaries are fuzzy:

Consider an article that gradually transitions from "EV technology" to "energy policy":

Sentence 1: Tesla released a new generation of battery technology
Sentence 2: The new battery's energy density improved by 50%
Sentence 3: Higher energy density means longer driving range
Sentence 4: Range anxiety has been a barrier for consumers buying EVs
Sentence 5: The government introduced charging station subsidies to address this
Sentence 6: Subsidies cover both residential and commercial charging facilities
Sentence 7: Commercial charging uses time-of-use electricity pricing
Sentence 8: Time-of-use pricing is a key component of electricity market reform

What Kamradt sees (adjacent distances):

1→2: 0.08  (both about batteries)
2→3: 0.10  (battery → range, very close)
3→4: 0.12  (range → range anxiety, very close)
4→5: 0.15  (consumers → government policy, slightly far but not outstanding)
5→6: 0.09  (both about subsidies)
6→7: 0.13  (subsidies → pricing, somewhat far)
7→8: 0.11  (both about pricing)

No single distance clearly "spikes" — the topic slides gradually. Kamradt's binary search struggles to find a reasonable threshold, potentially producing suboptimal splits like [1-4][5-8] or [1-3][4-6][7-8].

What Cluster sees (global similarity matrix summary):

        S1    S2    S3    S4    S5    S6    S7    S8
S1      --   0.9   0.7   0.4   0.2   0.1   0.1   0.05
S2           --    0.8   0.5   0.2   0.15  0.1   0.05
S3                 --    0.6   0.3   0.2   0.15  0.1
S4                       --    0.5   0.4   0.3   0.2
S5                             --    0.8   0.6   0.4
S6                                   --    0.7   0.5
S7                                         --    0.8
S8                                               --

The global view clearly shows: sentences 1-3 are highly similar to each other (battery/range technology), sentences 5-8 are highly similar to each other (policy/pricing), and sentence 4 is a transition. DP optimization discovers that [1-3][4-8] or [1-4][5-8] maximizes intra-cluster similarity, producing a more reasonable split.

The essential difference: Kamradt only looks at "the gap between adjacent sentences" — in a gradual transition, each step's gap is small, like the boiling frog metaphor. Cluster looks at "the overall similarity within each group" — even when the transition is smooth, it can still detect that sentence 1 and sentence 8 are essentially unrelated.

Embedding Cost Comparison

This is one of the most important practical differences between the two strategies:

Dimension	Kamradt	Cluster
Embedding input	combined_sentence (with buffer context)	Raw sentences (no buffer)
Embedding call count	N texts, 1 batch call	N texts, 1 batch call
Average text length	Longer (~7 sentences, buffer_size=3)	Shorter (1 sentence)
Total token consumption	Higher (buffer causes input inflation)	Lower (no redundancy)
Post-embedding computation	O(N) — only adjacent distances	O(N²) — full similarity matrix
DP computation	None	O(N × max_cluster_size)

Concrete Numbers (1000 sentences, ~30 tokens each)

Kamradt:

Embedding input: 1000 combined_sentences, each ~7×30 = 210 tokens
Total token consumption: 1000 × 210 = 210,000 tokens
Distance computation: 999 dot products → negligible
Memory: 1000 × embedding_dim matrix

Cluster:

Embedding input: 1000 raw sentences, each ~30 tokens
Total token consumption: 1000 × 30 = 30,000 tokens
Similarity matrix: 1000 × 1000 = 1 million floats (~8MB)
DP computation: O(1000 × max_cluster_size) iterations

Conclusions:

Embedding API cost: Kamradt consumes ~7x more tokens (due to buffer concatenation), higher API cost
Compute resources: Cluster's O(N²) matrix and DP are more expensive on local CPU/memory
Network latency: Same for both (both use 1 batch call, or multiple calls based on batch_size)

Large-Scale Scenario (100,000 sentences)

Metric	Kamradt	Cluster
Total embedding tokens	~21 million tokens	~3 million tokens
API calls (batch_size=500)	200	200
Similarity computation	99,999 dot products	10 billion dot products (N² matrix)
Memory usage	~400MB (embedding matrix)	~40GB (N² similarity matrix) ⚠️

At 100K sentences, Cluster's N² matrix will blow up memory — this is its hard limitation. In practice, Cluster is better suited for medium-length documents (hundreds to thousands of sentences), while Kamradt can handle any length.

Split Quality Comparison

Scenario	Kamradt	Cluster
Clear topic boundaries	✅ Excellent, obvious distance spikes	✅ Excellent
Gradual topic transitions	⚠️ May fail to find split points	✅ Global optimization still finds best split
Short documents (<50 sentences)	✅ Fast	✅ Higher quality
Long documents (>10K sentences)	✅ Linear scaling	❌ Memory explosion
Very short sentences	⚠️ Needs buffer for context	⚠️ Short sentence embeddings are low quality

How to Choose?

Your Scenario	Recommended	Reason
Unknown document length, need general solution	Kamradt	Linear complexity, won't blow memory
Short documents (<2000 sentences), want optimal splits	Cluster	Globally optimal, higher quality
Embedding API charges per token	Cluster	No buffer inflation, 7x fewer tokens
Limited local compute resources	Kamradt	O(N) computation, memory-friendly
Fuzzy topic boundaries, need precise splits	Cluster	DP global optimization is more robust

Why Can't Other Methods Replace Embedding?

Alternative	Problem
Keyword overlap / TF-IDF	Cannot capture synonyms or contextual semantics ("automobile" and "vehicle" would be considered unrelated)
Rule-based delimiters (paragraphs, periods)	One paragraph may contain multiple topics; different paragraphs may discuss the same topic
LLM direct judgment	Too expensive, high latency, unsuitable for batch processing tens of thousands of sentences

Embedding maps text into a high-dimensional semantic space where semantically similar texts have small vector distances and dissimilar texts have large distances. This is currently the optimal approach for semantic similarity measurement, balancing cost, speed, and quality.

buffer_size: The Role of the Context Window

Semantic chunking has a key parameter buffer_size (default: 3) that determines how much context is concatenated when generating embeddings for each sentence.

# Concatenation logic
for each sentence[i]:
    combined = sentence[i-3] + sentence[i-2] + sentence[i-1]  # 3 before
              + sentence[i]                                     # current
              + sentence[i+1] + sentence[i+2] + sentence[i+3]  # 3 after

Key point: buffer_size does not affect the number of Embedding calls — only the length of each input text.

With 10 sentences, whether buffer_size is 1 or 10, you still embed 10 combined_sentences. The difference is how much context each text contains:

buffer_size	Avg sentences per text	Effect
1	~3	Less context, may misjudge
3 (default)	~7	Balance point
10	~21	Rich context, but may exceed model token limit

Note: Embedding models have input length limits (e.g., BGE-M3 max 8192 tokens). If buffer_size is too large, texts get truncated, potentially losing the current sentence's information.

Performance at Scale

Suppose a long document is split into 100,000 sentences:

Texts to embed = 100,000
With batch_size of 500, actual API calls = 100,000 ÷ 500 = 200 HTTP requests

The performance bottleneck is API call count (determined by total sentences and batch_size), independent of buffer_size.

Fallback Strategy: What If Embedding Is Unavailable?

Good system design should account for Embedding service unavailability. The common approach: when Embedding calls fail, automatically fall back to recursive chunking (pure rule-based splitting, no Embedding needed).

This means semantic chunking is an enhancement, not a dependency — the system still works without the Embedding service, just with lower split quality.

Summary

Question	Answer
Why is Embedding needed?	Judging semantic similarity requires vector representations
Can rules replace it?	No, rules cannot capture semantics
Can LLM replace it?	Theoretically yes, but cost and latency are unacceptable
Kamradt vs Cluster core difference?	Local adjacent comparison vs global optimal segmentation
Which has higher Embedding cost?	Kamradt: higher token consumption (buffer inflation); Cluster: higher compute cost (N² matrix)
Which for large documents?	Kamradt — linear complexity, won't blow memory
Which for optimal splits?	Cluster — global DP optimization, but limited to medium-length documents
What if service is unavailable?	Both fall back to rule-based chunking

The Embedding API is the "eyes" of semantic chunking — without it, the chunking algorithm is a blind person cutting a cake. The two strategies "see" text differently: Kamradt is like a line-by-line scanner, Cluster is like an editor with a bird's-eye view. Which to choose depends on your document scale and split quality requirements.

Multiple Independent Questions: Batch Into One Request or Split Into Many? — An Analysis of LLM Concurrent Processing

eyanpen — Sun, 03 May 2026 00:19:18 +0000

When you have 5 unrelated questions, should you pack them into one message to the LLM, or send 5 requests simultaneously? Which is faster?

The Short Answer

Splitting into multiple independent parallel requests is almost always faster.

This isn't a gut feeling — it's determined by the underlying inference mechanism of LLMs. Let's walk through the reasoning from first principles.

1. How LLMs Generate Text: Autoregressive Decoding

To understand this problem, you first need to know how LLMs "write."

LLMs (GPT-4, Claude, etc.) use autoregressive generation: they produce one token at a time, append that token back to the input, then generate the next token. This repeats until generation is complete.

The key insight: Generating N tokens requires N forward passes.

This means:

A 100-token answer requires 100 inference steps
A 500-token answer requires 500 inference steps
Total output length directly determines total latency

2. Batched Request: Output Volumes Stack, Latency Grows Linearly

Suppose you have 5 independent questions, each requiring ~200 tokens to answer.

Approach A: Combine into one request

You stuff all 5 questions into a single message:

Please answer the following questions separately:
1. xxx
2. xxx
3. xxx
4. xxx
5. xxx

The LLM needs to generate total output ≈ 5 × 200 = 1000 tokens. Due to autoregressive decoding, these 1000 tokens are generated sequentially — token #201 must wait for the first 200 to finish.

Total latency ≈ 1000 × per-token generation time

Plus additional overhead:

The LLM must maintain context switches between answers ("now answering question 3")
Longer KV Cache means increasing attention computation at each step
Actual output often exceeds 1000 tokens (formatting, transition phrases, etc.)

3. Split Requests: Parallel Inference, Latency Equals the Slowest One

Approach B: Split 5 questions into 5 independent requests, sent simultaneously

Each request independently generates ~200 tokens. If the server has sufficient concurrent processing capacity (all modern LLM services do), these 5 requests are processed in parallel.

Total latency ≈ max(individual request latencies) ≈ 200 × per-token generation time

Comparison:

Approach	Total output tokens	Actual latency (relative)
Combined request	~1000+	~1000 steps (sequential)
Split into 5 requests	~200 each	~200 steps (parallel)

Theoretical speedup ≈ 5x (equals the number of questions).

4. Why Does Parallelism Work? — Server-Side Continuous Batching

You might ask: doesn't the LLM server have capacity limits? Won't 5 simultaneous requests queue up?

Modern LLM inference engines (vLLM, TensorRT-LLM, TGI, etc.) all implement Continuous Batching:

Multiple requests share the same GPU matrix operation: GPUs excel at parallel computation. Combining tokens from 5 requests into one batch allows a single forward pass to generate one token for each request simultaneously.
Dynamic scheduling: Different requests have different output lengths. Shorter ones finish first, and their slots are immediately given to new requests.
Throughput vs. latency decoupling: Larger batches mean higher GPU utilization and more total tokens processed per unit time.

From the server's perspective:

5 short parallel requests → GPU does 5-way batched inference, producing 5 tokens per step
1 long request → GPU does single-sequence inference, producing 1 token per step

The GPU's parallel computing power is wasted when requests are combined.

5. The Prefill Phase Difference

LLM inference has two phases:

Prefill: Process the input prompt, computing KV Cache for all input tokens. This step can process all input tokens in parallel, with latency roughly linear to input length.
Decode: Generate output token by token. This step is sequential.

With combined requests:

Prefill phase: Longer input (all 5 questions concatenated), longer prefill time
Decode phase: Longer output, longer decode time

With split requests:

Each request's prefill is shorter, and all 5 prefills can run in parallel or pipelined
Each request's decode is shorter, and they run in parallel

Both phases favor splitting.

6. An Often-Overlooked Factor: Quality

Beyond speed, combining requests carries quality risks:

Attention dilution: When an LLM processes multiple unrelated tasks in one generation, its "focus" on each task decreases. Research shows that more irrelevant information in the prompt leads to lower answer quality (the "Lost in the Middle" phenomenon).
Format confusion: Answers to 5 questions easily suffer from numbering errors, omissions, or mismatched responses.
Error propagation: If the answer to question 2 goes wrong, the LLM may be influenced in subsequent answers (autoregressive "inertia").

Split requests completely isolate context, giving each question the LLM's "full attention."

7. When Is Combining Actually Better?

To be fair, there are a few scenarios where combining may be more appropriate:

Hidden correlations between questions: Even if you think they're independent, the LLM might give more consistent answers seeing the full picture (e.g., different sections of the same report).
Strict API rate limits: If your API quota is 3 requests per minute, you have no choice but to combine 5 questions.
Network latency far exceeds generation time: If each API call has 2 seconds of network round-trip but generation only takes 0.5 seconds, splitting 5 times (5 × 2s = 10s network overhead) might exceed the combined generation time. But this is rare in practice — modern API network latency is typically 100-300ms, far less than generation time.
Extremely short answers: If each question only needs a word or two, prefill overhead dominates, and combining can reduce redundant prefill costs.

8. How to Verify This Yourself

If you want to test this empirically:

import asyncio
import time
import aiohttp

async def ask_single(session, question):
    start = time.time()
    # Call LLM API
    resp = await session.post(API_URL, json={"prompt": question})
    result = await resp.json()
    return time.time() - start

async def benchmark():
    questions = ["Question 1", "Question 2", "Question 3", "Question 4", "Question 5"]

    async with aiohttp.ClientSession() as session:
        # Approach A: Combined
        start = time.time()
        combined = "Please answer separately:\n" + "\n".join(questions)
        await ask_single(session, combined)
        time_combined = time.time() - start

        # Approach B: Parallel
        start = time.time()
        await asyncio.gather(*[ask_single(session, q) for q in questions])
        time_parallel = time.time() - start

    print(f"Combined: {time_combined:.2f}s")
    print(f"Parallel: {time_parallel:.2f}s")
    print(f"Speedup: {time_combined / time_parallel:.1f}x")

In practice, 5 moderately complex independent questions typically achieve 3-5x speedup with parallel requests.

Summary

Dimension	Combined request	Split parallel requests
Generation speed	Slow (sequential output of all answers)	Fast (parallel generation, latency = slowest)
GPU utilization	Low (single-sequence inference)	High (batched parallel inference)
Answer quality	May degrade (attention dilution)	Better (isolated context)
API calls	1	N
Best for	Rate-limited / extremely short answers	Independent questions needing detailed answers

Core principle in one sentence: LLM's autoregressive mechanism means output is sequential; combining requests = forcing all outputs into a single serial stream; splitting requests = leveraging server-side parallelism to generate multiple outputs simultaneously. Splitting independent questions is the classic strategy of trading space (concurrent slots) for time.

What Is GraphRAG Really Doing? — A Deep Dive into Microsoft's Blog Post

eyanpen — Fri, 24 Apr 2026 11:57:01 +0000

Original: GraphRAG: Unlocking LLM discovery on narrative private data - Microsoft Research

In early 2024, Microsoft published a technical blog post. The core message boils down to one sentence: Traditional RAG falls short with complex data, and GraphRAG fills the gap using knowledge graphs + graph clustering.

This isn't an academic paper — it reads more like a "tech pitch" aimed at technical decision-makers and engineers. Let me break it down.

Where Does Traditional RAG Fall Short?

To understand what GraphRAG solves, we need to start with the pain points of traditional RAG. The article highlights two scenarios where traditional RAG struggles:

Information That Can't Be Connected

Imagine asking an AI: "What has Novorossiya done?"

Traditional RAG takes the word "Novorossiya" and runs a vector search. But among the 10 text chunks retrieved, none directly mentions that name — the answer is scattered across different documents, connected only through indirect relationships between entities. Vector search only finds text that "looks similar"; it can't handle this kind of reasoning that requires "jumping" between connections.

GraphRAG works differently: it locates the Novorossiya node in the knowledge graph, then traverses along relationship edges — actions, goals, related organizations — and assembles the complete answer.

Put simply, vector retrieval is "local matching," while real-world knowledge is often connected indirectly through chains of entity relationships.

Can't Answer "Big Questions"

Another example: "What are the top 5 themes in this dataset?"

Traditional RAG is stumped — the word "themes" is too broad. Vector search doesn't know which direction to look, and ends up matching some irrelevant text that happens to contain the word "theme." The answer naturally goes off track.

This is fundamentally a granularity problem: vector RAG retrieves at the text chunk level, but "overall themes" require a macro-level understanding of the entire dataset. No single chunk can support that kind of answer.

GraphRAG handles this easily with pre-built community clusters and community summaries, extracting themes directly from the macro structure.

How Does GraphRAG Work?

The entire process has two phases: offline indexing, then online question answering.

Offline Indexing: Three Steps

Raw Documents
    │
    ▼
┌─────────────────────────────┐
│ Step 1: Entity & Relationship│  LLM processes documents chunk
│ Extraction                   │  by chunk, extracting all
│                              │  entities (people, places,
│                              │  organizations, etc.) and
│                              │  their relationships
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│ Step 2: Knowledge Graph      │  Assemble extracted entities
│ Construction                 │  and relationships into a
│                              │  complete graph structure
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│ Step 3: Community Detection  │  Perform bottom-up hierarchical
│ & Summarization              │  clustering on the graph (e.g.,
│                              │  Leiden algorithm), generate
│                              │  LLM summary reports for each
│                              │  community
└─────────────────────────────┘

In short: first let the LLM extract all the people, events, things, and their relationships from the documents, assemble them into a large graph, then cluster the graph into groups and write a summary for each group.

Online Answering: Choose Strategy by Question Type

Question Type	How to Find the Answer
Specific questions (e.g., "What has Novorossiya done?")	Locate entity in graph → traverse relationships → collect related text → generate answer
Macro questions (e.g., "Top 5 themes")	Use community summaries directly → aggregate layer by layer → generate global answer

Technical Points Worth Digging Into

Why Use LLM for Graph Construction Instead of Traditional NLP?

The traditional approach uses NER (Named Entity Recognition) + relation extraction models, but these have hard limitations: you need to predefine entity types and relation types, they break when you switch domains, and they can't capture implicit relationships.

LLM advantages are clear:

Zero-shot capability — no need to train separately for each domain
Can read between the lines — for example, extracting the implicit "government attention" relationship from "the Attorney General's office reported the creation of Novorossiya"
Not constrained by schema — let the LLM discover entity and relationship types on its own

The trade-off is straightforward: LLM calls are expensive, and the indexing phase needs to process the entire dataset, so computational costs are significant.

Community Detection — GraphRAG's Killer Feature

Many approaches use knowledge graphs to enhance RAG, but what truly sets GraphRAG apart is community detection:

Uses algorithms like Leiden to partition the knowledge graph into multi-level communities (think of them as "topic clusters")
Pre-generates an LLM summary report for each community
Different community levels correspond to different levels of abstraction; choose the right granularity when answering questions

This is the secret behind its ability to answer "big questions" — no need to traverse the entire graph on the fly, just look up the pre-written summaries.

When generating community reports, the LLM receives CSV tables of entities and relationships within that community: an Entities table (entity ID, name, description), a Relationships table (source, target, description, combined_degree), and an optional Claims table. Relationships are sorted by combined_degree in descending order, prioritizing the most important ones, with truncation when the token limit is exceeded.

Provenance — Every Statement Is Traceable

GraphRAG places special emphasis on provenance. The complete evidence chain looks like this:

User Query
    → GraphRAG Answer + [Data: Entities (ID), Relationships (ID)]
        → Relationship IDs point to specific edges in the knowledge graph
            → Edges link back to specific passages in the original source documents

Answer → entities/relationships in the graph → original documents — fully traceable end to end. For enterprise applications, this capability is critical — you can verify every claim the AI makes.

How Were the Experiments Conducted?

Dataset

They used the VIINA dataset (violence information from news articles), chosen deliberately:

Involves multi-party conflict with fragmented information — complex enough
Includes news sources from both Russian and Ukrainian sides with opposing viewpoints and contradictory information
Data from June 2023, ensuring it's not in the LLM's training set
Thousands of articles, far exceeding context window limits — can't be handled without RAG

Evaluation Results

Four metrics were used for scoring:

Metric	What It Measures	How It's Evaluated
Comprehensiveness	How complete is the answer	LLM scorer pairwise comparison
Human Empowerment	Does it provide sources for verification	LLM scorer pairwise comparison
Diversity	Does it answer from multiple perspectives	LLM scorer pairwise comparison
Faithfulness	Does it hallucinate	SelfCheckGPT absolute measurement

The results are interesting: GraphRAG significantly outperforms traditional RAG on the first three metrics, but they're roughly equal on faithfulness. In other words, GraphRAG's improvement is mainly in "finding more comprehensively," not in "hallucinating less."

Don't Just Look at the Strengths — Know the Limitations Too

This is a pitch piece after all, so it naturally emphasizes the positives. A few caveats to keep in mind:

High indexing cost — Every document chunk requires an LLM call to extract entities and relationships. For large datasets, this could take hours or even days. With GPT-4 level models, API costs are considerable.

Incremental updates are a hard problem — The article doesn't mention what happens when data changes. In practice, new documents require re-extraction and merging, community structures may change as a result, requiring re-clustering and re-generating summaries. There's no good engineering solution for this yet.

Extraction quality depends on the LLM — LLM entity and relationship extraction isn't 100% accurate. It may miss implicit entities, get relationships wrong, and different models produce varying extraction quality with inconsistent results.

Queries will be slower — Graph traversal + LLM generation has a longer pipeline than simple vector retrieval + LLM generation, so latency is naturally higher.

Not every question needs it — The article itself acknowledges that for simple factual queries (like "What is Novorossiya?"), traditional RAG is sufficient. GraphRAG's advantages are concentrated in multi-hop reasoning and global summarization scenarios.

An Analogy to Build Your Intuition

Imagine you're a new employee at a company, and you want to understand "the most important project developments in the last three months."

Traditional RAG is like searching through a filing cabinet: You walk into the archive room and search using "project developments" as a keyword. You find dozens of files scattered across different drawers — meeting minutes, emails, reports. You have to piece the fragments together yourself.

GraphRAG is like asking a colleague who knows everything: They've not only read every document but also remember that "Zhang San's Project A and Li Si's Project B are actually related," and know that "last month's budget adjustment affected three departments." They can give you an organized, complete answer right away.

	Traditional RAG	GraphRAG
How it works	Search keywords, find relevant passages	Build a relationship network first, then answer along relationships
Good at	"What is X?" "How to do X?"	"What's the relationship between X and Y?" "What's the overall picture?"
Analogy	A librarian helping you find books	A detective connecting clues into a complete story
Weakness	Fragmented, lacks global perspective	Building the relationship network takes time and compute

Key Takeaways

GraphRAG doesn't solve the "search more accurately" problem — it solves the "search dimension" problem — expanding from text similarity to entity relationships and global structure.
The knowledge graph is the means; community clustering is the real innovation — Many approaches use graphs to enhance RAG, but community detection + pre-summarization is GraphRAG's unique weapon for global queries.
Provenance is the foundation of trust — Every assertion can be traced back to the original document. Enterprise applications can't do without this.
The trade-off is indexing cost — Using LLMs to process all data for graph construction is much more expensive than simple vectorization. This must be weighed when deploying in production.
Not a replacement, but a complement — Use GraphRAG for complex reasoning and global analysis, traditional RAG for simple factual queries. In real systems, combining both is the right approach.

The Biggest Pitfall in GraphRAG: One Entity, Seven Identities

eyanpen — Fri, 24 Apr 2026 11:54:16 +0000

You thought the hardest part of GraphRAG was "building the graph." In reality, the hardest part is "assigning entity types" — even when you've predefined a strict type schema.

1. A Real-World Dataset

We ran GraphRAG entity extraction on 3GPP TS 23.502 (5G Core Network signaling procedure specification). This document is about 700+ pages and one of the most critical standards in the telecom domain.

The results were painful:

A total of 8,873 distinct entities were extracted (deduplicated by title)
1,123 entities were assigned 2 or more types — 12.7% of the total
The most extreme case, PMIC, was classified into 7 different types: ARCHITECTURE_CONCEPT, DATA_TYPE, INFORMATION_ELEMENT, MANAGEMENT_ENTITY, NETWORK_ELEMENT, PROCEDURE, PROTOCOL

Note that this experiment already used a strictly predefined entity type schema, with the prompt explicitly constraining the LLM to only use the specified type set. In other words, this isn't chaos caused by "no constraints" — it's chaos that persists even after constraints are applied.

What's worse, these "type conflicts" don't occur across different documents — they happen within the same document and even within the same chunk. When the LLM reads a minimal text segment, even with explicit type constraints, it still assigns different types to the same entity.

We found 63 text_unit-level overlapping conflicts — the same entity annotated with two different types within the same text block. For example:

Entity	Labeled as	Also labeled as
AF	ORGANIZATION	NETWORK_FUNCTION
NRF	INTERFACE	NETWORK_FUNCTION
5G SECURITY CONTEXT	SECURITY_ELEMENT	ARCHITECTURE_CONCEPT
HPLMN	NETWORK_FUNCTION	ORGANIZATION
SERVICE REQUEST	INFORMATION_ELEMENT	PROCEDURE

This isn't the LLM making rookie mistakes, nor is the schema poorly designed. Think about it: AF (Application Function) genuinely is both a "network function" and an "organizational role"; NRF is both a "network function" and exposes "interfaces." These types are all in our predefined schema, and the LLM picks a "legal" type every time — it just picks different legal types for the same entity. The problem isn't that the LLM judged wrong, nor that the schema isn't strict enough — it's that real-world entities are inherently not single-typed.

2. Why Is This Problem So Hard?

2.1 Entities Are Inherently Multi-Faceted

In 3GPP specifications, the term AMF (Access and Mobility Management Function):

In architecture diagrams, it's a NETWORK_FUNCTION
In signaling procedures, it's a participant in a PROCEDURE
In deployment descriptions, it's a NETWORK_ELEMENT
In interface definitions, it's an endpoint of an INTERFACE

The same entity plays different roles in different contexts. This isn't a bug — it's reality.

2.2 LLM Type Judgment Depends on the Context Window

GraphRAG entity extraction is performed chunk by chunk. Each text_unit is roughly a few hundred tokens, and the LLM can only see that small segment.

The same entity PDU SESSION ESTABLISHMENT:

In a chunk describing signaling procedures, the LLM classifies it as PROCEDURE
In a chunk describing message formats, the LLM classifies it as INFORMATION_ELEMENT

Both judgments are correct, but they conflict when merged into the knowledge graph.

2.3 No Matter How Good the Schema, Type Boundaries Are Inherently Fuzzy

We already predefined a type schema, but who defines the boundary between ARCHITECTURE_CONCEPT and NETWORK_FUNCTION? In the 3GPP context, many concepts naturally span multiple categories. POLICY CONTROL is both a "procedure" (PROCEDURE) and an "architectural concept" (ARCHITECTURE_CONCEPT) — both types are in our schema, and the LLM isn't wrong to pick either one.

This isn't a problem of poorly written prompts or imprecise schema definitions — it's a fundamental tension between the granularity of type systems and the complexity of the real world. You can make the schema more fine-grained, but a finer schema only creates more boundary issues, not fewer.

2.4 Scale Amplifies the Problem

Our data shows that among entities with multiple types, the top 20 entities average 4–7 types and are associated with 10–200 descriptions. A core entity like AF has 209 descriptions, 192 text_unit references, and 4 types.

When a knowledge graph contains thousands of such "multi-faceted entities," downstream community detection, relationship reasoning, and summary generation are all affected — because the graph structure is polluted by type noise.

3. How Does the Industry Currently Address This?

Approach 1: Predefined Strict Type System (Schema-First) ⚠️ We Already Tried This

Method: Before extraction, manually define a strict entity type schema and explicitly constrain the LLM in the prompt to only use these types.

Representatives: Microsoft GraphRAG's default configuration, most enterprise knowledge graph projects.

Our actual results: All the data at the beginning of this article was produced under Schema-First mode. We predefined the type set and explicitly constrained it in the prompt — yet 1,123 entities still had multi-type conflicts, and 63 text_unit-level overlapping conflicts persisted.

Why it's not enough:

Schema can constrain the LLM to "only pick from these types," but can't constrain it to "pick only one for the same entity"
Domain concepts are inherently multi-faceted; AF in the 3GPP context genuinely is both NETWORK_FUNCTION and ORGANIZATION — no schema, however strict, changes this fact
Requires domain experts to design the schema — high cost, and you need to redesign for each new domain
Being too strict loses information — forcing AF to be NETWORK_FUNCTION discards its semantics as ORGANIZATION

Conclusion: Schema-First is a necessary condition but not a sufficient one. It reduces the "random naming" problem but doesn't solve the fundamental contradiction of "one entity, multiple identities."

Approach 2: Allow Multi-Types, Post-Processing Merge (Multi-Label + Post-Processing)

Method: Don't limit the number of types during extraction; allow an entity to have multiple types, then merge, deduplicate, and select a primary type through rules or models in post-processing.

Representatives: LlamaIndex's PropertyGraphIndex, some academic research.

Pros:

Preserves multi-faceted entity information
No information loss during extraction

Cons:

Post-processing logic is complex; rules are hard to enumerate exhaustively
"Selecting a primary type" itself requires domain knowledge
Graph complexity increases; query performance degrades

Suitable for: Exploratory analysis, early stages where domain boundaries are uncertain.

Approach 3: Hierarchical Typing

Method: Build a hierarchical type system where, for example, NETWORK_FUNCTION is a subtype of ARCHITECTURE_CONCEPT. Extract at the finest granularity; aggregate by hierarchy during queries.

Representatives: Wikidata's type system, YAGO knowledge base.

Pros:

Balances precision and flexibility
Supports queries at different granularities

Cons:

Designing the hierarchy itself is a major undertaking
LLMs struggle to accurately determine hierarchical relationships during extraction
Cross-domain hierarchies are hard to unify

Suitable for: Large-scale, long-term knowledge graph projects.

Approach 4: Abandon Explicit Types, Use Embeddings (Type-Free + Embedding)

Method: Don't assign discrete type labels to entities; instead, use vector embeddings to represent semantic features. Similar entities naturally cluster in vector space.

Representatives: Some recent research, such as GNN-based entity representation learning.

Pros:

Completely avoids the type conflict problem
Captures subtle semantic differences between entities

Cons:

Loses interpretability — you can't tell users "this is a network function"
Downstream community detection and summary generation need redesign
Difficult to debug

Suitable for: Research projects, scenarios with low interpretability requirements.

Approach 5: Context-Aware Dynamic Typing

Method: Don't fix types during extraction; instead, dynamically determine entity types based on query context. For example, when a user asks about architecture, AF is treated as NETWORK_FUNCTION; when asking about organization, it's treated as ORGANIZATION.

Representatives: Currently mostly in the academic exploration stage.

Pros:

Most aligned with reality — an entity's "identity" truly depends on context
No difficult type decisions needed during extraction

Cons:

Extremely high engineering complexity
Graph structure can't be determined during offline graph building; community detection algorithms are hard to apply
Increased query latency

Suitable for: A research direction for next-generation GraphRAG systems.

4. My Recommendation: Schema-First Foundation + Layered Types + Primary Type Voting + Context Preservation

Our experiments have proven that Schema-First is a necessary starting point — without it, types become even more chaotic. But it alone isn't enough. Based on our hands-on experience with 3GPP documents, I recommend layering a pragmatic post-processing approach on top of Schema-First:

Layer 0: Keep Schema-First (Already in Place)

Continue using the predefined type schema to constrain the LLM. This step is already done; its value lies in keeping types within a finite set, preventing the LLM from freely inventing meaningless types like THINGY or STUFF.

Layer 1: Preserve All Types During Extraction

On top of Schema-First, don't force a single type during extraction. If the LLM picks multiple types from the predefined set, keep them all. Preserve every (entity, type, text_unit) triple. This is the raw signal — once lost, it can't be recovered.

Layer 2: Statistical Voting for Primary Type

For each entity, count how many times it's annotated as each type across all text_units, and select the most frequent as the primary type.

Taking AF as an example:

NETWORK_FUNCTION: 150 occurrences → primary type
ORGANIZATION: 30 occurrences
ARCHITECTURE_CONCEPT: 20 occurrences
NETWORK_ELEMENT: 9 occurrences

The primary type is used for the knowledge graph's main structure, community detection, and default queries.

Layer 3: Preserve Alternative Types as Properties

Other types aren't discarded — they're stored as the entity's alternative_types property, available for use during queries as needed.

{
  "title": "AF",
  "primary_type": "NETWORK_FUNCTION",
  "alternative_types": ["ORGANIZATION", "ARCHITECTURE_CONCEPT", "NETWORK_ELEMENT"],
  "type_distribution": {
    "NETWORK_FUNCTION": 150,
    "ORGANIZATION": 30,
    "ARCHITECTURE_CONCEPT": 20,
    "NETWORK_ELEMENT": 9
  }
}

Layer 4: Type Conflict Detection and Manual Review

For text_unit-level overlapping conflicts (same entity labeled as different types within the same chunk), flag them as candidates for review. These 63 conflicts are the most worth manually checking — they often reveal blind spots in the type system design.

What's the Cost?

Increased storage: Each entity stores multiple types and distribution info; graph data volume increases by roughly 20–30%.
No change to extraction: No need to modify prompts or extraction pipelines; no additional cost.
Post-processing development needed: The voting, merging, and conflict detection pipeline requires additional development — roughly 2–3 days of engineering effort.
Slightly more complex queries: The query layer needs to decide whether to use the primary type or all types, but this logic can be encapsulated.
Can't be fully automated: Text_unit-level conflicts still require human judgment, but the volume is manageable (only 63 in our case).

5. Final Thoughts

GraphRAG papers and blog posts always focus on the flashy capabilities like "community detection" and "global queries," but when it comes to real-world deployment, entity type chaos is the first roadblock.

One TS 23.502 document, 8,873 entities, 1,123 with multi-type conflicts — and this is after applying Schema-First constraints. This isn't an edge case; it's the norm for all complex domain documents. Predefined type schemas are necessary but far from sufficient.

There's no silver bullet for this problem. But at least we can: build on Schema-First, avoid losing information during post-processing, use statistical methods to select primary types, preserve multi-faceted nature for downstream use, and keep the conflicts that truly need human judgment within a manageable scope.

This is the gap between "running a demo" and "going to production" in GraphRAG — and it's the most important one to fill.

Why Do We Need GraphRAG? — The Evolution from "Search" to "Understanding"

eyanpen — Fri, 24 Apr 2026 11:49:37 +0000

When AI stops just "looking things up" and starts truly "understanding" your question.

1. Let's Start with an Everyday Scenario

Imagine you're a new employee at a company. On your first day, you want to know "the most important project updates from the past three months."

You have two options:

Option A: Dig through the filing cabinet
You walk to the archive room, open the filing cabinet, and search by the keyword "project updates." You find dozens of documents, but they're scattered across different drawers — some are meeting minutes, some are emails, some are reports. You have to piece these fragments together yourself to get a complete answer.

Option B: Ask a colleague who "knows everything"
This colleague has not only read every document but also remembers that "Project A led by Zhang San and Project B led by Li Si are actually related," and knows that "last month's budget adjustment affected three departments' plans." They can give you an organized, complete answer right away.

Option A is traditional RAG (Retrieval-Augmented Generation).
Option B is what GraphRAG aims to achieve.

2. What Is RAG? It's Already Impressive — So Why Isn't It Enough?

What Is RAG

RAG stands for Retrieval-Augmented Generation. Simply put, it lets AI search through a pile of documents for relevant content before answering your question, then generates a response based on what it found.

It's like an open-book exam — AI can flip through references to find answers instead of relying purely on memory.

RAG's Limitations

RAG is genuinely useful, but it has a fundamental weakness: it can "find" but it can't "connect."

For example, suppose you ask:

"What impact has the company's business expansion in Asia-Pacific had on the supply chain?"

Traditional RAG would:

Search for documents containing keywords like "Asia-Pacific," "business expansion," "supply chain"
Find several relevant passages
Hand these passages to the AI to generate an answer

Where's the problem?

Information about "Asia-Pacific business expansion" might be in a strategic report
Information about "supply chain adjustments" might be in an operations report
The connection between these two reports — such as "because of Asia-Pacific expansion, a new Vietnamese supplier was added, causing logistics cost changes" — might not be explicitly stated in any single document

What traditional RAG finds are isolated "fragments." It's not good at connecting the implicit relationships between fragments.

3. How Does GraphRAG Solve This Problem?

Core Idea: Build a "Relationship Network" First

GraphRAG's key innovation is that before answering questions, it does something extra: it organizes all the information from documents into a "relationship network" (knowledge graph).

What does this relationship network look like? Think of it as a character relationship map:

Nodes (circles): Represent individual "things" — people, companies, projects, locations, concepts
Edges (arrows): Represent relationships between them — "responsible for," "belongs to," "affects," "collaborates with"

A simple example:

[Zhang San] --responsible for--> [Project A]
[Project A] --depends on--> [Project B]
[Project B] --led by--> [Li Si]
[Project A] --budget from--> [Asia-Pacific Department]
[Asia-Pacific Department] --partners with--> [Vietnamese Supplier]

With this network, when you ask "What's the relationship between Zhang San's project and the Vietnamese supplier?", the AI can "walk" through the network and discover:

Zhang San → Project A → Asia-Pacific Department → Vietnamese Supplier

Even if no single document ever directly mentions "the relationship between Zhang San and the Vietnamese supplier," the AI can reason out the answer through this path.

Plain-Language Summary

	Traditional RAG	GraphRAG
How it works	Searches keywords, finds relevant passages	Builds a relationship network first, then follows relationships to answer
Good at	"What is X?" "How do I do X?"	"What's the relationship between X and Y?" "What's the big picture?"
Analogy	A librarian helping you find books	A detective connecting clues into a complete story
Weakness	Fragmented, lacks global perspective	Building the relationship network takes time and compute

4. What Can GraphRAG Do for Us?

Scenario 1: Enterprise Knowledge Management

A large company has thousands of internal documents: policies, procedures, meeting minutes, technical docs...

Traditional approach: Employees search by keywords, browse through many documents, summarize on their own
GraphRAG approach: AI has already "understood" the relationships between all documents. Employees can directly ask "What was the root cause of increased customer complaints last quarter?" and the AI can provide a connected analysis across product changes, customer service records, supplier issues, and more

Scenario 2: Healthcare

A patient's medical records, test reports, and medication history are scattered across different systems.

Traditional approach: Doctors review each one individually, relying on experience
GraphRAG approach: AI builds a network connecting patient information, medications, diseases, and test results. It can flag that "Drug A the patient is currently taking and newly prescribed Drug B may interact because they both act on the same metabolic pathway"

Scenario 3: Financial Risk Control

A bank needs to assess the risk of a loan.

Traditional approach: Review the borrower's credit report and financial data
GraphRAG approach: AI discovers that the borrower's company and another company that has already defaulted share the same ultimate beneficial owner, and this connection is hidden within multiple layers of equity structures — uncovering these "hidden relationships" is exactly where GraphRAG excels

Scenario 4: Everyday Q&A Assistant

You're using an AI assistant to learn about a complex topic like "climate change."

Traditional approach: AI gives you a general overview of climate change
GraphRAG approach: AI can tell you "climate change affects agricultural yields, which in turn affects food prices, which ultimately affects social stability in developing countries" — this kind of multi-hop reasoning (from A to B to C to D) is GraphRAG's core advantage

5. GraphRAG Isn't a Silver Bullet

After all these benefits, let's be honest about its limitations:

Building the relationship network has costs: Converting large volumes of documents into a knowledge graph requires time and compute resources. For small-scale, simple Q&A scenarios, traditional RAG may be sufficient.
The quality of the relationship network is critical: If the AI misunderstands a relationship during graph construction, subsequent reasoning will also be wrong. Just like a detective who connects clues incorrectly will reach the wrong conclusion.
Not every question needs it: If you just want to look up "What's the company's expense reimbursement process?", traditional search can answer that perfectly well — no need to deploy GraphRAG.

6. Summary

The essence of GraphRAG is evolving AI from "keyword search" to "relationship reasoning."

It's not meant to replace traditional RAG but to add a layer of "understanding relationships" on top of it. It's like upgrading from "looking up a dictionary" to "reading an encyclopedia" — a dictionary tells you what each word means; an encyclopedia also tells you how those words are connected.

For scenarios that involve processing large amounts of complex information, discovering hidden connections, and requiring a global perspective, GraphRAG is a direction worth paying attention to.