Forem: Einsteinder

Stop Over-Engineering Your AI Agent. Then Teach It What It Doesn't Know

Einsteinder — Thu, 30 Apr 2026 20:53:05 +0000

We built an AI agent that searches across a user's emails, Slack, Jira, and other channels to answer questions like "find the Acme project update" or "list all release versions." The search was broken in ways we didn't expect — and the fixes were counterintuitive.

Every instinct said add more intelligence: smarter classification, better query rewriting, self-grading loops, bigger models. Every one of those instincts was wrong.

The real path had two parts:

Get out of the AI's way. Most of our "improvements" were making things worse. Removing them fixed more than adding them.
Teach the AI what it can't derive. Once we stripped the pipeline down, we found failures that no amount of model capability could fix. Those needed human knowledge encoded as part of the system.

This post is about both halves.

Part 1: Get Out of the AI's Way

Wall #1: Dense-only search doesn't find what users actually search for

Users typed a client name — returned zero results. They typed a short project code — zero results. Dense vector search is great at semantic similarity but terrible at exact entity matching. A 3-letter abbreviation doesn't embed well. Proper nouns don't cluster near related content. And "Acme Corp" as a cosine similarity vector doesn't match the actual email about Acme Corp as reliably as you'd expect.

The instinct: Train better embeddings. Fine-tune the model. Add query expansion.

What actually worked: Add a boring keyword search alongside the semantic one. Fuse the results. This is hybrid search with Reciprocal Rank Fusion (RRF), and every serious production system uses it. Dense retrieval and sparse (keyword) retrieval solve different problems. You need both.

query → dense search + keyword search (parallel) → RRF fusion → results

That's it. No training, no fine-tuning, no fancy rewriters. Just two searches merged by rank.

Wall #2: Pre-search LLM calls were making results worse

Our original pipeline looked like this:

query → LLM call #1: classify the query type (1-2s)
      → LLM call #2: rewrite into 2-3 sentences (1-2s)
      → search → results

It felt intelligent. It was actually terrible.

The classifier would guess the query type ("this is a notification query") and that guess would control downstream filtering. When it guessed wrong — which happened constantly on short or ambiguous queries — the filter threw out valid results.

The rewriter was worse. Give it a client name and it would helpfully expand the query into something like "update on the project, including progress, milestones, and team members involved." The original keyword now lives inside a padded sentence instead of being the focused term the user typed. The embedding drifts. Keyword matching weakens. Results get worse.

The instinct: Make the classifier smarter. Make the rewriter better. Add fallbacks.

What actually worked: Delete both. Pass the user's original query directly to hybrid search.

Latency dropped from ~4-6s to ~2-3s. Zero-result failures went away. Quality went up.

The principle: Every LLM call before retrieval is a place where your pipeline can add noise. If it isn't provably adding signal, it's subtracting it.

Wall #3: Your tool output is for the LLM, not for humans

Search results were formatted to be readable: full previews, metadata fields, relevance scores, conversation IDs, the works. Roughly 300 tokens per result. Ten results per search.

The agent would make two searches and suddenly its context was full of ~6,000 tokens of formatted noise. By the third search call, the model would start narrating its intentions ("Let me search more specifically...") instead of actually making tool calls. It had lost coherence.

The instinct: Give the LLM a bigger context window. Upgrade the model.

What actually worked: Cut tool output from ~300 tokens per result to ~50. Stop sending conversation IDs and relevance percentages to the LLM. Keep just what's needed to reason about the next step.

# Before (~300 tokens per result)
[1] OUTLOOK: [repo] Release v0.4.0 (PR #332)
    Date: 2026-03-11 10:25
    Relevance Score: 98%
    Participants: github-actions[bot] <notifications@...>
    Content Preview:
    Summary: Bump project version from 0.3.2 to 0.4.0...
    Outlook Conversation ID: AAQkADE1YTA2Y2Q4...

# After (~50 tokens per result)
[src:332] (outlook) Release v0.4.0 (PR #332) | 2026-03-11 | From: github-actions
    Bump project version from 0.3.2 to 0.4.0

The 6x reduction is the difference between the agent finishing its job and running out of coherence mid-response. Design tool output for the reader that's consuming it — which, in an agentic system, isn't you.

That's the "get out of the way" part. Three changes. All subtractions. All improved the system.

Now here's where it gets interesting.

Part 2: Teach the AI What It Can't Derive

After stripping the pipeline down, we tested a harder query: "Find Alex's phone number."

The number existed in the system. It lived in the email signature of a message from Alex. The agent searched "Alex phone number" — found emails mentioning phone numbers but not the actual digits. It searched "Alex Chen phone" — same thing. It tried six or seven variations. It eventually gave up.

The number was right there. The agent couldn't find it.

Here's why: the document that contained the phone number never used the word "phone." It was an email signature — just a name, a title, and some digits. The user's query vocabulary and the document's vocabulary had nothing in common. No amount of keyword search, embedding similarity, or query rewriting can bridge that gap. It's the fundamental limit of lexical + semantic retrieval.

This is where a human would instantly succeed. A person asked "find Alex's phone number" would think: "Phone numbers are in email signatures. Let me pull up an email from Alex and read the bottom." That's a two-step reasoning chain that has nothing to do with the words "phone number."

The AI doesn't know this. Not because it's dumb — because it can't derive from first principles that phone numbers live in signatures. That's human knowledge about how email works.

The instinct: wait for smarter models

We almost did this. "Once models get better at multi-hop reasoning, they'll figure it out."

The problem: there will always be a next failure mode. Chasing the model capability curve is a losing race. And even if the next model reasoned through this specific case, it would fail on the next thing you didn't anticipate.

The fix: encode the knowledge, keep the agent

We did two things. First, a simple tool — get_communication_detail(id) — that fetches the full body of a specific communication. The compact format hides most details from the agent. This tool lets the agent drill down when the preview isn't enough. Detail on demand.

Second — and more importantly — we defined the reasoning chain as a skill:

---
name: find-contact-info
triggers:
  - phone number
  - contact info
  - how to reach
---

## Steps
1. Search for the person across all channels
2. Search for emails sent BY that person
3. Fetch full content of top result to read the signature

A skill is a declarative definition of a proven workflow. The steps use the same tools the agent already has, but they're sequenced in a way the agent couldn't reliably derive on its own. When the user asks about contact info, the system can invoke this skill instead of hoping the agent reasons through it.

The skill isn't hardcoded logic — it's encoded knowledge. The tools still do the searching. The LLM still fills in adaptive parameters at each step. But the order of operations — the human insight that "to find contact info you should look at emails from the person" — lives in the skill definition.

Why this matters more than it sounds

Skills are a bet on a different model of AI development. Instead of "AI that figures everything out," it's "AI that accumulates learned workflows." When the system notices a user manually performing a pattern — search a person, fetch an email, extract contact info — it can propose turning that pattern into a skill. The user approves. The next time anyone hits that problem, the skill handles it.

This is the loop that makes the agent get better over time without needing a better model underneath. The model's job is to handle the long tail — open-ended reasoning, novel situations, conversations. The skills handle the repeatable patterns that humans have already solved.

This is also why we resisted implementing the contact lookup as a hardcoded Python tool. A tool is opaque and one-off. A skill is portable, discoverable, and composable. Future you can read the skill definition and know exactly what it does. Future users can propose new skills for patterns the system missed.

The Two-Part Rule

Looking back, every improvement fit into one of two categories:

Subtractions from the pipeline:

Dense-only search → hybrid search (delete the assumption that semantic is enough)
Pre-search LLM calls → direct query (delete intelligence that was noise)
Verbose tool output → compact format (delete metadata the LLM can't use)

Additions of human knowledge:

Detail-on-demand tool (humans know "read the full thing" is sometimes the answer)
Contact lookup skill (humans know signatures contain contact info)

The instinct at every turn was to do the opposite — add intelligence to the pipeline, hope the model figures out the chains. That instinct was wrong.

The rule: First, delete everything in your pipeline that isn't provably helping. Then, for the failures that remain, ask whether the AI is missing knowledge a human has. If yes, encode it — as a tool, a skill, or structured data. Don't wait for a smarter model.

A Practical Checklist

If you're building an agentic search or RAG system, check these in order:

Subtractions:

[ ] Do you have both keyword and semantic retrieval fused together? (If no, add it.)
[ ] Are there LLM calls before retrieval that classify or rewrite the query? (Consider removing.)
[ ] Is your tool output designed for human readability or LLM consumption? (Cut everything the LLM doesn't need.)
[ ] Does your agent have enough context budget to iterate 3-5 times without losing coherence? (If no, reduce per-result tokens.)

Additions:

[ ] When your agent fails on a task, ask: is this a vocabulary mismatch or missing world knowledge? (Those need encoded knowledge, not smarter prompts.)
[ ] Does your agent have a way to drill down from summaries to full content when needed? (Add a detail-fetch tool.)
[ ] Can you extract multi-step patterns from user interactions and encode them as skills? (This is where the compounding returns are.)

The instinct is always to make the AI smarter. The practice is to make the system around it smaller and more specific.

Multi-Leader Replication

Einsteinder — Thu, 18 Jul 2024 16:57:25 +0000

	single-leader	multi-leader
Performance	significant latency	network delay hidden from users
Tolerance of datacenter outages	failover	each datacenter operating independently

Solution for multi-leader confliction:

last write wins (LWW)
predefined the replica ranking
merge values together
record the conflict, resolve later

Problems with Replication Lag

Einsteinder — Thu, 18 Jul 2024 15:26:32 +0000

Here are some example problems and possible solutions for replication lag

1. Post not instantly show

when user post the comment under a passage, he want to see it immediately, otherwise, he would think something wrong. When we user asynchronous replication, the comment may not be replicated to follower nodes. So we need to implement the read-after-write consistency to avoid it.

we assume the leader node always be synchronized with the action, based on it we can expect user will read something they've edit. for example, a user will edit his own profile instead of others, they can always read it from the leader node to make sure the read-after-write consistency.
keep track of the last updated time, for example user edited on 3:15, during 3:15 - 3:16 we always read from leader node.
client can remember of timestamp of write, read from whoever has it or wait util it has

2. Monotonic Reads

Users may not expect to see something they've seen is disappeared.

It happens when user read from different replicas, one have small lag, and the other have a greater lag.

The solution for this would be ask user read from same replica always, it may not read the latest data, but they will see the data sequentially

DFS and BFS

Einsteinder — Mon, 15 Jul 2024 17:22:16 +0000

Today I will start refreshing my computer science knowledge by writing tech blogs.

Start from Leetcode 865.
It's exactly same with Leetcode 1123

read the question clearly before starting coding, I thought there was only one deepest node in the tree, so I didn't think this was a problem of the LCA. the key to solving this problem is to check the left and right subtrees if they have the same level, if yes that means their parent is the smallest subtree. There are two conditions: a) when the 2 nodes are at the same deepest level, then their parent node is the smallest subtree.

b) if there is only one node is the deepest level, then it self is the smallest subtree

The first way I can think of is recursive, here is the code:

class Solution:
    max_level = 0
    res = None

    def subtreeWithAllDeepest(self, root: TreeNode) -> TreeNode:
        level = 0
        self.dfs(root,level)
        return self.res


    def dfs(self,node,current_level):
        if not node:
            return current_level
        else:
            l_next_level = self.dfs(node.left,current_level+1)

            r_next_level = self.dfs(node.right,current_level+1)

            max_cur_level = max(l_next_level,r_next_level)

            self.max_level = max(self.max_level,max_cur_level)

            if self.max_level == l_next_level and self.max_level==r_next_level:
                self.res = node

            return  max_cur_level

Since we need to keep track of the same level of the deepest nodes and find their common ancestor, it seems hard to simulate it with a iterative dfs, if you can, please leave the comment.

I try to use BFS to solve the same problem:

    def bfs(self,node):
        leftmost_node = None
        rightmost_node = None
        q = [node]
        while q:
            q_size = len(q)
            for i in range(q_size):
                the_node = q.pop()
                if i == 0:leftmost_node = the_node
                if i == q_size - 1 : rightmost_node = the_node
                if the_node.left: q.insert(0,the_node.left)
                if the_node.right:q.insert(0,the_node.right)
        self.res = self.lca(node,leftmost_node,rightmost_node)
        return 
    def lca(self,root,p,q):
        if not root:return
        if root==p or root ==q: return root
        left = self.lca(root.left,p,q)
        right = self.lca(root.right,p,q)
        if not left:
            return right
        elif not right:
            return left
        else:
            return root

Based on this problem, let's explore more about dfs and bfs.

Let's see Leetcode 200
This is a classic problem that can be solved by dfs and bfs. For dfs, the idea is we need to start from all points on the map and explore all the islands that can be reached. The basic operation is to check the four directions around the point to see if there are "1", if yes go into that point and check its four directions, and repeat this until no "1" is found around. Here is the code:

class Solution:
    directions = [
        [0,1], #right
        [1,0], #down
        [-1,0],#up
        [0,-1],#left
    ]
    def numIslands(self, grid: List[List[str]]) -> int:
        count = 0
        for i in range(len(grid)):
            for j in range(len(grid[0])):
                if grid[i][j] == "1":
                    grid[i][j] = "0"
                    count += 1
                    self.dfs(grid,i,j)
        return count


    def dfs(self,grid,i,j):
        for m,n in self.directions:
            if  0<= i+m < len(grid) and 0<= j+n <len(grid[0]) and grid[i+m][j+n] == "1":
                grid[i+m][j+n] = "0"
                self.dfs(grid,i+m,j+n)

In this code, we changed the visited points in place to "0", if we don't want to change the the given grid, we can also store the visited points like this:

    def numIslands_no_self(self, grid: List[List[str]]) -> int:
        def dfs(i,j):
            visited.add((i, j))
            for m,n in self.directions:
                if  0<= i+m < len(grid) and 0<= j+n <len(grid[0]) and grid[i+m][j+n] == "1" and (i+m,j+n) not in visited:
                    dfs(i+m,j+n)

        if not grid or not grid[0]: return 0    
        visited = set()        
        count = 0
        for row in range(len(grid)):
            for col in range(len(grid[0])):
                print(54,visited)
                if grid[row][col] == "1" and (row,col) not in visited:
                    count += 1
                    dfs(row,col)

        return count

You may wonder why I named the function "no_self". At the very beginning, I used a global variable to save the visited points, logically speaking, it's no different from the local variable, but when I submitted it, it couldn't pass the test. I figured that caused by the global variable is long-lived in memory, when a new test case comes in, the old visited points are still there, and that's why it cannot go through. So I recommend we all use the local variable here.

Next, I will try to use BFS to solve the problem. The key of BFS will queue, and iteration will be like the wave spread from the middle.

    def numIslands(self, grid: List[List[str]]) -> int:
        def bfs():
            if not grid or not grid[0]: return 0
            count = 0
            q = []
            for row in range(len(grid)):
                for col in range(len(grid[0])):

                    if grid[row][col]=="1":
                        count += 1
                        q.insert(0,(row,col))


                    while q:
                        i,j = q. pop()
                        if grid[i][j] == "1":
                            grid[i][j] = "0"
                            for m,n in self.directions:
                                if  0<= i+m < len(grid) and 0<= j+n <len(grid[0]) and grid[i+m][j+n] == "1":
                                    q.insert(0,(i+m,j+n))
            return count
        return bfs()

Let's try another one Leetcode 130
the key to this problem is to find all O at the edge of the board and all O connected to them cannot be converted. All other O needs to convert to X, here is the implementation:

class Solution:
    directions = [
        [0,-1],
        [0,1],
        [1,0],
        [-1,0]
    ]
    def solve(self, board: List[List[str]]) -> None:
        """
        Do not return anything, modify board in-place instead.
        """
        def dfs(i,j):
            if board[i][j] in ["#","X"]: return
            board[i][j] = "#"
            for m,n in self.directions:
                if 0 <= i+m <len(board) and 0<= j+n <len(board[0]):
                    dfs(i+m,j+n)



        if not board or not board[0]: return
        for i in range(len(board)):
            for j in range(len(board[0])):
                if i == 0 or i == len(board) - 1 or j == 0 or j == len(board[0]) - 1:
                    dfs(i,j)

        for i in range(len(board)):
            for j in range(len(board[0])):
                if board[i][j] == "O":
                    board[i][j] = "X"
                if board[i][j] == "#":
                    board[i][j] = "O"