Forem: Armel BOBDA

Deep Dive: Building Observable AI with Opik

Armel BOBDA — Fri, 20 Feb 2026 13:28:21 +0000

How Pause turns every AI decision into a traceable, scoreable, searchable artifact.

The Problem

Most AI applications treat observability as an afterthought — a console.log here, a dashboard metric there. When something goes wrong, you're left staring at opaque LLM calls with no idea why the model said what it said.

Pause is an AI-powered financial guardian that intercepts impulse purchases with personalized interventions. Every interaction involves risk assessment, strategy selection, coupon discovery, behavioral reflection, and user feedback — a multi-step pipeline where any stage can fail silently. Without deep observability, "self-improving AI" is just a marketing claim.

We needed Opik to make the intelligence visible, auditable, and measurable. This deep dive documents our approach for the Best Use of Opik category.

Architecture Overview

Link: mermaid.live

Key insight: Traces aren't static log entries. They're living documents that get richer over time as users make decisions and the learning pipeline processes outcomes.

Pattern 1: Automatic Tracing via Vercel AI SDK

Every Guardian streamText() call is automatically traced through the OpenTelemetry pipeline. No manual instrumentation needed.

Setup (instrumentation.ts):

export async function register() {
  if (process.env.OPIK_API_KEY) {
    const { OpikExporter } = await import("opik-vercel");
    registerOTel({
      serviceName: "pause-guardian",
      traceExporter: new OpikExporter({
        tags: ["hackathon", "pause"],
      }),
    });
  }
}

Usage in the Guardian route (route.ts):

const result = streamText({
  model,
  system: systemPrompt,
  messages,
  tools,
  experimental_telemetry: getGuardianTelemetry(
    interactionId,
    { score: riskResult.score, reasoning: riskResult.reasoning },
    tier,
    isAutoApproved,
    undefined,
    undefined,
    purchaseContext,
    prediction
  ),
});

The getGuardianTelemetry() function is the centralized point where every trace gets its name, metadata, and identity. This prevents the "unnamed ai.generateText" traces that make Opik dashboards unusable.

Pattern 2: 16 Named Trace Types — A Taxonomy, Not Just Labels

We defined 16 distinct trace names that map to every possible Guardian outcome:

export const TRACE_NAMES = {
  ANALYST_AUTO_APPROVED:       "guardian:analyst:auto_approved",
  NEGOTIATOR_ACCEPTED_SAVINGS: "guardian:negotiator:accepted_savings",
  NEGOTIATOR_SKIPPED_SAVINGS:  "guardian:negotiator:skipped_savings",
  NEGOTIATOR_ACCEPTED:         "guardian:negotiator:accepted",
  NEGOTIATOR_OVERRIDE:         "guardian:negotiator:override",
  THERAPIST_WAIT:              "guardian:therapist:wait",
  THERAPIST_ACCEPTED:          "guardian:therapist:accepted",
  THERAPIST_OVERRIDE:          "guardian:therapist:override",
  THERAPIST_WIZARD_BOOKMARK:   "guardian:therapist:wizard_bookmark",
  THERAPIST_WIZARD_ABANDONED:  "guardian:therapist:wizard_abandoned",
  BREAK_GLASS:                 "guardian:break_glass",
  SYSTEM_FAILURE_ANALYST_ONLY: "system:failure:analyst_only",
  SYSTEM_FAILURE_BREAK_GLASS:  "system:failure:break_glass",
  LEARNING_REFLECTION:         "learning:reflection",
  LEARNING_SKILLBOOK_UPDATE:   "learning:skillbook_update",
  CHAT_KNOWLEDGE:              "chat:knowledge",
} as const satisfies Record<string, TraceName>;

Why this matters: A judge (or developer) can filter Opik by guardian:negotiator:override and immediately see every time a user rejected the savings offer. Filter by learning:reflection to see every learning cycle. The namespace convention (guardian:, learning:, system:) creates natural groupings.

The resolveTraceName() function dynamically selects the right name based on tier, outcome, and degradation state — so traces are always semantically accurate, even during partial failures.

Pattern 3: The Child Trace Pattern — Working Around Immutability

The problem: Opik traces are immutable after creation. But we don't know the Guardian's reasoning summary until after streaming completes. The trace is already created by the time we have the data we need.

The solution: Create a child trace linked via interactionId metadata:

export async function writeTraceMetadata(
  interactionId: string,
  metadata: Record<string, unknown>
): Promise<void> {
  const client = getOpikClient();
  if (!client) return;

  // Find the parent trace by interactionId
  const traces = await client.searchTraces({
    filterString: `metadata.interactionId = "${interactionId}"`,
    maxResults: 1,
  });

  if (traces.length > 0) {
    // Create a child trace with the metadata
    const metadataTrace = client.trace({
      name: "guardian:metadata_update",
      input: {
        interactionId,
        parentTraceId: traces[0].id,
        ...metadata,
      },
    });
    metadataTrace.end();
    await client.flush();
  }
}

This pattern is used for:

Reasoning summaries — written after streaming completes
Reflection results — written after the learning pipeline runs
Skillbook snapshots — written after skills are updated
Satisfaction feedback — written days later from Ghost Cards

Each child trace is a timestamped addition to the interaction's story.

Pattern 4: Feedback Scores — Turning User Decisions into Metrics

Every user decision is converted to a numerical score and attached to the original Guardian trace:

export const INTERVENTION_ACCEPTANCE_SCORES: Record<
  string,
  { value: number; reason: string }
> = {
  accepted:        { value: 1.0, reason: "User accepted Guardian suggestion" },
  accepted_savings:{ value: 1.0, reason: "User accepted savings offer" },
  wait:            { value: 1.0, reason: "User chose to wait as suggested" },
  skipped_savings: { value: 0.5, reason: "User skipped savings but accepted unlock" },
  override:        { value: 0.0, reason: "User overrode Guardian intervention" },
  wizard_bookmark: { value: 1.0, reason: "User engaged deeply with reflection wizard" },
};

And a second score layer from retrospective Ghost Card feedback:

export const REGRET_FREE_SCORES: Record<
  string,
  { value: number; reason: string } | null
> = {
  worth_it:  { value: 1.0, reason: "User reports purchase was worth it" },
  regret_it: { value: 0.0, reason: "User reports regret about purchase" },
  not_sure:  null,  // No score — insufficient signal
};

The attachment mechanism searches for the original trace by interactionId, then logs the score:

client.logTracesFeedbackScores([{
  id: traceId,
  name: scoreName,
  value: scoreValue,
  ...(reason && { reason }),
}]);

Why this matters: A single trace in Opik now carries the full lifecycle — from initial risk assessment, through the intervention strategy, to the user's immediate decision, and finally their retrospective satisfaction days later. You can filter for intervention_acceptance = 0.0 to find every failed intervention and trace exactly why the strategy didn't work.

Pattern 5: Learning Pipeline Traces — Observing Self-Improvement

When the ACE learning pipeline runs, each stage creates its own trace:

Reflection trace — what the Reflector learned:

client.trace({
  name: "learning:reflection",
  input: {
    interactionId,
    parentTraceId: traces[0].id,
    reflectionAnalysis: reflectionOutput.analysis,
    helpfulSkillIds: reflectionOutput.helpful_skill_ids,
    harmfulSkillIds: reflectionOutput.harmful_skill_ids,
    newLearningsCount: reflectionOutput.new_learnings.length,
  },
  tags: ["learning", "reflection"],
});

Skillbook update trace — what changed in the knowledge base:

client.trace({
  name: "learning:skillbook_update",
  input: {
    interactionId,
    operationCount: updateBatch.operations.length,
    skillCountBefore,
    skillCountAfter,
    delta: skillCountAfter - skillCountBefore,
    operationsByType,
    reasoning: updateBatch.reasoning,
    operations: updateBatch.operations.map((op) => ({
      type: op.type,
      section: op.section,
      skill_id: op.skill_id,
    })),
    skillbook_snapshot: buildSkillbookSnapshot(skillbook),
  },
  tags: ["learning", "skillbook_update"],
});

Why this matters for judges: You can open Opik, filter to learning:skillbook_update, and watch the Skillbook grow. Each trace shows the delta — "3 skills before, 5 after, +2 added." This is the tangible proof that the AI is learning, not just a before/after screenshot.

Pattern 6: Degradation Tracing — Even Failures Are Observable

When the Guardian hits a service failure, the degradation itself is traced:

export async function logDegradationTrace(
  interactionId: string,
  degradationLevel: "analyst_only" | "break_glass",
  failureReason: string,
  riskMeta?: { score: number; reasoning: string },
): Promise<void> {
  const client = getOpikClient();
  if (!client) return;

  const trace = client.trace({
    name: degradationLevel === "analyst_only"
      ? TRACE_NAMES.SYSTEM_FAILURE_ANALYST_ONLY
      : TRACE_NAMES.SYSTEM_FAILURE_BREAK_GLASS,
    input: {
      interactionId,
      failureReason,
      degraded: true,
      degradationLevel,
      reasoning_summary: buildReasoningSummary({ ... }),
    },
  });
  trace.end();
  await client.flush();
}

System failures get their own namespace (system:failure:*) so they never pollute behavioral data. You can filter Opik to see only degradation events and understand system reliability independently from AI quality.

The Fire-and-Forget Philosophy

Every Opik operation follows one rule: telemetry failures must never disrupt the user.

// Every Opik call is wrapped like this:
try {
  const client = getOpikClient();
  if (!client) return;  // Gracefully disabled in dev

  // ... trace operations ...
  await client.flush();
} catch {
  // Telemetry failures must never disrupt the main flow
}

This means:

No Opik API key? App works normally, traces silently skipped
Opik API down? User flow completes, traces lost (acceptable trade-off)
Flush fails? Background retry via batch queue, user unaware

The getOpikClient() singleton returns null when OPIK_API_KEY isn't set, so every call site naturally degrades.

Reasoning Summary Sanitization

Every trace includes a human-readable reasoning_summary field. But since Pause deals with financial behavior, we sanitize clinical terminology:

const BANNED_SUMMARY_TERMS: Record<string, string> = {
  therapy: "reflection",
  therapist: "high-risk",
  diagnosis: "assessment",
  patient: "user",
  treatment: "approach",
  clinical: "structured",
  session: "interaction",
};

Summaries are capped at 200 characters with word-boundary truncation:

if (summary.length > MAX_SUMMARY_LENGTH) {
  const spaceIdx = summary.lastIndexOf(" ", MAX_SUMMARY_LENGTH);
  summary = `${summary.slice(0, spaceIdx === -1 ? MAX_SUMMARY_LENGTH : spaceIdx)}...`;
}

This ensures Opik traces are always professional and audit-safe — no accidental clinical language in the observability layer.

What We Learned

1. Trace immutability changes your architecture

Opik traces are write-once. We expected to update traces as the interaction progressed. Instead, we developed the child trace pattern with interactionId linkage. This turned out to be better than mutation — each child trace is a timestamped event, creating a natural timeline.

2. Name your traces from Day 1

We wired getGuardianTelemetry() in the first sprint. Every feature added afterward was automatically traced with a meaningful name. Projects that retrofit tracing end up with a mix of ai.generateText.123abc and proper names — unusable for analysis.

3. Feedback scores are the killer feature

Raw traces show what the AI did. Feedback scores show whether it worked. The ability to filter Opik for intervention_acceptance = 0.0 and see exactly why users rejected interventions is what makes observability actionable, not just informational.

4. Fire-and-forget is non-negotiable

We chose Node runtime over Edge specifically because after() callbacks are more reliable on Node. Opik trace fidelity is worth the marginally slower cold starts. But even with Node, wrapping everything in try-catch ensures a flaky network never blocks a user from unlocking their card.

5. Separate behavioral data from system health

Using system:failure:* vs guardian:* namespaces means you can analyze AI quality and system reliability independently. A spike in system:failure:break_glass traces means infrastructure problems, not bad AI strategies.

The Opik + ACE Connection

Opik doesn't just observe the AI — it observes the AI learning. The learning:reflection and learning:skillbook_update traces create a visible paper trail from "user rejected strategy A" → "Reflector analyzed why" → "SkillManager added strategy B" → "next interaction used strategy B successfully."

This is the bridge to the ACE Self-Learning Deep Dive — where we explain how the Skillbook grows from seed strategies into a personalized knowledge base, with every step traced in Opik.

File Reference

File	Role
`apps/web/instrumentation.ts`	Global OTel + OpikExporter setup
`apps/web/src/lib/server/opik.ts`	Client singleton, telemetry builder, trace operations, feedback scores
`apps/web/src/lib/guardian/trace-names.ts`	16 named trace types with TypeScript const assertion
`apps/web/src/lib/server/learning.ts`	Learning pipeline with reflection + skillbook update traces
`apps/web/src/app/api/ai/guardian/route.ts`	Guardian streaming with `experimental_telemetry` injection
`apps/web/src/app/api/ai/feedback/route.ts`	Feedback score attachment to traces
`packages/ace/src/observability/opik_integration.ts`	ACE framework's own Opik integration layer

Built with Next.js 16 + Opik by Comet ML + Vercel AI SDK v6 + ACE Framework

Deep Dive: Self-Improving AI with ACE Skillbooks

Armel BOBDA — Fri, 20 Feb 2026 12:58:04 +0000

How Pause's Guardian learns from every interaction — and proves it.

The Problem

"Self-improving AI" is one of the most overclaimed phrases in tech demos. Most projects mean one of two things: batch fine-tuning between releases, or A/B testing between user cohorts. Neither is real-time, per-user, per-interaction learning.

Pause needed something different. The Guardian AI intercepts impulse purchases with personalized strategies — but a strategy that works for late-night electronics doesn't work for morning coffee runs. A generic "Do you really need this?" prompt gets overridden immediately. The AI had to learn what works for each user and adapt on every interaction.

We integrated the ACE (Agentic Context Engine) framework to build a Skillbook that grows from 4 seed strategies into a personalized knowledge base — with every learning step traceable in Opik.

Architecture Overview

Link: mermaid.live

Two feedback loops, one Skillbook. Immediate outcomes (accepted/overridden) and retrospective satisfaction ("Was it worth it?") both feed into the same ACE pipeline, giving the Guardian two temporal perspectives on every decision.

The Skillbook: A Living Knowledge Base

The Skillbook is a persistent JSON registry of learned strategies. Each skill has an ID, content description, and effectiveness scores:

interface Skill {
  id: string;
  section: string;
  content: string;
  helpful: number;   // Times this skill led to acceptance
  harmful: number;   // Times this skill led to override
  neutral: number;   // Times outcome was ambiguous
  status: "active" | "invalid";
}

Storage: The Skillbook is stored as a JSONB column in Neon PostgreSQL, serialized via Skillbook.toDict() and loaded via Skillbook.fromDict(). No normalized skills table — the entire Skillbook is a single atomic document, simplifying concurrent updates. Two deserialization methods are used: Skillbook.loads() parses a JSON string (used when converting JSONB to string first), while Skillbook.fromDict() accepts a JS object directly (used when Drizzle returns JSONB as a parsed object).

Injection: Before every Guardian interaction, the Skillbook is loaded and formatted for the LLM:

// apps/web/src/lib/server/ace.ts
export async function loadUserSkillbook(userId: string): Promise<string> {
  const result = await withTimeout(
    db.select({ skills: skillbookTable.skills })
      .from(skillbookTable)
      .where(eq(skillbookTable.userId, userId))
      .limit(1),
    DB_TIMEOUT_MS
  );

  let instance: Skillbook;
  if (result[0]) {
    instance = Skillbook.loads(JSON.stringify(result[0].skills));
  } else {
    instance = new Skillbook();
  }

  const context = wrapSkillbookContext(instance);

  // Guard against prompt bloat
  if (context.length > MAX_CONTEXT_CHARS) {
    return `${context.substring(0, MAX_CONTEXT_CHARS)}\n\n[Skillbook truncated]`;
  }
  return context;
}

The wrapSkillbookContext() function formats skills into a structured text block that the LLM can reference when selecting strategies. Skills with high helpful scores rise to the top. Skills marked harmful are deprioritized.

The Adapter Layer: All ACE Flows Through One Module

Every ACE import in the application flows through a single adapter:

apps/web/src/lib/server/ace.ts

This module re-exports Skillbook, Reflector, SkillManager, VercelAIClient, and wrapSkillbookContext from the vendored @pause/ace package. It also provides two loading functions:

Function	Returns	Used By
`loadUserSkillbook(userId)`	Formatted context string	Guardian route (prompt injection)
`loadUserSkillbookInstance(userId)`	Raw `Skillbook` instance + version	Learning pipeline (mutation)

Why a single adapter? The ACE package is a vendored fork with Turbopack-specific modifications. Centralizing imports means upstream changes (import paths, API signatures) only need fixing in one file.

The Learning Pipeline: Three Stages

When a user makes a decision (accept, override, wait), the learning pipeline runs asynchronously via Next.js after():

Stage 1: Reflection

The Reflector analyzes why the interaction succeeded or failed:

export async function runReflection(params: {
  interactionId: string;
  userId: string;
  question: string;
  generatorAnswer: string;
  outcome: string;
}): Promise<LearningPipelineResult | null> {
  const { skillbook, version } = await loadUserSkillbookInstance(userId);
  const reflector = getReflector();

  const reflectionOutput = await Promise.race([
    reflector.reflect({
      question: params.question,
      generatorAnswer: params.generatorAnswer,
      feedback: feedbackSignalMap[params.outcome],
      skillbook,
    }),
    new Promise<never>((_, reject) => {
      setTimeout(() => reject(new Error("Reflection timed out")), 10_000);
    }),
  ]);

  return { reflectionOutput, interactionId, userId, skillbook, version };
}

The Reflector produces a ReflectorOutput:

interface ReflectorOutput {
  analysis: string;                    // What happened and why
  helpful_skill_ids: string[];         // Skills that contributed to success
  harmful_skill_ids: string[];         // Skills that contributed to failure
  new_learnings: Array<{
    section: string;
    content: string;
    atomicity_score: number;
  }>;
}

Outcome mapping converts database enum values to natural-language feedback for the Reflector:

const feedbackSignalMap = {
  accepted:  "correct — user accepted the Guardian's suggestion",
  overridden:"incorrect — user overrode the Guardian's suggestion",
  wait:      "correct — user chose to wait as suggested",
  abandoned: "neutral — user abandoned without deciding",
  // ...
};

Stage 2: Skill Curation

The SkillManager converts reflections into concrete Skillbook operations:

export async function runSkillUpdate(
  result: LearningPipelineResult
): Promise<UpdateBatch | null> {
  const skillManager = getSkillManager();

  const updateBatch = await skillManager.curate({
    reflectionAnalysis: result.reflectionOutput.analysis,
    skillbook: result.skillbook,
  });

  // Apply to in-memory Skillbook
  result.skillbook.applyUpdate(updateBatch);

  // Persist with optimistic locking
  await persistSkillbookUpdate(result.userId, result.skillbook, result.skillbookVersion);

  return updateBatch;
}

An UpdateBatch contains typed operations:

interface UpdateBatch {
  reasoning: string;       // Why these changes were made
  operations: Array<{
    type: "ADD" | "UPDATE" | "TAG" | "REMOVE";
    section: string;
    content?: string;
    skill_id?: string;
  }>;
}

Stage 3: Persistence with Optimistic Locking

The Skillbook is persisted using optimistic locking to handle concurrent interactions:

async function persistSkillbookUpdate(
  userId: string,
  updatedSkillbook: Skillbook,
  expectedVersion: number
): Promise<boolean> {
  const result = await db
    .update(skillbookTable)
    .set({
      skills: updatedSkillbook.toDict(),
      version: sql`${skillbookTable.version} + 1`,
    })
    .where(
      and(
        eq(skillbookTable.userId, userId),
        eq(skillbookTable.version, expectedVersion)
      )
    );

  return result.rowCount > 0;
}

If a version conflict occurs (another interaction updated the Skillbook while this one was reflecting), the pipeline reloads the fresh Skillbook, re-applies the same UpdateBatch, and retries — up to 3 attempts.

The Dual Feedback Loop

Immediate Feedback (Seconds)

User overrides → feedbackSignalMap["overridden"] → Reflector →
  "Strategy 'impulse-check-generic' was too generic, user felt lectured" →
  SkillManager → TAG harmful, ADD "time-cost-framing" strategy

Retrospective Feedback (Hours/Days)

Ghost Cards resurface past purchases for reflection:

"Still happy with those headphones?" → "Worth it" →
  satisfactionToFeedbackSignal("worth_it", "accepted") →
  "positive_reinforcement — user confirms good decision" →
  Reflector → TAG helpful on the strategy that was used

Why both loops matter: Immediate feedback captures decision quality — did the user listen? Retrospective feedback captures outcome quality — was the decision actually good? A user might override the Guardian (immediate negative signal) but later regret the purchase (retrospective negative signal), confirming the Guardian was right. Or they might accept (immediate positive) but regret (retrospective negative), revealing the strategy was persuasive but unhelpful.

The Counterfactual Proof

Rising acceptance rates alone could be data ordering artifacts. The real proof of learning is the counterfactual:

Interaction A (Day 1):
  Trigger: Late-night electronics, $80
  Strategy: "impulse-check-generic" → "Do you really need this?"
  Result: Override (user annoyed)

Learning cycle runs...

Interaction B (Day 2):
  Trigger: Late-night electronics, $60 (same pattern)
  Strategy: "time-cost-framing" → "That's 4 hours of your hourly rate"
  Result: Wait 24h (user paused)

Same trigger type. Different strategy. Different outcome. This is visible in Opik traces — filter by guardian:therapist:override to see Interaction A, then guardian:therapist:wait to see Interaction B. The learning:skillbook_update trace between them shows the pivot.

Vendored ACE Package

The ACE framework is vendored from kayba-ai/ace-ts@f790a4a at packages/ace/ with three modifications for Pause's build toolchain:

Change	Why
Stripped `.js` extensions from all imports	Turbopack can't resolve `.js` → `.ts` in workspace packages
Stubbed `sentence-transformers` dynamic import	Python-only dependency; Turbopack statically resolves `import()` even behind guards
Migrated to Zod v4 (`z.record(z.string(), z.any())`)	Zod v4 requires explicit key type in `z.record()`

The vendored package has full Biome linting (with targeted overrides for upstream patterns) and TypeScript compliance with verbatimModuleSyntax: true.

ACE + Opik: The Observable Learning Loop

Every ACE operation generates an Opik trace (see the Opik Deep Dive):

ACE Stage	Opik Trace	What's Captured
Reflection	`learning:reflection`	Analysis text, helpful/harmful skill IDs, new learnings count
Skill curation	`learning:skillbook_update`	Operation count, before/after skill count, delta, reasoning, full operations list
Satisfaction feedback	`learning:satisfaction_feedback`	Original outcome, mapped signal, reflection analysis

The Skillbook snapshot is included in update traces, so you can reconstruct the Skillbook's state at any point in time by walking the trace history.

The ACE package also has its own Opik integration layer (packages/ace/src/observability/) with:

OpikIntegration class for framework-level tracing
maybeTrack() decorator for conditional method tracing
Graceful degradation when Opik is unavailable

What We Learned

1. Optimistic locking is essential for Skillbook persistence

Multiple interactions can trigger learning simultaneously. Without version-based locking, later writes silently overwrite earlier learnings. The retry-with-reload pattern ensures no learning is lost.

2. Two temporal feedback loops are more powerful than one

Immediate feedback tells you what the user decided. Retrospective feedback tells you if it was the right decision. The Ghost Card loop catches cases where the Guardian's strategy was persuasive but ultimately unhelpful — a signal you'd never get from click-through rates alone.

3. The adapter layer saved us repeatedly

When the upstream ACE API changed (Zod v4, import paths), fixes were isolated to one file. When Turbopack broke on .js extensions, the fix was in the vendored package — the adapter layer shielded the application code entirely.

4. Seed scripts prove learning more convincingly than live demos

The demo-rookie → demo-pro seed script transition creates a controlled before/after that demonstrates learning in seconds. Without seeds, you'd need 10+ live interactions — impossible in a 5-minute demo. The seed scripts use Skillbook class methods (Skillbook.dumps(), Skillbook.fromDict()) for type-safe serialization.

5. Strategy selection is LLM-driven, not hardcoded

The system prompt includes the Skillbook context with effectiveness scores. The LLM chooses which strategy to use based on skill ratings — we don't hardcode "if score > X, use strategy Y." This means the learning is genuinely emergent, not just a threshold lookup.

The Learning Arc in Numbers

State	Skills	Dominant Strategy	Acceptance Rate
Rookie (seed)	4 generic	`impulse-check-generic`	~30%
After 5 interactions	6-7 mixed	Transitioning	~50%
Pro (seed)	8-10 learned	`time-cost-framing`	~70%+

The Skillbook grows from generic seeds into personalized strategies through real LLM-powered reflection — not hardcoded progression.

File Reference

File	Role
`packages/ace/`	Vendored ACE framework (Skillbook, Reflector, SkillManager, Agent)
`packages/ace/src/observability/`	ACE's own Opik integration layer
`apps/web/src/lib/server/ace.ts`	Adapter layer — all ACE imports flow through here
`apps/web/src/lib/server/learning.ts`	Learning pipeline: reflection, skill curation, persistence
`apps/web/src/lib/server/ghost-cards.ts`	Ghost Card satisfaction feedback signals
`apps/web/src/app/api/ai/feedback/route.ts`	Feedback collection + async learning trigger
`apps/web/src/app/api/ai/ghost-cards/[id]/route.ts`	Retrospective satisfaction feedback
`packages/db/src/schema/`	Skillbook table with JSONB column + version for optimistic locking

Built with Next.js 16 + ACE Framework + Opik + Vercel AI SDK v6 + Neon PostgreSQL

Building a CLI Tool with Cognee: Lessons from 5 Epics

Armel BOBDA — Sun, 25 Jan 2026 21:27:28 +0000

I just finished building Sentinel, a CLI tool that uses Cognee to detect energy conflicts in personal schedules. Five development epics. 860+ tests. Four critical bugs were found and fixed.

Along the way, I learned a lot about working with Cognee that isn't in the documentation. This article shares those lessons so you can avoid my mistakes.

What I Built

Sentinel analyses schedule text and builds a knowledge graph to find "energy collisions"—situations where a draining activity (dinner with a difficult relative) precedes a demanding one (important presentation).

$ sentinel paste < schedule.txt
✓ Extracted 7 entities
Found 6 relationships.
✓ Graph saved to ~/.local/share/sentinel/graph.db

$ sentinel check
⚠️  COLLISION DETECTED                    Confidence: 85%

[Aunt Susan] --DRAINS--> (drained)
                            |
                     CONFLICTS_WITH
                            |
                      (focused) <--REQUIRES-- [Strategy Presentation]

HTML export with collision highlighting:

The tool uses Cognee for entity extraction and relationship building, then applies custom collision detection logic on top.

Here's what I learned.

Lesson 1: Use CYPHER, Not GRAPH_COMPLETION

This one cost me hours of debugging.

The mistake:

# DON'T DO THIS for graph extraction
results = await cognee.search(
    SearchType.GRAPH_COMPLETION,
    query_text="*"
)

My unit tests (with mocked Cognee) passed. Production extracted zero entities.

The problem: GRAPH_COMPLETION returns LLM-generated prose, not structured graph data:

"The schedule contains a dinner event with Aunt Susan on Sunday,
which is described as emotionally draining..."

Useful for chat interfaces. Useless for graph algorithms.

The fix:

from cognee.api.v1.search import SearchType

# Get nodes
node_results = await cognee.search(
    query_text="MATCH (n) RETURN n",
    query_type=SearchType.CYPHER,
)

# Get edges
edge_results = await cognee.search(
    query_text="MATCH (a)-[r]->(b) RETURN a, r, b",
    query_type=SearchType.CYPHER,
)

Takeaway: If you need structured graph data for programmatic use, always use SearchType.CYPHER with explicit Cypher queries.

Lesson 2: Cognee Results Are Deeply Nested

When you get Cypher results back, don't expect a flat list of nodes.

Actual structure:

results = [
    {
        'search_result': [
            [
                [node1_data],  # <-- Your actual node is here
                [node2_data],
                ...
            ]
        ]
    }
]

Access pattern:

def extract_nodes(results):
    if not results:
        return []

    nodes = []
    search_result = results[0].get('search_result', [])

    if search_result:
        node_list = search_result[0]  # First level unwrap
        for node_wrapper in node_list:
            if isinstance(node_wrapper, list) and node_wrapper:
                node = node_wrapper[0]  # Second level unwrap
                nodes.append(node)

    return nodes

Takeaway: Write robust extraction helpers and test them against real Cognee output, not mocks.

Lesson 3: Filter to Entity Nodes Only

Cognee's graph contains multiple node types. Not all of them are what you want.

Node Type	What It Is	Keep?
`Entity`	Actual entities from your text	✅ Yes
`DocumentChunk`	Text segments	❌ No
`EntityType`	Category definitions	❌ No
`TextDocument`	Source document metadata	❌ No
`TextSummary`	LLM-generated summaries	❌ No

Filter pattern:

def extract_entities(nodes):
    return [
        node for node in nodes
        if node.get('type') == 'Entity'
    ]

Without this filter, your graph will be cluttered with infrastructure nodes that aren't useful for domain logic.

Lesson 4: Properties Are JSON Strings

Cognee returns node properties as JSON strings, not Python dicts:

# What you get
node = {
    'id': 'abc-123',
    'name': 'Aunt Susan',
    'type': 'Entity',
    'properties': '{"description": "Family member", "entity_type": "PERSON"}'
}

Parse them:

import json

def parse_properties(node):
    props = node.get('properties', '{}')
    if isinstance(props, str):
        try:
            return json.loads(props)
        except json.JSONDecodeError:
            return {}
    return props if isinstance(props, dict) else {}

Lesson 5: The LLM Will Generate Unexpected Relation Types

This was my biggest surprise. I expected Cognee to use consistent relation type names. Instead:

What I expected:

DRAINS, REQUIRES, INVOLVES, SCHEDULED_AT

What I got (sampling from multiple runs):

drains, depletes, exhausts, causes_fatigue,
emotionally_draining, negatively_impacts,
is_emotionally_draining, energy_draining,
leads_to_exhaustion, causes_exhaustion...

Eleven variations for one concept. Per run.

Why this happens: Cognee's LLM extraction has no ontology constraints. The model generates semantically correct but lexically variable relation names.

The fix: Build a normalisation layer. I wrote a 3-tier matching system:

# Tier 1: Exact match dictionary (85+ entries)
RELATION_MAP = {
    "drains": "DRAINS",
    "depletes": "DRAINS",
    "exhausts": "DRAINS",
    # ...
}

# Tier 2: Keyword matching (stems)
KEYWORDS = {
    "DRAINS": ["drain", "exhaust", "deplet", "fatigue"],
    # ...
}

# Tier 3: Fuzzy matching (RapidFuzz)
from rapidfuzz import fuzz, process
# Match against candidate phrases

Takeaway: Don't assume LLM output will be consistent. Build robust normalisation for any categorical data coming from Cognee.

I wrote a full deep-dive on this pattern: Taming LLM Output Chaos: A 3-Tier Normalisation Pattern

Lesson 6: Custom Prompts Change Everything

Cognee's cognify() function accepts a custom_prompt parameter. This was the key to getting domain-specific relationships.

Default behavior:

Generic entity extraction
Relations like involves, about, scheduled_at
No energy-domain relationships (DRAINS, REQUIRES)

With custom prompt:

EXTRACTION_PROMPT = """
You are extracting a PERSONAL ENERGY knowledge graph.

**REQUIRED RELATIONSHIP TYPES** (use ONLY these):
- DRAINS: Activity depletes energy/focus
- REQUIRES: Activity needs energy/focus
- CONFLICTS_WITH: Energy state conflicts with requirement
- SCHEDULED_AT: Activity occurs at time
- INVOLVES: Activity includes person/thing

**COLLISION PATTERN** (create when applicable):
[draining_activity] --DRAINS--> (energy_state) --CONFLICTS_WITH-->
[requiring_activity] --REQUIRES--> (resource)

**EXAMPLE**:
Input: "Sunday: Draining dinner. Monday: Important presentation."
Graph:
- [dinner] --DRAINS--> (emotional_energy)
- (emotional_energy) --CONFLICTS_WITH--> [presentation]
- [presentation] --REQUIRES--> (sharp_focus)
"""

await cognee.cognify(custom_prompt=EXTRACTION_PROMPT)

Results:

Before custom prompt: ~20% collision detection rate
After custom prompt: ~70% edge type accuracy (still needed normalisation)
After prompt + normalisation: 100% collision detection

Takeaway: Don't fight Cognee's defaults. Guide them with domain-specific prompts that include examples and explicit relationship ontologies.

Lesson 7: Node IDs Vary Too (Semantic Consolidation)

Even with good prompts and relation normalisation, I had one more problem:

Run 1: [dinner] --DRAINS--> (emotional_exhaustion)
Run 2: [dinner] --DRAINS--> (low_energy)
Run 3: [dinner] --DRAINS--> (drained_state)

Same concept, different node labels. My BFS collision detection couldn't find paths because it was doing exact string matching on node IDs.

The fix: Semantic node consolidation using RapidFuzz:

from rapidfuzz import fuzz

def group_similar_nodes(nodes, threshold=70):
    groups = []
    for node in nodes:
        merged = False
        for group in groups:
            if fuzz.WRatio(node.label, group[0].label) >= threshold:
                group.append(node)
                merged = True
                break
        if not merged:
            groups.append([node])
    return groups

def consolidate(graph):
    groups = group_similar_nodes(graph.nodes)
    # Pick canonical representative, rewrite edge references
    # ...

Takeaway: LLM variability affects both relation types AND node identity. Handle both.

Lesson 8: Mocked Tests Will Lie to You

I had 178 tests passing. All green. Two critical bugs in production.

Bug 1: SearchType.GRAPH_COMPLETION returned prose instead of graph data. My mock returned what I expected Cognee to return, not what it actually returns.

Bug 2: Rich console interpreted [node labels] as style markup. My tests didn't render through the actual Rich console.

The fix: Live API tests.

@pytest.mark.live
async def test_real_entity_extraction():
    """Verify actual Cognee behavior."""
    engine = CogneeEngine()
    graph = await engine.ingest("Dinner with Aunt Susan on Sunday")

    assert len(graph.nodes) > 0, "No entities extracted"
    labels = {n.label.lower() for n in graph.nodes}
    assert any("susan" in l for l in labels)

Run them manually before marking stories "done":

# Requires API key
uv run pytest tests/live/ -m live -v

# Skip in CI
uv run pytest -m "not live"

Takeaway: For LLM integrations, unit tests with mocks are necessary but not sufficient. Add live API tests for critical paths.

Lesson 9: Suppress Cognee's Logging (But Keep a Debug Mode)

Cognee produces verbose output during normal operation. Great for debugging, annoying for users.

Solution: Lazy import with suppression:

from contextlib import redirect_stdout, redirect_stderr
from io import StringIO

def get_engine():
    with redirect_stdout(StringIO()), redirect_stderr(StringIO()):
        import warnings
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            from sentinel.core.engine import CogneeEngine
    return CogneeEngine()

But keep a debug flag:

@click.option('--debug', '-d', is_flag=True)
def main(debug):
    if debug:
        engine = CogneeEngine()  # Normal import, verbose
    else:
        engine = get_engine()  # Suppressed

The Journey: 5 Epics in Numbers

Metric	Value
Development epics	5
Stories completed	37
Tests written	860+
Critical bugs found	4
Relation type mappings	85+
Collision detection rate	15% → 100%

The architecture that worked:

User Input
    ↓
Cognee Extraction (custom prompt)
    ↓
3-Tier Relation Mapping (exact → keyword → fuzzy)
    ↓
Semantic Node Consolidation (RapidFuzz grouping)
    ↓
BFS Collision Detection
    ↓
Rich Terminal Output

Each layer handles a different source of LLM variability.

Key Takeaways for Cognee Users

Use SearchType.CYPHER for structured graph data, not GRAPH_COMPLETION.
Expect nested results. Write robust extraction helpers.
Filter to Entity nodes. Cognee returns infrastructure nodes too.
Parse JSON properties. They come as strings.
Normalise relation types. The LLM will surprise you.
Use custom prompts. Domain-specific ontologies need explicit guidance.
Consolidate semantically equivalent nodes. IDs vary like relation types.
Add live API tests. Mocks don't catch integration bugs.
Suppress verbose logging. But keep a debug mode.

Resources

Sentinel on GitHub - The complete implementation
Cognee Documentation - Official docs
RapidFuzz - Fuzzy string matching library

Why Your Calendar App Misses the Real Conflicts - The "why" behind Sentinel
Taming LLM Output Chaos: A 3-Tier Normalization Pattern - Deep-dive on handling LLM variability

Built for the Cognee Mini Challenge 2026 - January Edition. Happy building!

Taming LLM Output Chaos: A 3-Tier Normalisation Pattern

Armel BOBDA — Sun, 25 Jan 2026 21:12:01 +0000

You ask the LLM for "DRAINS" relationships.

You get:

drains, depletes, exhausts, causes_fatigue,
emotionally_draining, negatively_impacts,
energy_draining, leads_to_exhaustion,
saps_vitality, wears_out, causes_drain...

Eleven variations. Per concept. Per run.

If your application depends on consistent output from an LLM, you have a problem. I learned this the hard way while building Sentinel, a CLI tool that uses Cognee to detect energy conflicts in personal schedules.

This article shares the pattern I developed to normalise chaotic LLM output into predictable, application-ready data.

The Problem: LLMs Don't Follow Instructions

I was building a knowledge graph where activities could DRAINS energy or REQUIRES focus. Simple enough. I wrote a custom extraction prompt:

**REQUIRED RELATIONSHIP TYPES** (use ONLY these exact names):
- DRAINS: Activity depletes energy/focus/motivation
- REQUIRES: Activity needs energy/focus/resources

The LLM nodded along and generated... whatever it felt like.

My collision detection algorithm expected DRAINS. Cognee's LLM returned is_emotionally_draining. My BFS traversal found nothing. Tests passed (mocks are liars). Production was broken.

The harsh truth: Prompting alone gets you ~70% consistency. For the remaining 30%, you need a normalisation layer.

Why Prompting Isn't Enough

I tried harder. I added examples. I used few-shot prompting. I YELLED IN CAPS.

**CRITICAL**: Use ONLY these relationship types:
- DRAINS (not "depletes", not "exhausts", not "causes_fatigue")

Result: The LLM now generated DRAINS 85% of the time. But also drains_energy, energy_draining, and my personal favourite: negatively_impacts_emotional_state.

The LLM understands semantics, not syntax. It knows these concepts are equivalent. It doesn't care about your string matching.

My detection rate: 15-20% of actual collisions found.

The Solution: 3-Tier Normalisation

The pattern I landed on processes LLM output through three increasingly fuzzy matching tiers:

┌─────────────────────────────────────────────────────────┐
│                    LLM Output                           │
│              "totally_exhausting_day"                   │
└─────────────────────┬───────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────┐
│  TIER 1: Exact Match (O(1))                             │
│  ───────────────────────────                            │
│  Dictionary lookup. Instant.                            │
│  "drains" → DRAINS ✓                                    │
│  "totally_exhausting_day" → miss, try next tier         │
└─────────────────────┬───────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────┐
│  TIER 2: Keyword Match (O(n))                           │
│  ────────────────────────────                           │
│  Check if input contains known stems.                   │
│  "totally_exhausting_day" contains "exhaust" → DRAINS ✓ │
└─────────────────────┬───────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────┐
│  TIER 3: Fuzzy Match (O(n×m))                           │
│  ─────────────────────────────                          │
│  RapidFuzz similarity scoring against candidates.       │
│  "reduces vitality" ~= "reduces energy" (82%) → DRAINS  │
└─────────────────────────────────────────────────────────┘

Key insight: Most lookups hit Tier 1. Only novel variations fall through to expensive fuzzy matching. Performance stays fast; coverage stays complete.

Implementation

Tier 1: Exact Match Dictionary

The fast path. Cover every variation you've ever seen:

RELATION_TYPE_MAP: dict[str, str] = {
    # DRAINS mappings
    "drains": "DRAINS",
    "depletes": "DRAINS",
    "exhausts": "DRAINS",
    "tires": "DRAINS",
    "fatigues": "DRAINS",
    "causes_fatigue": "DRAINS",
    "drains_energy": "DRAINS",
    "is_emotionally_draining": "DRAINS",
    "emotionally_draining": "DRAINS",
    "causes_exhaustion": "DRAINS",
    "negatively_impacts": "DRAINS",
    "leads_to_exhaustion": "DRAINS",
    # ... 70+ more entries across all types

    # REQUIRES mappings
    "requires": "REQUIRES",
    "needs": "REQUIRES",
    "demands": "REQUIRES",
    "depends_on": "REQUIRES",
    "requires_focus": "REQUIRES",
    "needs_energy": "REQUIRES",
    # ...
}

Lookup:

def map_relation_type(raw_type: str) -> str | None:
    normalized = raw_type.lower()

    # Tier 1: Exact match (O(1))
    if normalized in RELATION_TYPE_MAP:
        return RELATION_TYPE_MAP[normalized]

    # Fall through to Tier 2...

When to grow this dictionary: Every time you see a new variation in logs, add it. This dictionary is your institutional knowledge of LLM quirks.

Tier 2: Keyword Matching

When an exact match fails, check for semantic keywords. Use word stems to catch variations:

SEMANTIC_KEYWORDS: dict[str, list[str]] = {
    "DRAINS": [
        "drain",     # matches: draining, drained, drains
        "exhaust",   # matches: exhausting, exhausted, exhaustion
        "deplet",    # matches: depleting, depleted, depletion
        "fatigue",   # matches: fatiguing, fatigued
        "tire",      # matches: tiring, tired
        "sap",       # matches: sapping, sapped
        "stress",    # matches: stressing, stressed, stressful
    ],
    "REQUIRES": [
        "require",   # matches: requiring, required, requirement
        "need",      # matches: needing, needed, needs
        "demand",    # matches: demanding, demanded
        "depend",    # matches: depending, dependent, dependency
    ],
    "CONFLICTS_WITH": [
        "conflict",  # matches: conflicting, conflicted
        "clash",     # matches: clashing, clashed
        "interfer",  # matches: interfering, interference
        "hinder",    # matches: hindering, hindered
    ],
}

def _keyword_match_relation(relation_type: str) -> str | None:
    normalized = relation_type.lower()

    for canonical_type, keywords in SEMANTIC_KEYWORDS.items():
        for keyword in keywords:
            if keyword in normalized:
                return canonical_type

    return None

Why stems work: The LLM might say energy_depleting or causes_depletion or depleted_state. The stem deplet catches all three.

Tier 3: Fuzzy Matching with RapidFuzz

For truly novel outputs, use semantic similarity scoring:

from rapidfuzz import fuzz, process

FUZZY_CANDIDATES: dict[str, list[str]] = {
    "DRAINS": [
        "drains",
        "drains energy",
        "emotionally draining",
        "causes exhaustion",
        "leads to fatigue",
        "reduces energy",
        "saps energy",
        "wears out",
    ],
    "REQUIRES": [
        "requires",
        "needs",
        "demands",
        "depends on",
        "prerequisite for",
        "essential for",
    ],
    # ...
}

DEFAULT_FUZZY_THRESHOLD: int = 50  # Tune based on your domain

def _fuzzy_match_relation(
    relation_type: str,
    threshold: int = DEFAULT_FUZZY_THRESHOLD
) -> str | None:
    # Normalize: lowercase and spaces instead of underscores
    normalized = relation_type.lower().replace("_", " ")

    best_match: str | None = None
    best_score: float = 0.0

    for canonical_type, candidates in FUZZY_CANDIDATES.items():
        result = process.extractOne(
            normalized,
            candidates,
            scorer=fuzz.WRatio
        )
        if result is not None:
            match_str, score, _ = result
            if score > best_score and score >= threshold:
                best_score = score
                best_match = canonical_type

    return best_match

Why RapidFuzz? It's fast (written in C++), handles partial matches well, and WRatio balances token order flexibility with precision.

Putting It Together

The main mapping function chains all three tiers:

def _map_cognee_relation_to_edge(relation: dict) -> Edge | None:
    relation_type = relation.get("type", "").lower()

    # Tier 1: Exact match (fastest)
    edge_type = RELATION_TYPE_MAP.get(relation_type)
    if edge_type:
        return Edge(
            relationship=edge_type,
            metadata={"match_tier": "exact"},
            # ...
        )

    # Tier 2: Keyword match
    edge_type = _keyword_match_relation(relation_type)
    if edge_type:
        return Edge(
            relationship=edge_type,
            metadata={"match_tier": "keyword"},
            # ...
        )

    # Tier 3: Fuzzy match (slowest, most flexible)
    edge_type = _fuzzy_match_relation(relation_type)
    if edge_type:
        return Edge(
            relationship=edge_type,
            metadata={"match_tier": "fuzzy"},
            # ...
        )

    # No match - log and filter out
    logger.warning("Unknown relation type: %s", relation_type)
    return None

Notice the metadata. Tracking which tier matched helps you:

Debug why something matched (or didn't)
Know when to add new exact mappings (if Tier 3 hits the same input repeatedly)
Monitor your coverage over time

Code simplified for clarity. The actual implementation handles full Edge construction with source/target IDs and confidence scores.

Testing the Pattern

Parametrized tests cover all tiers:

import pytest

@pytest.mark.parametrize("input_type,expected_type,expected_tier", [
    # Tier 1: Exact matches
    ("drains", "DRAINS", "exact"),
    ("requires", "REQUIRES", "exact"),
    ("emotionally_draining", "DRAINS", "exact"),

    # Tier 2: Keyword matches
    ("energy_depleting", "DRAINS", "keyword"),
    ("absolutely_exhausting", "DRAINS", "keyword"),
    ("strongly_dependent", "REQUIRES", "keyword"),

    # Tier 3: Fuzzy matches (no exact or keyword match)
    ("reduces vitality", "DRAINS", "fuzzy"),
    ("critical for success", "REQUIRES", "fuzzy"),
])
def test_3_tier_mapping(input_type, expected_type, expected_tier):
    result = _map_cognee_relation_to_edge({
        "type": input_type,
        "source_id": "a",
        "target_id": "b",
    })

    assert result is not None
    assert result.relationship == expected_type
    assert result.metadata["match_tier"] == expected_tier

Critical: Add real LLM outputs to this test as you discover them. Your test suite becomes documentation of LLM behaviour.

Results

Before the 3-tier pattern:

Collision detection rate: 15-20%
Unmatched relation types: 11+ per run

After:

Collision detection rate: 100%
Unmatched relation types: 0
Performance impact: Negligible (most lookups hit Tier 1's O(1) dictionary)

The journey from broken to working required:

Custom extraction prompt (got edge types right)
3-tier normalisation (got relation mapping right)
Semantic node consolidation (got node matching right)

Each layer catches what the previous layer misses.

Note: Semantic node consolidation is a separate module that merges equivalent nodes (e.g., "emotional_exhaustion" and "low_energy") using similar RapidFuzz techniques. The 3-tier pattern handles relation types; consolidation handles node identity. Both are needed for reliable graph traversal.

When to Use This Pattern

Use 3-tier normalisation when:

Your application depends on categorical LLM output
You're building knowledge graphs, classification systems, or structured extraction
Prompt engineering alone isn't achieving consistency
You need to handle novel variations gracefully

Consider alternatives when:

Output is free-form text (summarization, chat)
You control the training/fine-tuning
Exact matching is acceptable (you can reject unknowns)

Key Takeaways

Prompting is necessary but not sufficient. Even great prompts get ~70-85% consistency.
Build the fast path first. Exact match dictionary handles most cases in O(1).
Use word stems for keyword matching. Catches inflections without regex complexity.
Fuzzy matching is your safety net. RapidFuzz with tunable thresholds handles novel outputs.
Track which tier matched. Metadata enables debugging and coverage monitoring.
Grow your dictionaries from production. Every new variation you see should become an exact match.

Resources

Sentinel on GitHub - The project where this pattern was developed
Cognee - Knowledge graph library for LLM applications
RapidFuzz - Fast fuzzy string matching

Why Your Calendar App Misses the Real Conflicts - The "why" behind Sentinel (general audience)
Building a CLI Tool with Cognee: Lessons from 5 Epics - Cognee-specific integration lessons

Built while participating in the Cognee Mini Challenge 2026 - January Edition. The full implementation is in src/sentinel/core/engine.py.

Forem: Armel BOBDA

Deep Dive: Building Observable AI with Opik

The Problem

Architecture Overview

Pattern 1: Automatic Tracing via Vercel AI SDK

Pattern 2: 16 Named Trace Types — A Taxonomy, Not Just Labels

Pattern 3: The Child Trace Pattern — Working Around Immutability

Pattern 4: Feedback Scores — Turning User Decisions into Metrics

Pattern 5: Learning Pipeline Traces — Observing Self-Improvement

Pattern 6: Degradation Tracing — Even Failures Are Observable

The Fire-and-Forget Philosophy

Reasoning Summary Sanitization

What We Learned

1. Trace immutability changes your architecture

2. Name your traces from Day 1

3. Feedback scores are the killer feature

4. Fire-and-forget is non-negotiable

5. Separate behavioral data from system health

The Opik + ACE Connection

File Reference

Deep Dive: Self-Improving AI with ACE Skillbooks

The Problem

Architecture Overview

The Skillbook: A Living Knowledge Base

The Adapter Layer: All ACE Flows Through One Module

The Learning Pipeline: Three Stages

Stage 1: Reflection

Stage 2: Skill Curation

Stage 3: Persistence with Optimistic Locking

The Dual Feedback Loop

Immediate Feedback (Seconds)

Retrospective Feedback (Hours/Days)

The Counterfactual Proof

Vendored ACE Package

ACE + Opik: The Observable Learning Loop

What We Learned

1. Optimistic locking is essential for Skillbook persistence

2. Two temporal feedback loops are more powerful than one

3. The adapter layer saved us repeatedly

4. Seed scripts prove learning more convincingly than live demos

5. Strategy selection is LLM-driven, not hardcoded

The Learning Arc in Numbers

File Reference

Building a CLI Tool with Cognee: Lessons from 5 Epics

What I Built

Lesson 1: Use CYPHER, Not GRAPH_COMPLETION

Lesson 2: Cognee Results Are Deeply Nested

Lesson 3: Filter to Entity Nodes Only

Lesson 4: Properties Are JSON Strings

Lesson 5: The LLM Will Generate Unexpected Relation Types

Lesson 6: Custom Prompts Change Everything

Lesson 7: Node IDs Vary Too (Semantic Consolidation)

Lesson 8: Mocked Tests Will Lie to You

Lesson 9: Suppress Cognee's Logging (But Keep a Debug Mode)

The Journey: 5 Epics in Numbers

Key Takeaways for Cognee Users

Resources

Related Articles

Taming LLM Output Chaos: A 3-Tier Normalisation Pattern

The Problem: LLMs Don't Follow Instructions

Why Prompting Isn't Enough

The Solution: 3-Tier Normalisation

Implementation

Tier 1: Exact Match Dictionary

Tier 2: Keyword Matching

Tier 3: Fuzzy Matching with RapidFuzz

Putting It Together

Testing the Pattern

Results

When to Use This Pattern

Key Takeaways

Resources

Related Articles