Forem: Stephen Goldberg

Build a Conversational AI Agent on Harper in 5 Minutes

Stephen Goldberg — Fri, 03 Apr 2026 17:21:07 +0000

Building AI agents usually means stitching together a database, a vector store, a caching layer, an API server, and a deployment pipeline. Five services, five sets of credentials, and a weekend gone.

We built an agent that does all of that on Harper — database, vector search, semantic cache, API, and deployment in one runtime. The full source is open. Here's how to get it running.

What You Get

A conversational chat agent powered by Claude with:

Semantic memory — every message is embedded and stored. Ask a question from three conversations ago and it remembers, powered by Harper's built-in HNSW vector index.
Semantic caching — ask the same question twice (or a rephrased version) and it answers instantly from Harper at $0.00 LLM cost. Over time, a popular agent builds a dense cache that handles most queries for free.
Web search — Anthropic's built-in server-side search, no extra API key.
Local embeddings — bge-small-en-v1.5 runs locally via llama.cpp inside Harper. No embedding API, no extra cost.
A chat UI — served directly from Harper at /Chat with per-response cost, latency, and token tracking.
A live savings counter — tracks how much money the cache has saved across all conversations.

Live demo →

Get It Running

Prerequisites: Node.js 22+ and an Anthropic API key.

git clone https://github.com/stephengoldberg/agent-example-harper.git
cd agent-example-harper
npm install

Add your key:

echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env

Start it:

npm run dev

Open http://localhost:9926/Chat. That's it.

On first run, the embedding model (~24 MB) downloads automatically. Every run after that starts in seconds.

See the Cache in Action

Ask a question — say, "Who is the president of the United States?" You'll see the metadata strip under the response:

LATENCY 5.2s · TOKENS 2,400 in / 150 out · COST $0.0098 · WEB 1 search

Now ask it again, or rephrase it — "who's the current US president?" The response comes back instantly:

LATENCY 0.03s · CACHE Served from Harper semantic cache — $0.00 · saved $0.0098

Same answer, zero LLM cost, sub-50ms. Harper's HNSW vector index found that the new question is semantically equivalent (cosine similarity ≥ 0.88) and served the cached response without calling Claude at all.

The savings panel in the sidebar tracks this across every conversation.

How It Works

Everything runs inside Harper — a unified runtime that combines a database, vector index, cache, and API server in one process:

Harper Agent — a JavaScript Resource class that handles POST /Agent. It's your agent logic, running in-process. No Express, no API framework.
Vector Store — @indexed(type: "HNSW", distance: "cosine") in the schema. One line gives you a full vector index. Harper searches it natively with conditions and comparator: 'lt' — no in-memory math, no full scans.
Semantic Cache — two layers. First: exact text match (free, instant). Second: Harper's HNSW index finds semantically similar past questions within a cosine distance threshold. Both run inside the database, not in your application code.
Local SLM — bge-small-en-v1.5 runs via llama.cpp in the same Node.js process. Embeddings cost nothing and never leave the machine.
Claude + Web Search — the only external call. Only happens on cache misses. Anthropic's built-in server-side search means no Google API key.

The entire schema is 24 lines of GraphQL. The agent logic is ~200 lines of JavaScript. There's no ORM, no database driver, no vector database client, no caching library, and no route definitions.

Deploy to Harper Fabric

npm run deploy

One command. Your agent is now running globally on Harper Fabric. No Docker, no Kubernetes, no CI/CD pipeline.

Why Harper

The traditional stack for an AI agent with memory and caching looks like: Postgres + Pinecone + Redis + Express + Docker + a deployment pipeline. That's six services to configure, connect, and maintain.

With Harper, it's one runtime:

What	Traditional	Harper
Database	Postgres/MongoDB	Built in
Vector search	Pinecone/Weaviate	Built in (`@indexed(type: "HNSW")`)
Semantic cache	Redis + custom logic	Built in (native HNSW conditions search)
API server	Express/Fastify	Auto-generated from schema
Embeddings	OpenAI/Voyage API ($)	Local SLM, $0
Deployment	Docker + K8s	`harperdb deploy .`

The cache is the real story at scale. Every question that hits the cache saves you the full cost of a Claude API call — tokens, web searches, and all. The savings compound: the more your agent is used, the less it costs per query.

Clone the repo, add your Anthropic key, and try it.

Build Your Own AI Ops Assistant — Part 6: Ship It

Stephen Goldberg — Thu, 26 Feb 2026 21:25:13 +0000

This is Part 6 of a 6-part series. Part 5 covers the knowledge loop.

Deploy to Production & What to Build Next

You've built an AI assistant that searches your internal tools in parallel, synthesizes answers with Claude, learns from feedback, and lives in Slack. Now let's deploy it to production, look at real costs, and talk about the extensions that take Harper Eye from good to indispensable.

Step 1: Deploy to Harper Fabric

If you've been developing locally, deploying to production is a single command. Make sure your .env has your Fabric credentials:

CLI_TARGET=your-cluster.your-org.harperfabric.com:9925
CLI_USERNAME=your-username
CLI_PASSWORD=your-password

And your CONFIG.env has all your API keys and tokens.

Deploy:

npm run deploy

This runs:

npx -y dotenv-cli -- harperdb deploy . restart=rolling replicated=true

What happens behind the scenes:

Your application code gets pushed to the Fabric cluster
Harper reads schema.graphql and creates/updates all tables and indexes
Environment variables from CONFIG.env are loaded
The cluster does a rolling restart with zero downtime
Your Resource Classes become live HTTPS endpoints

The whole process takes about 5-8 seconds. Your application is now live at:

https://your-cluster.your-org.harperfabric.com/

Update your Slack app's event URLs to point at this endpoint, and you're done.

Step 2: Verify Everything Works

Run through the checklist:

# Health check
curl https://your-cluster.your-org.harperfabric.com/HealthCheck

# API query
curl -X POST https://your-cluster.your-org.harperfabric.com/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'user:pass' | base64)" \
  -d '{"query": "How does replication work?", "mode": "ask"}'

Then in Slack:

/harper-ask how does sharding work?

If you get a structured response with sources from your actual internal systems, you're live. You've just shipped a tool that would cost thousands per month from a vendor.

The Real Cost Breakdown

I've been running Harper Eye in production for our team. Here are the actual numbers:

Monthly Costs

Resource	Cost	Notes
Harper Fabric	~$25	Single instance handles everything: HTTP, database, vector search
Claude API (Anthropic)	~$30-50	~200-400 queries/month at ~$0.10-0.15/query (Sonnet)
Gemini Embeddings	$0	Free tier: 1,500 requests/min. We use maybe 1,000/month
Confluence API	$0	Included in existing Atlassian subscription
Zendesk API	$0	Included in existing subscription
Datadog API	$0	Included in existing subscription
GitHub API	$0	Included (authenticated requests: 5,000/hr)
Slack API	$0	Free for workspace apps
Total	~$55-75/mo	For the entire organization

Cost Per Query

With the knowledge base doing its job, many queries hit the cache and never touch Claude:

Query Type	Cost	Percentage of Queries
KB exact hit	~$0.001 (just the embedding call)	~30% after 2 months
Full orchestration	~$0.10-0.15 (embedding + Claude)	~70% initially, dropping over time
Blended average	~$0.07/query	Getting cheaper every month

The knowledge loop is doing real work. Every verified answer that gets cached means one fewer Claude API call in the future. After two months, about 30% of our queries return instant cached results. That percentage keeps climbing.

What This Replaces

SaaS Alternative	Monthly Cost
Glean or Dashworks (AI search)	$800-2,000
PagerDuty AIOps add-on	$500-1,000
Incident.io or similar	$400-800
Pinecone (vector DB)	$70-200
Total replaced	$1,770-4,000/mo

Savings: $1,700-3,900/month. That's $20,000-47,000/year.

And your custom system is better — it knows your terminology, your architecture, your people. It learns from your team's feedback. It doesn't forget when you cancel a subscription.

Extensions Worth Building

Once you have the core running, these extensions are each a day or less of work:

PagerDuty Webhooks

Create resources/PagerDutyWebhook.js that receives PagerDuty incident webhooks. When a new incident fires, Harper Eye automatically runs the full orchestration and posts the analysis to your Slack incident channel, before any human even looks at it.

export class PagerDutyWebhook extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const event = data?.event;
    if (event?.event_type === 'incident.triggered') {
      const incident = event.data;

      // Auto-analyze the incident
      const result = await orchestrate(
        `PagerDuty incident: ${incident.title}. Service: ${incident.service?.summary}`,
        { mode: 'incident' }
      );

      // Post to your incident channel
      const slack = getSlackClient();
      await slack.chat.postMessage({
        channel: config.slack.incidentChannel(),
        blocks: [
          ...formatPagerDutyHeader(incident),
          ...formatIncidentResponse(result),
        ],
      });
    }
    return { ok: true };
  }
}

The result: PagerDuty fires at 2 am, and by the time your on-call engineer opens Slack, there's already an AI analysis with root cause candidates, relevant runbooks, customer impact assessment, and which colleague to escalate to.

Web Dashboard

The site/ directory serves static HTML at /app/. Build a simple dashboard for:

Knowledge base management: view, edit, and delete KB entries
Query analytics: most asked questions, resolution times, source utilization
Feedback trends: which topics get the most negative feedback (signals for documentation gaps)
Expert map: who on your team is the go-to person for which topics

No React. No build step. Vanilla HTML + CSS + JS that calls your /Api endpoint. Harper serves it as static files.

Knowledge Capture from Slack

Add intent classification to @-mentions so engineers can say @harper-eye save this in a Slack thread, and Harper Eye automatically extracts the key Q&A from the thread discussion and saves it to the knowledge base. The tribal knowledge from a debugging session is preserved forever without anyone having to write a wiki page.

Source Relevance Learning

Track which data sources actually contribute to useful answers. If Zendesk results never get cited for architecture questions, stop searching Zendesk for those queries. This saves API calls and reduces noise in Claude's context.

The Project Structure (Final)

harper-eye/
├── config.yaml              # Harper app config
├── schema.graphql           # 7 tables, 5 vector indexes
├── CONFIG.env               # Secrets (gitignored)
├── .env                     # Deploy creds (gitignored)
├── package.json
│
├── resources/               # HTTP endpoints (Resource Classes)
│   ├── SlackEvents.js       # Slack commands + @mentions
│   ├── SlackInteractivity.js # Feedback button handlers
│   ├── PagerDutyWebhook.js  # PagerDuty auto-analysis
│   ├── Api.js               # REST API for web dashboard
│   ├── HealthCheck.js       # GET /HealthCheck
│   └── Debug.js             # Debug endpoint
│
├── lib/                     # Core business logic
│   ├── orchestrator.js      # AI orchestration (the brain)
│   ├── knowledge-base.js    # KB CRUD + vector search + feedback
│   ├── embeddings.js        # Gemini embedding generation
│   ├── config.js            # Config loader
│   ├── slack-formatter.js   # Slack Block Kit formatting
│   └── slack-mentions.js    # @-mention expert suggestions
│
├── mcp/                     # Data source wrappers
│   ├── confluence.js        # Confluence REST API
│   ├── zendesk.js           # Zendesk REST API
│   ├── datadog.js           # Datadog REST API
│   ├── github.js            # GitHub REST API
│   └── harper-docs.js       # Documentation site search
│
└── site/                    # Static web UI
    ├── index.html           # Knowledge base dashboard
    ├── dashboard.html       # Analytics
    └── help.html            # Help page

Total lines of core logic: about 700. Total external infrastructure: zero (beyond Harper itself). Total time to build: 3 days.

What You've Built

Let's take a final inventory. Look at these two images side by side:

That's not a mockup. That's a production system that 35 engineers use every day, running for under $100/month. Here's what you now have:

An AI assistant that:

Searches 6+ internal data sources in parallel (2-4 seconds)
Synthesizes results with Claude into structured, cited responses
Lives in Slack with slash commands, @mentions, and threaded follow-ups
Returns verified cached answers instantly (sub-second)
Learns from team feedback without manual curation
Automatically degrades and purges bad answers
Knows which engineers are experts on which topics
Costs under $100/month for your entire organization

Running on a stack of:

Harper: one platform for HTTP, database, and vector search
Claude: AI synthesis with structured JSON output
Gemini: embedding generation (free tier)
Vanilla JS: Built into Harper, no framework, no build step

The compounding effects over time:

Month 1: ~0% KB cache hits. Every query goes through full orchestration.
Month 2: ~15-20% cache hits. Common questions return instantly.
Month 3: ~25-35% cache hits. Negative feedback has pruned bad answers.
Month 6: ~40-50% cache hits. Your AI costs are dropping while quality improves.
Month 12+: The knowledge base becomes your most valuable internal asset. It's the institutional memory that survives employee turnover.

The Bigger Picture

Here's what I've learned from building and running Harper Eye:

Custom beats generic, always. No vendor product will ever understand your architecture, your terminology, your people, or your incident history the way a custom tool does. The gap isn't about AI capability; Claude is the same Claude whether you use it through a vendor or directly. The gap is about context. Your context.

The feedback loop is everything. An AI assistant without a learning mechanism is a party trick. One that gets smarter every time someone uses it is an institution. The knowledge loop, verified answers, negative feedback, automatic degradation, is what makes Harper Eye more valuable the longer you run it.

Infrastructure should disappear. I didn't want to manage Postgres, Pinecone, Redis, Express, and a deployment pipeline. Harper let me put all of that in a single schema.graphql and config.yaml. The less time you spend on infrastructure, the more time you spend on the logic that actually makes your tool useful.

Build it yourself, but build it with the right tools. I built Harper Eye in 3 days, not because I'm fast, but because Claude wrote most of the code and Harper eliminated most of the infrastructure. The combination of AI-assisted development and a unified platform is what makes this feasible for a single engineer.

If you've followed along this far, you have everything you need. The code is real. The architecture is battle-tested. The costs are provably lower than the alternatives.

Build it. Run it. Let your team use it. Watch it get smarter.

And stop paying thousands a month for SaaS that doesn't know your name.

*If you build your own version, I'd love to hear about it. Reach out on the Harper Discord.

Build Your Own AI Ops Assistant — Part 5: The Knowledge Loop

Stephen Goldberg — Thu, 26 Feb 2026 18:32:10 +0000

This is Part 5 of a 6-part series. Part 4 covers the Slack integration.

Vector Search, Feedback & Self-Improvement

This is the part that separates a demo from a production system. Most AI assistants are stateless; they forget everything between conversations. Harper Eye gets smarter every time your team uses it. Good answers get cached and returned instantly. Bad answers get flagged, degraded, and eventually purged.

Here's what the knowledge loop looks like after a few weeks of real usage:

29 knowledge base articles, all created automatically when engineers clicked "Helpful." 24 feedback items — teaching the system what to avoid. Zero degraded entries, meaning the self-healing pipeline is working. That's institutional knowledge being built by your team without anyone having to write a wiki page. The knowledge loop is the compound interest of institutional knowledge.

How the Knowledge Loop Works

The beauty of this system: no one has to manually curate the knowledge base. Engineers just click thumbs up or thumbs down. The embeddings and similarity scores handle the rest.

Step 1: The Knowledge Base — Storage & Search

This is the core of the knowledge loop. Create lib/knowledge-base.js:

import { tables } from 'harperdb';
import crypto from 'crypto';
import { generateEmbedding } from './embeddings.js';

const SIMILARITY_HIGH = 0.85;   // Exact match — return cached
const SIMILARITY_LOW = 0.70;    // Partial match — inject as context
const NEGATIVE_FEEDBACK_THRESHOLD = 0.80;
const NEGATIVE_RATIO_DOWNGRADE = 0.3;  // 30%+ negative → downgrade
const NEGATIVE_RATIO_DELETE = 0.5;     // 50%+ negative → delete
const MIN_NEGATIVES_TO_DELETE = 2;     // Need at least 2 negatives

/**
 * Search the knowledge base for verified answers similar to the query.
 * Returns: { match: 'exact' | 'partial' | 'none', entry?, score?, degraded? }
 */
export async function searchKnowledgeBase(queryText, precomputedEmbedding = null) {
  let embedding;
  try {
    embedding = precomputedEmbedding ?? await generateEmbedding(queryText);
  } catch (err) {
    console.error('Embedding failed, skipping KB search:', err.message);
    return { match: 'none' };
  }

  try {
    // Vector similarity search using Harper's built-in HNSW index
    const results = tables.KnowledgeEntry.search({
      sort: { attribute: 'queryEmbedding', target: embedding },
      limit: 1,
    });

    const entries = [];
    for await (const entry of results) {
      entries.push(entry);
    }

    if (entries.length === 0) return { match: 'none' };

    const topEntry = entries[0];
    const storedEmbedding = topEntry.queryEmbedding;
    if (!storedEmbedding?.length) return { match: 'none' };

    // Compute cosine similarity
    const score = cosineSimilarity(embedding, storedEmbedding);

    if (score >= SIMILARITY_HIGH) {
      // Check if the entry has been degraded by negative feedback
      const negativeCount = topEntry.negativeCount ?? 0;
      const total = (topEntry.useCount ?? 0) + negativeCount;

      if (negativeCount > 0 && total > 0) {
        const ratio = negativeCount / total;
        if (ratio >= NEGATIVE_RATIO_DOWNGRADE) {
          // Downgrade: don't return as exact, let Claude regenerate
          return {
            match: 'partial',
            entry: serializeEntry(topEntry),
            score,
            degraded: true,
          };
        }
      }

      // Genuine exact match — increment use count and return
      await updateUseCount(topEntry);
      return { match: 'exact', entry: serializeEntry(topEntry), score };
    }

    if (score >= SIMILARITY_LOW) {
      return { match: 'partial', entry: serializeEntry(topEntry), score };
    }

    return { match: 'none' };
  } catch (err) {
    console.error('KB search failed:', err.message);
    return { match: 'none' };
  }
}

Let me highlight what's happening with Harper's vector search:

tables.KnowledgeEntry.search({
  sort: { attribute: 'queryEmbedding', target: embedding },
  limit: 1,
});

That's it. One line. Harper's built-in HNSW index handles approximate nearest-neighbor search across all stored embeddings. No Pinecone client, no pgvector extension, no separate vector database. You defined @indexed(type: "HNSW", distance: "cosine") in your schema and Harper handles the rest.

Step 2: Storing Verified Knowledge

When an engineer clicks "Helpful," we store the question-answer pair with a vector embedding. Future similar questions will match against it.

/**
 * Store a verified answer in the knowledge base.
 * Called when user clicks thumbs-up.
 */
export async function storeKnowledgeEntry({
  query,
  answer,
  sources,
  originalIncidentId,
  approvedByUserId,
  channelId,
}) {
  const embedding = await generateEmbedding(query);

  const entry = {
    id: crypto.randomUUID(),
    query,
    queryEmbedding: embedding,
    answer: typeof answer === 'string' ? answer : JSON.stringify(answer),
    sources: Array.isArray(sources) ? sources : [],
    originalIncidentId,
    approvedByUserId,
    approvedAt: new Date().toISOString(),
    channelId,
    useCount: 0,
    lastUsedAt: null,
    negativeCount: 0,
  };

  await tables.KnowledgeEntry.put(entry);
  console.log(`KB entry stored: ${entry.id} for "${query.slice(0, 80)}"`);
  return entry;
}

Once stored, the embedding is automatically added to the HNSW index. The next time someone asks a semantically similar question, searchKnowledgeBase() will find it.

Step 3: The Negative Feedback Loop

This is where the system self-heals. Negative feedback does two things:

Warns Claude about past failures so it doesn't repeat the same mistake
Degrades bad KB entries so they stop being returned as cached answers

/**
 * Search for past negative feedback on similar queries.
 * Used by the orchestrator to warn Claude about past failures.
 */
export async function searchNegativeFeedback(queryText, precomputedEmbedding = null) {
  let embedding;
  try {
    embedding = precomputedEmbedding ?? await generateEmbedding(queryText);
  } catch (err) {
    return [];
  }

  try {
    const results = tables.NegativeFeedback.search({
      sort: { attribute: 'queryEmbedding', target: embedding },
      limit: 3,
    });

    const matches = [];
    for await (const entry of results) {
      const stored = entry.queryEmbedding;
      if (!stored?.length) continue;

      const score = cosineSimilarity(embedding, stored);
      if (score >= NEGATIVE_FEEDBACK_THRESHOLD) {
        matches.push({
          category: entry.category,
          details: entry.details,
          originalQuery: entry.originalQuery,
          score,
        });
      }
    }
    return matches;
  } catch (err) {
    console.error('Negative feedback search failed:', err.message);
    return [];
  }
}

/**
 * Store negative feedback with embedding.
 * If linked to a KB entry, degrades that entry.
 */
export async function storeNegativeFeedback({
  queryId,
  originalQuery,
  category,
  details,
  userId,
  channelId,
  knowledgeEntryId,
}) {
  const embedding = await generateEmbedding(originalQuery);

  const record = {
    id: crypto.randomUUID(),
    queryId,
    originalQuery,
    queryEmbedding: embedding,
    category,
    details: details || null,
    userId,
    channelId,
    knowledgeEntryId: knowledgeEntryId || null,
    createdAt: new Date().toISOString(),
  };

  await tables.NegativeFeedback.put(record);

  // Degrade the linked KB entry if one exists
  if (knowledgeEntryId) {
    await degradeKnowledgeEntry(knowledgeEntryId);
  }

  return record;
}

The degradation logic is where it gets interesting:

async function degradeKnowledgeEntry(entryId) {
  try {
    const entry = await tables.KnowledgeEntry.get(entryId);
    if (!entry) return;

    const newNegativeCount = (entry.negativeCount ?? 0) + 1;
    const total = (entry.useCount ?? 0) + newNegativeCount;
    const ratio = newNegativeCount / total;

    if (newNegativeCount >= MIN_NEGATIVES_TO_DELETE && ratio >= NEGATIVE_RATIO_DELETE) {
      // This answer is doing more harm than good — remove it
      await tables.KnowledgeEntry.delete(entryId);
      console.log(`KB entry ${entryId} DELETED (ratio: ${ratio.toFixed(2)})`);
    } else {
      // Just increment the counter
      await tables.KnowledgeEntry.put({
        ...entry,
        negativeCount: newNegativeCount,
      });
      console.log(`KB entry ${entryId} negative count → ${newNegativeCount} (ratio: ${ratio.toFixed(2)})`);
    }
  } catch (err) {
    console.error('Failed to degrade KB entry:', err.message);
  }
}

The thresholds are tuned for safety:

30% negative ratio → the "exact" match gets downgraded to "partial," so Claude regenerates a fresh answer using the old one as reference
50% negative ratio AND at least 2 negatives → the entry is deleted entirely
This means a single bad vote can't kill an entry that's been helpful 10 times

The result: bad knowledge expires naturally. Good knowledge compounds. No human curation required.

Step 4: Cosine Similarity

The math behind the matching:

function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

We compute this manually after Harper's HNSW search because Harper returns the nearest vectors but doesn't expose the similarity score directly. The HNSW index does the heavy lifting (finding the right candidates from potentially thousands of entries); we just compute the final score on the top result.

Step 5: The Use Count Tracker

Every time a cached answer is returned, we increment its use count. This serves two purposes: analytics (which answers are most valuable?) and feedback ratio calculation (a bad answer that's been used 100 times needs more than 2 complaints to be deleted).

async function updateUseCount(entry) {
  try {
    const existing = await tables.KnowledgeEntry.get(entry.id);
    if (!existing) return;
    await tables.KnowledgeEntry.put({
      ...existing,
      useCount: (existing.useCount ?? 0) + 1,
      lastUsedAt: new Date().toISOString(),
    });
  } catch (err) {
    console.error('Failed to update KB use count:', err.message);
  }
}

function serializeEntry(entry) {
  return {
    id: entry.id,
    query: entry.query,
    answer: entry.answer,
    sources: entry.sources,
    approvedByUserId: entry.approvedByUserId,
    approvedAt: entry.approvedAt,
    useCount: entry.useCount,
    negativeCount: entry.negativeCount ?? 0,
  };
}

Notice serializeEntry strips the embedding vector before returning. Those are 768-element float arrays; there is no need to send them downstream or serialize them in responses.

The Full Picture

Here's what your Harper Eye knowledge pipeline looks like now:

Scenario	What Happens	Speed
New question, no KB match	Full orchestration: embed → parallel search → Claude → respond	5-15s
Similar question asked before (verified)	KB exact match → return cached answer	<1s
Similar question but answer was bad	KB match downgraded → Claude regenerates with old answer as context	5-15s
Same mistake pattern detected	Negative feedback injected into Claude prompt → avoids repeating error	5-15s
Terrible answer accumulates complaints	KB entry auto-deleted → system reverts to fresh orchestration	Automatic

Every interaction makes the system better. Every thumbs-up builds institutional knowledge. Every thumbs-down teaches the system what to avoid. This is the compound interest of an AI assistant that works for your company, not a vendor's.

What's Next

In Part 6, we deploy to production, walk through the actual cost breakdown with real numbers, and explore extensions — PagerDuty webhooks for automatic incident analysis, a web dashboard for knowledge base management, and expert routing that knows who on your team to page.

Build Your Own AI Ops Assistant — Part 4: Slack Integration

Stephen Goldberg — Thu, 26 Feb 2026 16:41:41 +0000

This is Part 4 of a 6-part series. Part 3 covers the AI orchestrator.

Slash Commands, Events & Rich Responses

Your AI brain is built. Now let's put it where your team actually works: Slack. By the end of this part, your engineers will be able to type /incident replication lag on us-east or @harper-eye how does sharding work? and get a response that looks like this:

That's a real response to a real question from one of our engineers. Every @-mention, every source citation, every step — generated by the orchestrator from Part 3, formatted with Block Kit, and delivered right where the conversation is already happening.

Why Slack-First Matters

You could build the world's best web dashboard and your team wouldn't use it. Not because the dashboard is bad, but because context switching kills adoption. Your engineers live in Slack. When something breaks at 2 am, they're already in a channel. Making them open a separate tool, log in, type a query, wait for a response, then copy-paste the answer back into Slack, you've already lost them.

The Slack integration isn't a nice-to-have. It is the product for most of your team.

Step 1: Create a Slack App

Go to api.slack.com/apps and create a new app. Choose "From an app manifest" and use this JSON:

{
  "display_information": {
    "name": "Harper Eye",
    "description": "AI-powered ops assistant",
    "background_color": "#1a1a2e"
  },
  "features": {
    "bot_user": {
      "display_name": "Harper Eye",
      "always_online": true
    },
    "slash_commands": [
      {
        "command": "/incident",
        "url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
        "description": "Analyze an incident with AI",
        "usage_hint": "describe the issue",
        "should_escape": false
      },
      {
        "command": "/harper-ask",
        "url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
        "description": "Ask Harper a question",
        "usage_hint": "your question about the platform",
        "should_escape": false
      }
    ]
  },
  "oauth_config": {
    "scopes": {
      "bot": [
        "app_mentions:read",
        "chat:write",
        "commands",
        "channels:history",
        "groups:history",
        "users:read"
      ]
    }
  },
  "settings": {
    "event_subscriptions": {
      "request_url": "https://your-cluster.your-org.harperfabric.com/SlackEvents",
      "bot_events": ["app_mention"]
    },
    "interactivity": {
      "is_enabled": true,
      "request_url": "https://your-cluster.your-org.harperfabric.com/SlackInteractivity"
    }
  }
}

Replace your-cluster.your-org.harperfabric.com with your actual Fabric URL. Install the app to your workspace and grab the tokens from the "OAuth & Permissions" and "Basic Information" pages. Add them to your CONFIG.env:

SLACK_BOT_TOKEN=xoxb-...
SLACK_VERIFICATION_TOKEN=...
SLACK_SIGNING_SECRET=...

Step 2: Build the Slack Event Handler

This is the main Slack endpoint. It handles URL verification, slash commands, and @-mention events — all in a single Resource Class.

Create resources/SlackEvents.js:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';
import { orchestrate } from '../lib/orchestrator.js';
import { generateEmbedding } from '../lib/embeddings.js';
import { searchKnowledgeBase } from '../lib/knowledge-base.js';
import {
  formatIncidentResponse,
  formatPlainText,
  formatAskResponse,
  formatAskPlainText,
} from '../lib/slack-formatter.js';
import { WebClient } from '@slack/web-api';

let slackClient;
function getSlackClient() {
  if (!slackClient) slackClient = new WebClient(config.slack.botToken());
  return slackClient;
}

export class SlackEvents extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const body = data;

    // Slack URL verification — MUST return before any auth check
    if (body?.type === 'url_verification') {
      return { challenge: body.challenge };
    }

    // Verify request came from Slack
    if (!verifySlackRequest(body)) {
      const err = new Error('Invalid request');
      err.statusCode = 401;
      throw err;
    }

    // Slash commands: /incident and /harper-ask
    if (body?.command === '/incident') return handleSlashCommand(body, 'incident');
    if (body?.command === '/harper-ask') return handleSlashCommand(body, 'ask');

    // Event callbacks (@mentions)
    if (body?.type === 'event_callback') return handleEventCallback(body);

    return { ok: true };
  }
}

There are two critical things to notice here:

1. URL verification returns BEFORE auth. When you first configure your Slack app's event URL, Slack sends a challenge request. If you verify the signature first and the verification fails (which it can during initial setup), Slack never gets its challenge response and rejects your URL. Always handle url_verification first.

2. The post() method signature. Remember from Part 2: data is the parsed request body (second argument), not target (first argument). This is the most common mistake when building Harper Resource Classes.

Step 3: Handle Slash Commands

Slash commands have a critical constraint: Slack requires a response within 3 seconds. Your orchestrator takes 5-15 seconds. So the pattern is: acknowledge immediately, process asynchronously, then post the results via Slack's response_url.

function handleSlashCommand(body, mode = 'incident') {
  const { text, user_id, channel_id, response_url } = body;
  const isAsk = mode === 'ask';

  if (!text?.trim()) {
    return {
      response_type: 'ephemeral',
      text: isAsk
        ? 'Usage: `/harper-ask <your question>`'
        : 'Usage: `/incident <describe the issue>`',
    };
  }

  // Acknowledge immediately (returns within 3s)
  // Process asynchronously — fire and forget
  processQueryAsync({
    query: text.trim(),
    userId: user_id,
    channelId: channel_id,
    responseUrl: response_url,
    mode,
  });

  return {
    response_type: 'in_channel',
    text: ':mag: Searching across knowledge base...',
  };
}

The processQueryAsync function runs the orchestrator, formats the response, and posts it back:

async function processQueryAsync({
  query, userId, channelId, responseUrl, threadTs, updateTs, mode = 'incident',
}) {
  try {
    // Fast path: check KB before hitting Claude
    let result;
    let kbHit = false;

    try {
      const embedding = await generateEmbedding(query);
      if (embedding) {
        const kbResult = await searchKnowledgeBase(query, embedding);
        if (kbResult.match === 'exact' && kbResult.entry?.answer) {
          const cached = JSON.parse(kbResult.entry.answer);
          result = {
            summary: cached.summary ?? cached.answer ?? null,
            sources: cached.sources ?? [],
            steps: cached.steps ?? [],
            customerImpact: cached.customerImpact ?? null,
            escalation: cached.escalation ?? null,
            fromKnowledgeBase: true,
            knowledgeBaseEntryId: kbResult.entry.id,
          };
          kbHit = true;
        }
      }
    } catch (kbErr) {
      console.error('KB fast path failed:', kbErr.message);
    }

    // Full path: orchestrate if no KB hit
    if (!kbHit) {
      result = await orchestrate(query, { channelId, mode });
    }

    // Save to audit trail
    const queryId = crypto.randomUUID();
    await tables.IncidentQuery.put({
      id: queryId,
      slackUserId: userId,
      slackChannelId: channelId,
      query,
      response: JSON.stringify(result),
      sourcesUsed: result.sources?.map((s) => s.url ?? s.title) ?? [],
      createdAt: new Date().toISOString(),
    });
    result.queryId = queryId;

    // Format for Slack
    const isAsk = mode === 'ask';
    const blocks = isAsk ? formatAskResponse(result) : formatIncidentResponse(result);
    const fallbackText = isAsk ? formatAskPlainText(result) : formatPlainText(result);

    if (responseUrl) {
      // Slash command → post via response_url
      await fetch(responseUrl, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          response_type: 'in_channel',
          replace_original: true,
          blocks,
          text: fallbackText,
        }),
      });
    } else if (updateTs) {
      // @mention → update the ack message
      const slack = getSlackClient();
      await slack.chat.update({
        channel: channelId,
        ts: updateTs,
        blocks,
        text: fallbackText,
      });
    }
  } catch (err) {
    console.error('Error processing query:', err);
    const errorText = `:warning: Something went wrong: ${err.message}`;
    if (responseUrl) {
      await fetch(responseUrl, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ response_type: 'ephemeral', replace_original: true, text: errorText }),
      });
    }
  }
}

Notice the fast path: before ever calling the orchestrator (and spending API tokens), we check if the knowledge base already has a verified answer. If someone asked this question last week and another engineer confirmed the answer was good, we return it instantly. This is the knowledge loop at work; we'll build it fully in Part 5.

Step 4: Handle @-Mentions with Threaded Conversations

@-mentions are more powerful than slash commands because they support threaded follow-up conversations. Your engineer can ask a question, read the response, then reply in the thread with a follow-up, and Harper Eye has the full conversation context.

async function handleEventCallback(payload) {
  const event = payload.event;

  // Ignore bot messages to prevent loops
  if (event.bot_id || event.subtype === 'bot_message') {
    return { ok: true };
  }

  if (event.type === 'app_mention') {
    // Strip the @mention from the query text
    const query = event.text.replace(/<@[A-Z0-9]+>/g, '').trim();

    if (!query) {
      const slack = getSlackClient();
      await slack.chat.postMessage({
        channel: event.channel,
        thread_ts: event.ts,
        text: 'Ask me anything! Example: `@harper-eye how does replication work?`',
      });
      return { ok: true };
    }

    // Check for thread context (follow-up questions)
    const threadTs = event.thread_ts ?? event.ts;
    let conversationHistory = [];

    if (event.thread_ts) {
      // This is a reply in an existing thread — fetch conversation context
      conversationHistory = await getThreadConversationContext(
        event.channel,
        event.thread_ts
      );
    }

    // Post acknowledgment in the thread
    const slack = getSlackClient();
    const ack = await slack.chat.postMessage({
      channel: event.channel,
      thread_ts: threadTs,
      text: ':mag: Searching across knowledge base...',
    });

    // Process async, update the ack message when done
    processQueryAsync({
      query,
      userId: event.user,
      channelId: event.channel,
      threadTs,
      updateTs: ack.ts,
      conversationHistory,
    });
  }

  return { ok: true };
}

The thread context fetcher pairs user messages with bot responses to build conversation history:

async function getThreadConversationContext(channelId, threadTs) {
  try {
    const slack = getSlackClient();
    const result = await slack.conversations.replies({
      channel: channelId,
      ts: threadTs,
      limit: 20,
    });

    const history = [];
    let lastUserQuery = null;

    for (const msg of result.messages ?? []) {
      if (!msg.bot_id && !msg.subtype && msg.text) {
        lastUserQuery = msg.text.replace(/<@[A-Z0-9]+>/g, '').trim();
      } else if (msg.bot_id && lastUserQuery) {
        const summary = extractSummaryFromBlocks(msg.blocks) || msg.text;
        if (summary?.length > 20) {
          history.push({ query: lastUserQuery, summary });
        }
        lastUserQuery = null;
      }
    }
    return history;
  } catch (err) {
    console.error('Failed to fetch thread context:', err.message);
    return [];
  }
}

Step 5: Format Responses with Block Kit

Slack's Block Kit gives you rich, structured messages with headers, dividers, formatted text, and interactive buttons. Here's the formatter that turns orchestrator results into Slack blocks.

Create lib/slack-formatter.js:

/**
 * Format an incident analysis result into Slack Block Kit blocks.
 */
export function formatIncidentResponse(result) {
  const blocks = [];

  blocks.push({
    type: 'header',
    text: { type: 'plain_text', text: 'Incident Analysis', emoji: true },
  });

  // Knowledge base indicator
  if (result.fromKnowledgeBase) {
    blocks.push({
      type: 'context',
      elements: [{
        type: 'mrkdwn',
        text: ':brain: _Retrieved from knowledge base (verified past answer)._',
      }],
    });
  }

  // Summary
  if (result.summary) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Situation Summary*\n${result.summary}` },
      }
    );
  }

  // Customer Impact
  if (result.customerImpact) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Customer Impact*\n${result.customerImpact}` },
      }
    );
  }

  // Steps
  if (result.steps?.length) {
    const stepsText = result.steps.map((s, i) => `${i + 1}. ${s}`).join('\n');
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Recommended Steps*\n${stepsText}` },
      }
    );
  }

  // Sources with clickable links
  if (result.sources?.length) {
    const sourcesText = result.sources.map((s) => {
      if (typeof s === 'string') return `- ${s}`;
      if (s.url && s.title) return `- <${s.url}|${s.title}>`;
      return `- ${s.title ?? s.url ?? JSON.stringify(s)}`;
    }).join('\n');
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Sources*\n${sourcesText}` },
      }
    );
  }

  // Escalation
  if (result.escalation) {
    blocks.push(
      { type: 'divider' },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `:rotating_light: *Escalation*\n${result.escalation}` },
      }
    );
  }

  // Feedback buttons
  const buttonValue = JSON.stringify({
    queryId: result.queryId ?? null,
    knowledgeEntryId: result.knowledgeBaseEntryId ?? null,
  });

  blocks.push(
    { type: 'divider' },
    {
      type: 'actions',
      block_id: 'feedback_actions',
      elements: [
        {
          type: 'button',
          text: { type: 'plain_text', text: ':thumbsup: Helpful', emoji: true },
          action_id: 'feedback_positive',
          value: buttonValue,
          style: 'primary',
        },
        {
          type: 'button',
          text: { type: 'plain_text', text: ':thumbsdown: Not Helpful', emoji: true },
          action_id: 'feedback_negative',
          value: buttonValue,
        },
      ],
    }
  );

  return blocks;
}

Step 6: Handle Button Interactions

When someone clicks "Helpful" or "Not Helpful," Slack sends a payload to your interactivity endpoint. Create resources/SlackInteractivity.js:

import { Resource, tables } from 'harperdb';
import { config } from '../lib/config.js';
import { storeKnowledgeEntry, storeNegativeFeedback } from '../lib/knowledge-base.js';
import { WebClient } from '@slack/web-api';

let slackClient;
function getSlackClient() {
  if (!slackClient) slackClient = new WebClient(config.slack.botToken());
  return slackClient;
}

export class SlackInteractivity extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    // Slack sends interactivity payloads as form-encoded with a "payload" field
    const payloadStr = data?.payload ?? data;
    const payload = typeof payloadStr === 'string' ? JSON.parse(payloadStr) : payloadStr;

    if (payload.type === 'block_actions') {
      for (const action of payload.actions ?? []) {
        const meta = JSON.parse(action.value ?? '{}');

        if (action.action_id === 'feedback_positive') {
          await handlePositiveFeedback(meta, payload);
        }

        if (action.action_id === 'feedback_negative') {
          await handleNegativeFeedback(meta, payload);
        }
      }
    }

    return { ok: true };
  }
}

async function handlePositiveFeedback({ queryId }, payload) {
  if (!queryId) return;

  try {
    const entry = await tables.IncidentQuery.get(queryId);
    if (!entry) return;

    // Store as verified knowledge for future queries
    await storeKnowledgeEntry({
      query: entry.query,
      answer: entry.response,
      sources: entry.sourcesUsed ?? [],
      originalIncidentId: queryId,
      approvedByUserId: payload.user?.id,
      channelId: entry.slackChannelId,
    });

    // Acknowledge in Slack
    const slack = getSlackClient();
    await slack.chat.postMessage({
      channel: payload.channel?.id,
      thread_ts: payload.message?.ts,
      text: ':white_check_mark: Thanks! This answer has been saved to the knowledge base.',
    });
  } catch (err) {
    console.error('Failed to store positive feedback:', err.message);
  }
}

async function handleNegativeFeedback({ queryId, knowledgeEntryId }, payload) {
  // Open a modal asking what was wrong
  const slack = getSlackClient();
  await slack.views.open({
    trigger_id: payload.trigger_id,
    view: {
      type: 'modal',
      callback_id: 'negative_feedback_modal',
      private_metadata: JSON.stringify({ queryId, knowledgeEntryId }),
      title: { type: 'plain_text', text: 'What went wrong?' },
      blocks: [
        {
          type: 'input',
          block_id: 'category',
          element: {
            type: 'static_select',
            action_id: 'category_select',
            options: [
              { text: { type: 'plain_text', text: 'Inaccurate' }, value: 'inaccurate' },
              { text: { type: 'plain_text', text: 'Outdated' }, value: 'outdated' },
              { text: { type: 'plain_text', text: 'Too Generic' }, value: 'too_generic' },
              { text: { type: 'plain_text', text: 'Missing Info' }, value: 'missing_info' },
            ],
          },
          label: { type: 'plain_text', text: 'Category' },
        },
        {
          type: 'input',
          block_id: 'details',
          optional: true,
          element: {
            type: 'plain_text_input',
            action_id: 'details_input',
            multiline: true,
            placeholder: { type: 'plain_text', text: 'Any specific feedback?' },
          },
          label: { type: 'plain_text', text: 'Details' },
        },
      ],
      submit: { type: 'plain_text', text: 'Submit' },
    },
  });
}

When the user submits the modal, it comes back as a view_submission event. You'd handle that in the same post() method to call storeNegativeFeedback() — which we'll build fully in Part 5.

Deploy & Test

Deploy your updated code:

npm run deploy

Once deployed, verify your Slack app's event URL is pointing at your Fabric URL. Then try it:

/harper-ask how does replication work?

You should see the "🔍 Searching..." acknowledgment, followed by a rich Block Kit response with your answer, sources, and feedback buttons.

Try the @mention in a channel:

@harper-eye what happens when a node goes down?

Then reply in the thread:

@harper-eye what about if it's the primary node?

Harper Eye will include the previous Q&A as context for the follow-up, with no repetition and full continuity.

What You Have Now

Your team can now interact with Harper Eye entirely from Slack:

/incident <description> — incident analysis with remediation steps
/harper-ask <question> — general questions about your platform
@harper-eye <question> — @mention anywhere, supports threaded follow-ups
Feedback buttons — thumbs up saves to knowledge base, thumbs down collects structured feedback
Rich formatting — headers, sections, clickable source links, escalation callouts

All of it running on a single Harper instance. No Lambda functions, no SQS queues, no separate API gateway. One deploy target.

What's Next

In Part 5, we build the knowledge loop, the system that makes Harper Eye smarter every time your team uses it. Vector-indexed knowledge base, semantic similarity search, feedback-driven degradation, and the self-healing answer pipeline.

Build Your Own AI Ops Assistant — Part 3: The AI Brain

Stephen Goldberg — Wed, 25 Feb 2026 23:47:13 +0000

This is Part 3 of a 6-part series. Part 2 covers the foundation setup.

Claude Orchestrator & Multi-Source Search

This is where Harper Eye becomes genuinely powerful. By the end of this part, you'll have an AI assistant that searches all your internal data sources in parallel, synthesizes the results with Claude, and returns structured, cited responses like this:

About 400 lines of code, and every one of them earns its keep.

The Architecture of a Single Query

When someone asks Harper Eye a question, here's exactly what happens:

The key insight: everything that can run in parallel, does. The embedding generation happens once at the start, then gets reused for KB search, negative feedback search, expertise lookup, and code knowledge search. The six data sources all fire simultaneously. The total wall-clock time is dominated by the slowest source (usually 2-4 seconds) plus Claude's synthesis time, not the sum of all sources.

Here's the actual pipeline timing from a production query, you can see the parallelism in action:

Step 1: Build Your First Data Source Wrapper

Every external data source follows the same pattern: define the tools Claude could use, then implement simple fetch() calls to the source's REST API. No MCP subprocess servers, no complex client libraries, just HTTP.

Let's build the Confluence wrapper as our template. Create mcp/confluence.js:

import { config } from '../lib/config.js';

/**
 * Confluence search tool — direct HTTP calls to the REST API.
 * Provides tool definitions (Claude format) and execution functions.
 */

const tools = [
  {
    name: 'search_confluence',
    description: 'Full-text search across Confluence spaces for runbooks, postmortems, and architecture docs.',
    input_schema: {
      type: 'object',
      properties: {
        query: { type: 'string', description: 'Search query (CQL-compatible)' },
        space: { type: 'string', description: 'Optional space key to restrict search' },
        limit: { type: 'number', description: 'Max results (default 5)' },
      },
      required: ['query'],
    },
  },
  {
    name: 'get_confluence_page',
    description: 'Retrieve the body content of a specific Confluence page by ID.',
    input_schema: {
      type: 'object',
      properties: {
        pageId: { type: 'string', description: 'Confluence page ID' },
      },
      required: ['pageId'],
    },
  },
];

export function getConfluenceTools() {
  return tools;
}

export async function executeConfluenceTool(name, input) {
  const baseUrl = config.confluence.baseUrl();
  const auth = Buffer.from(
    `${config.confluence.email()}:${config.confluence.apiToken()}`
  ).toString('base64');
  const headers = {
    Authorization: `Basic ${auth}`,
    Accept: 'application/json',
  };

  switch (name) {
    case 'search_confluence':
      return searchConfluence(baseUrl, headers, input);
    case 'get_confluence_page':
      return getPageContent(baseUrl, headers, input);
    default:
      throw new Error(`Unknown Confluence tool: ${name}`);
  }
}

async function searchConfluence(baseUrl, headers, { query, space, limit = 5 }) {
  // Extract meaningful keywords for broader matching
  const keywords = query
    .toLowerCase()
    .replace(/[^a-z0-9\s\-_]/g, ' ')
    .split(/\s+/)
    .filter((w) => w.length >= 3 && !stopWords.has(w));

  let cql = keywords.length > 0
    ? keywords.map((kw) => `text ~ "${kw}"`).join(' AND ')
    : `text ~ "${query}"`;

  if (space) cql += ` AND space = "${space}"`;

  const url = `${baseUrl}/wiki/rest/api/search?cql=${encodeURIComponent(cql)}&limit=${limit}`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Confluence search failed: ${res.status}`);

  const data = await res.json();
  return data.results.map((r) => ({
    title: r.content?.title ?? r.title,
    id: r.content?.id ?? r.id,
    url: `${baseUrl}/wiki${r.url ?? r.content?._links?.webui ?? ''}`,
    excerpt: stripHtml(r.excerpt ?? ''),
    lastModified: r.lastModified,
  }));
}

async function getPageContent(baseUrl, headers, { pageId }) {
  const url = `${baseUrl}/wiki/rest/api/content/${pageId}?expand=body.storage,version,space`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Confluence get page failed: ${res.status}`);

  const data = await res.json();
  const body = stripHtml(data.body?.storage?.value ?? '');

  // Truncate to ~2000 tokens worth of text (roughly 8000 chars)
  const truncated = body.length > 8000
    ? body.slice(0, 8000) + '\n\n[Content truncated]'
    : body;

  return {
    title: data.title,
    id: data.id,
    space: data.space?.key,
    url: `${baseUrl}/wiki${data._links?.webui ?? ''}`,
    content: truncated,
  };
}

const stopWords = new Set([
  'the', 'what', 'how', 'when', 'where', 'does', 'this', 'that',
  'with', 'for', 'and', 'but', 'not', 'are', 'was', 'can', 'has',
  'have', 'from',
]);

function stripHtml(html) {
  return html
    .replace(/<[^>]*>/g, '')
    .replace(/&nbsp;/g, ' ')
    .replace(/&amp;/g, '&')
    .replace(/&lt;/g, '<')
    .replace(/&gt;/g, '>')
    .replace(/&quot;/g, '"')
    .replace(/\s+/g, ' ')
    .trim();
}

The pattern is always the same for every data source:

Define the tool schema (what Claude sees)
Export getXTools() for tool definitions
Export executeXTool(name, input) as the dispatcher
Implement each tool as a simple fetch() call
Return clean, structured data

Here are the other wrappers you'll build using this same pattern, I'll show the Zendesk one as another example since each API is slightly different:

Create mcp/zendesk.js:

import { config } from '../lib/config.js';

export async function executeZendeskTool(name, input) {
  const subdomain = config.zendesk.subdomain();
  const auth = Buffer.from(
    `${config.zendesk.email()}/token:${config.zendesk.apiToken()}`
  ).toString('base64');
  const headers = {
    Authorization: `Basic ${auth}`,
    Accept: 'application/json',
  };

  if (name === 'search_tickets') {
    return searchTickets(subdomain, headers, input);
  }
  throw new Error(`Unknown Zendesk tool: ${name}`);
}

async function searchTickets(subdomain, headers, { query, limit = 5 }) {
  const url = `https://${subdomain}.zendesk.com/api/v2/search.json?query=${encodeURIComponent(query)}&per_page=${limit}`;
  const res = await fetch(url, { headers });

  if (!res.ok) throw new Error(`Zendesk search failed: ${res.status}`);

  const data = await res.json();
  return (data.results ?? []).map((t) => ({
    id: t.id,
    subject: t.subject,
    status: t.status,
    priority: t.priority,
    description: (t.description ?? '').slice(0, 500),
    url: `https://${subdomain}.zendesk.com/agent/tickets/${t.id}`,
    createdAt: t.created_at,
    updatedAt: t.updated_at,
  }));
}

You'll follow the same pattern for Datadog (mcp/datadog.js), GitHub (mcp/github.js), and your documentation site (mcp/harper-docs.js). Each one is just fetch() calls to their respective REST APIs, returning clean JSON.

Step 2: Build the Embedding Pipeline

Embeddings are how Harper Eye understands meaning, not just keywords. When someone asks "why is my cluster slow?" and a knowledge base entry is titled "investigating replication latency," vector similarity catches the semantic match that keyword search would miss.

Create lib/embeddings.js:

import { config } from './config.js';

/**
 * Generate an embedding vector using Google Gemini.
 * Returns an array of 768 floats (cosine-similarity compatible).
 */
export async function generateEmbedding(text) {
  const apiKey = config.gemini.apiKey();
  const model = config.gemini.embeddingModel();

  const url = `https://generativelanguage.googleapis.com/v1beta/models/${model}:embedContent?key=${apiKey}`;

  const res = await fetch(url, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: `models/${model}`,
      content: { parts: [{ text }] },
      outputDimensionality: 768,
    }),
  });

  if (!res.ok) {
    const body = await res.text().catch(() => '');
    throw new Error(`Gemini embeddings failed: ${res.status} — ${body}`);
  }

  const data = await res.json();
  return data.embedding.values;
}

That's 25 lines. A single API call. And it gives you 768-dimensional vectors that work beautifully with Harper's HNSW indexes.

We use Gemini for embeddings because it's fast, the free tier is generous (1,500 requests/min), and the text-embedding-004 model produces high-quality vectors. You could swap this for OpenAI's embeddings or any other provider; the orchestrator doesn't care.

Step 3: Build the Orchestrator

This is the heart of Harper Eye. The orchestrator coordinates everything: checks the knowledge base, fetches all data sources in parallel, constructs the Claude prompt, parses the response, and records source relevance for future optimization.

Create lib/orchestrator.js:

import Anthropic from '@anthropic-ai/sdk';
import { config } from './config.js';
import { executeConfluenceTool } from '../mcp/confluence.js';
import { executeZendeskTool } from '../mcp/zendesk.js';
import { executeDatadogTool } from '../mcp/datadog.js';
import { executeGitHubTool } from '../mcp/github.js';
import { executeHarperDocsTool } from '../mcp/harper-docs.js';
import { searchKnowledgeBase, searchNegativeFeedback } from './knowledge-base.js';
import { generateEmbedding } from './embeddings.js';

Now for the system prompt. This is the most important prompt you'll ever write. It defines the personality, the constraints, and the output format of every response your team sees:

const SYSTEM_PROMPT = `You are Harper Eye, the internal assistant for [YOUR COMPANY].
You help engineers resolve incidents and answer questions about our platform.

CONTEXT RULES — CRITICAL:
- ALL answers must be specific to [YOUR COMPANY] and our platform.
  Never give generic advice.
- Use our internal terminology (define your key terms here).
- If you cannot find specific information in the provided data,
  say so — do not fill in with generic knowledge.
- Do NOT pad with generic steps like "check logs" or "restart the service"
  unless you can specify EXACTLY which logs, which service.

You will receive data pre-fetched from these sources:
- DOCUMENTATION: Official product docs (most authoritative)
- CONFLUENCE: Internal runbooks, postmortems, architecture docs
- DATADOG MONITORS: Currently alerting monitors and recent events
- DATADOG LOGS: Recent log entries matching the query
- ZENDESK: Customer-reported tickets
- GITHUB: Code search results
- SLACK HISTORY: Recent relevant messages from team channels
- TEAM EXPERTISE MAP: Who to contact for help on these topics

Analyze ALL provided data and respond with a JSON object:
{
  "summary": "What is happening, grounded in the data provided",
  "customerImpact": "Impact from Zendesk data, or null",
  "steps": ["Step 1 as a string", "Step 2", "Step 3"],
  "sources": [{"title": "...", "url": "...", "sourceType": "confluence|zendesk|datadog|github|docs"}],
  "escalation": "Who to loop in based on expertise data, or null"
}

CRITICAL FORMAT RULES:
- "steps" MUST be an array of plain strings. NOT objects.
- Every step should be specific and actionable, citing which source.
- Respond ONLY with the JSON object.`;

You'll want to customize this heavily for your company. The more specific your system prompt is about your architecture, your terminology, and your deprecated vs. current systems, the better Claude's responses will be.

Now the main orchestrate() function:

/**
 * Main orchestration loop.
 * @param {string} query - The user's question
 * @param {Object} options
 * @param {string} [options.channelId] - Slack channel for history search
 * @param {'incident'|'ask'} [options.mode] - Query mode
 * @param {Array} [options.conversationHistory] - Previous Q&A turns
 */
export async function orchestrate(query, {
  channelId,
  mode = 'incident',
  conversationHistory = [],
} = {}) {
  // Step 0: Generate embedding once, reuse everywhere
  let embedding;
  try {
    embedding = await generateEmbedding(query);
  } catch (err) {
    console.error('Embedding generation failed:', err.message);
    embedding = null;
  }

  // Step 1: Check knowledge base + negative feedback in parallel
  const [kbResult, negativeFeedback] = await Promise.all([
    embedding
      ? searchKnowledgeBase(query, embedding)
      : Promise.resolve({ match: 'none' }),
    embedding
      ? searchNegativeFeedback(query, embedding)
      : Promise.resolve([]),
  ]);

  // Step 1a: If KB has a high-confidence cached answer, return immediately
  if (kbResult.match === 'exact' && conversationHistory.length === 0) {
    try {
      const cachedAnswer = JSON.parse(kbResult.entry.answer);
      return {
        ...cachedAnswer,
        fromKnowledgeBase: true,
        knowledgeBaseScore: kbResult.score,
        knowledgeBaseEntryId: kbResult.entry.id,
      };
    } catch {
      // Parse failed — fall through to normal flow
    }
  }

  // Step 2: Build system prompt with KB context + negative feedback warnings
  let systemPrompt = SYSTEM_PROMPT;

  if (kbResult.match === 'partial') {
    // Inject partial match as reference context for Claude
    try {
      const pastAnswer = JSON.parse(kbResult.entry.answer);
      systemPrompt += `\n\nADDITIONAL CONTEXT — A similar past question was answered (similarity: ${kbResult.score.toFixed(2)}). Use as reference but verify:\nPast query: "${kbResult.entry.query}"\nPast answer: ${JSON.stringify(pastAnswer)}`;
    } catch { /* ignore parse errors */ }
  }

  if (negativeFeedback.length > 0) {
    // Warn Claude about past failures so it doesn't repeat them
    const warnings = negativeFeedback
      .map((fb) => `- "${fb.originalQuery}" was marked "${fb.category}"${fb.details ? `: "${fb.details}"` : ''}`)
      .join('\n');
    systemPrompt += `\n\nIMPORTANT — PAST NEGATIVE FEEDBACK:\n${warnings}\nAvoid repeating these mistakes.`;
  }

  // Step 3: Fetch ALL data sources in parallel
  const searchResults = await fetchAllSources(query, embedding, { channelId });

  // Step 4: Build context block from all results
  const contextParts = [];

  if (searchResults.harperDocs?.length)
    contextParts.push(`DOCUMENTATION:\n${JSON.stringify(searchResults.harperDocs)}`);
  if (searchResults.confluence?.length)
    contextParts.push(`CONFLUENCE:\n${JSON.stringify(searchResults.confluence)}`);
  if (searchResults.datadog?.monitors?.length)
    contextParts.push(`DATADOG MONITORS:\n${JSON.stringify(searchResults.datadog.monitors)}`);
  if (searchResults.datadog?.logs?.length)
    contextParts.push(`DATADOG LOGS:\n${JSON.stringify(searchResults.datadog.logs)}`);
  if (searchResults.zendesk?.length)
    contextParts.push(`ZENDESK TICKETS:\n${JSON.stringify(searchResults.zendesk)}`);
  if (searchResults.githubCode?.length)
    contextParts.push(`GITHUB CODE:\n${JSON.stringify(searchResults.githubCode)}`);
  if (searchResults.slackHistory?.length)
    contextParts.push(`SLACK HISTORY:\n${JSON.stringify(searchResults.slackHistory)}`);
  if (searchResults.expertise?.experts?.length)
    contextParts.push(`TEAM EXPERTISE MAP:\n${JSON.stringify(searchResults.expertise)}`);

  const contextBlock = contextParts.length > 0
    ? `\n\nData from all sources:\n\n${contextParts.join('\n\n')}`
    : '\n\nNo relevant data found in any source.';

  const userMessage = `Incident report: ${query}${contextBlock}\n\nAnalyze and respond with JSON.`;

  // Step 5: Call Claude
  const client = new Anthropic({ apiKey: config.claude.apiKey() });

  let response;
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      response = await client.messages.create({
        model: config.claude.model(),
        max_tokens: 4096,
        system: systemPrompt,
        messages: [{ role: 'user', content: userMessage }],
      });
      break;
    } catch (apiErr) {
      if (apiErr.status === 429 && attempt < 3) {
        const retryAfter = Number(apiErr.headers?.['retry-after']) || (attempt * 15);
        await new Promise((r) => setTimeout(r, retryAfter * 1000));
        continue;
      }
      throw apiErr;
    }
  }

  // Step 6: Parse response
  const textBlock = response.content.find((b) => b.type === 'text');
  const rawResponse = textBlock?.text ?? '{}';
  return parseResponse(rawResponse);
}

Step 4: The Parallel Source Fetcher

This is where the "fast" in "12-second response" comes from. Every data source fires at the same time. If Confluence takes 3 seconds and Datadog takes 1 second, the total is 3 seconds, not 4.

/**
 * Fetch data from all configured sources in parallel.
 * Each source is wrapped in try/catch so failures don't block others.
 */
async function fetchAllSources(query, embedding, { channelId } = {}) {
  const results = {
    harperDocs: [],
    confluence: [],
    datadog: { monitors: [], events: [], hosts: [], logs: [] },
    zendesk: [],
    githubCode: [],
    slackHistory: [],
    expertise: { experts: [], channels: [] },
  };

  const fetches = [];

  // Documentation
  fetches.push(
    executeHarperDocsTool('search_harper_docs', { query, limit: 5 })
      .then((r) => { results.harperDocs = r; })
      .catch((err) => console.error('Docs search failed:', err.message))
  );

  // Confluence — wrapped in config check so it's skipped if not configured
  try {
    config.confluence.baseUrl(); // throws if not configured
    fetches.push(
      executeConfluenceTool('search_confluence', { query, limit: 5 })
        .then((r) => { results.confluence = r; })
        .catch((err) => console.error('Confluence failed:', err.message))
    );
  } catch { /* not configured — skip */ }

  // Datadog
  try {
    config.datadog.apiKey();
    fetches.push(
      executeDatadogTool('get_alerting_monitors', {})
        .then((r) => { results.datadog.monitors = r; })
        .catch((err) => console.error('Datadog monitors failed:', err.message))
    );
    fetches.push(
      executeDatadogTool('search_datadog_logs', {
        query: `status:(error OR warn) ${query}`,
        hours: 2,
        limit: 15,
      })
        .then((r) => { results.datadog.logs = r; })
        .catch((err) => console.error('Datadog logs failed:', err.message))
    );
  } catch { /* not configured */ }

  // Zendesk
  try {
    config.zendesk.subdomain();
    fetches.push(
      executeZendeskTool('search_tickets', { query, limit: 5 })
        .then((r) => { results.zendesk = r; })
        .catch((err) => console.error('Zendesk failed:', err.message))
    );
  } catch { /* not configured */ }

  // GitHub
  try {
    config.github.token();
    fetches.push(
      executeGitHubTool('search_github_code', { query, limit: 3 })
        .then((r) => { results.githubCode = r; })
        .catch((err) => console.error('GitHub failed:', err.message))
    );
  } catch { /* not configured */ }

  await Promise.all(fetches);
  return results;
}

The try { config.x.y() } catch {} pattern is intentional. If Datadog isn't configured, the config getter throws, and we skip that source entirely. Zero if statements, zero feature flags. Just configure an env var and the source lights up.

Step 5: The Response Parser

Claude outputs JSON, but LLMs are messy. Sometimes they wrap it in markdown code blocks. Sometimes the JSON has weird types. You need a parser that handles all the edge cases gracefully:

function parseResponse(text) {
  const jsonStr = extractJson(text);

  try {
    const parsed = JSON.parse(jsonStr);
    return normalizeResult(parsed);
  } catch (err) {
    // Last resort: regex out the summary field
    const match = text.match(/"(?:answer|summary)"\s*:\s*"((?:[^"\\]|\\.)*)"/);
    if (match) {
      return {
        summary: match[1].replace(/\\n/g, '\n').replace(/\\"/g, '"'),
        customerImpact: null,
        steps: [],
        sources: [],
        escalation: null,
      };
    }
    // If all parsing fails, return the raw text
    return { summary: text, steps: [], sources: [], escalation: null };
  }
}

function extractJson(text) {
  // Strategy 1: markdown code block
  const codeBlock = text.match(/```
{% endraw %}
(?:json)?\s*([\s\S]*?)
{% raw %}
```/);
  if (codeBlock) return codeBlock[1].trim();

  // Strategy 2: outermost braces
  const first = text.indexOf('{');
  const last = text.lastIndexOf('}');
  if (first !== -1 && last > first) return text.slice(first, last + 1);

  // Strategy 3: raw text
  return text.trim();
}

function normalizeResult(parsed) {
  // Normalize steps to plain strings (Claude sometimes returns objects)
  let steps = [];
  if (Array.isArray(parsed.steps)) {
    steps = parsed.steps.map((s) => {
      if (typeof s === 'string') return s;
      if (typeof s === 'object' && s !== null) {
        return s.step ?? s.action ?? s.description ?? JSON.stringify(s);
      }
      return String(s);
    });
  }

  // Normalize escalation to string
  let escalation = parsed.escalation ?? null;
  if (escalation && typeof escalation === 'object') {
    escalation = escalation.message ?? escalation.text ?? JSON.stringify(escalation);
  }

  return {
    summary: parsed.summary ?? parsed.answer ?? null,
    customerImpact: parsed.customerImpact ?? null,
    steps,
    sources: Array.isArray(parsed.sources) ? parsed.sources : [],
    escalation,
    relatedTopics: Array.isArray(parsed.relatedTopics) ? parsed.relatedTopics : [],
  };
}

The normalization code may look defensive, but it's the difference between a demo that works sometimes and a production system that works always. Claude will occasionally return steps as objects instead of strings, or wrap escalation in a nested object. This parser handles all of it.

Step 6: Wire It Into the API

Now update resources/Api.js to use the real orchestrator:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';
import { orchestrate } from '../lib/orchestrator.js';

export class Api extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const context = this.getContext();
    const authHeader = context.headers.get('authorization');
    if (!authHeader) {
      const err = new Error('Authorization required');
      err.statusCode = 401;
      throw err;
    }

    const { query, mode = 'ask' } = data ?? {};
    if (!query?.trim()) {
      const err = new Error('Missing "query" in request body');
      err.statusCode = 400;
      throw err;
    }

    // Run the full orchestration pipeline
    const result = await orchestrate(query.trim(), { mode });

    // Save to audit trail
    const queryId = crypto.randomUUID();
    await tables.IncidentQuery.put({
      id: queryId,
      query: query.trim(),
      response: JSON.stringify(result),
      sourcesUsed: result.sources?.map((s) => s.url ?? s.title) ?? [],
      createdAt: new Date().toISOString(),
    });
    result.queryId = queryId;

    return result;
  }
}

Test it:

curl -X POST http://localhost:9926/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -d '{"query": "How does replication work in our platform?", "mode": "ask"}'

You should get back a structured JSON response with a summary, sources from your actual data systems, and actionable steps — all grounded in your real documentation and internal knowledge.

What Just Happened

Let's appreciate what you built. In about 400 lines of code:

Six data sources searched in parallel in 2-4 seconds
Claude synthesizes all results into a structured, cited response
Knowledge base is checked first for instant cache hits
Negative feedback is checked to avoid repeating past mistakes
Unconfigured sources are skipped gracefully, no code changes needed
Rate limiting is handled with exponential backoff
Response parsing handles every edge case Claude might throw at you

The orchestrator is the most valuable part of the entire system. Everything else, Slack, the web UI, the feedback loop, is just plumbing to get queries in and results out. This is the brain.

What's Next

In Part 4, we wire this into Slack, slash commands, @-mention handling, threaded conversations, and rich Block Kit responses with feedback buttons. Your team will be able to ask Harper Eye questions without leaving the tool they already live in.

Build Your Own AI Ops Assistant — Part 2: Foundation

Stephen Goldberg — Wed, 25 Feb 2026 23:16:35 +0000

This is Part 2 of a 6-part series on building your own AI-powered internal assistant. Part 1 covers the why and architecture overview.

Harper Setup, Schema & Your First Endpoint

In Part 1, you saw what Harper Eye can do, structured incident analysis in Slack, expert routing, and a self-improving knowledge base. Now we build the foundation. In this part, you'll go from zero to a running application with a database, vector-indexed tables, and a deployed, live REST API. Takes about 30 minutes.

By the end of this article, you'll have the infrastructure for everything you see in this dashboard:

Step 1: Install Harper & Create Your Project

If you want to develop locally first (recommended), install Harper:

npm install -g harperdb

Now scaffold your project:

mkdir harper-eye && cd harper-eye
npm init -y

Update your package.json with the dependencies we'll need:

{
  "name": "harper-eye",
  "version": "1.0.0",
  "description": "AI-powered ops assistant — searches internal tools, synthesizes with Claude, learns from feedback",
  "type": "module",
  "engines": {
    "harperdb": "^4.4"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.39.0",
    "@slack/web-api": "^7.8.0"
  },
  "scripts": {
    "start": "harperdb run .",
    "dev": "harperdb dev .",
    "deploy": "npx -y dotenv-cli -- harperdb deploy . restart=rolling replicated=true"
  }
}

npm install

Notice what's not there: no Express, no database driver, no vector DB client, no Redis. Harper handles HTTP routing, data storage, and vector search natively. Your node_modules stays small.

Step 2: Define Your Database Schema

This is where Harper shines. Instead of writing migrations or setting up a separate database, you define your schema in a single GraphQL file and Harper creates everything automatically.

Create schema.graphql in your project root:

type IncidentQuery @table @export {
  id: ID @primaryKey
  slackUserId: String
  slackChannelId: String
  conversationId: String
  query: String
  response: String
  sourcesUsed: [String]
  createdAt: String
  resolvedAt: String
}

type KnowledgeEntry @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  answer: String
  sources: [String]
  originalIncidentId: String
  approvedByUserId: String
  approvedAt: String
  channelId: String
  useCount: Int
  lastUsedAt: String
  negativeCount: Int
}

type NegativeFeedback @table @export {
  id: ID @primaryKey
  queryId: String
  originalQuery: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  category: String
  details: String
  userId: String
  channelId: String
  knowledgeEntryId: String
  createdAt: String
}

type AuditLog @table @export {
  id: ID @primaryKey
  action: String
  userEmail: String
  query: String
  details: String
  source: String
  queryId: String
  entryId: String
  createdAt: String
}

type SlackExpertise @table @export {
  id: ID @primaryKey
  userId: String
  userName: String
  topic: String
  topicEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  channelId: String
  channelName: String
  answerCount: Int
  questionCount: Int
  lastActiveAt: String
  indexedAt: String
}

type SourceRelevance @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  mode: String
  sourceSignals: String
  createdAt: String
}

type CodeKnowledge @table @export {
  id: ID @primaryKey
  repo: String
  filePath: String
  fileName: String
  fileType: String
  content: String
  contentEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  description: String
  tier: Int
  lastIndexedAt: String
  commitSha: String
  url: String
}

Let me break down what's happening here, because there's a lot of power in these few lines:

@table tells Harper to create a database table. @export exposes it as a REST endpoint automatically. You get full CRUD (PUT, GET, DELETE) for free.

@primaryKey designates the unique identifier.

@indexed(type: "HNSW", distance: "cosine"): this is the magic line. It tells Harper to build an HNSW (Hierarchical Navigable Small World) vector index on that field, using cosine distance for similarity. This is the same algorithm that powers vector search in dedicated vector databases like Pinecone, but it's built into Harper natively. No separate service, no extra cost.

We have four vector-indexed fields across our tables:

Table	Vector Field	Purpose
`KnowledgeEntry`	`queryEmbedding`	Find similar past questions to return cached answers
`NegativeFeedback`	`queryEmbedding`	Find similar past complaints to avoid repeating mistakes
`SlackExpertise`	`topicEmbedding`	Find engineers who are experts on similar topics
`CodeKnowledge`	`contentEmbedding`	Find relevant source code by semantic meaning
`SourceRelevance`	`queryEmbedding`	Learn which data sources matter for which query types

That's five HNSW indexes running natively inside Harper, with zero additional infrastructure.

Step 3: Configure Your Application

Create config.yaml in your project root. This is Harper's application configuration, it tells Harper what to load and how to serve it:

# Load environment variables from CONFIG.env
loadEnv:
  files: 'CONFIG.env'

# Enable the REST API for all exported resources
rest: true

# Read GraphQL schemas to define database tables
graphqlSchema:
  files: 'schema.graphql'

# Load JavaScript modules as HTTP endpoints
jsResource:
  files: 'resources/*.js'

# Serve static web UI files at /app/
static:
  files: 'site/**'
  urlPath: 'app'
  index: true
  extensions: ['html']

Five lines of YAML and you have: environment variable loading, REST API generation, schema-driven database creation, custom HTTP endpoints, and static file serving. No boilerplate. No middleware chain. No app.use() rituals.

Now create your CONFIG.env file for secrets (this file is gitignored, never commit it):

# Claude (Anthropic)
CLAUDE_API_KEY=sk-ant-your-key-here
CLAUDE_MODEL=claude-sonnet-4-20250514

# Gemini (Embeddings)
GEMINI_API_KEY=your-gemini-key-here
GEMINI_EMBEDDING_MODEL=text-embedding-004

# Slack
SLACK_BOT_TOKEN=xoxb-your-bot-token
SLACK_VERIFICATION_TOKEN=your-verification-token
SLACK_SIGNING_SECRET=your-signing-secret

# Confluence (optional — add your data sources as you have them)
CONFLUENCE_BASE_URL=https://yourcompany.atlassian.net
CONFLUENCE_EMAIL=your-email@company.com
CONFLUENCE_API_TOKEN=your-confluence-token

# Zendesk (optional)
ZENDESK_SUBDOMAIN=yourcompany
ZENDESK_EMAIL=your-email@company.com
ZENDESK_API_TOKEN=your-zendesk-token

# Datadog (optional)
DATADOG_API_KEY=your-datadog-api-key
DATADOG_APP_KEY=your-datadog-app-key

# GitHub (optional)
GITHUB_TOKEN=ghp_your-github-token
GITHUB_ORG=your-org-name

Don't worry if you don't have all of these yet. Harper Eye is designed to gracefully skip any source that isn't configured, you can start with just Claude and one data source, then add more over time.

Create a config loader at lib/config.js:

/**
 * Config loader — reads values from environment variables (loaded from CONFIG.env).
 * Each getter throws if the value is missing, so unconfigured sources
 * are caught gracefully by the orchestrator's try/catch blocks.
 */
function env(key) {
  return () => {
    const val = process.env[key];
    if (!val) throw new Error(`Missing env: ${key}`);
    return val;
  };
}

export const config = {
  claude: {
    apiKey: env('CLAUDE_API_KEY'),
    model: () => process.env.CLAUDE_MODEL || 'claude-sonnet-4-20250514',
  },
  gemini: {
    apiKey: env('GEMINI_API_KEY'),
    embeddingModel: () => process.env.GEMINI_EMBEDDING_MODEL || 'text-embedding-004',
  },
  slack: {
    botToken: env('SLACK_BOT_TOKEN'),
    verificationToken: env('SLACK_VERIFICATION_TOKEN'),
    signingSecret: env('SLACK_SIGNING_SECRET'),
  },
  confluence: {
    baseUrl: env('CONFLUENCE_BASE_URL'),
    email: env('CONFLUENCE_EMAIL'),
    apiToken: env('CONFLUENCE_API_TOKEN'),
  },
  zendesk: {
    subdomain: env('ZENDESK_SUBDOMAIN'),
    email: env('ZENDESK_EMAIL'),
    apiToken: env('ZENDESK_API_TOKEN'),
  },
  datadog: {
    apiKey: env('DATADOG_API_KEY'),
    appKey: env('DATADOG_APP_KEY'),
  },
  github: {
    token: env('GITHUB_TOKEN'),
    org: () => process.env.GITHUB_ORG || '',
  },
};

This pattern is intentional. Each config getter is a function that throws if the value is missing. That means the orchestrator can wrap source calls in try/catch and gracefully skip any source that isn't configured. You never have to comment out code or manage feature flags; just add or remove environment variables.

Step 4: Build Your First Resource Class Endpoint

Resource Classes are how you define HTTP endpoints in Harper. If you've used Express, think of them as route handlers, but with a few important differences.

Create the resources/ directory and add a health check endpoint:

mkdir -p resources lib mcp site

Create resources/HealthCheck.js:

import { Resource } from 'harperdb';

export class HealthCheck extends Resource {
  static loadAsInstance = false;

  async get() {
    return {
      status: 'ok',
      service: 'harper-eye',
      timestamp: new Date().toISOString(),
    };
  }
}

That's it. That's a complete HTTP endpoint. The class name HealthCheck becomes the URL path /HealthCheck. The get() method handles GET requests. The return value is automatically serialized to JSON.

Critical rules for Resource Classes (these will save you hours of debugging):

Always use static loadAsInstance = false: this tells Harper to create instances per-request. Without this, you'll get confusing behavior.
Instance methods, not static methods: write async get(), not static async get(). The static methods are Harper's internal entry points.
The first argument to post() is NOT the request body — it's the "target" (resource context). The body is the second argument:

// ✅ CORRECT
async post(target, data) {
  const body = data; // This is your POST body
}

// ❌ WRONG — target is not the body
async post(request) {
  const body = request.body; // This won't work
}

Headers come from this.getContext(), not from a request parameter:

async post(target, data) {
  const context = this.getContext();
  const authHeader = context.headers.get('authorization');
}

To return errors, throw with a statusCode property, don't construct a Response object:

// ✅ CORRECT
const err = new Error('Unauthorized');
err.statusCode = 401;
throw err;

// ❌ WRONG
return new Response('Unauthorized', { status: 401 });

Step 5: Build the REST API Endpoint

Now let's build the endpoint that the web dashboard will use. This accepts questions via HTTP POST and returns AI-generated answers.

Create resources/Api.js:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';

export class Api extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const context = this.getContext();

    // Basic auth check (we'll build proper auth later)
    const authHeader = context.headers.get('authorization');
    if (!authHeader) {
      const err = new Error('Authorization required');
      err.statusCode = 401;
      throw err;
    }

    const { query, mode = 'ask' } = data ?? {};

    if (!query?.trim()) {
      const err = new Error('Missing "query" in request body');
      err.statusCode = 400;
      throw err;
    }

    // For now, return a placeholder — we'll wire in the orchestrator in Part 3
    const queryId = crypto.randomUUID();
    const result = {
      queryId,
      summary: `Received your question: "${query}" — orchestrator coming in Part 3!`,
      sources: [],
      steps: [],
    };

    // Save the query to the audit trail
    await tables.IncidentQuery.put({
      id: queryId,
      query: query.trim(),
      response: JSON.stringify(result),
      sourcesUsed: [],
      createdAt: new Date().toISOString(),
    });

    return result;
  }

  async get() {
    // Return recent queries for the dashboard
    const queries = [];
    for await (const entry of tables.IncidentQuery.search({
      sort: { attribute: 'createdAt', descending: true },
      limit: 20,
    })) {
      queries.push({
        id: entry.id,
        query: entry.query,
        createdAt: entry.createdAt,
      });
    }
    return { queries };
  }
}

Notice how we're using tables.IncidentQuery directly, no connection strings, no query builders, no ORM. Harper tables are imported directly from the harperdb module and are available anywhere in your application code. You put() to write, get() to read by ID, and search() to query.

Step 6: Run It Locally

Start the Harper dev server:

npm run dev

Harper will read your config.yaml, create the tables defined in schema.graphql, and start serving your Resource Classes. You should see output like:

Harper is running
Application loaded from /path/to/harper-eye

Test your health check:

curl http://localhost:9926/HealthCheck

{
  "status": "ok",
  "service": "harper-eye",
  "timestamp": "2025-02-24T10:30:00.000Z"
}

Test the API endpoint:

curl -X POST http://localhost:9926/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -d '{"query": "How does replication work?", "mode": "ask"}'

{
  "queryId": "a1b2c3d4-...",
  "summary": "Received your question: \"How does replication work?\" — orchestrator coming in Part 3!",
  "sources": [],
  "steps": []
}

Your data is already being stored. Check it:

curl http://localhost:9926/IncidentQuery \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)"

You'll see the query you just sent, stored in the database with a UUID, timestamp, and the response. This is the audit trail — every question your team ever asks Harper Eye gets recorded here.

Step 7: Deploy to Harper Fabric

Local development is great, but let's get this live. Sign up for Harper Fabric at fabric.harper.fast if you haven't already, then create a cluster.

Create a .env file (also gitignored) with your deploy credentials:

CLI_TARGET=your-cluster.your-org.harperfabric.com:9925
CLI_USERNAME=your-username
CLI_PASSWORD=your-password

Deploy:

npm run deploy

That's it. Harper deploys your application code, creates the tables from your schema, loads your environment variables, and starts serving. The deploy takes about 5-8 seconds for a restart.

Your application is now live at:

https://your-cluster.your-org.harperfabric.com/

Test it:

curl https://your-cluster.your-org.harperfabric.com/HealthCheck

What You Have So Far

Let's take stock. In about 30 minutes, you've built and deployed:

A database with 7 tables, including 5 HNSW vector indexes for semantic search
A REST API with automatic CRUD for all tables, plus two custom endpoints
An audit trail that records every query
A production deployment on Harper Fabric with a public HTTPS endpoint

Your project structure looks like this:

harper-eye/
├── config.yaml          # Harper app config
├── schema.graphql       # Database tables + vector indexes
├── CONFIG.env           # Secrets (gitignored)
├── .env                 # Deploy credentials (gitignored)
├── package.json
├── lib/
│   └── config.js        # Config loader
├── resources/
│   ├── HealthCheck.js   # GET /HealthCheck
│   └── Api.js           # POST /Api (query endpoint)
├── mcp/                 # (empty — data source wrappers in Part 3)
└── site/                # (empty — web UI in Part 6)

Total lines of code: about 120. Total external services: zero (Harper handles everything). Total monthly cost so far: whatever your Harper Fabric plan costs — which is as low as free.

What's Next

In Part 3, we build the AI brain, the orchestrator that searches all your data sources in parallel, feeds results to Claude, and returns structured, cited responses. That's where Harper Eye goes from "a database with an API" to "the most useful tool your engineering team has ever had."

We'll build:

The MCP tool wrappers for Confluence, Zendesk, Datadog, and GitHub
The embedding generation pipeline using Gemini
The parallel source fetcher
The Claude orchestration loop with structured JSON output
The response parser with fallback strategies

It's about 400 lines of code, and it's the most satisfying code you'll write all year.

Build Your Own AI Ops Assistant — Part 2: Foundation

Stephen Goldberg — Wed, 25 Feb 2026 23:16:35 +0000

This is Part 2 of a 6-part series on building your own AI-powered internal assistant. Part 1 covers the why and architecture overview.

Harper Setup, Schema & Your First Endpoint

By the end of this article, you'll have the infrastructure for everything you see in this dashboard:

Step 1: Install Harper & Create Your Project

If you want to develop locally first (recommended), install Harper:

npm install -g harperdb

Now scaffold your project:

mkdir harper-eye && cd harper-eye
npm init -y

Update your package.json with the dependencies we'll need:

{
  "name": "harper-eye",
  "version": "1.0.0",
  "description": "AI-powered ops assistant — searches internal tools, synthesizes with Claude, learns from feedback",
  "type": "module",
  "engines": {
    "harperdb": "^4.4"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.39.0",
    "@slack/web-api": "^7.8.0"
  },
  "scripts": {
    "start": "harperdb run .",
    "dev": "harperdb dev .",
    "deploy": "npx -y dotenv-cli -- harperdb deploy . restart=rolling replicated=true"
  }
}

npm install

Notice what's not there: no Express, no database driver, no vector DB client, no Redis. Harper handles HTTP routing, data storage, and vector search natively. Your node_modules stays small.

Step 2: Define Your Database Schema

This is where Harper shines. Instead of writing migrations or setting up a separate database, you define your schema in a single GraphQL file and Harper creates everything automatically.

Create schema.graphql in your project root:

type IncidentQuery @table @export {
  id: ID @primaryKey
  slackUserId: String
  slackChannelId: String
  conversationId: String
  query: String
  response: String
  sourcesUsed: [String]
  createdAt: String
  resolvedAt: String
}

type KnowledgeEntry @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  answer: String
  sources: [String]
  originalIncidentId: String
  approvedByUserId: String
  approvedAt: String
  channelId: String
  useCount: Int
  lastUsedAt: String
  negativeCount: Int
}

type NegativeFeedback @table @export {
  id: ID @primaryKey
  queryId: String
  originalQuery: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  category: String
  details: String
  userId: String
  channelId: String
  knowledgeEntryId: String
  createdAt: String
}

type AuditLog @table @export {
  id: ID @primaryKey
  action: String
  userEmail: String
  query: String
  details: String
  source: String
  queryId: String
  entryId: String
  createdAt: String
}

type SlackExpertise @table @export {
  id: ID @primaryKey
  userId: String
  userName: String
  topic: String
  topicEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  channelId: String
  channelName: String
  answerCount: Int
  questionCount: Int
  lastActiveAt: String
  indexedAt: String
}

type SourceRelevance @table @export {
  id: ID @primaryKey
  query: String
  queryEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  mode: String
  sourceSignals: String
  createdAt: String
}

type CodeKnowledge @table @export {
  id: ID @primaryKey
  repo: String
  filePath: String
  fileName: String
  fileType: String
  content: String
  contentEmbedding: [Float] @indexed(type: "HNSW", distance: "cosine")
  description: String
  tier: Int
  lastIndexedAt: String
  commitSha: String
  url: String
}

Let me break down what's happening here, because there's a lot of power in these few lines:

@table tells Harper to create a database table. @export exposes it as a REST endpoint automatically. You get full CRUD (PUT, GET, DELETE) for free.

@primaryKey designates the unique identifier.

We have four vector-indexed fields across our tables:

Table	Vector Field	Purpose
`KnowledgeEntry`	`queryEmbedding`	Find similar past questions to return cached answers
`NegativeFeedback`	`queryEmbedding`	Find similar past complaints to avoid repeating mistakes
`SlackExpertise`	`topicEmbedding`	Find engineers who are experts on similar topics
`CodeKnowledge`	`contentEmbedding`	Find relevant source code by semantic meaning
`SourceRelevance`	`queryEmbedding`	Learn which data sources matter for which query types

That's five HNSW indexes running natively inside Harper, with zero additional infrastructure.

Step 3: Configure Your Application

Create config.yaml in your project root. This is Harper's application configuration, it tells Harper what to load and how to serve it:

# Load environment variables from CONFIG.env
loadEnv:
  files: 'CONFIG.env'

# Enable the REST API for all exported resources
rest: true

# Read GraphQL schemas to define database tables
graphqlSchema:
  files: 'schema.graphql'

# Load JavaScript modules as HTTP endpoints
jsResource:
  files: 'resources/*.js'

# Serve static web UI files at /app/
static:
  files: 'site/**'
  urlPath: 'app'
  index: true
  extensions: ['html']

Now create your CONFIG.env file for secrets (this file is gitignored, never commit it):

# Claude (Anthropic)
CLAUDE_API_KEY=sk-ant-your-key-here
CLAUDE_MODEL=claude-sonnet-4-20250514

# Gemini (Embeddings)
GEMINI_API_KEY=your-gemini-key-here
GEMINI_EMBEDDING_MODEL=text-embedding-004

# Slack
SLACK_BOT_TOKEN=xoxb-your-bot-token
SLACK_VERIFICATION_TOKEN=your-verification-token
SLACK_SIGNING_SECRET=your-signing-secret

# Confluence (optional — add your data sources as you have them)
CONFLUENCE_BASE_URL=https://yourcompany.atlassian.net
CONFLUENCE_EMAIL=your-email@company.com
CONFLUENCE_API_TOKEN=your-confluence-token

# Zendesk (optional)
ZENDESK_SUBDOMAIN=yourcompany
ZENDESK_EMAIL=your-email@company.com
ZENDESK_API_TOKEN=your-zendesk-token

# Datadog (optional)
DATADOG_API_KEY=your-datadog-api-key
DATADOG_APP_KEY=your-datadog-app-key

# GitHub (optional)
GITHUB_TOKEN=ghp_your-github-token
GITHUB_ORG=your-org-name

Create a config loader at lib/config.js:

/**
 * Config loader — reads values from environment variables (loaded from CONFIG.env).
 * Each getter throws if the value is missing, so unconfigured sources
 * are caught gracefully by the orchestrator's try/catch blocks.
 */
function env(key) {
  return () => {
    const val = process.env[key];
    if (!val) throw new Error(`Missing env: ${key}`);
    return val;
  };
}

export const config = {
  claude: {
    apiKey: env('CLAUDE_API_KEY'),
    model: () => process.env.CLAUDE_MODEL || 'claude-sonnet-4-20250514',
  },
  gemini: {
    apiKey: env('GEMINI_API_KEY'),
    embeddingModel: () => process.env.GEMINI_EMBEDDING_MODEL || 'text-embedding-004',
  },
  slack: {
    botToken: env('SLACK_BOT_TOKEN'),
    verificationToken: env('SLACK_VERIFICATION_TOKEN'),
    signingSecret: env('SLACK_SIGNING_SECRET'),
  },
  confluence: {
    baseUrl: env('CONFLUENCE_BASE_URL'),
    email: env('CONFLUENCE_EMAIL'),
    apiToken: env('CONFLUENCE_API_TOKEN'),
  },
  zendesk: {
    subdomain: env('ZENDESK_SUBDOMAIN'),
    email: env('ZENDESK_EMAIL'),
    apiToken: env('ZENDESK_API_TOKEN'),
  },
  datadog: {
    apiKey: env('DATADOG_API_KEY'),
    appKey: env('DATADOG_APP_KEY'),
  },
  github: {
    token: env('GITHUB_TOKEN'),
    org: () => process.env.GITHUB_ORG || '',
  },
};

Step 4: Build Your First Resource Class Endpoint

Resource Classes are how you define HTTP endpoints in Harper. If you've used Express, think of them as route handlers, but with a few important differences.

Create the resources/ directory and add a health check endpoint:

mkdir -p resources lib mcp site

Create resources/HealthCheck.js:

import { Resource } from 'harperdb';

export class HealthCheck extends Resource {
  static loadAsInstance = false;

  async get() {
    return {
      status: 'ok',
      service: 'harper-eye',
      timestamp: new Date().toISOString(),
    };
  }
}

Critical rules for Resource Classes (these will save you hours of debugging):

Always use static loadAsInstance = false: this tells Harper to create instances per-request. Without this, you'll get confusing behavior.
Instance methods, not static methods: write async get(), not static async get(). The static methods are Harper's internal entry points.
The first argument to post() is NOT the request body — it's the "target" (resource context). The body is the second argument:

// ✅ CORRECT
async post(target, data) {
  const body = data; // This is your POST body
}

// ❌ WRONG — target is not the body
async post(request) {
  const body = request.body; // This won't work
}

Headers come from this.getContext(), not from a request parameter:

async post(target, data) {
  const context = this.getContext();
  const authHeader = context.headers.get('authorization');
}

To return errors, throw with a statusCode property, don't construct a Response object:

// ✅ CORRECT
const err = new Error('Unauthorized');
err.statusCode = 401;
throw err;

// ❌ WRONG
return new Response('Unauthorized', { status: 401 });

Step 5: Build the REST API Endpoint

Now let's build the endpoint that the web dashboard will use. This accepts questions via HTTP POST and returns AI-generated answers.

Create resources/Api.js:

import { Resource, tables } from 'harperdb';
import crypto from 'crypto';
import { config } from '../lib/config.js';

export class Api extends Resource {
  static loadAsInstance = false;

  async post(target, data) {
    const context = this.getContext();

    // Basic auth check (we'll build proper auth later)
    const authHeader = context.headers.get('authorization');
    if (!authHeader) {
      const err = new Error('Authorization required');
      err.statusCode = 401;
      throw err;
    }

    const { query, mode = 'ask' } = data ?? {};

    if (!query?.trim()) {
      const err = new Error('Missing "query" in request body');
      err.statusCode = 400;
      throw err;
    }

    // For now, return a placeholder — we'll wire in the orchestrator in Part 3
    const queryId = crypto.randomUUID();
    const result = {
      queryId,
      summary: `Received your question: "${query}" — orchestrator coming in Part 3!`,
      sources: [],
      steps: [],
    };

    // Save the query to the audit trail
    await tables.IncidentQuery.put({
      id: queryId,
      query: query.trim(),
      response: JSON.stringify(result),
      sourcesUsed: [],
      createdAt: new Date().toISOString(),
    });

    return result;
  }

  async get() {
    // Return recent queries for the dashboard
    const queries = [];
    for await (const entry of tables.IncidentQuery.search({
      sort: { attribute: 'createdAt', descending: true },
      limit: 20,
    })) {
      queries.push({
        id: entry.id,
        query: entry.query,
        createdAt: entry.createdAt,
      });
    }
    return { queries };
  }
}

Step 6: Run It Locally

Start the Harper dev server:

npm run dev

Harper will read your config.yaml, create the tables defined in schema.graphql, and start serving your Resource Classes. You should see output like:

Harper is running
Application loaded from /path/to/harper-eye

Test your health check:

curl http://localhost:9926/HealthCheck

{
  "status": "ok",
  "service": "harper-eye",
  "timestamp": "2025-02-24T10:30:00.000Z"
}

Test the API endpoint:

curl -X POST http://localhost:9926/Api \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -d '{"query": "How does replication work?", "mode": "ask"}'

{
  "queryId": "a1b2c3d4-...",
  "summary": "Received your question: \"How does replication work?\" — orchestrator coming in Part 3!",
  "sources": [],
  "steps": []
}

Your data is already being stored. Check it:

curl http://localhost:9926/IncidentQuery \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)"

You'll see the query you just sent, stored in the database with a UUID, timestamp, and the response. This is the audit trail — every question your team ever asks Harper Eye gets recorded here.

Step 7: Deploy to Harper Fabric

Local development is great, but let's get this live. Sign up for Harper Fabric at fabric.harper.fast if you haven't already, then create a cluster.

Create a .env file (also gitignored) with your deploy credentials:

CLI_TARGET=your-cluster.your-org.harperfabric.com:9925
CLI_USERNAME=your-username
CLI_PASSWORD=your-password

Deploy:

npm run deploy

That's it. Harper deploys your application code, creates the tables from your schema, loads your environment variables, and starts serving. The deploy takes about 5-8 seconds for a restart.

Your application is now live at:

https://your-cluster.your-org.harperfabric.com/

Test it:

curl https://your-cluster.your-org.harperfabric.com/HealthCheck

What You Have So Far

Let's take stock. In about 30 minutes, you've built and deployed:

A database with 7 tables, including 5 HNSW vector indexes for semantic search
A REST API with automatic CRUD for all tables, plus two custom endpoints
An audit trail that records every query
A production deployment on Harper Fabric with a public HTTPS endpoint

Your project structure looks like this:

harper-eye/
├── config.yaml          # Harper app config
├── schema.graphql       # Database tables + vector indexes
├── CONFIG.env           # Secrets (gitignored)
├── .env                 # Deploy credentials (gitignored)
├── package.json
├── lib/
│   └── config.js        # Config loader
├── resources/
│   ├── HealthCheck.js   # GET /HealthCheck
│   └── Api.js           # POST /Api (query endpoint)
├── mcp/                 # (empty — data source wrappers in Part 3)
└── site/                # (empty — web UI in Part 6)

Total lines of code: about 120. Total external services: zero (Harper handles everything). Total monthly cost so far: whatever your Harper Fabric plan costs — which is as low as free.

What's Next

We'll build:

The MCP tool wrappers for Confluence, Zendesk, Datadog, and GitHub
The embedding generation pipeline using Gemini
The parallel source fetcher
The Claude orchestration loop with structured JSON output
The response parser with fallback strategies

It's about 400 lines of code, and it's the most satisfying code you'll write all year.

I Built an AI Ops Assistant That Replaced $2,400/month in SaaS — In 3 Days, for Under $100/month

Stephen Goldberg — Wed, 25 Feb 2026 18:09:26 +0000

Last month, one of our engineers typed this into Slack:

/incident what would cause basic authentication to fail even with a properly formed authentication header when the user is present and the roles are correct

Nine seconds later — before anyone else had even read the message — Harper Eye posted a full incident analysis in the thread. It had identified the root cause (a misconfigured authorizeLocal setting in harperdb-config.yaml), cited three internal sources (our documentation, codebase, and Slack history), listed specific remediation steps, and @-mentioned the five engineers with the most expertise in authentication — by name, with clickable Slack handles.

No one had to context-switch. No one had to dig through four different dashboards. No one had to remember which runbook lived where. The answer was right there in the channel, with clickable links to every source.

That moment crystallized something I'd been feeling for a while: the most valuable internal tool your company will ever have is one that connects your existing knowledge and puts it in the hands of every engineer, instantly.

And you don't need to buy it from a vendor. You can build it yourself.

The Hidden Cost of Scattered Engineering Knowledge

If you run an engineering team, this scenario is your daily reality:
Your knowledge is scattered across six or more systems, documentation wikis, support tickets, monitoring dashboards, chat history, code repos, and incident postmortems. When something breaks at 2 am, your on-call engineer has to manually search each one, piece together context from fragments, and hope they find the right runbook before customers start complaining.

The real cost isn't the monthly SaaS bills (though we'll get to those). It's the compounding tax on every engineer, every day:

Time to resolution goes up. A junior engineer who joined last quarter doesn't know that "replication lag on us-east" was caused by a misconfigured sharding key three months ago. That tribal knowledge lives in a Slack thread they'll never find.

Expertise walks out the door. When your senior engineer leaves, their mental model of the system — the one that connects a Datadog alert to a Confluence runbook to a specific config file — goes with them.

Duplicate work compounds silently. The same question gets answered differently by different people in different channels. There's no feedback loop. No institutional memory.

Most companies try to solve this by buying SaaS tools. An AI chatbot here, an incident management platform there, a knowledge base product, a Slack integration. Before you know it, you're paying $2,000-5,000/month across multiple vendors, and none of them actually know your systems, your architecture, your people, or your internal terminology.

What Harper Eye Actually Does (And Why It's Different)

Harper Eye is not a chatbot with a Confluence integration. It's an AI orchestrator that treats your entire internal knowledge surface as a single searchable brain.

Here's what happens when someone asks Harper Eye a question, and I mean exactly what happens, because I'll show you every line of code in this series:

1. Instant Knowledge Base Check

Before anything else, Harper Eye checks if this question (or something semantically similar) has been asked and answered before. If a verified answer exists with high confidence, it returns it instantly, no API calls, no waiting. Sub-second response.

2. Parallel Multi-Source Search

If there's no cached answer, Harper Eye searches all your data sources simultaneously, not sequentially, not one-at-a-time. In parallel:

Internal documentation (Confluence, wikis, runbooks)
Support tickets (Zendesk — what are customers actually reporting?)
Monitoring data (Datadog — what's alerting right now? What do host metrics show?)
Source code (GitHub — what does the code actually do?)
Chat history (Slack — what have engineers said about this topic recently?)
Team expertise (Who on your team has answered similar questions before?)

This parallel search typically completes in 2-4 seconds across all sources. Here's what the pipeline timing actually looks like for a real query:

That's a real query, not a benchmark. 135 milliseconds for the embedding, 7 milliseconds for the knowledge base search, 3.5 seconds for all six data sources in parallel, and 5.5 seconds for Claude to synthesize. Nine seconds from question to fully cited answer.

3. AI Synthesis with Claude

All that raw data gets fed to Claude with a system prompt that is deeply specific to your company. Not generic advice — your architecture, your terminology, your config files, your API endpoints. Claude synthesizes it into a structured response:

Look at that response. It's not generic. It names the specific replication engine (Plexus), references the exact config file (harperdb-config.yaml), cites Slack conversations about a real customer deployment, and names five specific engineers to contact — all grounded in actual internal data, not LLM hallucination.

4. The Knowledge Loop (This Is the Magic)

After every response, engineers can give feedback — thumbs up or thumbs down. When someone marks an answer as helpful, it gets stored in a vector-indexed knowledge base. The next time a similar question comes in, Harper Eye returns that verified answer instantly.

When someone marks an answer as not helpful, that negative signal gets stored too. If an answer accumulates enough negative feedback, Harper Eye automatically degrades it — forcing a fresh search instead of returning stale information. Eventually, bad answers get purged entirely.

Your AI assistant gets smarter every time your team uses it. No retraining, no manual curation. Just a feedback loop that compounds institutional knowledge over time.

5. Expert Routing — It Knows Your People

This is the feature that no vendor tool offers. Harper Eye indexes your Slack conversations to build a map of which engineers answer questions about which topics. Not who's assigned to a team — who actually demonstrates expertise by helping others.

That's a live skills matrix with 35 engineers across 17 topics, built entirely by mining Slack history. When an incident requires escalation, Harper Eye doesn't guess; it tells you exactly who to loop in, based on who's actually answered questions about that topic before.

The Architecture: Surprisingly Simple

Here's the part that surprised me most: the entire system is about 700 lines of core logic running on a single platform.

The entire application runs on Harper; it's simultaneously the application server, the database, and the vector search engine. There's no separate Postgres for storage, no Pinecone for embeddings, no Redis for caching, no Express for HTTP. One platform, one deploy target, one bill.

Here's what the stack looks like:

Layer	Technology	Why
Runtime + Database + Vector Search	Harper (Fabric)	One platform handles HTTP endpoints, data storage, and HNSW vector indexing. No infrastructure to manage.
AI Synthesis	Claude (Anthropic API)	Structured JSON output, handles massive context windows for multi-source analysis.
Embeddings	Google Gemini	768-dimension vectors for semantic search. Free tier covers most usage.
Data Sources	Direct HTTP APIs	Simple `fetch()` calls to Confluence, Zendesk, Datadog, GitHub, and Slack APIs.
Frontend	Vanilla HTML/CSS/JS	Static files served by Harper. No build step, no React, no webpack.

Proof This Works: Real Numbers from Production

This isn't a prototype. Harper Eye has been running in production at Harper, serving a team of 35 engineers. Here are the real numbers:

222 queries answered. 29 knowledge base articles built automatically from positive feedback. 24 feedback items shaping future responses. 12+ engineers actively using it across the team. All of that from a system that costs us less than $100/month to run.

The Cost Breakdown That Changes the Math

Here's what a typical "AI-powered internal knowledge" stack costs from vendors:

Tool	Monthly Cost	What You Get
Guru / Glean / Dashworks	$800-2,000	AI search across your tools
PagerDuty AIOps add-on	$500-1,000	AI incident triage
Incident.io or similar	$400-800	Incident management + Slack bot
Vector DB (Pinecone, etc.)	$70-200	Embedding storage
Total	$1,770-4,000/mo	Separate tools that don't talk to each other

Here's what Harper Eye costs to run:

Resource	Monthly Cost
Harper Fabric (1 instance)	~$25
Claude API (Anthropic)	~$30-50 (depends on query volume)
Gemini embeddings	~$0 (free tier)
Confluence/Zendesk/Datadog APIs	$0 (included in existing subscriptions)
Total	~$55-75/mo

That's not a typo. Under $100/month to replace thousands in SaaS, with a system that's more integrated, more specific to your company, and gets smarter over time.

The gap is even wider when you factor in what vendors can't do: search your actual Slack history, know which engineers are experts on which topics, learn from your team's feedback, or speak in your company's specific terminology.

Why I Built This Instead of Buying It

I want to be direct about the motivation. I'm a technical co-founder at Harper. I could have justified buying a tool; there are plenty of good ones out there, and the budget would have been approved.

But here's the thing: every SaaS tool you buy is someone else's opinion about how your problem should be solved. It's built for the average customer, not for your specific architecture, your specific team, your specific knowledge topology.

Harper Eye knows that our replication engine is called Plexus (not NATS, that's deprecated). It knows that "Custom Functions" is a legacy term, and the current pattern is Resource Classes. It knows that when someone asks about "Fabric," they mean our managed cloud platform, not a generic concept. And when it escalates an incident, it doesn't tell you to "contact the on-call team" — it @-mentions @Kris and @David by name, because it knows from Slack data that they've answered 28 and 32 questions about clustering, respectively.

No vendor product will ever know that. And more importantly, you know things about your company that no vendor ever will. The question isn't whether AI can help your team, it's whether you want that AI to work for a vendor, or for you.

Let's build it.

What We're Building in This Series

Over the next five parts, you'll build your own version of Harper Eye from scratch. Every line of code is real, tested, and production-ready. By the end, you'll have a system that does everything you've seen in the screenshots above, running on your infrastructure, for your team.

Part 2: Foundation — Harper Setup, Schema & Your First Endpoint

Set up Harper, define your database schema with vector-indexed tables, and build your first Resource Class endpoint. By the end, you'll have a running application with a REST API.

Part 3: The AI Brain — Claude Orchestrator & Multi-Source Search

The core of the system. Build the orchestrator that searches all your data sources in parallel, feeds results to Claude, and returns structured, cited responses. This is where the real magic happens.

Part 4: Slack Integration — Slash Commands, Events & Rich Responses

Wire it into Slack so your team can use slash commands, @-mention the bot, and get beautifully formatted responses with feedback buttons — all inside the tool they already live in.

Part 5: The Knowledge Loop — Vector Search, Feedback & Self-Improvement

Build the system that makes Harper Eye smarter over time. Embedding generation, semantic similarity search, feedback collection, knowledge degradation, and expert routing.

Part 6: Ship It — Deploy to Production & What to Build Next

Deploy to Harper Fabric, walk through the real cost breakdown, and explore extensions like PagerDuty webhooks, a web dashboard, and automated knowledge capture from Slack threads.

Each part stands on its own — you'll have a working, deployed system after every article. But together, they build something that fundamentally changes how your team operates.

Prerequisites

To follow along, you'll need:

Node.js (v20+ recommended)
Harper Fabric account (free to sign up at fabric.harper.fast)
Anthropic API key (for Claude — console.anthropic.com)
Slack workspace where you can create an app
At least one data source to connect (Confluence, Zendesk, Datadog, or GitHub — even one is enough to start)

No prior experience with Harper is required. If you can write JavaScript and have used a REST API, you have everything you need.

That’s a Wrap - HarperDB 2020

Stephen Goldberg — Tue, 22 Dec 2020 17:42:16 +0000

Though this year has been quite different for everyone, the team at HarperDB has made great strides from both a product and organization standpoint. Before 2021 arrives, we wanted to take a moment to summarize what we’ve accomplished this year, how we overcame challenges, and what we look forward to next year. HarperDB has an energetic and dynamic team, always looking to stay ahead of the curve. Our product is breaking records on data read and write speeds, simplifying the data pipeline, and changing the way people think about data management as a whole. We launched HarperDB Cloud (our managed and hosted offering) in April, released many product feature updates, and worked with top tier companies to provide innovative solutions on a large scale… it’s all pretty exciting, to say the least.

Of course, there have been challenges; we had to make cuts, reduce costs, and made a lot of difficult decisions. We’ve been working remotely since March, and our team has never been more productive, specifically our development team. The entire HarperDB staff has stayed healthy and stuck together during these times, and we are coming out stronger on the other end. We have a lot to be grateful for, and I think it’s safe to say that HarperDB is currently in the best position we’ve ever been.

We asked each of our team members to share a few sentences about how this year has been, what they worked on, and what they’ll be focusing on next year:

Kyle Bernhardy, Co-Founder & CTO

Both personally & professionally this year we had to navigate experiences we have never faced before. Building a cloud platform from the ground up; replacing our storage engine with a net new technology; facing the uncertainty of a pandemic; bringing our team remote in the middle of a major development cycle; maintaining company culture, connection & collaboration; delivering solutions & support to our customers & community; continuing to innovate. These are just some of the challenges from 2020. Facing these challenges created the opportunities of a vastly more accessible, performant product with a sizzling front end Studio, a resilient team that has come together and leaned into individual strengths to bolster each other with creativity, intelligence & support; a growing community of developers & customers who believe in our vision. There have been cycles of fear & anxiety but also cycles of focus, growth & restoration that have got us through this year with more challenges & opportunities to tackle together. The highlight of my year & what I am most proud to come through 2020 is the HarperDB team. It is our individual members together who create solutions that delight customers and make our mission & vision possible.

My focus for 2020 was on building a solid foundation for the HarperDB product & engineering team. Personally, I built a new data storage engine from the ground up with LMDB as the underlying data store. This gave us upwards 700x performance improvements over our previous state & boosted us in outperforming a number of competitive products on the market as well as set us up for future benefits to be implemented in upcoming releases. I have been researching how our inter-node communication layer & distribution can be enhanced & have been deeply involved in PoC development for future projects. On the team front to guide the engineering team through development & release cycles and refine our processes to be better suited to remote collaboration.

My focus will continue to be focused on advanced R&D and fostering the engineering team. Early next year I will be finishing & releasing across the board data engine enhancements which will provide better indexing, read & write performance. Following that I will build out backup/import functionality and assist in customer project success. I am most looking forward to seeing us execute our vision on the business side and getting HarperDB in the hands of more devs & use cases.

Zach Fowler, Co-Founder & CPO

To be perfectly honest this year at HarperDB has been the best so far, save not having the team a few steps away. I have come to appreciate me as not a worker bee but as a contributor to a larger community. Highlights have been working and supporting clients, the amazing things they are building with HarperDB. Watching the community build tools for HarperDB and being an endless source of ideas and features.

2020 was busy and the team carried out many extraordinary endeavors. HarperDB Cloud as a Service was a gigantic undertaking, and we implemented it in short order with great success!

I did a lot of work in the containerization of HarperDB and the ecosystems that support containers, specifically Kubernetes. This is an ongoing challenge but is very rewarding learning and adapting HarperDB for these use cases. This work has prepared HarperDB for opportunities in amazing partnerships that we should see come to fruition in the coming months.

In the new year I am excited to continue to watch the community expand. I will be continuing to focus on improving and increasing HarperDB deployment channels; expanding the Kubernetes offering, adding alternative cloud providers, Azure, GCP, Packet. It is going to be a good year, very proud to have come this far. Cheers! And Happy New Year.

Fred Yoon, Co-Founder & COO

It has undoubtedly been a challenging year for everyone due to the uncertainties of Covid. Overall for me, I had a smooth transition working from home. However, I didn't realize how hard it could be with the entire family at the house at all times. I'm proud of the entire HarperDB team as everyone has seemingly adapted well to the new challenges of working exclusively from home while maintaining their productivity. It has allowed us to end the year on a high note by successfully closing a Series Seed financing round and positioning the company for success in 2021.

This year, some of my main goals and accomplishments were to help the organization navigate through Covid-19 operationally and financially, nurture and finalize strategic partnerships with Lumen and Section.io, and complete our Series Seed financing.

2021 will be more of Covid and hopefully the transition to the end of the pandemic and more great opportunities for HarperDB. Next year, my goals will be centered around successful partnerships and helping close out the significant sales opportunities developed this year. I look forward to a very prosperous New Year for our company.

Mark Hardy, VP Solution Delivery

Overall I feel fortunate to have my family together and safe - I missed the face to face conversations with friends and colleagues and customers but overall the team at HarperDB made some amazing strides with the HDB Cloud product and improving performance and capabilities for our clients. It is unfortunate that my college kids were not able to have the on campus experience that they had expected. They have great attitudes and have been very positive during the crisis. So I am grateful for their perseverance. Highlights of the year includes many camping trips during the summer months with friends and family. With the support of our partner Casne, we developed a capability for industrial IOT which allows for the collection and management of extremely high frequency datasets for use in analytics and plant management. We also made great strides in introducing HarperDB within the DoD community. Both of these efforts will provide the market with a unique capability to support a variety of use cases in 2021. I am very excited to market the high resolution data collection solution to industrial clients and to assist in helping our DoD clients enhance their cyber capabilities.

Jaxon Repp, VP Product

This year has been pretty, pretty, pretty good. I worked on the Studio, HarperDB Cloud, Developer Marketplace, DB Migrator, and Clustering. Next year, I intend to reduce the amount of coding I do, if/when we get to the point of hiring other people.

Coding is a skill set I possess, but as we head into the new year most projects will be incremental improvements to our already-awesome suite of products. It will no longer going to be greenfield development, which is where the art is (for me, at least).

I want to spend more time with customers and developers on a comprehensive roadmap that puts us in the same league as Oracle and SQL Server. I want to be able to partner with universities to make us the database that the next generation of developers cut their teeth on. I want to never have to hedge when answering a customer question because we are always faster, more flexible, more performant, and more cost-effective than any other solution they might be considering.I want ubiquitous data to save the world, and I want HarperDB to be the way that data gets it done.

Jacob Cohen, Director of Product Management

This year has certainly been weird, but I think I've made the best of it. I've gotten used to working from home and have grown to enjoy the flexibility of it, however, I do miss seeing everyone. I've taken the opportunity to learn new skills.

In the middle of this year, I was promoted to Director of Product Management. My previous role was Solutions Architect, which I am still involved in with the sales team, but it helped me better understand the needs of our customers/users so that I can better guide the direction of the product. One of the more fun things I created this year was our video tutorial series. Additionally, I am in the process of revamping our documentation, which became available with the release of HarperDB 2.3.0.

I will continue working with the engineering and sales teams to ensure that HarperDB is delivering the proper features and functionality to deliver cutting edge data solutions. HarperDB has some exciting features on our roadmap and I'm looking forward to seeing them come to fruition.

Aron Johnson, Automation Engineer

Given the social/economic/pandemic challenges of 2020, I've been very fortunate this year all things considered. The main thing I have worked on and accomplished is helping to build, maintain, and improve HarperDB Cloud. Next year, I’ll continue to improve HarperDB Cloud. Continue to support our developers so they can make HarperDB even better. I am most excited about seeing HarperDB being used by more people, and used in new and interesting ways. Growing the team. Getting vaccinated, and seeing my friends and family more.

David Cockerill, Software Engineer

This year has not been that bad for me. While I miss the camaraderie of working in the office I prefer working from home. I enjoy the flexibility that it provides and the location independence. Due to social distancing restrictions I have spent more time exercising outside. My running has progressed significantly and I ran my first half marathon. One of my biggest highlights of the year was adopting a dog, it has been quite the adjustment but a worthy one. This year for me was all about the HarperDB Studio, specifically the back end. I spent most my time building out the serverless architecture that supports the Studio. I have really enjoyed learning more about cloud computing and am excited to continue that exploration. Next year I will be working on automating some of our internal processes and will continue supporting HarperDB Cloud offerings.

Sam Johnson, Software Engineer

As one of the few HarperDB'ers who, pre-pandemic, strongly preferred to not work from home, this year has been a challenge and major adjustment. That said, I have been really lucky that the transition allowed me to spend significantly more time with my baby girl (who just turned ONE!) while still managing to keep some semblance of productivity. Highlight of the year has absolutely been getting to see Olivia transition from awesome, happy newborn to awesome, happy kiddo. This year I worked on hardening our core product. Mainly overhauling and adding new features to our SQL engine, implementing a new, more robust approach for permissions, and adding other features like bulk load operations from S3 and NoSQL Upsert. This coming year I will be primarily working on improving HarperDB's SQL operations, focusing on performance improvements and new features and functionality - currently working on building a much faster server layer for HDB that should be ready for release in early 2021. Outside of work, I am excited to be able to do a bit more traveling and catch up on all the beers I've had to skip with my friends and family in 2020.

Jason Bathrick, Business Development

My dogs aren't barking but HarperDB is!

2020 brought some serious change on the sales side over at HarperDB. During Q1, it was my goal to introduce HarperDB to as many companies in Denver as I possibly could and that meant face to face meetings, lunches, and networking events. It was exciting and fun to meet with businesses and share new data management concepts and help them solve challenges with HarperDB. With Covid, the entire business development market was flipped. Instead of running and gunning, I've been glued to my chair since March. My dad bod is a testament to the lack of movement but as a sales guy, you gotta roll with the punches.

One of the achievements I am most proud of is helping forge a partnership with the US Army. HarperDB is being deployed to collect sensor data in the field and distribute/aggregate the data in real-time for situational awareness purposes. When I entered the data management industry, I never expected to work on innovative projects that help keep our soldiers safe. I really think it's awesome and am proud HarperDB is a staple in the project.

Secondly, we have been working with a major Telecommunications Company on innovative ways to leverage HarperDB to bring value to their end customers. Through trial and error, and bringing a lot of very smart and talented individuals together to strategize, we have come up with business applications that are game changers, The future is very bright for HarperDB!

For 2021, I would like to get HarperDB more involved in Digital Manufacturing and Design projects. HarperDB's ability for data filtering, real-time alerting, and our ability to move operational data into enterprise systems can really enhance operations and reduce spend. I would also like to find more cutting edge technology projects in the government sector and work on partnering with a major aerospace and defense company.

Kaylan Stock, Director Marketing

Thankful for my health, my family’s health, my boyfriend Matt, and our pet clan (Alby, Ike, & Harold). This year hasn’t been easy, but it’s been a year of growth, self reflection, and learning to appreciate what I have. My favorite part was finally achieving my goal of buying a home and just in time to have an office space now that HDB is a fully remote company. 🙂 This year at Harper, I worked with my new partner in crime Margo on the marketing launch of HarperDB Cloud. I have spent a ton of time blogging, helping coordinate events, running social media, and coordinating the marketing around HDB Cloud! I also helped run a complete website makeover in the beginning of year. 2021 will be HarperDB’s year! We will be focusing on content, brand & product awareness along with driving user adoption through developer evangelism!

Margo McCabe, Sr Director DevRel & Partnerships

What a year! While I miss the same things we all do (seeing people, traveling, leaving my house), I really do feel fortunate all things considered. My family and friends have stayed healthy, and we even jumped on the puppy bandwagon with our new rescue pup named Jagger. My focus has been around expanding the HarperDB user base, procuring dev resources, gauging feedback, connecting our users and product team, and raising product awareness, as well as partnerships on a larger scale. It’s incredibly rewarding being able to see firsthand how HarperDB has helped developers and tech leaders of all skill levels across numerous applications. In 2021 I’ll stay on the same track, continuing to increase community engagement with our informative livestream events, developer examples, and new tech podcast series (we’re looking for guest speakers!).

It may seem like it, but I didn’t force any of them to write that. We have a great team, and it really has been a strategic and productive year for HarperDB and we are eager to show you what’s yet to come. It goes without saying that we wouldn’t be where we are today if it weren’t for our incredible partners, customers, and HarperDB users, as well as the innovative minds and resources of the developer community as a whole.

Happy holidays, stay safe, and well.

Stephen Goldberg, CEO & Co-Founder

Deep Dive: NewSQL Databases

Stephen Goldberg — Fri, 10 Jul 2020 16:12:56 +0000

Overview

One of my colleagues, @margo_hdb, recently posted a great article Database Architectures & Use Cases - Explained here on dev.to. In response a user asked for a deeper dive on NewSQL databases, so I thought I would put one together.

The term NewSQL was coined in 2011 by 451 Group analyst Matthew Aslett.

Wikipedia defines the term below:

“ NewSQL is a class of relational database management systems that seek to provide the scalability of NoSQL systems for online transaction processing (OLTP) workloads while maintaining the ACID guarantees of a traditional database system.
Many enterprise systems that handle high-profile data (e.g., financial and order processing systems) are too large for conventional relational databases, but have transactional and consistency requirements that are not practical for NoSQL systems. The only options previously available for these organizations were to either purchase more powerful computers or to develop custom middleware that distributes requests over conventional DBMS. Both approaches feature high infrastructure costs and/or development costs. NewSQL systems attempt to reconcile the conflicts.”

The “NewSQL” Confusion

The problem in my mind is that the term is so new, and not widely adopted, that what is defined as a NewSQL database can be really confusing. DB-Engines, doesn’t even list NewSQL databases as a category. I understand why, because I think until the term is fully cemented, listing it would create more confusion than clarity.

Here is one list I found of “NewSQL” databases

Amazon Aurora
Apache Trafodion
HarperDB
Altibase
c-treeACE
CLustrix DB
CockroachDB
MemSQL
NuoDB
VoltDB
Google Spanner

I obviously, and selfishly like that it includes HarperDB, and I think a lot of the players mentioned deserve to be on that list, but Amazon Aurora is by no means a NewSQL database. It is basically just an RDBMS like Postgres and MySQL.

The Storage Engine

Why do I think that Aurora should not be on the list? Because to me, what makes a “NewSQL” database a “NewSQL” database is the underlying storage engine. I don’t think you would find Postgres or MongoDB, two of the most popular databases in the world, on a list of NewSQL databases. That’s because while they are both capable of supporting SQL (columns and rows) and NoSQL (documents/json), their underlying storage engines are not designed to fulfill the definition of “NewSQL” above.

Wikipedia defines a “Storage Engine” as follows:

“A database engine (or storage engine) is the underlying software component that a database management system (DBMS) uses to create, read, update and delete (CRUD) data from a database. Most database management systems include their own application programming interface (API) that allows the user to interact with their underlying engine without going through the user interface of the DBMS.”

If you remove the element of time from the innovation lifecycle of a product and compress it from beginning to end into one moment, most products at a high-level are building very similar things. They may build them in a different order and in different ways, but they are trying to accomodate the same market feedback. For example if you look at the first automobiles on the road they were wildly different, but over time began to look pretty similar with different high-level features that distinguished them.

As a result almost all databases will eventually accommodate SQL, NoSQL, GraphQL, etc… These are really just interfaces for how you interact with the storage engine of a product. Kind of like all cars eventually having seat-belts, four wheels, gas pedals.

Most people look at the following categories when evaluating a database:

What interfaces does it have? (REST, JDBC, ODBC, CLI, Web-Sockets, etc…)
Is it ACID compliant?
Does it support SQL and/or NoSQL?
What is the cost?
How is the performance?
What are the security features?
Can it scale?
Is it easy to use?
How popular is it?
How long will it take to migrate?
How long will it take for my dev team to get up to speed?
What does the ecosystem look like?

These are all really valid points, but what’s not on that list is the underlying storage engine. Obviously some very sophisticated data experts will look at that, but the vast majority of people do not. What most folks do not realize is that the underlying storage engine impacts nearly all of those questions.

Additionally, once a database has picked an underlying storage engine it is a nearly Herculean effort to migrate to another. MongoDB for example, realized the need to provide SQL, and instead of changing their underlying storage engine which could never accommodate SQL in any performant manner, appended to their storage engine which created a host of issues which I have outlined in this article: Multimodel Databases- A Mistake.

For an example with HarperDB, it took us almost a year to migrate from one storage engine (file system) to another (LMDB) and we were only able to do that because we are not nearly as widely used as MongoDB. In contrast, for us to add in Geospatial search capability into HarperDB took only a few weeks.

So to use my analogy above, automobile manufactures can start from scratch when building a new car. When you buy a new car, the legacy technology in your previous car has no impact on the new car. They may want to keep a consistent user experience across the two cars, but each new car can be redesigned from the ground up entirely without the old car causing issues.

The same is not true of databases. If you upgrade from AcmeDB 1.0 with a document store to AcmeDB 2.0 with a columnar store your data is going to look VASTLY different. You could have data corruption, loss of features, new cost, down-time, migration time, security issues, etc. Whereas if you buy car 1 with a gas engine and switch to car 2 with an electric engine, it has no impact on car 2.

That is why IMHO, in order to categorize something as a NewSQL database, the underlying storage engine has to have been specifically designed for or able to accommodate the definition of NewSQL in terms of scale, flexibility, and analytical capability.

So what makes it a “NewSQL” Database?

Databases are typically designed for specific use cases. Timescale for example was purpose built to solve time-series analysis needs that are challenging in other DBMS systems, i.e. analyzing data over time. It’s really good at it. That said, no one would use Timescale to power a simple web page when a document store like MongoDB could do it much easier, or even Postgres which Timescale sits on top of would be a better choice. If however you try to do time-series analysis in MongoDB, good luck to you, and please make sure you have a budget the size of a small country’s GDP.

So when we look at what defines a “NewSQL” database we see a few key things:

“…a class of relational database management systems that seek to provide the scalability of NoSQL systems… “
“….while maintaining the ACID guarantees…”
“Many enterprise systems … are too large for conventional relational databases, but … are not practical for NoSQL systems.”
“The only options previously available for these organizations were to either purchase more powerful computers”
“…or to develop custom middleware…”
“Both approaches feature high infrastructure costs and/or development costs. NewSQL systems attempt to reconcile the conflicts.”

Thus a NewSQL system must accommodate the following:

Scale with the level and flexibility of a NoSQL system
Be ACID compliant
Be cost effective and efficient
Not solve the challenge through massive computing needs
Not require middleware either internalized or externalized

So let’s dig deeper. The reason I droned on so much about the storage engine is because they are all impacted by them significantly.

Document Stores, what are more commonly called NoSQL databases, are fantastic at write scale. They are also highly flexible because they are not enforcing schema, and as a result simply store a new version of your record for every transaction. The scale comes from the fact that unlike a relational database, they don’t need to update other tables on write, check data consistency, guarantee a transaction, etc… Basically each write is independent, has nothing to do with other rows of data, and just creates a new document each time. Typically one attribute, the hash, is indexed by default, and if you want to add additional indexes then they actually create copies of these records. You can then search by those other attributes; however, you are searching a copy of that record, not the original. This means you have data duplication, eventual consistency, which drives up your compute costs, slows down writes, and creates a lot of overhead.

This means though that they can’t do performant joins, they are not ACID compliant, and it’s challenging to do highly scalable analytics. Thus, they cannot be NewSQL databases. Again some document stores try to achieve this, but read my other blog to understand why that’s a bad idea.

RDBMS systems on the other hand, struggle with high-scale data ingestion, but are fantastic at read-scale. They struggle with high-scale data ingestion because they are capable of maintaining relationships of data, they have a lot of background processes to guarantee transactions and consistency, and each write might affect numerous rows of data causing row-locking etc… This causes a ton of overhead on write, whereas Document Stores have the overhead on read. It can also lead to crashes on high-scale writes. For example, if I am trying to write to a table that has a billion rows of data, and it has foreign key to a table with a billion rows of data, the RDBMS has to inspect both tables prior to committing the write. As a one-off this is not a big deal, but if I am doing this with a 1,000 threads writing a 1,000 rows per second it begins to become problematic. RDBMS tight enforce of schema, which is much needed at times for certain use cases, is what also makes them inflexible.

People have tried to solve this by making RDBMS systems in-memory, but this just pushes the problem down the road by requiring really expensive computing needs, and when you run out of memory leading to massive crashes, data inconsistency, downtime etc… Our team previously managed a multi-terabyte in-memory database for streaming analytics. When we had enough memory for all our data it was AWESOME; however, the moment we began to run close to our memory limits it was a nightmare. It was like fighting off a zombie horde. We were always racing to prevent a crash, because the moment we crashed, it was nearly impossible to catch back up to the stream. Restarting the DB and loading everything into memory took forever, and we were processing hundreds of thousands of rows of data a second. So minutes were a long time. Additionally, making an RDBMS an in-memory RDBMS doesn’t make it any more flexible or change how the storage engine works.

In my mind then, to be a “NewSQL” database, the storage engine needs to be optimized to minimize the overhead of read AND write as much as possible. It needs to do this in a flexible manner that allows for dynamic data, without driving up costs.

Without going into too much of a sales pitch, that is exactly how we designed HarperDB. We optimized it for read, by making every attribute fully indexed dynamically upon write. We did this without duplicating data. This allowed us to achieve high read performance without the pitfalls of multimodal or document stores. We optimized it for write by ensuring each data element was atomic. This allows for us to ensure high scale writes without bottlenecks because updating one attribute is a totally different transaction than another. If you want to learn more you can read this somewhat dated blog about our storage engine. It is dated, because instead of the file system we now use LMDB. Pingcap has a similar, more complex, model.

Our solution and Pincap’s are not the ONLY way to solve the “NewSQL” challenge, but in my mind in order to be “NewSQL” the approach needs to be new to accommodate these needs. That said, this is just my view of the NewSQL database landscape and I would love to hear your thoughts! Am I wrong? What am I missing?

Tutorial: Setting up HarperDB Cloud

Stephen Goldberg — Wed, 01 Jul 2020 20:48:08 +0000

Overview

The goal of this tutorial is to give a very simple explanation of how to set up a HarperDB Cloud account and a free instance of HarperDB.

Signing Up

Go to https://studio.harperdb.io/sign-up
Fill out the form with your details

Click Sign Up For Free
You will then receive an email that looks like this:

Make sure to copy your temporary password and then click Log into HarperDB Studio
Type in your email and temporary password
You will then be asked to create a new password
Note: This is only for the management studio. Your database creds are separate for each db instance. Additionally, you will be able to invite other users to your organization once you login. *
Now click on your org

Creating an Instance

Now click on Create New HarperDB Cloud Instance

Click on Create HarperDB Cloud Instance
Enter your Credentials

Note: these are NOT the same as your Studio account. This is for the super user you are creating on this database instance. Username should NOT be in the form of an email. Instance Name is for reference only.

Pick your instance specs. You can start with a free version and always upgrade later. Also pick the AWS region closest to you to reduce latency.

Click Confirm Instance Details
Review your instance details

Note: The Instance URL is how you will access HarperDB with REST calls

Select I agree
Click Add Instance. Note: Your HarperDB Cloud instance will be provisioned immediately, but it takes a few minutes to setup
In 5 to 10 minutes you will receive an email letting you know that your instance is ready. In the background we are spinning up a cloudformation template, a network load balancer, a vpn, and an ec2 instance all with HarperDB running on it.

Log back into the Studio
Click back into your org
Click on your instance

You are all set! Make sure to check out the example code for help getting started!