Forem: EmitHQ

Webhook Delivery Architecture: How We Built for Reliability

EmitHQ — Sun, 29 Mar 2026 20:54:34 +0000

Webhook delivery looks simple. POST a JSON body to a URL, check for a 2xx, move on.

Then your Redis instance restarts and you lose 4,000 queued deliveries. A customer's endpoint goes down for six hours and your retry logic hammers it 50 times a second. Someone registers http://169.254.169.254/latest/meta-data/ as their endpoint URL and starts reading your cloud metadata.

These are the problems that determine webhook delivery reliability — and they only show up in production. Here's how EmitHQ handles each one, with the actual TypeScript from our open-source codebase.

The Architecture

Customer API call
       │
       ▼
  ┌─────────┐     ┌────────────┐     ┌──────────────┐
  │ Hono API │────▶│ PostgreSQL │────▶│ BullMQ Queue │
  │ (verify  │     │ (persist   │     │ (deliver     │
  │  auth,   │     │  message + │     │  to endpoint,│
  │  RLS)    │     │  attempts) │     │  retry, DLQ) │
  └─────────┘     └────────────┘     └──────────────┘
                        ▲                    │
                        │                    ▼
                   Source of truth    Customer endpoint

The arrow from PostgreSQL to BullMQ is one-way on purpose. The database is the source of truth. The queue is a best-effort delivery mechanism. If the queue loses a job, the database still has the message and every pending delivery attempt.

Persist Before Enqueue

When a customer sends a webhook message through the API, we write it to PostgreSQL inside a transaction — along with one delivery_attempt row per active endpoint — before touching the queue:

// messages.ts — inside a tenant-scoped transaction
const [message] = await tx
  .insert(messages)
  .values({ appId, eventType, payload, eventId })
  .onConflictDoNothing() // UNIQUE(app_id, event_id)
  .returning();

if (!message) {
  // Idempotency hit — this event_id was already processed
  const existing = await tx.query.messages.findFirst({
    where: and(eq(messages.appId, appId), eq(messages.eventId, eventId)),
  });
  return existing; // 200, not 202
}

// Fan out: one delivery attempt per active endpoint
await tx.insert(deliveryAttempts).values(attemptRows);

// Queue is best-effort — failure here is non-fatal
await enqueueDelivery(attemptRows).catch(() => {});

The onConflictDoNothing() on UNIQUE(app_id, event_id) gives us database-level idempotency. If the same event arrives twice — network retry, client bug, load balancer replay — the second insert silently skips and we return the original message.

The catch(() => {}) on enqueueDelivery is deliberate. If Redis is down, the message and its delivery attempts are already in PostgreSQL. A recovery process can re-enqueue pending attempts from the database.

Standard Webhooks Signing

Every outbound delivery is signed using the Standard Webhooks specification — the same HMAC-SHA256 scheme used by Zapier, Twilio, and Supabase:

// webhook-signer.ts
export function signWebhook(
  webhookId: string,
  timestamp: number,
  payload: string,
  secret: string,
): string {
  const rawSecret = Buffer.from(secret.startsWith('whsec_') ? secret.slice(6) : secret, 'base64');
  const toSign = `${webhookId}.${timestamp}.${payload}`;
  return `v1,${createHmac('sha256', rawSecret).update(toSign).digest('base64')}`;
}

Three headers go out with every delivery: webhook-id (message ID), webhook-timestamp (Unix seconds), and webhook-signature (the v1,{base64} HMAC).

Secrets use the whsec_ prefix format — the prefix is stripped, and the remainder is base64-decoded to raw key bytes. Each endpoint gets its own signing secret. Independent rotation, no cross-endpoint impact.

On the verification side, we use crypto.timingSafeEqual — never string equality. String comparison leaks timing information that can be used to forge signatures byte by byte. Verification also rejects timestamps outside a 5-minute tolerance window, preventing replay attacks.

Webhook Retry Logic: Full Jitter

When a delivery fails with a 5xx, timeout, or connection error, the webhook retry logic kicks in. But we don't use simple exponential backoff — we use full jitter:

// backoff.ts
const RETRY_DELAYS_MS = [
  5_000, // ~5s
  30_000, // ~30s
  120_000, // ~2m
  900_000, // ~15m
  3_600_000, // ~1h
  14_400_000, // ~4h
  86_400_000, // ~24h
];

export function computeBackoffDelay(attemptsMade: number): number {
  const index = Math.min(attemptsMade - 1, RETRY_DELAYS_MS.length - 1);
  const cap = RETRY_DELAYS_MS[index];
  return Math.floor(Math.random() * cap);
}

The delay is random(0, cap) — not cap + random jitter. This is full jitter as described in the AWS architecture blog. Standard exponential backoff with decorrelated jitter still clusters retries near the cap. Full jitter spreads them uniformly across the entire window. The result: a recovering endpoint gets a steady trickle of retries instead of a synchronized burst.

Eight attempts over a ~29-hour window. After that, the message moves to the dead-letter queue.

Not everything gets retried. Status codes 400, 401, 403, 404, and 410 are permanent failures — the endpoint rejected the request for a reason that won't fix itself:

// delivery-worker.ts
const NON_RETRIABLE_CODES = new Set([400, 401, 403, 404, 410]);

if (NON_RETRIABLE_CODES.has(result.statusCode)) {
  throw new UnrecoverableError(`Non-retriable status ${result.statusCode}`);
}

BullMQ's UnrecoverableError bypasses the entire retry schedule and moves the job straight to failed. No wasted retries against a 404.

Circuit Breaker and Dead-Letter Queue

If an endpoint fails 10 times consecutively, we stop hitting it:

// delivery-worker.ts
const CIRCUIT_BREAKER_THRESHOLD = 10;

const currentFailures = (endpoint.failureCount ?? 0) + 1;
if (currentFailures >= CIRCUIT_BREAKER_THRESHOLD) {
  await adminDb.update(endpoints).set({
    disabled: true,
    disabledReason: 'circuit_breaker: consecutive failure threshold reached',
    failureCount: currentFailures,
  });
  // Operational webhook sent: endpoint.disabled
}

Any successful delivery resets failureCount to 0. The circuit breaker is per-endpoint — one failing endpoint doesn't affect the others. Disabled endpoints can be re-enabled through the API or dashboard, which resets the failure counter and resumes delivery.

When all 8 retry attempts are exhausted, the delivery attempt is marked exhausted and lands in the dead-letter queue. From there, it can be replayed through the API or dashboard:

// replay.ts
export async function replayDelivery(attemptId: string) {
  await adminDb.update(deliveryAttempts).set({
    status: 'pending',
    attemptNumber: 1,
    nextAttemptAt: new Date(),
  });
  await deliveryQueue.add('deliver', jobData, {
    jobId: `replay:${attemptId}:${Date.now()}`,
  });
}

The replay: prefix and timestamp in the job ID prevent BullMQ deduplication from ignoring the re-enqueued job.

SSRF Protection

Customers provide their own endpoint URLs. Without validation, an attacker could register http://169.254.169.254/latest/meta-data/ — the AWS metadata endpoint — and have your delivery worker fetch their cloud credentials.

Blocking the IP at registration time isn't enough. DNS rebinding attacks work by having a hostname resolve to a public IP during validation, then switching to an internal IP before the actual delivery request.

EmitHQ validates at both points. At endpoint creation, we resolve the hostname and check the IP against blocked ranges (RFC 1918, loopback, link-local, cloud metadata). At delivery time, we resolve again and re-check — catching any DNS rebinding that happened between registration and delivery.

Blocked ranges include 169.254.169.254, metadata.google.internal, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, and 127.0.0.0/8.

Read the Code

The code is at github.com/Not-Another-Ai-Co/EmitHQ under AGPL-3.0. Start with packages/core/src/workers/delivery-worker.ts — it ties together every pattern in this post. The signing is at packages/core/src/signing/webhook-signer.ts, the retry schedule at packages/core/src/queue/backoff.ts.

If something doesn't hold up, open an issue. If it does, emithq.com — one API call to sign up, no credit card.

Why We Built EmitHQ: The $49–$490 Webhook Pricing Gap

EmitHQ — Sun, 29 Mar 2026 20:53:23 +0000

I was building a SaaS product that needed to send webhooks to customers. The kind of thing where your billing system fires invoice.paid and three different customer endpoints — their CRM, their Slack bot, their data pipeline — all need to receive it reliably.

So I looked at the existing webhook platforms.

The Pricing Cliff

The free tiers are generous. Svix gives you 50,000 messages a month. Hookdeck offers 10,000 events. Plenty to build against.

But then you need to go paid.

Svix's first paid tier is $490/month. There is no $50 plan. No $100 plan. You go from free to $490 in one step.

Hookdeck has a $39/month Team plan, but it caps default throughput at 5 events per second. Their next real tier is Growth at $499/month. Need more throughput? That's an add-on at roughly $1 per event per second — so 50 evt/s costs an extra ~$45/month on top of your plan.

Convoy starts at $99/month, which is more reasonable. But their Elastic License v2.0 restricts offering it as a managed service. If you're just using Convoy to send your own webhooks, the license is fine. If you want to embed webhook delivery into a platform you sell to others, you'll need to look elsewhere.

That leaves a gap. A SaaS team doing 200K events a month — past the free tier, nowhere near enterprise — has no good option between building it themselves and paying $490.

Why the Gap Exists

These aren't bad companies. Svix created the Standard Webhooks specification, now adopted by companies including Zapier, Twilio, Kong, ngrok, and Supabase. That's a genuine contribution — dozens of companies ship interoperable webhook signatures because of that work. Hookdeck has processed over 100 billion webhooks. Convoy pioneered open-source webhook delivery.

But they've raised venture capital. Svix took $13M in funding led by a16z. When your investors expect 10x returns, you optimize for enterprise contracts, not $49/month plans. The pricing reflects the business model, not the cost of delivery.

Webhook infrastructure is cheap to run. At 50 million events per month — enough to serve 400+ customers — here's what the infrastructure actually costs:

Service	Monthly Cost
Cloudflare Workers	~$3
Neon PostgreSQL	~$15
Upstash Redis	~$200
Railway	~$100
QStash	~$100
Total	~$418

That's a 99%+ gross margin at scale. The $490 price tag isn't about infrastructure costs. It's about market positioning.

The DIY Trap

So teams try to build their own webhook service from scratch. An HTTP POST with a JSON body. The first version takes a day. The production-ready version takes three months.

You need HMAC-SHA256 signing with timing-safe comparison — string equality is vulnerable to timing attacks. You need exponential backoff with jitter, not fixed intervals, because thundering herd problems will take down your customer's endpoints and yours. You need per-endpoint circuit breakers that auto-disable after consecutive failures. Idempotency keys so duplicate deliveries don't double-charge someone. A dead-letter queue with manual replay. Multi-tenant isolation that doesn't rely on WHERE clauses alone.

Figure six weeks for a senior engineer to get it production-ready. That's before the first outage teaches you what you missed. And then you maintain it forever — every infrastructure change is a change to your delivery system, every on-call engineer needs to understand it.

What We Built

EmitHQ is an open-source webhook platform built for SaaS teams that have outgrown free tiers but can't justify enterprise pricing. It handles outbound delivery to customer endpoints and inbound reception from providers like Stripe and GitHub, starting at $49/month with no throughput add-ons.

Three paid tiers:

Starter: $49/month — 500K events, 50 evt/s, configurable retries
Growth: $149/month — 2M events, 200 evt/s
Scale: $349/month — 10M events, 1,000 evt/s

Retries don't count against your quota — penalizing retries would punish you for your customers' downtime. Overage on paid tiers is $0.50/K (Starter), $0.40/K (Growth), or $0.30/K (Scale), billed at the end of the month.

We use the same Standard Webhooks spec that Svix created for outbound signing — the same webhook-id, webhook-timestamp, webhook-signature headers, the same HMAC-SHA256 algorithm. If your customers already verify Standard Webhooks signatures, switching to EmitHQ requires zero changes to their verification code. Migration cost is zero on their side.

How It Works

Two architectural decisions define EmitHQ's reliability, and both are visible in the source code.

Persist Before Enqueue

Every message is written to PostgreSQL before it enters the Redis delivery queue. If the queue loses a job — Redis restart, memory pressure, network partition — the database still has the message. This is the difference between "we'll try to deliver it" and "we guarantee we recorded it."

Full-Jitter Exponential Backoff

Failed deliveries retry with randomized delays: ~30 seconds, then ~2 minutes, then ~15 minutes, scaling up to ~12 hours. Eight attempts over a 29-hour window. The jitter prevents synchronized retries from overwhelming a recovering endpoint. Each endpoint gets its own retry schedule — one failing endpoint doesn't affect the others.

Per-Endpoint Circuit Breakers

After 10 consecutive failures, EmitHQ auto-disables the endpoint and sends you an operational webhook (endpoint.disabled) so you know about it. A half-open probe resumes delivery automatically after a 30-second cooldown. Both thresholds — failure count and cooldown period — are configurable per endpoint.

Tenant Isolation

PostgreSQL Row-Level Security enforces isolation at the database level. Every query runs with SET LOCAL app.current_tenant set by middleware before any data access. Even a bug in application code can't cross tenant boundaries — the database rejects the query before it executes.

Open Source

EmitHQ's server is AGPL-3.0. The SDKs are MIT. You can read every line of delivery logic, retry calculation, and signing implementation.

You can self-host it. No phone-home, no license keys, no feature gates. AGPL means if you modify the server and offer it as a service, you share your changes. But running it for your own infrastructure? No restrictions.

This matters because webhook delivery is a trust decision. You're routing your customers' data through someone else's infrastructure. Knowing exactly how that infrastructure works — and having the option to run it yourself — isn't a feature. It's a prerequisite.

Try It

The code is on GitHub. Read the delivery worker, the retry logic, the signing implementation. If it holds up, sign up at emithq.com — one API call, no credit card, no sales demo. If it doesn't, the issue tracker is open.