Forem: Mary Olowu

Add Webhooks to Your SaaS in 10 Minutes (Without Queues or Retries)

Mary Olowu — Mon, 04 May 2026 18:50:28 +0000

TL;DR — One API call subscribes a customer endpoint. Centrali signs each delivery with HMAC-SHA256, retries 5 times over ~40 minutes on failure, logs every attempt, and exposes a one-line replay endpoint. No queue. No retry logic. No Svix. The whole subscribe call is right below — scroll to it if you just want the shape.

Your customers want webhooks. You know the checklist:

A queue so user requests don't block on HTTPS calls to third-party servers
HMAC signing so customers can verify the request came from you
Retry with exponential backoff, jitter, a max attempt count
A circuit breaker so flaky endpoints don't cost you compute
A delivery log with replay for the "we never got that event" support ticket
A subscription model with rotatable secrets, event filters, active/inactive state

It's two weeks of work, plus ongoing maintenance. This post shows how to skip all of it.

The whole thing, in one SDK call

import { CentraliSDK, RecordEvents } from '@centrali-io/centrali-sdk';

const sub = await centrali.webhookSubscriptions.create({
  name: 'customer-acme-order-events',
  url: 'https://customer-acme.example.com/webhooks',
  events: [RecordEvents.CREATED, RecordEvents.UPDATED],
  recordSlugs: ['orders'],
});

// `secret` is returned on create only — copy it now, reads omit it.
console.log('Signing secret:', sub.data.secret!);

That's the whole setup. HMAC signing, retry, circuit breaker, delivery log, replay — all behind that one call. The rest of this post explains what just got handled for you, and shows how to wire up the customer side.

What you'd build yourself (and won't have to)

Before the tutorial, the honest version of what ships with a production-ready webhook system:

An outbound queue. You can't block the user's API request on an HTTPS call to a customer server. Something has to pull from a queue (Redis, SQS, BullMQ) and dispatch asynchronously.
HMAC signing. Sign the raw body with the subscription's secret, attach as a header. Secrets must be rotatable and scoped per subscription.
Retry with backoff. When a delivery fails (5xx or timeout), retry — but not immediately, and not forever. Exponential backoff is the baseline. Most teams get this subtly wrong: no jitter, too many attempts, too aggressive early.
Circuit breaker. When a customer's endpoint has been failing for minutes, stop attempting. Resume when it's healthy. Otherwise you waste compute on doomed requests and compound the outage.
Delivery log. Every attempt: HTTP status, response body, error, timestamp. This is the contract with support. When a customer asks "did you send it?", the log is the only answer that matters.
Manual replay. Customer deploys a fix, wants to catch up on the last hour. They need a replay endpoint. It shouldn't duplicate into your retry queue.
A subscription model. Customers create subscriptions, scoped to events and record types. Activate/deactivate. Rotate the secret without losing history.

Each item is a day of work. Together they're a sprint. Maintained well, they're an ongoing tax on your platform team.

What Centrali gives you

One line each, mapped to the list above:

Outbound queue: built in, no worker to deploy. Dispatch happens on record changes automatically.
HMAC signing: SHA-256 over the raw body, base64-encoded, header is X-Signature. Secret is whsec_…, auto-generated per subscription, rotatable.
Retry with backoff: 5 attempts over ~40 minutes — delays of 30s, 2m, 10m, 30m between attempts.
Circuit breaker: per-URL, opens when an endpoint is consistently failing, resets after a cool-down. Flaky endpoints stop costing you compute.
Delivery log: every attempt stored — HTTP status, error, payload, response body — visible in the console and queryable via API.
Manual replay: one endpoint — POST /webhook-subscriptions/deliveries/{id}/retry. New delivery is linked to the original via replayedFrom.
Subscription model: collection-scoped (via recordSlugs), event-type-filtered (record_created, record_updated, record_deleted), active/inactive, rotatable secret.

Tutorial: ship an outbound webhook in 5 minutes

The demo: a SaaS that tracks orders. When an order is created or updated, subscribed customers get a webhook.

Step 1: Create a collection

If you don't have one already, create an orders collection in the Centrali console. Any collection works — Centrali emits record events for every collection in the workspace.

Step 2: Subscribe a customer endpoint

You already saw the subscribe call at the top of the post. The response wraps the subscription under sub.data — the two fields that matter:

sub.data.id — subscription ID for updates and delivery queries
sub.data.secret — signing secret, returned once on create. Save it and hand it to your customer so they can verify incoming requests. If lost, rotate in the console: old-secret deliveries stay in the log, new deliveries use the new secret.

The console view shows the subscription with URL, event filters, a masked rotatable secret, signature header, and algorithm:

Prefer raw HTTP? Same shape over REST — POST /data/workspace/{your-workspace}/api/v1/webhook-subscriptions with a bearer token and a body matching the SDK call. Translating fetch or any HTTP client is one-to-one.

Step 3: Trigger an event

Any record change in the orders collection now fans out to subscribed endpoints. Create an order:

await centrali.createRecord('orders', {
  orderNumber: 'ORD-1045',
  customerEmail: 'pia@terradome.studio',
  total: 210,
  currency: 'USD',
  status: 'pending',
  itemCount: 3,
});

Centrali dispatches a POST to the subscribed URL within seconds. The request body looks like this:

{
  "event": "record_created",
  "workspaceSlug": "demo",
  "recordSlug": "orders",
  "recordId": "7d5a87d5-b10c-48bb-85d8-c42dbdaab417",
  "data": {
    "orderNumber": "ORD-1045",
    "customerEmail": "pia@terradome.studio",
    "total": 210,
    "currency": "USD",
    "status": "pending",
    "itemCount": 3,
    "placedAt": "2026-04-22T04:59:00Z"
  },
  "timestamp": "2026-04-22T04:59:02Z"
}

And the headers include the HMAC signature:

X-Signature: xXyHjYrTLS0bKh9qypUv8U5fj5UwBpIn2u0loyEoSQg=
Content-Type: application/json
User-Agent: Centrali-Webhooks/1.0

Step 4: Verify the signature on the customer side

The signature is HMAC-SHA256(signingSecret, rawBody), base64-encoded. The critical word is raw: compute it over the exact bytes you received, before any JSON parsing or middleware touches them.

Here's an Express.js handler that does it right:

import express from 'express';
import crypto from 'crypto';

const app = express();
const WEBHOOK_SECRET = process.env.CENTRALI_WEBHOOK_SECRET;

// Use raw body middleware for the webhook route only
app.post(
  '/webhooks/centrali',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.get('X-Signature');
    if (!signature) return res.status(400).send('missing signature');

    const expected = crypto
      .createHmac('sha256', WEBHOOK_SECRET)
      .update(req.body) // req.body is a Buffer of the raw bytes
      .digest('base64');

    const valid = crypto.timingSafeEqual(
      Buffer.from(signature),
      Buffer.from(expected)
    );

    if (!valid) return res.status(401).send('invalid signature');

    const event = JSON.parse(req.body.toString('utf8'));
    console.log('got event:', event.event, event.recordId);

    // Do work, then 2xx to acknowledge
    res.status(200).send('ok');
  }
);

The two gotchas that trip up every first implementation:

Parse after verification. If you let express.json() touch the request first, the raw bytes are gone and the signature won't match. Use express.raw() for this route only.
Constant-time comparison. === leaks timing information. Use crypto.timingSafeEqual or your language's equivalent.

Normally you'd be writing the signing code too — on your server. Here your server doesn't ship webhook code at all. The only HMAC work is on the customer's end, which is where it belongs.

What happens when the customer's endpoint is down

The honest failure mode is what sells the feature. Create a second subscription pointed at a deliberately broken endpoint:

await centrali.webhookSubscriptions.create({
  name: 'customer-globex-order-events',
  url: 'https://httpbin.org/status/500',
  events: [RecordEvents.CREATED, RecordEvents.UPDATED],
  recordSlugs: ['orders'],
});

Trigger any order event and watch the delivery log. The first attempt fails with HTTP 500. The next four attempts happen at +30s, +2m, +10m, +30m. The log stays visible the whole time, with live nextAttemptAt and attemptCount values:

After the fifth attempt, the delivery's status flips to failed and the error is preserved:

No retry code in your app. No job queue. No alerts you wired up and forgot about. Just a log you point support at when someone asks "did it send?".

Replay a failed delivery once the endpoint is healthy

The customer fixes their endpoint. They want the missed events. Replay any failed delivery by ID:

await centrali.webhookSubscriptions.deliveries.retry(deliveryId);

A new delivery is created, pointed at the current subscription URL. The new delivery's detail view shows the original delivery ID under replayedFrom — you always know where a replay came from:

If replay was the entire story, you'd still need a queue. But combined with automatic retry plus the delivery log, the customer flow becomes:

Customer endpoint goes down.
Centrali tries for ~40 minutes, then gives up (logged).
Customer fixes their endpoint.
Customer (or you, from their support request) hits the replay endpoint once.
Back to normal — no lost events.

What's in the subscription model

If you've used Svix or Hookdeck, this shape will feel familiar:

Field	Purpose
`name`	Display name — usually the customer's name + event scope
`url`	HTTPS endpoint that receives deliveries
`events`	Event types to subscribe to — `record_created`, `record_updated`, `record_deleted`
`recordSlugs`	Collection filter — deliver only for these record slugs
`status`	`active` or `inactive` — pause without deleting
`signingSecret`	Shared secret for HMAC signing (auto-generated, rotatable)
`signatureHeader`	Header name for the signature (default: `X-Signature`)
`algorithm`	HMAC algorithm (default: `sha256`)
`encoding`	Signature encoding (default: `base64`)

One subscription per customer-per-scope. Rotating the secret doesn't lose history — existing deliveries in the log remain readable, future deliveries use the new secret.

When you should still build it yourself

If webhooks are core to your product — you're Zapier, or Segment, or you're Svix — you need more than what's in a backend platform. Per-customer delivery portals, detailed per-endpoint SLAs, webhook-as-a-product pricing, platform-wide rate budgets. Those companies exist for a reason and the category is real.

If webhooks are a feature of your product — one of many things your API does, and your customers want them — you don't need dedicated webhook infrastructure. You need the seven things above, and you need them to work without turning into a team's full-time job.

That's what the platform you're already using gives you.

Follow along: Create a free workspace and subscribe your first endpoint in 5 minutes. No deployment required.

What Is Record TTL? Database Time-to-Live Explained

Mary Olowu — Tue, 28 Apr 2026 18:22:25 +0000

Cross-posted from the Centrali blog. The canonical version with code highlighting and updates lives there.

TL;DR: Record TTL (time-to-live) is a database feature that automatically deletes records once they expire. You set an expiration time on a record — by duration ("delete in 1 hour") or by timestamp ("delete on March 1") — and the database removes it without a cron job. This is database TTL; it's the same idea as DNS TTL but applied to rows in a table instead of cached DNS lookups.

A note before we go further: if you searched for "TTL" and ended up here looking for DNS TTL (how long a DNS resolver caches an A/AAAA/CNAME record), this isn't that post — try Cloudflare's DNS TTL reference. This post is about record TTL in databases: making rows in your data store auto-delete on a schedule.

What Is Record TTL?

Record TTL is a per-record expiration time. The database tracks an expiresAt timestamp on the record (either set explicitly or computed from a duration), and once that timestamp passes:

The record disappears from query results immediately (read-time filtering).
A background sweep deletes it from storage permanently.

The two writes you make as a developer are:

ttlSeconds — a duration. "Delete this record 3,600 seconds from now." The database calculates the absolute expiresAt for you.
expiresAt — an absolute timestamp. "Delete this record at 2026-09-01T00:00:00Z."

Both end up storing the same field. Use ttlSeconds when the deadline is relative ("expire 24 hours after creation"); use expiresAt when the deadline is fixed ("expire on the sale end date").

How TTL Works (Mechanically)

Three things happen between "you wrote a TTL" and "the record is gone":

Write time — you create or update a record with ttlSeconds or expiresAt. The database stores the absolute expiration timestamp on the record.
Read time — every query is implicitly filtered against now(). Records past their expiresAt are excluded from results, even if they haven't been physically deleted yet.
Sweep time — a background job (every few minutes, depending on the database) finds expired records and deletes them. Some systems also publish an event ("record.expired") so your app can react before the row is gone.

The key property: expired records become invisible to your application instantly, even if the on-disk delete lags by minutes. You don't need to filter WHERE expiresAt > now() in every query — the database does it for you.

When to Use Record TTL

TTL is the right tool whenever you're tempted to write a "cleanup script that runs every night." Common cases:

Session tokens — expire after 24 hours of inactivity, or 30 days from creation.
Verification codes — one-time codes for email verification, magic links, password reset. Expire in 15 minutes.
Promo codes and flash deals — auto-disable when the deadline hits (or the flash window closes).
Draft content — autosave drafts that should evaporate after N days unless promoted to a published post.
Rate-limit windows — track requests-per-minute by storing a record per request with a 60-second TTL.
Audit logs with retention rules — "delete after 90 days" without writing the cleanup job yourself.
Cache-like records — when you need durable storage with cache-like eviction.

The negative test: TTL is the wrong tool when you need soft expiration — i.e., the record should be flagged as expired but kept around for analytics or recovery. For that, use a status field (status: 'archived') and filter explicitly. TTL means gone.

Default TTL vs. Per-Record TTL

Most databases that support record TTL let you set it at two levels:

Default TTL on the collection (or table) — every new record auto-inherits the duration. Set this for collections that are always time-bounded (sessions, verification codes).
Per-record TTL — override the default (or set TTL on a record in a non-TTL collection). Set this for one-off cases — flash promos in a long-lived "Promotions" table, for example.

A reasonable mental model: default TTL is for the table's intent; per-record TTL is for the exception.

Database TTL vs. DNS TTL

Same name, different concept. They share the underlying idea — "this thing is valid for N seconds, then forget it" — but they apply to different layers:

	DNS TTL	Database (record) TTL
What expires	A DNS record cached at a resolver	A row stored in your database
Who enforces it	DNS resolvers worldwide	Your database server
Typical duration	Seconds to hours	Minutes to months
Tunable per-record?	Yes (per DNS record)	Yes (per row, if the DB supports it)
What "expired" means	Resolver re-fetches from authoritative server	Row is hidden from queries and eventually deleted

If you're in the DNS world, TTL is about cache freshness. In the database world, TTL is about automatic cleanup. The mechanics aren't connected — your A-record TTL has nothing to do with your sessions table TTL.

Implementing Record TTL

Most managed databases now offer some form of record TTL: MongoDB has expireAfterSeconds indexes; DynamoDB has TTL attributes; Redis has EXPIRE. The exact API differs, but the shape is the same: a field on the record (or an index on a field) tells the database when to delete.

In Centrali, record TTL is built into the storage layer. You set ttlSeconds or expiresAt when creating a record, or set a default at the collection level:

import { CentraliSDK } from '@centrali-io/centrali-sdk';

const centrali = new CentraliSDK({
  workspaceId: 'my-workspace',
  clientId: process.env.CENTRALI_CLIENT_ID,
  clientSecret: process.env.CENTRALI_CLIENT_SECRET,
});

// Per-record TTL: this verification code expires in 15 minutes
await centrali.createRecord('VerificationCodes', {
  userId: 'user-123',
  code: '482910',
}, { ttlSeconds: 900 });

For a step-by-step walkthrough — sessions with sliding expiration, promo codes with fixed deadlines, draft content with TTL clearing on publish — see the companion post: How to Auto-Expire Records with Centrali TTL. Full reference lives in the Record TTL docs.

Centrali sits in a broader category — a backend for ingesting third-party webhooks, storing them as data, and sending your own webhooks and workflows. Record TTL is one feature inside the storage layer; if you're working with stored Stripe webhook events or schemaless data, the same TTL rules apply to those records.

Common Questions

Does TTL delete data immediately when it expires?
Records become invisible to queries the instant expiresAt passes. Physical deletion happens on a background sweep, typically within a few minutes. Don't rely on instant disk deletion for security-sensitive data — use a separate purge if you need it gone now.

Can I extend a TTL after the record is created?
Yes — update the record with a new ttlSeconds or expiresAt. This is how "sliding expiration" works for sessions: every authenticated request resets the TTL.

Can I remove a TTL after setting one?
Yes — most TTL implementations support clearing the expiration so the record becomes permanent. In Centrali, pass { clearTtl: true } on the update.

Is TTL the same as data retention policy?
TTL is a mechanism for retention. A retention policy is the rule ("keep audit logs for 90 days"); TTL is one way to enforce it. Other ways: scheduled jobs, archival to cold storage, manual purges.

What's the smallest practical TTL?
Depends on the database, but seconds-level TTL is common. Sub-second TTL usually doesn't make sense — by the time the write replicates, the record may already be expired.

Summary

Record TTL is the database equivalent of "set it and forget it" cleanup: assign an expiration to a record, and the database handles the rest. Use it for any data that's intrinsically time-bounded — sessions, codes, promos, drafts, rate-limit windows. Don't confuse it with DNS TTL; same name, different layer.

If you want to see TTL in action with concrete code, the auto-expire records guide walks through three full use cases.

Originally published on centrali.io/blog/what-is-record-ttl. Centrali is the backend for webhooks in and out — ingest third-party webhooks, store them as data, and send your own workflows from one SDK.

How to Test Stripe Webhooks Locally (Stripe CLI + Replay + Logs)

Mary Olowu — Wed, 22 Apr 2026 07:43:19 +0000

Most Stripe webhook bugs are not business logic bugs. They are reproducibility bugs. If you can't replay the exact event path in under 30 seconds, debugging takes hours instead of minutes — because every attempt involves refreshing the Stripe Dashboard, re-triggering a test, and squinting at logs to figure out whether your handler ran at all.

This post walks through the local loop that makes this painless: forward → trigger → replay → inspect. Four steps, one terminal window, no guessing.

The loop, in full

# Terminal 1 — forward Stripe events to your local handler
stripe listen --forward-to "http://localhost:3000/api/stripe-webhook"

# Terminal 2 — trigger real Stripe test events
stripe trigger payment_intent.succeeded
stripe trigger charge.failed
stripe trigger invoice.payment_failed

# When something fails, replay the exact event
stripe events resend evt_1NfP6Q2eZvKYlo2CsKT4a5oS

That is the whole loop. Everything below is how to make it reliable.

Step 1 — Give your dev webhook its own path

Don't forward Stripe events to your production webhook path. Use a dedicated dev path:

stripe listen --forward-to "http://localhost:3000/api/stripe-webhook-dev"

Why separate? Because you want to test with lax validation (no signature verification, verbose logging) without loosening your production handler. Keep two paths:

/api/stripe-webhook — production. Strict signature verification, minimal logs.
/api/stripe-webhook-dev — dev only. Signature check optional, logs every field.

When stripe listen starts, it prints a signing secret:

> Ready! Your webhook signing secret is whsec_abc123...

Copy that into your dev environment variable (STRIPE_WEBHOOK_SECRET_DEV). This is NOT the same as the secret from your Stripe Dashboard — the CLI generates its own. This trips up everyone the first time.

Step 2 — Trigger real events, not made-up payloads

Do not hand-craft JSON payloads. Stripe's trigger command sends a real event through your account's test data, which means you're testing against payloads that match production exactly.

# Core payment flows
stripe trigger payment_intent.succeeded
stripe trigger charge.succeeded
stripe trigger charge.failed

# Subscription lifecycle
stripe trigger customer.subscription.created
stripe trigger customer.subscription.updated
stripe trigger customer.subscription.deleted

# Invoice / billing
stripe trigger invoice.payment_succeeded
stripe trigger invoice.payment_failed

Each trigger writes a real event to your Stripe test account. That means every event has a real id (starts with evt_) that you can replay later.

Run all the events your handler cares about at least once. Anything you haven't triggered locally is a surprise waiting in production.

Step 3 — Replay the exact event, on demand

This is the step most people skip, and it's where debugging speed compounds. When a test fails, you can replay the exact same event by ID:

stripe events resend evt_1NfP6Q2eZvKYlo2CsKT4a5oS

Same payload. Same signature. Same timestamp. Your handler sees a byte-for-byte identical request.

This is gold for two reasons:

Idempotency testing. Your handler should be safe to call with the same event ID twice. Replay it five times in a row and confirm you create one record, not five. If you create five, you have a bug.
Deterministic debugging. When something fails at 2am, you can add a console.log, restart your server, and replay the same event. No hunting for a new trigger, no hoping you can reproduce the path.

Step 4 — Validate via logs, not hope

For every event, verify:

✓ Your handler returned 200 (Stripe will retry otherwise)
✓ The event ID was logged
✓ A record was created (first time)
✓ A replay was skipped (second time — idempotency)
✓ Unknown event types no-op safely (don't throw)

A minimal handler that makes this visible:

export default async function handler(req, res) {
  const event = verifySignature(req); // or skip on dev path

  console.log(`[stripe] ${event.type} ${event.id}`);

  const existing = await findEvent(event.id);
  if (existing) {
    console.log(`[stripe] skipped duplicate ${event.id}`);
    return res.json({ received: true, skipped: true });
  }

  switch (event.type) {
    case 'payment_intent.succeeded':
      await handlePaymentSucceeded(event.data.object);
      break;
    case 'charge.failed':
      await handleChargeFailed(event.data.object);
      break;
    default:
      console.log(`[stripe] no-op for ${event.type}`);
  }

  await recordEvent(event);
  res.json({ received: true });
}

Three rules this enforces:

Log the event ID on every call. When something goes wrong, the event ID is the only key that connects Stripe's dashboard to your logs.
Return 200 even for no-ops. Stripe retries non-200 responses. A throw on an unknown event type will get retried for 3 days and fill up your logs.
Make duplicate detection visible. When you replay for testing, you want to SEE that the skip branch fired.

Pre-production checklist

Before you switch Stripe from your CLI forwarder to your real endpoint:

[ ] Every event type you care about has been triggered locally at least once
[ ] Each one has been replayed and the idempotency branch ran
[ ] At least one unknown event type was sent — handler returned 200, did not throw
[ ] Signature verification works in prod mode with the real Dashboard secret (not the CLI secret)
[ ] Handler returns 200 in under 5 seconds for all event types (Stripe times out at 30s but backs off aggressively if you're slow)
[ ] Logs include event ID on every line relevant to the event

What this costs

Nothing. The Stripe CLI is free. stripe trigger uses your test mode data. stripe events resend uses events you already generated. You don't need a tunnel service (ngrok, localtunnel) — stripe listen does the forwarding itself.

What to pair this with

If you want stored event history you can query later (replay a 2-week-old event, audit a customer's event sequence, etc.), you need something between Stripe and your handler that logs every event permanently. Stripe's own Events API only goes back 30 days and can't filter by fields you care about.

Centrali's webhook triggers do this out of the box — every inbound event is stored in a collection you can query later. But the testing loop above works with any handler, framework, or platform. The important part is the loop.

If this was useful, I write more about webhook reliability and Stripe integration patterns. Follow me for more.

Stop Writing Custom Scrapers: Index Static Content into Meilisearch with One Config

Mary Olowu — Wed, 22 Apr 2026 07:42:56 +0000

TL;DR — content-mill is an open-source CLI and library that reads static content — MkDocs sites, markdown directories, JSON files, HTML pages — and indexes it into Meilisearch, driven by a YAML config. You define the document shape; it handles extraction, templating, chunking, and atomic zero-downtime re-indexing. You still tune templates and debug extraction for your own content — that part's on you — but you stop maintaining bespoke scraper code.
npm install @centrali-io/content-mill

If you've ever tried to make your docs, blog posts, or changelogs searchable with Meilisearch, you know the drill: write a custom scraper, parse the content, transform it into the right shape, push it to an index, and hope you don't break search during re-indexing.

I got tired of writing that glue code for every project, so I built content-mill — a CLI and library that indexes static content into Meilisearch, driven by a YAML config.

The problem

Meilisearch is fantastic for search, but getting your content into it is surprisingly manual. Every docs site, every changelog, every collection of markdown files needs its own extraction pipeline. And if you want zero-downtime re-indexing? That's more code on top.

Most existing solutions are either tightly coupled to a specific framework (like DocSearch for Algolia) or expect you to run a full crawler. Lighter-weight options exist — usually ad-hoc scripts people write once per project — but nothing I could find that's reusable across source types and explicit about document shape.

What content-mill does

You describe your content sources and the document shape you want in a YAML config:

meili:
  host: http://localhost:7700
  apiKey: ${MEILI_MASTER_KEY}

sources:
  - name: docs
    type: mkdocs
    config: ./mkdocs.yml
    index: docs
    document:
      primaryKey: id
      fields:
        id: "{{ slug }}"
        title: "{{ heading }}"
        content: "{{ body }}"
        section: "{{ nav_section }}"
        url: "{{ path }}"
        type: "docs"
      searchableAttributes: [title, content]
      filterableAttributes: [section, type]

Then run:

npx @centrali-io/content-mill index --config content-mill.yml

Once the config matches your content, re-running is a single command. You'll still spend time tuning templates and sanity-checking extraction (use --dry-run for that) — but you're not maintaining scraper code anymore. content-mill handles extraction, templating, and atomic index swapping, so search never goes down during re-indexing.

Four source types, one interface

content-mill ships with adapters for the content formats you're most likely already using:

mkdocs — Reads your mkdocs.yml, follows the nav tree, and parses each markdown page. You get nav_section context so you know which part of the docs each page belongs to.
markdown-dir — Recursively reads .md files from a directory. Supports YAML frontmatter, so you can pull version numbers, dates, or any metadata into your search index. Great for changelogs and blog posts.
json — Reads a JSON array (or directory of JSON files). Every key in each object becomes a template variable. Perfect for structured data you already have lying around.
html — Reads .html files, strips scripts/styles/nav/footer, and gives you clean text. Useful for indexing a built static site.

Templating: you control the document shape

The key design decision is that you define what your Meilisearch documents look like. Source adapters extract raw variables (slug, heading, body, path, frontmatter.*, etc.), and you map them to fields using {{ template }} syntax:

fields:
  id: "{{ slug }}-{{ chunk_index }}"
  title: "{{ chunk_heading }}"
  content: "{{ chunk_body }}"
  excerpt: "{{ body | truncate(200) }}"
  url: "{{ path }}#{{ chunk_heading | slugify }}"

Filters like truncate, slugify, lower, upper, and strip_md can be chained with pipes. This means you're not locked into someone else's schema — your search index looks exactly the way your frontend expects.

Chunking for granular results

Whole-page results are often too broad for docs search. content-mill can split pages by heading level:

chunking:
  strategy: heading
  level: 2

This turns one long page into multiple documents — one per ## section — each with its own chunk_heading, chunk_body, and chunk_index. Your search results can now link directly to the relevant section instead of dumping users at the top of a page.

Zero-downtime re-indexing

Every indexing run uses Meilisearch's index swap:

Documents go into a temp index (docs_tmp)
Atomic swap with the live index (docs)
Old index gets cleaned up

If something fails mid-way, your live index is untouched. No maintenance windows needed.

CI/CD in two lines

# GitHub Actions
- name: Index docs
  env:
    MEILI_MASTER_KEY: ${{ secrets.MEILI_MASTER_KEY }}
  run: npx @centrali-io/content-mill index --config content-mill.yml

Hook this into your release pipeline and your search index stays in sync with every deploy.

Use as a library

Don't need the CLI? Import it directly:

import { loadConfig, indexAll } from '@centrali-io/content-mill';

const config = loadConfig('./content-mill.yml');
await indexAll(config, { dryRun: false });

Or build the config object in code if you prefer programmatic control.

Why not docs-scraper, DocSearch, or a custom crawler?

docs-scraper (the Meilisearch-native option) is a Scrapy-based web crawler. Works well for live sites, heavy for "I already have markdown in a repo."
Algolia DocSearch is excellent, but framework-specific and indexes into Algolia — not useful if you've chosen Meilisearch.
Custom scrapers work fine for one project. Painful when you have three of them to maintain across different repos.

content-mill is intentionally narrow: static content in, Meilisearch out, config-driven shape in between. If you're not already on Meilisearch, use something else.

Getting started

npm install @centrali-io/content-mill

Create a content-mill.yml with your Meilisearch connection and source definitions
Run with --dry-run first to preview the extracted documents
Run for real and check your Meilisearch dashboard

The full config reference and source type examples are in the README on GitHub.

content-mill is MIT-licensed and open source. If you use Meilisearch and have static content to index, try it — and if your source type isn't covered (AsciiDoc, RST, Notion export, whatever), open an issue and I'll look at adding an adapter.

Stop Writing Custom Scrapers: Index Any Static Content into Meilisearch with One Config File

Mary Olowu — Wed, 25 Mar 2026 05:16:16 +0000

I got tired of writing that glue code for every project, so I built content-mill — a CLI and library that indexes static content into Meilisearch from a single YAML config.

The Problem

Most existing solutions are either tightly coupled to a specific framework (like DocSearch for Algolia) or require you to write a full crawler. If you just have some markdown files and a Meilisearch instance, there's nothing lightweight that bridges the gap.

What content-mill Does

You describe your content sources and the document shape you want in a YAML config:

meili:
  host: http://localhost:7700
  apiKey: ${MEILI_MASTER_KEY}

sources:
  - name: docs
    type: mkdocs
    config: ./mkdocs.yml
    index: docs
    document:
      primaryKey: id
      fields:
        id: "{{ slug }}"
        title: "{{ heading }}"
        content: "{{ body }}"
        section: "{{ nav_section }}"
        url: "{{ path }}"
        type: "docs"
      searchableAttributes: [title, content]
      filterableAttributes: [section, type]

Then run:

npx @centrali-io/content-mill index --config content-mill.yml

That's it. content-mill reads your sources, extracts content, applies your field templates, and pushes everything to Meilisearch with atomic index swapping (so search never goes down during re-indexing).

Four Source Types, One Interface

content-mill ships with adapters for the content formats you're most likely already using:

mkdocs — Reads your mkdocs.yml, follows the nav tree, and parses each markdown page. You get nav_section context so you know which part of the docs each page belongs to.

markdown-dir — Recursively reads .md files from a directory. Supports YAML frontmatter, so you can pull version numbers, dates, or any metadata into your search index. Great for changelogs and blog posts.

json — Reads a JSON array (or directory of JSON files). Every key in each object becomes a template variable. Perfect for structured data you already have lying around.

html — Reads .html files, strips scripts/styles/nav/footer, and gives you clean text. Useful for indexing a built static site.

Templating: You Control the Document Shape

fields:
  id: "{{ slug }}-{{ chunk_index }}"
  title: "{{ chunk_heading }}"
  content: "{{ chunk_body }}"
  excerpt: "{{ body | truncate(200) }}"
  url: "{{ path }}#{{ chunk_heading | slugify }}"

Chunking for Granular Results

Whole-page results are often too broad for docs search. content-mill can split pages by heading level:

chunking:
  strategy: heading
  level: 2

Zero-Downtime Re-indexing

Every indexing run uses Meilisearch's index swap:

Documents go into a temp index (docs_tmp)
Atomic swap with the live index (docs)
Old index gets cleaned up

If something fails mid-way, your live index is untouched. No maintenance windows needed.

CI/CD in Two Lines

# GitHub Actions
- name: Index docs
  env:
    MEILI_MASTER_KEY: ${{ secrets.MEILI_MASTER_KEY }}
  run: npx @centrali-io/content-mill index --config content-mill.yml

Hook this into your release pipeline and your search index stays in sync with every deploy.

Use as a Library

Don't need the CLI? Import it directly:

import { loadConfig, indexAll } from '@centrali-io/content-mill';

const config = loadConfig('./content-mill.yml');
await indexAll(config, { dryRun: false });

Or build the config object in code if you prefer programmatic control.

Getting Started

npm install @centrali-io/content-mill

Create a content-mill.yml with your Meilisearch connection and source definitions
Run with --dry-run first to preview the extracted documents
Run for real and check your Meilisearch dashboard

The full config reference and source type examples are in the README on GitHub.

content-mill is MIT licensed and open source. If you're using Meilisearch and have static content to index, I'd love to hear how it works for your use case. Issues and PRs welcome on GitHub.

Tags: #meilisearch #search #typescript #opensource

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

Mary Olowu — Sun, 15 Mar 2026 23:46:07 +0000

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

The Core Idea

When you're building rate limits for event-driven triggers, you face a fundamental problem: how do you set a threshold that catches loops without blocking legitimate high-volume workloads?

The answer is that loops and legitimate traffic have fundamentally different growth characteristics:

Legitimate triggers scale linearly with user actions.

1 user creates 1 order → 1 trigger execution
50 users create 50 orders per minute → 50 trigger executions per minute
The ratio is always 1:1. Trigger executions track user actions.

Recursive loops scale exponentially from a single user action.

1 user creates 1 record → trigger fires → function creates another record → trigger fires again
After 10 seconds: 100+ executions
After 60 seconds: 700+ executions
All from 1 user action. The trigger is its own input.

This isn't a subtle distinction. It's the difference between a line and an exponential curve. And it means your rate limit doesn't need to be clever — it just needs to sit in the massive gap between the two curves.

Why This Matters for Rate Limit Design

A rate limit of 100 executions per 60 seconds:

Never blocks legitimate traffic. Even a high-volume e-commerce system processing 80 orders per minute sits under the limit.
Always catches loops. A recursive loop hits 100 executions in under 8 seconds.

The gap between "highest legitimate volume" and "slowest possible loop" is enormous. You don't need machine learning or anomaly detection. You just need basic arithmetic.

The Math

A recursive trigger loop doubles (at minimum) with each iteration. If one trigger execution creates one record, and that record fires one trigger:

Time	Executions (cumulative)
Iteration 1	1
Iteration 2	2
Iteration 3	4
Iteration 10	1,024
Iteration 16	65,536

Even with network latency and compute overhead slowing each iteration to 100ms, you hit 100 executions in ~7 seconds. With faster execution (10ms per iteration), you hit 100 in under a second.

Meanwhile, the highest legitimate trigger volume we've seen across our platform is ~80 executions per minute per trigger — and that's a busy e-commerce workspace during a flash sale.

The gap is 10x-100x. Your rate limit has a lot of room.

What About Burst Traffic?

The natural objection: "What about a bulk import? A user imports 500 records at once, and each fires a trigger."

This is a valid concern but a different problem:

Bulk imports via API publish a single aggregate event (records_bulk_created), not 500 individual events. Event-driven triggers don't match on the aggregate event, so they don't fire at all.
Batch operations from compute functions do publish individual events. But even 500 trigger executions from a batch operation is a one-time burst, not a sustained loop. If your rate limit window is 60 seconds, the burst registers once. A loop registers continuously.
If batch-triggered functions need to fire triggers, the rate limit should be configurable per-trigger. Default 100/60s works for 99% of cases. The 1% that needs more can raise it.

Implementing the Test

The simplest implementation is a Redis counter with a TTL:

async function isWithinRateLimit(triggerId: string): Promise<boolean> {
  const key = `trigger_rate:${triggerId}`;
  const count = await redis.incr(key);
  if (count === 1) await redis.expire(key, 60);
  return count <= 100;
}

That's it. Six lines. The INCR is atomic (no race conditions across instances), the EXPIRE handles cleanup, and the threshold separates linear from exponential with a 10x margin.

Beyond Rate Limiting

Rate limiting is the safety net, not the whole solution. For a complete defense:

Block obvious loops at configuration time. When a user creates a trigger on record_created for collection X, and the function calls api.createRecord('X', ...), reject it with a clear error. This is prevention, not detection.
Track causality at runtime. Propagate a sourceTriggerId through event chains so you can identify self-loops without waiting for the rate limit to trip. The user gets a "recursive loop detected" message instead of a vague "rate limit exceeded."
Rate limit as the catch-all. For cross-trigger chains (A→B→A) and exotic patterns that bypass the first two layers.

We wrote a detailed post about implementing all three layers: How We Stopped Recursive Trigger Loops From Melting Our Compute Fleet.

The Takeaway

If your platform has event-driven triggers, ask yourself: can a trigger's output become its own input? If yes, you need loop protection. And the simplest, most reliable loop protection is a rate limit set in the gap between linear user-driven traffic and exponential recursive behavior.

That gap is enormous. Use it.

Building event-driven infrastructure? We'd love to hear about your trigger architecture challenges. Reach out on [Twitter/X] @centraliio or drop a comment.

Forem: Mary Olowu

Add Webhooks to Your SaaS in 10 Minutes (Without Queues or Retries)

The whole thing, in one SDK call

What you'd build yourself (and won't have to)

What Centrali gives you

Tutorial: ship an outbound webhook in 5 minutes

Step 1: Create a collection

Step 2: Subscribe a customer endpoint

Step 3: Trigger an event

Step 4: Verify the signature on the customer side

What happens when the customer's endpoint is down

Replay a failed delivery once the endpoint is healthy

What's in the subscription model

When you should still build it yourself

Related reading

What Is Record TTL? Database Time-to-Live Explained

What Is Record TTL?

How TTL Works (Mechanically)

When to Use Record TTL

Default TTL vs. Per-Record TTL

Database TTL vs. DNS TTL

Implementing Record TTL

Common Questions

Summary

How to Test Stripe Webhooks Locally (Stripe CLI + Replay + Logs)

The loop, in full

Step 1 — Give your dev webhook its own path

Step 2 — Trigger real events, not made-up payloads

Step 3 — Replay the exact event, on demand

Step 4 — Validate via logs, not hope

Pre-production checklist

What this costs

What to pair this with

Stop Writing Custom Scrapers: Index Static Content into Meilisearch with One Config

The problem

What content-mill does

Four source types, one interface

Templating: you control the document shape

Chunking for granular results

Zero-downtime re-indexing

CI/CD in two lines

Use as a library

Why not docs-scraper, DocSearch, or a custom crawler?

Getting started

Stop Writing Custom Scrapers: Index Any Static Content into Meilisearch with One Config File

The Problem

What content-mill Does

Four Source Types, One Interface

Templating: You Control the Document Shape

Chunking for Granular Results

Zero-Downtime Re-indexing

CI/CD in Two Lines

Use as a Library

Getting Started

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

The Core Idea

Why This Matters for Rate Limit Design

The Math

What About Burst Traffic?

Implementing the Test

Beyond Rate Limiting

The Takeaway