Forem: Mary Olowu

Stop Writing Custom Scrapers: Index Any Static Content into Meilisearch with One Config File

Mary Olowu — Wed, 25 Mar 2026 05:16:16 +0000

If you've ever tried to make your docs, blog posts, or changelogs searchable with Meilisearch, you know the drill: write a custom scraper, parse the content, transform it into the right shape, push it to an index, and hope you don't break search during re-indexing.

I got tired of writing that glue code for every project, so I built content-mill — a CLI and library that indexes static content into Meilisearch from a single YAML config.

The Problem

Meilisearch is fantastic for search, but getting your content into it is surprisingly manual. Every docs site, every changelog, every collection of markdown files needs its own extraction pipeline. And if you want zero-downtime re-indexing? That's more code on top.

Most existing solutions are either tightly coupled to a specific framework (like DocSearch for Algolia) or require you to write a full crawler. If you just have some markdown files and a Meilisearch instance, there's nothing lightweight that bridges the gap.

What content-mill Does

You describe your content sources and the document shape you want in a YAML config:

meili:
  host: http://localhost:7700
  apiKey: ${MEILI_MASTER_KEY}

sources:
  - name: docs
    type: mkdocs
    config: ./mkdocs.yml
    index: docs
    document:
      primaryKey: id
      fields:
        id: "{{ slug }}"
        title: "{{ heading }}"
        content: "{{ body }}"
        section: "{{ nav_section }}"
        url: "{{ path }}"
        type: "docs"
      searchableAttributes: [title, content]
      filterableAttributes: [section, type]

Then run:

npx @centrali-io/content-mill index --config content-mill.yml

That's it. content-mill reads your sources, extracts content, applies your field templates, and pushes everything to Meilisearch with atomic index swapping (so search never goes down during re-indexing).

Four Source Types, One Interface

content-mill ships with adapters for the content formats you're most likely already using:

mkdocs — Reads your mkdocs.yml, follows the nav tree, and parses each markdown page. You get nav_section context so you know which part of the docs each page belongs to.

markdown-dir — Recursively reads .md files from a directory. Supports YAML frontmatter, so you can pull version numbers, dates, or any metadata into your search index. Great for changelogs and blog posts.

json — Reads a JSON array (or directory of JSON files). Every key in each object becomes a template variable. Perfect for structured data you already have lying around.

html — Reads .html files, strips scripts/styles/nav/footer, and gives you clean text. Useful for indexing a built static site.

Templating: You Control the Document Shape

The key design decision is that you define what your Meilisearch documents look like. Source adapters extract raw variables (slug, heading, body, path, frontmatter.*, etc.), and you map them to fields using {{ template }} syntax:

fields:
  id: "{{ slug }}-{{ chunk_index }}"
  title: "{{ chunk_heading }}"
  content: "{{ chunk_body }}"
  excerpt: "{{ body | truncate(200) }}"
  url: "{{ path }}#{{ chunk_heading | slugify }}"

Filters like truncate, slugify, lower, upper, and strip_md can be chained with pipes. This means you're not locked into someone else's schema — your search index looks exactly the way your frontend expects.

Chunking for Granular Results

Whole-page results are often too broad for docs search. content-mill can split pages by heading level:

chunking:
  strategy: heading
  level: 2

This turns one long page into multiple documents — one per ## section — each with its own chunk_heading, chunk_body, and chunk_index. Your search results can now link directly to the relevant section instead of dumping users at the top of a page.

Zero-Downtime Re-indexing

Every indexing run uses Meilisearch's index swap:

Documents go into a temp index (docs_tmp)
Atomic swap with the live index (docs)
Old index gets cleaned up

If something fails mid-way, your live index is untouched. No maintenance windows needed.

CI/CD in Two Lines

# GitHub Actions
- name: Index docs
  env:
    MEILI_MASTER_KEY: ${{ secrets.MEILI_MASTER_KEY }}
  run: npx @centrali-io/content-mill index --config content-mill.yml

Hook this into your release pipeline and your search index stays in sync with every deploy.

Use as a Library

Don't need the CLI? Import it directly:

import { loadConfig, indexAll } from '@centrali-io/content-mill';

const config = loadConfig('./content-mill.yml');
await indexAll(config, { dryRun: false });

Or build the config object in code if you prefer programmatic control.

Getting Started

npm install @centrali-io/content-mill

Create a content-mill.yml with your Meilisearch connection and source definitions
Run with --dry-run first to preview the extracted documents
Run for real and check your Meilisearch dashboard

The full config reference and source type examples are in the README on GitHub.

content-mill is MIT licensed and open source. If you're using Meilisearch and have static content to index, I'd love to hear how it works for your use case. Issues and PRs welcome on GitHub.

Tags: #meilisearch #search #typescript #opensource

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

Mary Olowu — Sun, 15 Mar 2026 23:46:07 +0000

Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping

The Core Idea

When you're building rate limits for event-driven triggers, you face a fundamental problem: how do you set a threshold that catches loops without blocking legitimate high-volume workloads?

The answer is that loops and legitimate traffic have fundamentally different growth characteristics:

Legitimate triggers scale linearly with user actions.

1 user creates 1 order → 1 trigger execution
50 users create 50 orders per minute → 50 trigger executions per minute
The ratio is always 1:1. Trigger executions track user actions.

Recursive loops scale exponentially from a single user action.

1 user creates 1 record → trigger fires → function creates another record → trigger fires again
After 10 seconds: 100+ executions
After 60 seconds: 700+ executions
All from 1 user action. The trigger is its own input.

This isn't a subtle distinction. It's the difference between a line and an exponential curve. And it means your rate limit doesn't need to be clever — it just needs to sit in the massive gap between the two curves.

Why This Matters for Rate Limit Design

A rate limit of 100 executions per 60 seconds:

Never blocks legitimate traffic. Even a high-volume e-commerce system processing 80 orders per minute sits under the limit.
Always catches loops. A recursive loop hits 100 executions in under 8 seconds.

The gap between "highest legitimate volume" and "slowest possible loop" is enormous. You don't need machine learning or anomaly detection. You just need basic arithmetic.

The Math

A recursive trigger loop doubles (at minimum) with each iteration. If one trigger execution creates one record, and that record fires one trigger:

Time	Executions (cumulative)
Iteration 1	1
Iteration 2	2
Iteration 3	4
Iteration 10	1,024
Iteration 16	65,536

Even with network latency and compute overhead slowing each iteration to 100ms, you hit 100 executions in ~7 seconds. With faster execution (10ms per iteration), you hit 100 in under a second.

Meanwhile, the highest legitimate trigger volume we've seen across our platform is ~80 executions per minute per trigger — and that's a busy e-commerce workspace during a flash sale.

The gap is 10x-100x. Your rate limit has a lot of room.

What About Burst Traffic?

The natural objection: "What about a bulk import? A user imports 500 records at once, and each fires a trigger."

This is a valid concern but a different problem:

Bulk imports via API publish a single aggregate event (records_bulk_created), not 500 individual events. Event-driven triggers don't match on the aggregate event, so they don't fire at all.
Batch operations from compute functions do publish individual events. But even 500 trigger executions from a batch operation is a one-time burst, not a sustained loop. If your rate limit window is 60 seconds, the burst registers once. A loop registers continuously.
If batch-triggered functions need to fire triggers, the rate limit should be configurable per-trigger. Default 100/60s works for 99% of cases. The 1% that needs more can raise it.

Implementing the Test

The simplest implementation is a Redis counter with a TTL:

async function isWithinRateLimit(triggerId: string): Promise<boolean> {
  const key = `trigger_rate:${triggerId}`;
  const count = await redis.incr(key);
  if (count === 1) await redis.expire(key, 60);
  return count <= 100;
}

That's it. Six lines. The INCR is atomic (no race conditions across instances), the EXPIRE handles cleanup, and the threshold separates linear from exponential with a 10x margin.

Beyond Rate Limiting

Rate limiting is the safety net, not the whole solution. For a complete defense:

Block obvious loops at configuration time. When a user creates a trigger on record_created for collection X, and the function calls api.createRecord('X', ...), reject it with a clear error. This is prevention, not detection.
Track causality at runtime. Propagate a sourceTriggerId through event chains so you can identify self-loops without waiting for the rate limit to trip. The user gets a "recursive loop detected" message instead of a vague "rate limit exceeded."
Rate limit as the catch-all. For cross-trigger chains (A→B→A) and exotic patterns that bypass the first two layers.

We wrote a detailed post about implementing all three layers: How We Stopped Recursive Trigger Loops From Melting Our Compute Fleet.

The Takeaway

If your platform has event-driven triggers, ask yourself: can a trigger's output become its own input? If yes, you need loop protection. And the simplest, most reliable loop protection is a rate limit set in the gap between linear user-driven traffic and exponential recursive behavior.

That gap is enormous. Use it.

Building event-driven infrastructure? We'd love to hear about your trigger architecture challenges. Reach out on [Twitter/X] @centraliio or drop a comment.