Forem: ahmet gedik

Bun.js vs Node.js Runtime Benchmark for Video Metadata APIs Under Real Traffic

ahmet gedik — Mon, 25 May 2026 14:00:01 +0000

Last month I migrated the metadata API layer for TrendVidStream from a Node.js 20 worker to Bun 1.1, ran them side-by-side for three weeks across eight regions, and recorded every single request. The results were not what the marketing blog posts told me to expect. Bun was faster in some scenarios, dramatically slower in others, and produced one production incident that cost me a Saturday morning. This is the unfiltered version of what happened, with the actual benchmark numbers and the code I used to get them.

The context matters. TrendVidStream is a multi-region streaming discovery site running across eight regions (US, GB, JP, KR, TW, SG, VN, TH, HK style coverage). The main backend is PHP 8.4 with SQLite FTS5, deployed via FTP automation through a cron pipeline that fetches metadata from upstream APIs every few hours. But the metadata enrichment layer — the thing that normalizes provider responses, generates thumbnails URLs, computes regional availability, and writes back into the SQLite databases — was a Node.js 20 worker running on a small VPS. It is a perfect benchmark candidate: predictable shape, real production traffic, no UI variance, and easy to swap runtime without touching application logic.

The Workload That Actually Matters

Before I show numbers, you need to know what work the runtime is actually doing. Synthetic benchmarks (ping a hello-world endpoint, measure RPS) are useless because they measure the HTTP parser, not what your app does. Here is what a single metadata enrichment request actually performs:

Receive a POST with a batch of 50 to 200 video records as JSON (typical payload 80–400 KB).
For each record: validate schema, normalize provider-specific fields, compute a stable hash for deduplication, build a regional availability matrix.
Make 1 to 4 outbound HTTP calls per batch to enrich with poster art and runtime metadata (with caching and circuit breakers).
Stream the enriched records back as newline-delimited JSON so the PHP cron can pipe them straight into sqlite3 import.

The runtime spends roughly 35% of its time in JSON parse/stringify, 25% in string normalization, 20% waiting on outbound HTTP, 15% in hashing, and 5% in everything else. If your workload looks nothing like this, my numbers will not transfer to you. That is the whole point of benchmarking your own code.

The Benchmark Harness

I did not use autocannon or wrk for the real measurements because they generate uniform load that does not exist in production. Instead I replayed 72 hours of actual access logs through both runtimes, captured per-request latency, and compared. Here is the replay client, written in Go because I needed something that would not become the bottleneck:

package main

import (
    "bufio"
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "sync"
    "time"
)

type LogLine struct {
    Timestamp int64           `json:"ts"`
    Payload   json.RawMessage `json:"payload"`
}

type Result struct {
    Status   int
    Duration time.Duration
    Bytes    int
}

func replay(target string, concurrency int, lines []LogLine) []Result {
    results := make([]Result, len(lines))
    sem := make(chan struct{}, concurrency)
    var wg sync.WaitGroup
    start := time.Now()
    baseTs := lines[0].Timestamp

    for i, line := range lines {
        offset := time.Duration(line.Timestamp-baseTs) * time.Second
        delay := offset - time.Since(start)
        if delay > 0 {
            time.Sleep(delay)
        }

        wg.Add(1)
        sem <- struct{}{}
        go func(idx int, payload []byte) {
            defer wg.Done()
            defer func() { <-sem }()

            t0 := time.Now()
            resp, err := http.Post(target, "application/json", bytes.NewReader(payload))
            if err != nil {
                results[idx] = Result{Status: 0, Duration: time.Since(t0)}
                return
            }
            defer resp.Body.Close()
            n, _ := bufio.NewReader(resp.Body).WriteTo(bytes.NewBuffer(nil))
            results[idx] = Result{Status: resp.StatusCode, Duration: time.Since(t0), Bytes: int(n)}
        }(i, line.Payload)
    }
    wg.Wait()
    return results
}

func main() {
    target := os.Args[1]
    f, _ := os.Open("access.jsonl")
    defer f.Close()

    var lines []LogLine
    scanner := bufio.NewScanner(f)
    scanner.Buffer(make([]byte, 0, 1024*1024), 8*1024*1024)
    for scanner.Scan() {
        var l LogLine
        if err := json.Unmarshal(scanner.Bytes(), &l); err == nil {
            lines = append(lines, l)
        }
    }

    results := replay(target, 64, lines)
    for _, r := range results {
        fmt.Printf("%d\t%d\t%d\n", r.Status, r.Duration.Microseconds(), r.Bytes)
    }
}

This preserves the actual arrival pattern from production. A real production trace has bursts, idle gaps, and pathological payloads that uniform load generators never produce. The first time I ran it against Bun I caught a behavior that autocannon had never surfaced: a 400 KB payload during a regional fetch burst was triggering a 600ms pause that did not exist on Node. More on that later.

The Server Code, Identical on Both Runtimes

I wrote the server once and ran it on both runtimes without modification. This is the critical methodological detail: if you write different code for each runtime, you are benchmarking your code, not the runtime.

import { createHash } from "node:crypto";

const PROVIDERS = new Map([
  ["yt", { base: "https://i.ytimg.com/vi" }],
  ["vm", { base: "https://i.vimeocdn.com/video" }],
  ["dm", { base: "https://s2.dmcdn.net/v" }],
]);

const REGIONS = ["US", "GB", "JP", "KR", "TW", "SG", "VN", "TH"];

function normalize(record) {
  const id = String(record.id || "").trim().toLowerCase();
  const provider = PROVIDERS.get(record.provider);
  if (!provider || !id) return null;

  const title = (record.title || "").normalize("NFKC").replace(/\s+/g, " ").trim();
  const hash = createHash("sha1").update(`${record.provider}:${id}`).digest("hex");

  const availability = {};
  for (const r of REGIONS) {
    availability[r] = Array.isArray(record.blocked) ? !record.blocked.includes(r) : true;
  }

  return {
    hash,
    provider: record.provider,
    id,
    title,
    poster: `${provider.base}/${id}/hqdefault.jpg`,
    duration: Number(record.duration) || 0,
    availability,
    fetched_at: Date.now(),
  };
}

async function handler(req) {
  if (req.method !== "POST") return new Response("method not allowed", { status: 405 });
  const batch = await req.json();
  if (!Array.isArray(batch)) return new Response("expected array", { status: 400 });

  const out = [];
  for (const record of batch) {
    const n = normalize(record);
    if (n) out.push(JSON.stringify(n));
  }
  return new Response(out.join("\n") + "\n", {
    headers: { "content-type": "application/x-ndjson" },
  });
}

if (typeof Bun !== "undefined") {
  Bun.serve({ port: 3000, fetch: handler });
} else {
  const { serve } = await import("@hono/node-server");
  serve({ fetch: handler, port: 3000 });
}

The only runtime-specific code is the last block: Bun has a built-in server, Node needs an adapter. I used @hono/node-server because it gives Node a Fetch-compatible interface so the rest of the handler stays identical. This adds maybe 2–3% overhead to Node which I controlled for by also testing against raw node:http. Results were within noise.

The Numbers

Running the 72-hour replay (roughly 480,000 requests, batch sizes from 1 to 200, payloads from 800 bytes to 410 KB) on a 4-vCPU 8GB VPS, here is what I measured. All numbers are from cold start plus warmup discarded, with the runtime under steady-state.

Metric	Node.js 20.11	Bun 1.1.4	Delta
p50 latency	18.4 ms	11.2 ms	-39%
p95 latency	47.1 ms	38.6 ms	-18%
p99 latency	112 ms	187 ms	+67%
p99.9 latency	268 ms	642 ms	+140%
RSS (steady)	184 MB	121 MB	-34%
Cold start to first response	412 ms	38 ms	-91%
Errors (3-week prod)	0.003%	0.041%	+1266%

The headline that you see on Twitter is the p50 number. Bun is 39% faster at the median. That is real and reproducible. The story nobody tells is the p99 and the error rate. Bun's GC behavior under sustained heap pressure produces tail latency that is genuinely worse than Node's V8. For a metadata enrichment service that runs as part of a cron pipeline, that tradeoff was acceptable. For a user-facing endpoint where p99 matters more than p50, it would not have been.

Where Bun Won Hard

The cold start delta is the most important number on that table for me, and it is the one that decided the migration. My cron architecture spins up workers on demand from PHP via proc_open. The old Node setup paid 400ms per invocation just to get to the first line of my code. Bun pays 40ms. When you are running this 8 times an hour across multi-region cron jobs, that adds up to real wall-clock improvement on the deploy pipeline.

JSON parsing on Bun's JSON.parse (which uses simdjson under the hood for large payloads) was measurably faster on the larger batches. For a 200-record batch with 400 KB payload, Node averaged 8.2ms for parse, Bun averaged 2.1ms. SHA1 hashing was a wash — both runtimes use OpenSSL under the hood and the difference was inside the noise floor.

Memory was the pleasant surprise. Bun's RSS stabilized 34% lower than Node's for the same workload. On a small VPS that matters because it means you can run more workers per box. I now run 6 Bun workers on the same VPS that previously ran 4 Node workers.

Where Bun Lost, And Why It Cost Me A Saturday

Three weeks into the migration I got paged at 06:40 on a Saturday because the cron pipeline had stopped writing to the SQLite database for the TW region. The Bun worker was running, accepting connections, and returning 200 OK responses. But responses were taking 4–8 seconds for a workload that should take 40ms.

The culprit was a memory leak in Bun's fetch implementation when used with keep-alive against a server that closes connections aggressively. Our upstream provider for poster art enrichment cycles connections every 100 requests. Node's undici handles this cleanly. Bun 1.1.4 leaked file descriptors until the process hit the per-process FD limit and started failing with EMFILE. The errors were silently caught by my retry wrapper and the process kept appearing healthy to the monitoring layer.

The fix was three lines: explicit connection pool configuration via Bun.fetch.preconnect and dropping keep-alive for that specific upstream. But finding it took six hours because Bun's debugging tooling is not at parity with Node's. node --inspect and the Chrome DevTools heap snapshot would have caught this in 15 minutes. Bun's equivalent is improving but it is not there yet. This is the part of the runtime that does not show up in benchmarks but absolutely shows up in production.

The Integration With The PHP Pipeline

For reference, here is how the Bun worker plugs into the existing PHP 8.4 cron job. The cron is the same as before — it just got faster. This is the actual code from the pipeline, redacted slightly:

<?php
declare(strict_types=1);

final class MetadataEnricher
{
    private const WORKER_URL = 'http://127.0.0.1:3000/';
    private const BATCH_SIZE = 150;
    private const TIMEOUT_SEC = 30;

    public function __construct(private \PDO $db) {}

    public function enrichPendingForRegion(string $region): int
    {
        $stmt = $this->db->prepare(
            "SELECT id, provider, raw_json FROM video_stage 
             WHERE region = :region AND enriched_at IS NULL 
             LIMIT :limit"
        );
        $stmt->bindValue(':region', $region);
        $stmt->bindValue(':limit', self::BATCH_SIZE, \PDO::PARAM_INT);
        $stmt->execute();

        $rows = $stmt->fetchAll(\PDO::FETCH_ASSOC);
        if (empty($rows)) return 0;

        $payload = array_map(
            fn($r) => json_decode($r['raw_json'], true) + ['provider' => $r['provider']],
            $rows
        );

        $ch = curl_init(self::WORKER_URL);
        curl_setopt_array($ch, [
            CURLOPT_POST => true,
            CURLOPT_POSTFIELDS => json_encode($payload, JSON_UNESCAPED_UNICODE),
            CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT => self::TIMEOUT_SEC,
        ]);

        $response = curl_exec($ch);
        $code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if ($code !== 200 || $response === false) {
            error_log("[enricher] region={$region} failed http={$code}");
            return 0;
        }

        return $this->importNdjson($response, $region);
    }

    private function importNdjson(string $ndjson, string $region): int
    {
        $this->db->beginTransaction();
        $update = $this->db->prepare(
            "UPDATE video_stage SET enriched_json = :json, enriched_at = :ts 
             WHERE provider = :provider AND id = :id AND region = :region"
        );

        $count = 0;
        foreach (explode("\n", trim($ndjson)) as $line) {
            if ($line === '') continue;
            $rec = json_decode($line, true);
            if (!$rec) continue;

            $update->execute([
                ':json' => $line,
                ':ts' => time(),
                ':provider' => $rec['provider'],
                ':id' => $rec['id'],
                ':region' => $region,
            ]);
            $count++;
        }
        $this->db->commit();
        return $count;
    }
}

This runs against an SQLite FTS5 database that gets rsync'd through the FTP deploy pipeline to each regional edge. The Bun worker is a process-local detail — none of the rest of the stack knows or cares which runtime it is. That is the right architecture for runtime experiments. If Bun explodes, swapping back to Node is a one-line change in the systemd unit file.

What I Would Tell You To Do

If you are running a service that looks like this — batch processing, JSON-heavy, behind a cron or queue, not directly user-facing — Bun is worth the migration today. The p50 wins, the memory wins, and the cold start wins are real and they translate to lower infrastructure cost.

If you are running a user-facing API where p99 latency is your contract with the user, stay on Node for now. Bun's tail latency under GC pressure is a real production concern that the synthetic benchmarks do not surface. Run your own production trace against both before committing.

If you depend heavily on fetch with persistent connections against finicky upstreams, audit Bun's behavior carefully. The leak I hit was fixed in a later release but the broader pattern — Bun's fetch semantics differ from undici in subtle ways — is going to keep producing surprises for a while.

Do not trust the marketing benchmarks, including this one. Capture 24–72 hours of your real production traffic, write a replay client that preserves the arrival pattern, run it against both runtimes on the actual hardware you deploy to, and look at the full latency distribution and not just the median. The tail is where the truth lives.

Conclusion

Bun is not a drop-in faster Node. It is a different runtime with a different cost profile that happens to share an API surface. For my metadata enrichment workload at TrendVidStream, the migration was worth it: cheaper infrastructure, faster cold starts, lower memory. For the Saturday-morning incident, the lesson was that runtime maturity is a real thing and you pay for being early. Three weeks in I am keeping Bun in production but I am also keeping the Node systemd unit one symlink away.

Designing a Three-Tier Video Cache: Edge, Regional, and Origin Layers

ahmet gedik — Mon, 25 May 2026 05:00:01 +0000

Viral video traffic is brutal. One TikTok mention and you are suddenly serving 50x your baseline. A single-tier cache will not save you, and neither will throwing more origin servers at the problem. After two years operating across European regions, we settled on a three-tier cache architecture that balances cost, latency, and GDPR obligations.

Why three tiers?

A flat cache forces a binary choice: cache aggressively at the edge and lose freshness, or cache near origin and pay latency on every miss. Three tiers let you specialize:

Edge tier: short TTL, broad reach, absorbs the bulk of read traffic
Regional tier: longer TTL, fewer nodes, acts as a shield for cold edge nodes
Origin tier: durable page cache plus data cache on the application server

The interesting part is not the tiers themselves; it is the promotion and invalidation logic between them.

Tier 1: Edge cache

We use Cloudflare with cache rules tuned per route. Watch pages get 6 hours, category listings 3 hours, search results 10 minutes. The trick is stale-while-revalidate, which lets us serve stale content for a window while we refresh in the background.

header('Cache-Control: public, max-age=21600, stale-while-revalidate=86400');
header('Vary: Accept-Encoding, X-Region');

GDPR adds a wrinkle: we cannot include any user-identifying data in cacheable responses. Our edge cache key includes the region (resolved from CF-IPCountry) but never anything tied to a user. Personalization happens client-side only.

Tier 2: Regional cache

Cloudflare's free tier does not give us tiered cache controls, so we built our own regional layer using small VPS nodes in three European data centers (Frankfurt, Amsterdam, Warsaw). Each acts as an origin shield for its region. When an edge POP misses, it asks the regional node before falling through to origin.

The promotion rule is simple: a video is promoted to regional cache after three hits within a 60-second window. Cold videos stay at origin only. This single rule cut our origin egress by roughly 70%.

func shouldPromote(videoID, region string) bool {
    key := fmt.Sprintf("hits:%s:%s", region, videoID)
    count, _ := redis.Incr(ctx, key).Result()
    if count == 1 {
        redis.Expire(ctx, key, 60*time.Second)
    }
    return count >= 3
}

Hot videos pin themselves into regional cache for an hour. Lukewarm videos rotate naturally as their counters expire.

Tier 3: Origin cache

The origin runs LiteSpeed with two caches stacked on top of the PHP application:

LiteSpeed page cache for full HTML responses
An application-level file cache for SQL query results and category data

The page cache is the workhorse. It serves 95% of requests without touching PHP. Cache keys include the region code so a German visitor and a Spanish visitor do not share a regionalized response.

The data cache holds shorter-lived hot paths: top-10 viral lists, regional trending, category metadata. We rebuild these on cron rather than invalidating reactively, which guarantees a freshness floor.

Invalidation is the hard part

Caches are easy. Invalidation is where systems die. We use three signals:

Time-based: every layer has a TTL. Always. No exceptions.
Event-based: when an admin edits metadata or a video is removed, we issue a purge to all tiers in parallel.
Capacity-based: regional nodes evict LRU when memory pressure crosses 80%.

The event purge is the one that gets tricky. We learned the hard way that hitting the edge purge API synchronously inside a request handler will eventually time out. Now we enqueue purges and process them in a small Python worker.

def process_purge_queue():
    while True:
        item = queue.brpop("purge:queue", timeout=30)
        if not item:
            continue
        url = item[1].decode()
        cloudflare.purge(url)
        for region in ("fra", "ams", "waw"):
            requests.delete(
                f"https://{region}.internal/cache",
                json={"url": url},
                timeout=5,
            )
        origin.purge_local(url)

If a purge fails on one tier, we retry with exponential backoff. If it fails permanently, we log and rely on the TTL as a backstop. Defense in depth is the only thing that keeps invalidation honest.

GDPR considerations

European compliance shaped the design more than I expected. Three rules ended up baked into every tier:

No personal data in any cache key or response body. Personalization is client-side.
Geolocation uses country-level granularity only (CF-IPCountry), never city or coordinate.
A user data deletion request fires a purge across all tiers for any URL containing their handle.

If you are building anything similar for a European audience, work the deletion path first. Most teams build caches they cannot purge by user, and that is a compliance landmine.

Numbers that matter

After moving to the three-tier model, we measured the change at ViralVidVault over a month:

Edge hit ratio: 88% to 94%
Origin requests per second under viral load: 1,200 to 180
p95 latency for cold European visitors: 480ms to 140ms
Egress bandwidth cost: down 62%

Closing notes

You do not need exotic infrastructure for a tiered video cache. We run the regional tier on three small VPS nodes. The hard parts are the promotion rules, the invalidation pipeline, and treating GDPR as a first-class constraint instead of a checkbox.

If I were starting over, I would build the invalidation worker before the cache tiers. Everything else is downstream of that.

Building a Multi-Region Health-Check Aggregator for Our Video CDN Fleet

ahmet gedik — Sun, 24 May 2026 18:37:58 +0000

It was 03:14 on a Saturday when our pager went off. Watch time across our Tokyo PoP had dropped 38% in 15 minutes, but every monitoring tool we had was green. Cloudflare's dashboard said "all good." UptimeRobot was pinging the apex domain successfully. Our origin LiteSpeed boxes were serving requests in under 80ms. And yet thousands of Japanese viewers were silently failing to load the HLS manifest for popular videos.

The problem turned out to be a single PoP in Osaka where Cloudflare was caching a stale master.m3u8 with a now-deleted variant URL. From any vantage point outside that specific edge, the file looked fine. From inside Japan on a JP residential ISP, it was broken. Our existing health checks — the kind that ping /healthz from a single US-east AWS region — could not see this. We were blind.

At TopVideoHub we aggregate trending video across nine regions in APAC, and "the manifest serves correctly in Tokyo" is just as important to us as "the database accepts writes." So we built a multi-region health-check aggregator that probes real CDN paths from real network locations, rolls the results up into a single SQLite store, and alerts on per-region anomalies rather than global outages. This post walks through what we measure, how the probes are structured, how the aggregator stores and exposes the data, and what surprised us when we put it in production.

What we actually need to measure

The first mistake we made was treating "is the site up?" as a single boolean. With a video site behind Cloudflare, there are at least four independent failure modes per region:

Origin reachability. Can LiteSpeed actually be reached from the edge? Usually yes, but a Cloudflare misroute can break this for a specific ASN combination.
HTML cache correctness. Is the cached /category/anime page returning the right vary set and a 200, not a stale 500 from an earlier deploy?
HLS manifest integrity. Does master.m3u8 parse, list variants, and do those variants resolve?
Search relevance. Is the Japanese/Korean/Chinese query returning results, or has the FTS5 tokenizer been silently broken by a schema migration?

A green ping doesn't help with any of those. We needed probes that exercise real product surface area, from real geographies, with assertions specific to each surface.

The metrics we ended up collecting per probe:

latency_ms — TTFB to first byte
total_ms — full body received
http_status
cf_cache_status from the cf-cache-status header
cf_pop from cf-ray (last three letters identify the PoP)
body_sha256 — content fingerprint so we can detect divergence between regions
assertion_failures — a JSON blob of which content checks failed

That last column is the one that made the system actually useful. A 200 OK with a 4 KB body is "healthy" by classical monitoring. A 200 OK with a 4 KB body when the page should be 180 KB is a silent disaster, and assertions catch it.

Architecture in one paragraph

Probes run as small Go binaries on cheap VPS instances we already keep around in nine regions for IP diversity. Each probe hits a list of URLs supplied by the central aggregator, runs HTTP requests with strict timeouts, performs content assertions, and POSTs a JSON batch back to the aggregator every 60 seconds. The aggregator is a PHP 8.4 endpoint that validates the payload, writes it to a SQLite database with two tables (probe_results, probe_alerts), and a small ingestion worker rolls hourly aggregates into a materialized view. The whole thing serves a /ops/healthboard page from LiteSpeed with the standard page cache disabled — we want fresh numbers.

No Kubernetes, no Prometheus, no Grafana. The total runtime cost is about $24/month including the probe VPS fleet.

The probe worker

Each probe is a single Go binary because we wanted accurate timing without GC-induced jitter masking real latency, and because we wanted to ship one static file via scp to whatever VPS we had handy. The structure is intentionally boring.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "regexp"
    "strings"
    "time"
)

type Target struct {
    URL         string   `json:"url"`
    Kind        string   `json:"kind"` // "page", "manifest", "search"
    MustContain []string `json:"must_contain"`
    MinBytes    int      `json:"min_bytes"`
}

type Result struct {
    Region    string   `json:"region"`
    URL       string   `json:"url"`
    Kind      string   `json:"kind"`
    Status    int      `json:"status"`
    LatencyMs int64    `json:"latency_ms"`
    TotalMs   int64    `json:"total_ms"`
    CFPop     string   `json:"cf_pop"`
    CFCache   string   `json:"cf_cache_status"`
    BodySHA   string   `json:"body_sha256"`
    BodyBytes int      `json:"body_bytes"`
    Failures  []string `json:"assertion_failures"`
    Timestamp int64    `json:"ts"`
}

var rayRe = regexp.MustCompile(`-([A-Z]{3})$`)

func probe(region string, t Target, client *http.Client) Result {
    r := Result{Region: region, URL: t.URL, Kind: t.Kind, Timestamp: time.Now().Unix()}
    start := time.Now()
    req, _ := http.NewRequest("GET", t.URL, nil)
    req.Header.Set("User-Agent", "TVH-Probe/1.0")
    req.Header.Set("Accept-Language", regionLang(region))

    resp, err := client.Do(req)
    if err != nil {
        r.Failures = append(r.Failures, "transport: "+err.Error())
        return r
    }
    defer resp.Body.Close()
    r.LatencyMs = time.Since(start).Milliseconds()

    body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
    if err != nil {
        r.Failures = append(r.Failures, "read: "+err.Error())
        return r
    }
    r.TotalMs = time.Since(start).Milliseconds()
    r.Status = resp.StatusCode
    r.BodyBytes = len(body)
    sum := sha256.Sum256(body)
    r.BodySHA = hex.EncodeToString(sum[:])
    r.CFCache = resp.Header.Get("cf-cache-status")

    if ray := resp.Header.Get("cf-ray"); ray != "" {
        if m := rayRe.FindStringSubmatch(ray); len(m) == 2 {
            r.CFPop = m[1]
        }
    }

    if r.BodyBytes < t.MinBytes {
        r.Failures = append(r.Failures, fmt.Sprintf("size %d<min %d", r.BodyBytes, t.MinBytes))
    }
    bodyStr := string(body)
    for _, needle := range t.MustContain {
        if !strings.Contains(bodyStr, needle) {
            r.Failures = append(r.Failures, "missing: "+needle)
        }
    }
    return r
}

func regionLang(region string) string {
    switch region {
    case "JP":
        return "ja,en;q=0.8"
    case "KR":
        return "ko,en;q=0.8"
    case "TW", "HK":
        return "zh-TW,zh;q=0.8,en;q=0.6"
    case "VN":
        return "vi,en;q=0.8"
    case "TH":
        return "th,en;q=0.8"
    }
    return "en"
}

func main() {
    region := os.Getenv("PROBE_REGION")
    if region == "" {
        fmt.Fprintln(os.Stderr, "PROBE_REGION required")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 15 * time.Second}

    resp, err := http.Get(os.Getenv("PROBE_TARGETS_URL"))
    if err != nil {
        fmt.Fprintln(os.Stderr, "fetch targets:", err)
        os.Exit(1)
    }
    var targets []Target
    json.NewDecoder(resp.Body).Decode(&targets)
    resp.Body.Close()

    var results []Result
    for _, t := range targets {
        results = append(results, probe(region, t, client))
    }

    payload, _ := json.Marshal(map[string]any{
        "region":  region,
        "results": results,
    })
    post, _ := http.NewRequest("POST", os.Getenv("PROBE_SINK"), strings.NewReader(string(payload)))
    post.Header.Set("Content-Type", "application/json")
    post.Header.Set("Authorization", "Bearer "+os.Getenv("PROBE_TOKEN"))
    client.Do(post)
}

Three things to call out about this probe:

The Accept-Language header changes per region. Cloudflare and our LiteSpeed origin both vary on it in subtle ways, and the Japanese homepage of an Asia-Pacific aggregator looks different from the English one. Probing without the right language is testing a code path no real user takes.
We read the body with io.LimitReader capped at 8 MiB. If something on the origin starts streaming a 2 GB file because of a bad migration, we don't want every probe VPS to OOM.
The cf-ray header's trailing three characters identify the Cloudflare PoP. Capturing this lets us answer "which edge are we actually hitting from this region?" — the answer is sometimes surprising (a probe in Bangkok served by SIN, etc.).

This binary is driven by cron every 60 seconds on each VPS. We deliberately do not run it as a long-lived daemon; one-shot cron means a crash on Tuesday doesn't go unnoticed until Friday.

The aggregator

The receiver is plain PHP 8.4. We already run LiteSpeed in front of our PHP for the main site, so adding one more endpoint costs us nothing. The whole receiver is under 150 lines.

<?php
declare(strict_types=1);

final class ProbeAggregator
{
    public function __construct(private \PDO $pdo) {}

    public function ingest(string $rawBody, string $authHeader): array
    {
        if (!hash_equals('Bearer ' . ($_ENV['PROBE_TOKEN'] ?? ''), $authHeader)) {
            http_response_code(401);
            return ['error' => 'unauthorized'];
        }

        $payload = json_decode($rawBody, true, 8, JSON_THROW_ON_ERROR);
        if (!is_array($payload['results'] ?? null)) {
            http_response_code(400);
            return ['error' => 'no results'];
        }

        $region = (string)($payload['region'] ?? 'XX');
        $stmt = $this->pdo->prepare(<<<SQL
            INSERT INTO probe_results
              (region, url, kind, status, latency_ms, total_ms,
               cf_pop, cf_cache, body_sha, body_bytes, failures, ts)
            VALUES (:region, :url, :kind, :status, :latency_ms, :total_ms,
                    :cf_pop, :cf_cache, :body_sha, :body_bytes, :failures, :ts)
        SQL);

        $this->pdo->beginTransaction();
        $inserted = 0;
        foreach ($payload['results'] as $r) {
            $stmt->execute([
                ':region'     => $region,
                ':url'        => (string)$r['url'],
                ':kind'       => (string)$r['kind'],
                ':status'     => (int)$r['status'],
                ':latency_ms' => (int)$r['latency_ms'],
                ':total_ms'   => (int)$r['total_ms'],
                ':cf_pop'     => (string)($r['cf_pop'] ?? ''),
                ':cf_cache'   => (string)($r['cf_cache_status'] ?? ''),
                ':body_sha'   => (string)($r['body_sha256'] ?? ''),
                ':body_bytes' => (int)$r['body_bytes'],
                ':failures'   => json_encode($r['assertion_failures'] ?? []),
                ':ts'         => (int)$r['ts'],
            ]);
            $inserted++;
        }
        $this->pdo->commit();

        $this->detectAnomalies($region);

        return ['ok' => true, 'inserted' => $inserted];
    }

    private function detectAnomalies(string $region): void
    {
        $stmt = $this->pdo->prepare(<<<SQL
            SELECT url,
                   AVG(CASE WHEN status >= 200 AND status < 400 THEN 1 ELSE 0 END) AS ok_rate,
                   COUNT(*) AS samples
            FROM probe_results
            WHERE region = :region
              AND ts > strftime('%s','now') - 600
            GROUP BY url
            HAVING samples >= 5 AND ok_rate < 0.6
        SQL);
        $stmt->execute([':region' => $region]);

        $ins = $this->pdo->prepare(<<<SQL
            INSERT INTO probe_alerts (region, url, ok_rate, ts)
            VALUES (:region, :url, :ok_rate, :ts)
            ON CONFLICT(region, url) DO UPDATE SET
              ok_rate = excluded.ok_rate, ts = excluded.ts
        SQL);
        foreach ($stmt as $row) {
            $ins->execute([
                ':region'  => $region,
                ':url'     => $row['url'],
                ':ok_rate' => (float)$row['ok_rate'],
                ':ts'      => time(),
            ]);
        }
    }
}

The anomaly detector is intentionally dumb: any URL with five or more samples in the last ten minutes whose success rate drops below 60% gets upserted into probe_alerts. A separate cron reads that table and posts to a Discord webhook. Smarter detection (EWMA, change-point detection) is on the backlog, but the dumb threshold has caught every real outage we've had so far. Premature sophistication in alerting almost always means false positives at 4 AM, and false positives are how teams learn to mute pages.

SQLite schema and queries

We use SQLite for the same reason we use it everywhere else on the site: it's one file, it backs up via rsync, and it scales further than people expect. At ~7.5 million probe rows per month across nine regions, the database file is about 480 MB. Query latency for the dashboard is single-digit milliseconds.

PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA temp_store = MEMORY;
PRAGMA mmap_size = 268435456;

CREATE TABLE probe_results (
    id          INTEGER PRIMARY KEY,
    region      TEXT NOT NULL,
    url         TEXT NOT NULL,
    kind        TEXT NOT NULL,
    status      INTEGER NOT NULL,
    latency_ms  INTEGER NOT NULL,
    total_ms    INTEGER NOT NULL,
    cf_pop      TEXT,
    cf_cache    TEXT,
    body_sha    TEXT,
    body_bytes  INTEGER,
    failures    TEXT,
    ts          INTEGER NOT NULL
) STRICT;

CREATE INDEX idx_probe_region_ts ON probe_results(region, ts DESC);
CREATE INDEX idx_probe_url_ts    ON probe_results(url, ts DESC);

CREATE TABLE probe_alerts (
    region   TEXT NOT NULL,
    url      TEXT NOT NULL,
    ok_rate  REAL NOT NULL,
    ts       INTEGER NOT NULL,
    PRIMARY KEY (region, url)
) STRICT;

CREATE TABLE probe_hourly (
    region        TEXT NOT NULL,
    url           TEXT NOT NULL,
    hour_ts       INTEGER NOT NULL,
    samples       INTEGER NOT NULL,
    ok_count      INTEGER NOT NULL,
    p50_latency   INTEGER NOT NULL,
    p95_latency   INTEGER NOT NULL,
    PRIMARY KEY (region, url, hour_ts)
) STRICT;

The interesting query is the one that powers the dashboard. We want, per region and per URL, the rolling 5-minute success rate plus a flag for "divergence" — the case where this region sees a different body_sha than the global mode.

WITH recent AS (
    SELECT region, url, status, total_ms, body_sha
    FROM probe_results
    WHERE ts > strftime('%s','now') - 300
),
sha_counts AS (
    SELECT url, body_sha, COUNT(*) AS n
    FROM recent
    GROUP BY url, body_sha
),
majority_sha AS (
    SELECT url, body_sha
    FROM sha_counts s
    WHERE n = (SELECT MAX(n) FROM sha_counts s2 WHERE s2.url = s.url)
)
SELECT r.region,
       r.url,
       COUNT(*) AS samples,
       AVG(CASE WHEN r.status BETWEEN 200 AND 399 THEN 1.0 ELSE 0.0 END) AS ok_rate,
       SUM(CASE WHEN r.body_sha != m.body_sha THEN 1 ELSE 0 END) AS divergent
FROM recent r
JOIN majority_sha m USING (url)
GROUP BY r.region, r.url
ORDER BY r.region, r.url;

The body_sha divergence column is what would have caught the Osaka incident. When eight regions see SHA abc... for the manifest and JP sees SHA def..., that's not a flap, that's a cache poisoning event at one edge. We render the dashboard cell red regardless of HTTP status.

For the FTS5 search probe specifically — testing that "アニメ" still returns results from our CJK-tokenized index — we needed an assertion stronger than HTTP 200. The probe target manifest is generated by a small Python script that lives next to our content vault:

import json
from pathlib import Path

REGIONS = ["US", "GB", "JP", "KR", "TW", "SG", "VN", "TH", "HK"]

SEARCH_QUERIES = {
    "JP": ["アニメ", "音楽", "ゲーム実況"],
    "KR": ["케이팝", "먹방", "드라마"],
    "TW": ["遊戲", "美食", "音樂"],
    "HK": ["遊戲", "美食", "音樂"],
    "VN": ["bóng đá", "âm nhạc", "phim"],
    "TH": ["เพลง", "เกม", "ละคร"],
}

BASE = "https://topvideohub.com"

def build_targets(region: str) -> list[dict]:
    targets = [
        {
            "url": f"{BASE}/",
            "kind": "page",
            "must_contain": ["TopVideoHub", '<main id="content"'],
            "min_bytes": 40_000,
        },
        {
            "url": f"{BASE}/category/music",
            "kind": "page",
            "must_contain": ["category-music", "video-card"],
            "min_bytes": 30_000,
        },
    ]
    for q in SEARCH_QUERIES.get(region, ["music"]):
        targets.append({
            "url": f"{BASE}/search?q={q}",
            "kind": "search",
            "must_contain": ["search-results", q],
            "min_bytes": 8_000,
        })
    return targets

if __name__ == "__main__":
    out = Path("targets")
    out.mkdir(exist_ok=True)
    for r in REGIONS:
        (out / f"{r}.json").write_text(
            json.dumps(build_targets(r), ensure_ascii=False, indent=2),
            encoding="utf-8",
        )
    print(f"wrote {len(REGIONS)} target files")

Putting the query string itself into must_contain is a deliberate test of two things at once: that FTS5 returned something (we render the query back into the results page), and that the response wasn't a cached page for a different query. URL-keyed cache poisoning is real, particularly after a deploy that changes how query parameters affect cache keys.

Surfacing it through LiteSpeed and Cloudflare

The aggregator endpoint at /ops/probe-ingest and the dashboard at /ops/healthboard both live on the main LiteSpeed origin. We did three small but important things to keep them from breaking the main site:

CacheDisable yes for any URL starting with /ops/. The whole point of a health dashboard is fresh data, and the LiteSpeed page cache would happily serve a 2-minute-old snapshot otherwise.
A Cloudflare page rule that disables Cloudflare's CDN cache for /ops/* and forces the WAF into "high" mode. The ingest endpoint is rate-limited at the WAF layer to 30 requests/minute per probe ASN, which more than covers our once-a-minute cron with headroom.
Basic auth on /ops/healthboard enforced by the PHP app rather than Cloudflare Access. We want the page to load even if Cloudflare itself is the thing that broke — which has happened.

That last point bit us early. The first version of the dashboard required Cloudflare Access to view, and on the day Cloudflare's APAC dashboard had its own incident, we couldn't see our own monitoring because the SSO challenge couldn't load. Now the dashboard is reachable on a long random path with HTTP basic auth, and works even with Cloudflare in Under Attack Mode.

What broke first

The list of things that surprised us in the first month, roughly in order:

The JP probe VPS was on the same backbone as our Tokyo Cloudflare PoP. Latency from there looked great, success rate was 100% even when real users were complaining. We moved it to a different provider on a different transit and immediately started seeing the real picture. The lesson generalizes: a probe colocated with the thing it probes is barely a probe at all.
Our must_contain assertions tripped on cookie banner A/B tests. Whenever marketing flipped the consent banner copy, a third of probes started "failing." We moved to asserting on structural HTML (<main id="content">) instead of marketing copy. Anything a non-engineer can change without telling you should not be an assertion target.
The Korean probe started returning a translated page because our content negotiation hit a code path that, for Accept-Language: ko, applied an experimental Korean UI. The probe was working perfectly; the assertion was wrong. We added per-region must_contain lists.
SQLite VACUUM during ingest caused a 12-second stall. We moved VACUUM to a weekly cron at 03:00 UTC and added PRAGMA wal_autocheckpoint=1000 so checkpoint pauses stay small.
Discord webhook 429s during a real incident meant we got fewer alerts during the worst moment. We now batch alerts every 30 seconds and include a "this is alert #N for this URL in the last hour" counter.

None of these are exotic. They're the ordinary lessons of running synthetic monitoring against a real system. The point of the exercise is to surface them in your own architecture, not to copy ours.

Conclusion

The version of this system we run today does about 1.3 million probes a month across nine regions, has caught three Cloudflare cache anomalies that ordinary uptime monitoring missed, and costs less than a single APM seat at any commercial vendor. The trick was not the technology — Go binary, PHP receiver, SQLite store, LiteSpeed in front — but the discipline of writing assertions about what real users see, from places real users live, in the languages they actually speak. A monitoring system that only knows English HTTP 200s is not monitoring an Asia-Pacific video site; it's monitoring a US health-check endpoint. Build for the surface area you actually serve, store the raw data so future-you can ask new questions, and prefer dumb thresholds you trust to clever detection you don't.

Monitoring and Alerting for Video Platform Infrastructure with Go

ahmet gedik — Mon, 11 May 2026 22:00:01 +0000

Why Custom Monitoring

Off-the-shelf monitoring tools are powerful but heavy. For a video platform like DailyWatch running on shared LiteSpeed hosting, we needed a lightweight Go binary that checks endpoint health, tracks uptime, and exposes Prometheus metrics -- without installing a full monitoring stack.

Health Check Configuration

Define what to monitor:

package monitor

import "time"

type CheckConfig struct {
    Name        string        `json:"name"`
    URL         string        `json:"url"`
    Method      string        `json:"method"`
    Interval    time.Duration `json:"interval"`
    Timeout     time.Duration `json:"timeout"`
    ExpectCode  int           `json:"expect_code"`
    ExpectBody  string        `json:"expect_body"`
}

type CheckResult struct {
    Name       string        `json:"name"`
    URL        string        `json:"url"`
    Status     string        `json:"status"` // "up" or "down"
    StatusCode int           `json:"status_code"`
    Latency    time.Duration `json:"latency"`
    Error      string        `json:"error,omitempty"`
    CheckedAt  time.Time     `json:"checked_at"`
}

var DefaultChecks = []CheckConfig{
    {
        Name:       "homepage",
        URL:        "https://dailywatch.video",
        Method:     "GET",
        Interval:   5 * time.Minute,
        Timeout:    10 * time.Second,
        ExpectCode: 200,
    },
    {
        Name:       "search_api",
        URL:        "https://dailywatch.video/search?q=test&format=json",
        Method:     "GET",
        Interval:   10 * time.Minute,
        Timeout:    15 * time.Second,
        ExpectCode: 200,
        ExpectBody: "results",
    },
    {
        Name:       "sitemap",
        URL:        "https://dailywatch.video/sitemap.xml",
        Method:     "GET",
        Interval:   30 * time.Minute,
        Timeout:    10 * time.Second,
        ExpectCode: 200,
        ExpectBody: "<urlset",
    },
}

The Health Checker

Perform checks concurrently:

package monitor

import (
    "fmt"
    "io"
    "net/http"
    "strings"
    "sync"
    "time"
)

type HealthChecker struct {
    client  *http.Client
    results map[string]*CheckResult
    history map[string][]CheckResult
    mu      sync.RWMutex
}

func NewHealthChecker() *HealthChecker {
    return &HealthChecker{
        client:  &http.Client{},
        results: make(map[string]*CheckResult),
        history: make(map[string][]CheckResult),
    }
}

func (hc *HealthChecker) RunCheck(cfg CheckConfig) CheckResult {
    hc.client.Timeout = cfg.Timeout
    start := time.Now()

    req, err := http.NewRequest(cfg.Method, cfg.URL, nil)
    if err != nil {
        return hc.recordResult(cfg.Name, cfg.URL, CheckResult{
            Name: cfg.Name, URL: cfg.URL, Status: "down",
            Error: fmt.Sprintf("request creation failed: %v", err),
            CheckedAt: time.Now(),
        })
    }
    req.Header.Set("User-Agent", "DailyWatch-Monitor/1.0")

    resp, err := hc.client.Do(req)
    latency := time.Since(start)

    if err != nil {
        return hc.recordResult(cfg.Name, cfg.URL, CheckResult{
            Name: cfg.Name, URL: cfg.URL, Status: "down",
            Latency: latency, Error: err.Error(), CheckedAt: time.Now(),
        })
    }
    defer resp.Body.Close()

    result := CheckResult{
        Name:       cfg.Name,
        URL:        cfg.URL,
        StatusCode: resp.StatusCode,
        Latency:    latency,
        CheckedAt:  time.Now(),
    }

    // Check status code
    if cfg.ExpectCode > 0 && resp.StatusCode != cfg.ExpectCode {
        result.Status = "down"
        result.Error = fmt.Sprintf("expected %d, got %d", cfg.ExpectCode, resp.StatusCode)
        return hc.recordResult(cfg.Name, cfg.URL, result)
    }

    // Check body content
    if cfg.ExpectBody != "" {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) // 1MB max
        if !strings.Contains(string(body), cfg.ExpectBody) {
            result.Status = "down"
            result.Error = fmt.Sprintf("body missing expected string: %s", cfg.ExpectBody)
            return hc.recordResult(cfg.Name, cfg.URL, result)
        }
    }

    result.Status = "up"
    return hc.recordResult(cfg.Name, cfg.URL, result)
}

func (hc *HealthChecker) recordResult(name, url string, result CheckResult) CheckResult {
    hc.mu.Lock()
    defer hc.mu.Unlock()
    hc.results[name] = &result
    hc.history[name] = append(hc.history[name], result)
    // Keep last 100 results
    if len(hc.history[name]) > 100 {
        hc.history[name] = hc.history[name][len(hc.history[name])-100:]
    }
    return result
}

func (hc *HealthChecker) GetUptime(name string, window time.Duration) float64 {
    hc.mu.RLock()
    defer hc.mu.RUnlock()
    cutoff := time.Now().Add(-window)
    total, up := 0, 0
    for _, r := range hc.history[name] {
        if r.CheckedAt.After(cutoff) {
            total++
            if r.Status == "up" {
                up++
            }
        }
    }
    if total == 0 {
        return 0
    }
    return float64(up) / float64(total) * 100
}

Prometheus Metrics Endpoint

Expose metrics in Prometheus format for scraping:

package main

import (
    "fmt"
    "net/http"
    "time"
    "myapp/monitor"
)

func metricsHandler(checker *monitor.HealthChecker, checks []monitor.CheckConfig) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain; version=0.0.4")

        for _, cfg := range checks {
            result := checker.RunCheck(cfg)
            upVal := 0
            if result.Status == "up" {
                upVal = 1
            }
            uptime24h := checker.GetUptime(cfg.Name, 24*time.Hour)

            fmt.Fprintf(w, "# HELP video_platform_up Whether the endpoint is up (1) or down (0)\n")
            fmt.Fprintf(w, "video_platform_up{name=%q,url=%q} %d\n", cfg.Name, cfg.URL, upVal)
            fmt.Fprintf(w, "video_platform_latency_ms{name=%q} %.1f\n", cfg.Name, float64(result.Latency.Milliseconds()))
            fmt.Fprintf(w, "video_platform_uptime_24h{name=%q} %.2f\n", cfg.Name, uptime24h)
        }
    }
}

func main() {
    checker := monitor.NewHealthChecker()
    checks := monitor.DefaultChecks

    // Run checks on schedule
    go func() {
        for {
            for _, cfg := range checks {
                result := checker.RunCheck(cfg)
                if result.Status == "down" {
                    fmt.Printf("ALERT: %s is DOWN - %s\n", result.Name, result.Error)
                }
            }
            time.Sleep(5 * time.Minute)
        }
    }()

    http.HandleFunc("/metrics", metricsHandler(checker, checks))
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("OK"))
    })

    fmt.Println("Monitor running on :9090")
    http.ListenAndServe(":9090", nil)
}

Scrape with Prometheus

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'dailywatch'
    scrape_interval: 5m
    static_configs:
      - targets: ['localhost:9090']

This monitoring setup keeps us informed about the health of DailyWatch without any external dependencies. The single Go binary uses under 10MB of RAM and catches downtime within minutes. When an endpoint goes down, we see it in Grafana dashboards and can set up Alertmanager rules for Slack or email notifications.

This article is part of the Building DailyWatch series. Check out DailyWatch to see these techniques in action.

Monitoring and Alerting for Video Platform Infrastructure with Go

ahmet gedik — Mon, 11 May 2026 14:00:01 +0000

Monitoring a Multi-Region Video Vault

ViralVidVault serves viral video content from 7 European regions. When the Polish trending page goes down or the Swedish cron job fails, we need to know immediately -- not when a user complains. A custom Go monitor gives us exactly the health visibility we need.

Configuration

package monitor

import "time"

type Target struct {
    Name       string        `json:"name"`
    URL        string        `json:"url"`
    Interval   time.Duration `json:"interval"`
    Timeout    time.Duration `json:"timeout"`
    ExpectCode int           `json:"expect_code"`
    ExpectBody string        `json:"expect_body"`
    Region     string        `json:"region"`
}

type Result struct {
    Name       string        `json:"name"`
    Region     string        `json:"region"`
    Status     string        `json:"status"`
    StatusCode int           `json:"status_code"`
    Latency    time.Duration `json:"latency"`
    Error      string        `json:"error,omitempty"`
    Timestamp  time.Time     `json:"timestamp"`
}

var Targets = []Target{
    {Name: "homepage", URL: "https://viralvidvault.com", Interval: 5 * time.Minute, Timeout: 10 * time.Second, ExpectCode: 200},
    {Name: "viral_pl", URL: "https://viralvidvault.com/?region=PL", Interval: 10 * time.Minute, Timeout: 10 * time.Second, ExpectCode: 200, Region: "PL"},
    {Name: "viral_nl", URL: "https://viralvidvault.com/?region=NL", Interval: 10 * time.Minute, Timeout: 10 * time.Second, ExpectCode: 200, Region: "NL"},
    {Name: "viral_se", URL: "https://viralvidvault.com/?region=SE", Interval: 10 * time.Minute, Timeout: 10 * time.Second, ExpectCode: 200, Region: "SE"},
    {Name: "search_api", URL: "https://viralvidvault.com/search?q=viral&format=json", Interval: 15 * time.Minute, Timeout: 15 * time.Second, ExpectCode: 200, ExpectBody: "results"},
    {Name: "sitemap", URL: "https://viralvidvault.com/sitemap.xml", Interval: 30 * time.Minute, Timeout: 10 * time.Second, ExpectCode: 200, ExpectBody: "<urlset"},
}

The Checker Engine

package monitor

import (
    "fmt"
    "io"
    "net/http"
    "strings"
    "sync"
    "time"
)

type Checker struct {
    client  *http.Client
    latest  map[string]*Result
    history map[string][]Result
    mu      sync.RWMutex
    alerts  chan Result
}

func NewChecker() *Checker {
    return &Checker{
        client:  &http.Client{},
        latest:  make(map[string]*Result),
        history: make(map[string][]Result),
        alerts:  make(chan Result, 100),
    }
}

func (c *Checker) Check(t Target) Result {
    c.client.Timeout = t.Timeout
    start := time.Now()

    req, _ := http.NewRequest("GET", t.URL, nil)
    req.Header.Set("User-Agent", "ViralVidVault-Monitor/1.0")

    resp, err := c.client.Do(req)
    latency := time.Since(start)

    result := Result{
        Name:      t.Name,
        Region:    t.Region,
        Latency:   latency,
        Timestamp: time.Now(),
    }

    if err != nil {
        result.Status = "down"
        result.Error = err.Error()
        c.record(t.Name, result)
        return result
    }
    defer resp.Body.Close()

    result.StatusCode = resp.StatusCode

    if t.ExpectCode > 0 && resp.StatusCode != t.ExpectCode {
        result.Status = "down"
        result.Error = fmt.Sprintf("got %d, expected %d", resp.StatusCode, t.ExpectCode)
        c.record(t.Name, result)
        return result
    }

    if t.ExpectBody != "" {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 512*1024))
        if !strings.Contains(string(body), t.ExpectBody) {
            result.Status = "down"
            result.Error = "missing expected body content"
            c.record(t.Name, result)
            return result
        }
    }

    result.Status = "up"
    c.record(t.Name, result)
    return result
}

func (c *Checker) record(name string, r Result) {
    c.mu.Lock()
    defer c.mu.Unlock()

    // Detect state transitions for alerting
    prev := c.latest[name]
    if prev != nil && prev.Status == "up" && r.Status == "down" {
        c.alerts <- r
    }

    c.latest[name] = &r
    c.history[name] = append(c.history[name], r)
    if len(c.history[name]) > 200 {
        c.history[name] = c.history[name][100:]
    }
}

func (c *Checker) Uptime(name string, window time.Duration) float64 {
    c.mu.RLock()
    defer c.mu.RUnlock()
    cutoff := time.Now().Add(-window)
    var total, up int
    for _, r := range c.history[name] {
        if r.Timestamp.After(cutoff) {
            total++
            if r.Status == "up" {
                up++
            }
        }
    }
    if total == 0 {
        return 100.0
    }
    return float64(up) / float64(total) * 100
}

Prometheus Metrics and Alerting

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "time"
    "myapp/monitor"
)

func main() {
    checker := monitor.NewChecker()

    // Background check loop
    go func() {
        for {
            for _, t := range monitor.Targets {
                r := checker.Check(t)
                if r.Status == "down" {
                    log.Printf("DOWN: %s (%s) - %s", r.Name, r.Region, r.Error)
                }
            }
            time.Sleep(5 * time.Minute)
        }
    }()

    // Alert consumer
    go func() {
        for alert := range checker.Alerts() {
            log.Printf("ALERT: %s went DOWN at %s - %s",
                alert.Name, alert.Timestamp.Format(time.RFC3339), alert.Error)
            // Send to Slack, email, etc.
        }
    }()

    // Prometheus metrics
    http.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain; version=0.0.4")
        for _, t := range monitor.Targets {
            result := checker.Check(t)
            up := 0
            if result.Status == "up" { up = 1 }
            uptime := checker.Uptime(t.Name, 24*time.Hour)
            fmt.Fprintf(w, "vvv_endpoint_up{name=%q,region=%q} %d
", t.Name, t.Region, up)
            fmt.Fprintf(w, "vvv_endpoint_latency_ms{name=%q} %.0f
", t.Name, float64(result.Latency.Milliseconds()))
            fmt.Fprintf(w, "vvv_endpoint_uptime_24h{name=%q} %.2f
", t.Name, uptime)
        }
    })

    // Status JSON endpoint
    http.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        status := map[string]interface{}{}
        for _, t := range monitor.Targets {
            status[t.Name] = map[string]interface{}{
                "status":  checker.Latest(t.Name).Status,
                "latency": checker.Latest(t.Name).Latency.String(),
                "uptime":  fmt.Sprintf("%.1f%%", checker.Uptime(t.Name, 24*time.Hour)),
            }
        }
        json.NewEncoder(w).Encode(status)
    })

    log.Println("Monitor on :9090")
    log.Fatal(http.ListenAndServe(":9090", nil))
}

This monitor runs as a single Go binary using under 15MB RAM, checking all ViralVidVault endpoints every 5 minutes. The state transition alerting means we only get notified when something changes from up to down -- no alert fatigue from repeated "still down" messages.

This article is part of the Building ViralVidVault series. Check out ViralVidVault to see these techniques in action.

Building a REST API Client Library in Python for Video Data

ahmet gedik — Mon, 11 May 2026 06:00:01 +0000

Why Build a Client Library

When your video platform talks to the YouTube Data API from multiple scripts -- the cron fetcher, the metadata enricher, the broken link checker -- you end up duplicating HTTP logic everywhere. At DailyWatch, we consolidated all YouTube API interactions into a single Python SDK with retry logic, rate limiting, and caching built in.

The Core Client

import time
import hashlib
import json
import requests
from typing import Optional
from dataclasses import dataclass, field
from functools import lru_cache

@dataclass
class APIConfig:
    api_key: str
    base_url: str = "https://www.googleapis.com/youtube/v3"
    max_retries: int = 3
    base_delay: float = 1.0
    timeout: int = 15
    daily_quota: int = 10000
    cache_ttl: int = 3600  # 1 hour

class QuotaExhaustedError(Exception):
    pass

class YouTubeClient:
    """Reusable YouTube Data API client with retry, rate limiting, and caching."""

    def __init__(self, config: APIConfig):
        self.config = config
        self.session = requests.Session()
        self.session.headers.update({"Accept": "application/json"})
        self._quota_used = 0
        self._cache: dict[str, tuple[float, dict]] = {}

    def _cache_key(self, endpoint: str, params: dict) -> str:
        raw = f"{endpoint}:{json.dumps(params, sort_keys=True)}"
        return hashlib.md5(raw.encode()).hexdigest()

    def _get_cached(self, key: str) -> Optional[dict]:
        if key in self._cache:
            timestamp, data = self._cache[key]
            if time.time() - timestamp < self.config.cache_ttl:
                return data
            del self._cache[key]
        return None

    def _request(self, endpoint: str, params: dict, quota_cost: int = 1) -> dict:
        """Make an API request with retry, backoff, caching, and quota tracking."""
        # Check quota
        if self._quota_used + quota_cost > self.config.daily_quota:
            raise QuotaExhaustedError(
                f"Would exceed daily quota ({self._quota_used}/{self.config.daily_quota})"
            )

        # Check cache
        cache_key = self._cache_key(endpoint, params)
        cached = self._get_cached(cache_key)
        if cached is not None:
            return cached

        url = f"{self.config.base_url}/{endpoint}"
        params["key"] = self.config.api_key
        last_exception = None

        for attempt in range(self.config.max_retries):
            try:
                response = self.session.get(url, params=params, timeout=self.config.timeout)

                if response.status_code == 200:
                    data = response.json()
                    self._quota_used += quota_cost
                    self._cache[cache_key] = (time.time(), data)
                    return data

                if response.status_code == 403:
                    error_reason = response.json().get("error", {}).get("errors", [{}])[0].get("reason", "")
                    if error_reason == "quotaExceeded":
                        raise QuotaExhaustedError("YouTube API quota exceeded")

                if response.status_code in (429, 500, 503):
                    delay = self.config.base_delay * (2 ** attempt)
                    time.sleep(delay)
                    continue

                response.raise_for_status()

            except requests.ConnectionError as e:
                last_exception = e
                delay = self.config.base_delay * (2 ** attempt)
                time.sleep(delay)

        raise last_exception or RuntimeError(f"Failed after {self.config.max_retries} retries")

    @property
    def quota_remaining(self) -> int:
        return self.config.daily_quota - self._quota_used

High-Level Methods

Build domain-specific methods on top of the core:

    def get_trending(self, region: str = "US", max_results: int = 25, category_id: str = "") -> list[dict]:
        """Fetch trending videos for a region."""
        params = {
            "part": "snippet,statistics,contentDetails",
            "chart": "mostPopular",
            "regionCode": region,
            "maxResults": min(max_results, 50),
        }
        if category_id:
            params["videoCategoryId"] = category_id

        data = self._request("videos", params)
        return data.get("items", [])

    def get_video_details(self, video_ids: list[str]) -> list[dict]:
        """Batch fetch video details (up to 50 per call)."""
        results = []
        for i in range(0, len(video_ids), 50):
            batch = video_ids[i:i + 50]
            params = {
                "part": "snippet,statistics,contentDetails",
                "id": ",".join(batch),
            }
            data = self._request("videos", params)
            results.extend(data.get("items", []))
        return results

    def search_videos(self, query: str, region: str = "US", max_results: int = 10) -> list[dict]:
        """Search for videos. Costs 100 quota units per call."""
        params = {
            "part": "snippet",
            "q": query,
            "type": "video",
            "regionCode": region,
            "maxResults": min(max_results, 50),
        }
        data = self._request("search", params, quota_cost=100)
        return data.get("items", [])

    def get_categories(self, region: str = "US") -> dict[str, str]:
        """Get video category ID to name mapping."""
        params = {
            "part": "snippet",
            "regionCode": region,
        }
        data = self._request("videoCategories", params)
        return {
            item["id"]: item["snippet"]["title"]
            for item in data.get("items", [])
        }

Usage in Practice

# Initialize once, use everywhere
config = APIConfig(api_key="AIzaSy...")
client = YouTubeClient(config)

# Fetch trending from multiple regions
for region in ["US", "GB", "DE", "FR", "IN", "BR", "AU", "CA"]:
    videos = client.get_trending(region=region, max_results=25)
    print(f"[{region}] {len(videos)} trending videos (quota left: {client.quota_remaining})")

# Batch details (uses cache if recently fetched)
details = client.get_video_details(["dQw4w9WgXcQ", "jNQXAC9IVRw"])

# Search (expensive -- 100 quota units)
results = client.search_videos("python tutorial", region="US")

Key Design Decisions

Exponential backoff: Each retry doubles the wait time, preventing thundering herd
In-memory cache: Avoids redundant API calls within the same run
Quota tracking: Proactively prevents quota exhaustion instead of discovering it mid-pipeline
Batch-aware: The get_video_details method automatically chunks large ID lists

This client powers every API interaction at DailyWatch. Having retry logic and quota awareness in one place eliminated an entire class of intermittent failures from our pipeline.

This article is part of the Building DailyWatch series. Check out DailyWatch to see these techniques in action.

Monitoring and Alerting for Video Platform Infrastructure with Go

ahmet gedik — Sun, 10 May 2026 23:00:01 +0000

Monitoring Across Time Zones

TrendVidStream serves UAE (+4 UTC), Finland (+2), Czech Republic (+1), Denmark (+1), Belgium (+1), UK (+0), Switzerland (+1), and US (-5 to -8). A cron failure in Finland at 3am local is 1am UTC — monitoring must alert before users notice stale content.

Health Check Endpoint

package main

import (
    "database/sql"
    "encoding/json"
    "net/http"
    "time"
)

type HealthResponse struct {
    Overall string            `json:"overall"`
    Checks  map[string]string `json:"checks"`
    Uptime  string            `json:"uptime"`
    Regions []string          `json:"regions_active"`
}

var (
    startTime     = time.Now()
    activeRegions = []string{"AE", "FI", "CZ", "DK", "BE", "CH", "GB", "US"}
)

func healthHandler(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := map[string]string{}
        overall := "ok"

        if err := db.Ping(); err != nil {
            checks["database"] = "error: " + err.Error()
            overall = "degraded"
        } else {
            checks["database"] = "ok"
        }

        for _, region := range activeRegions {
            var count int
            db.QueryRow(
                "SELECT COUNT(*) FROM videos WHERE region=? AND fetched_at > datetime('now','-8 hours')",
                region,
            ).Scan(&count)
            if count == 0 {
                checks["region_"+region] = "stale"
                overall = "degraded"
            } else {
                checks["region_"+region] = "ok"
            }
        }

        w.Header().Set("Content-Type", "application/json")
        if overall != "ok" {
            w.WriteHeader(http.StatusServiceUnavailable)
        }
        json.NewEncoder(w).Encode(HealthResponse{
            Overall: overall, Checks: checks,
            Uptime: time.Since(startTime).Round(time.Second).String(),
            Regions: activeRegions,
        })
    }
}

Prometheus Exporter

import "github.com/prometheus/client_golang/prometheus"

var (
    fetchedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "tvs_videos_fetched_total",
        Help: "Total videos fetched per region.",
    }, []string{"region"})

    fetchDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name: "tvs_fetch_duration_seconds", Help: "Fetch duration per region.",
        Buckets: prometheus.DefBuckets,
    }, []string{"region"})

    regionFreshness = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "tvs_region_freshness_seconds",
        Help: "Seconds since last successful fetch per region.",
    }, []string{"region"})
)

func init() { prometheus.MustRegister(fetchedTotal, fetchDuration, regionFreshness) }

Uptime Monitor with Per-Timezone Context

var probes = []struct {
    URL    string
    Region string
    UTCOff int
}{
    {"https://trendvidstream.com/?region=AE", "AE", +4},
    {"https://trendvidstream.com/?region=FI", "FI", +2},
    {"https://trendvidstream.com/?region=CZ", "CZ", +1},
    {"https://trendvidstream.com/?region=DK", "DK", +1},
    {"https://trendvidstream.com/?region=GB", "GB",  0},
}

func (m *UptimeMonitor) Start(interval time.Duration) {
    for _, p := range probes {
        go func(probe struct{ URL, Region string; UTCOff int }) {
            ticker := time.NewTicker(interval)
            for range ticker.C {
                start := time.Now()
                resp, err := m.client.Get(probe.URL)
                latency := time.Since(start)

                status := 0
                if resp != nil { status = resp.StatusCode; resp.Body.Close() }

                if err != nil || status >= 500 {
                    localHour := (time.Now().UTC().Hour() + probe.UTCOff + 24) % 24
                    msg := fmt.Sprintf(
                        ":red_circle: *%s DOWN* (HTTP %d, %.0fms) — local %02d:xx",
                        probe.Region, status, latency.Seconds()*1000, localHour,
                    )
                    sendSlack(m.slack, msg)
                }
            }
        }(p)
    }
}

Including local time in alerts helps on-call engineers understand whether the affected region is in peak hours — a UAE failure at 21:00 local (prime time) is more urgent than a Finnish failure at 04:00.

Grafana Dashboard Queries

Panel	PromQL
Videos fetched rate	`rate(tvs_videos_fetched_total[5m])` per region
P95 fetch latency	`histogram_quantile(0.95, rate(tvs_fetch_duration_seconds_bucket[5m]))`
Region freshness	`tvs_region_freshness_seconds` — alert if > 28800 (8h)
Quota remaining	`10000 - sum(tvs_api_quota_used)`

The freshness alert is the most valuable metric. When a region goes stale due to API quota exhaustion, Grafana pages on-call before any user reports missing content on TrendVidStream.

This article is part of the Building TrendVidStream series. Check out TrendVidStream to see these techniques in action.

Building a REST API Client Library in Python for Video Data

ahmet gedik — Sun, 10 May 2026 11:00:01 +0000

Centralizing API Logic

When you fetch viral videos from 7 European regions, you end up calling the YouTube Data API from multiple scripts: the main fetcher, the virality scorer, the metadata enricher. At ViralVidVault, we built a Python client library that encapsulates retry logic, quota tracking, and region-aware caching.

The Client Core

import time
import json
import hashlib
import requests
from dataclasses import dataclass
from typing import Optional

@dataclass
class ClientConfig:
    api_keys: list[str]  # Multiple keys for rotation
    base_url: str = "https://www.googleapis.com/youtube/v3"
    max_retries: int = 3
    timeout: int = 15
    quota_per_key: int = 10000
    cache_ttl: int = 1800  # 30 min

class QuotaExhaustedError(Exception):
    pass

class YouTubeSDK:
    """Multi-key YouTube client with automatic key rotation and caching."""

    def __init__(self, config: ClientConfig):
        self.config = config
        self.session = requests.Session()
        self._key_index = 0
        self._quota_used: dict[str, int] = {k: 0 for k in config.api_keys}
        self._cache: dict[str, tuple[float, dict]] = {}

    @property
    def _current_key(self) -> str:
        return self.config.api_keys[self._key_index]

    def _rotate_key(self) -> bool:
        """Switch to the next available API key. Returns False if all exhausted."""
        original = self._key_index
        while True:
            self._key_index = (self._key_index + 1) % len(self.config.api_keys)
            if self._key_index == original:
                return False  # All keys exhausted
            if self._quota_used[self._current_key] < self.config.quota_per_key:
                return True

    def _cache_get(self, key: str) -> Optional[dict]:
        if key in self._cache:
            ts, data = self._cache[key]
            if time.time() - ts < self.config.cache_ttl:
                return data
            del self._cache[key]
        return None

    def _request(self, endpoint: str, params: dict, cost: int = 1) -> dict:
        cache_key = hashlib.md5(f"{endpoint}:{json.dumps(params, sort_keys=True)}".encode()).hexdigest()
        cached = self._cache_get(cache_key)
        if cached:
            return cached

        last_error = None
        for attempt in range(self.config.max_retries):
            if self._quota_used[self._current_key] + cost > self.config.quota_per_key:
                if not self._rotate_key():
                    raise QuotaExhaustedError("All API keys exhausted")

            params["key"] = self._current_key
            try:
                resp = self.session.get(
                    f"{self.config.base_url}/{endpoint}",
                    params=params,
                    timeout=self.config.timeout,
                )

                if resp.status_code == 200:
                    data = resp.json()
                    self._quota_used[self._current_key] += cost
                    self._cache[cache_key] = (time.time(), data)
                    return data

                if resp.status_code == 403:
                    # Quota exceeded for this key, rotate
                    self._quota_used[self._current_key] = self.config.quota_per_key
                    if not self._rotate_key():
                        raise QuotaExhaustedError("All keys quota exceeded")
                    continue

                if resp.status_code in (429, 500, 503):
                    time.sleep(2 ** attempt)
                    continue

                resp.raise_for_status()

            except requests.ConnectionError as e:
                last_error = e
                time.sleep(2 ** attempt)

        raise last_error or RuntimeError(f"Failed after {self.config.max_retries} attempts")

European Region Methods

Build domain-specific methods for multi-region fetching:

    EU_REGIONS = ["PL", "NL", "SE", "NO", "AT", "GB", "US"]

    def get_trending(self, region: str = "GB", max_results: int = 25) -> list[dict]:
        params = {
            "part": "snippet,statistics,contentDetails",
            "chart": "mostPopular",
            "regionCode": region,
            "maxResults": min(max_results, 50),
        }
        return self._request("videos", params).get("items", [])

    def get_trending_all_regions(self, max_per_region: int = 25) -> dict[str, list]:
        """Fetch trending from all European regions."""
        results = {}
        for region in self.EU_REGIONS:
            try:
                results[region] = self.get_trending(region, max_per_region)
            except QuotaExhaustedError:
                print(f"Quota exhausted at region {region}")
                break
        return results

    def get_video_batch(self, video_ids: list[str]) -> list[dict]:
        """Fetch details for up to 50 videos per call."""
        all_items = []
        for i in range(0, len(video_ids), 50):
            batch = video_ids[i:i + 50]
            params = {
                "part": "snippet,statistics,contentDetails",
                "id": ",".join(batch),
            }
            data = self._request("videos", params)
            all_items.extend(data.get("items", []))
        return all_items

    @property
    def total_quota_remaining(self) -> int:
        return sum(
            self.config.quota_per_key - used
            for used in self._quota_used.values()
        )

    def quota_report(self) -> dict:
        return {
            key[:8] + "...": {
                "used": self._quota_used[key],
                "remaining": self.config.quota_per_key - self._quota_used[key],
            }
            for key in self.config.api_keys
        }

Usage

config = ClientConfig(
    api_keys=["AIzaSy_key1", "AIzaSy_key2", "AIzaSy_key3"],
)
sdk = YouTubeSDK(config)

# Fetch all European regions
all_trending = sdk.get_trending_all_regions(max_per_region=25)
for region, videos in all_trending.items():
    print(f"[{region}] {len(videos)} videos")

# Check quota health
print(sdk.quota_report())
# {'AIzaSy_k...': {'used': 14, 'remaining': 9986}, ...}

Key Design Decisions

Multi-key rotation: When one key hits its quota, automatically switch to the next
Region-aware caching: Same region + same params = cache hit, avoiding duplicate calls
Cost-aware requests: Search costs 100 units vs 1 for videos.list
Graceful degradation: Partial results are better than a crash when quota runs out

This SDK powers every YouTube API interaction at ViralVidVault. The key rotation alone saved us from quota exhaustion during peak viral detection periods.

This article is part of the Building ViralVidVault series. Check out ViralVidVault to see these techniques in action.

Monitoring and Alerting for Video Platform Infrastructure with Go

ahmet gedik — Sun, 10 May 2026 01:00:00 +0000

Why Roll Your Own Monitoring?

Managed monitoring like Datadog is excellent — and expensive. For a self-hosted video platform like TopVideoHub, a 200-line Go binary can cover 90% of what you need: uptime checks, latency histograms, error rate alerts, and a Grafana dashboard wired to Prometheus.

Health Check Endpoints

package main

import (
    "database/sql"
    "encoding/json"
    "net/http"
    "time"
)

type HealthStatus struct {
    Status  string            `json:"status"`
    Checks  map[string]string `json:"checks"`
    Uptime  string            `json:"uptime"`
    Ts      time.Time         `json:"ts"`
}

var startTime = time.Now()

func healthHandler(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := map[string]string{}
        overall := "ok"

        if err := db.Ping(); err != nil {
            checks["database"] = "error: " + err.Error()
            overall = "degraded"
        } else {
            checks["database"] = "ok"
        }

        status := HealthStatus{
            Status: overall,
            Checks: checks,
            Uptime: time.Since(startTime).Round(time.Second).String(),
            Ts:     time.Now(),
        }

        w.Header().Set("Content-Type", "application/json")
        if overall != "ok" {
            w.WriteHeader(http.StatusServiceUnavailable)
        }
        json.NewEncoder(w).Encode(status)
    }
}

Prometheus Metrics Exporter

import "github.com/prometheus/client_golang/prometheus"

var (
    videosFetched = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "topvideohub_videos_fetched_total",
        Help: "Total videos fetched.",
    }, []string{"region"})

    fetchDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "topvideohub_fetch_duration_seconds",
        Help:    "Fetch duration per region.",
        Buckets: prometheus.DefBuckets,
    }, []string{"region"})

    regionFreshness = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "topvideohub_region_freshness_seconds",
        Help: "Seconds since last successful fetch per region.",
    }, []string{"region"})
)

func init() {
    prometheus.MustRegister(videosFetched, fetchDuration, regionFreshness)
}

Instrument the fetch loop:

func fetchWithMetrics(region string, fn func() (int, error)) {
    start := time.Now()
    count, err := fn()
    duration := time.Since(start).Seconds()

    fetchDuration.WithLabelValues(region).Observe(duration)
    if err == nil {
        videosFetched.WithLabelValues(region).Add(float64(count))
    }
}

Uptime Monitor

type UptimeMonitor struct {
    client *http.Client
    slack  string
}

func (m *UptimeMonitor) Watch(targets []string, interval time.Duration) {
    for _, target := range targets {
        go func(t string) {
            ticker := time.NewTicker(interval)
            for range ticker.C {
                start := time.Now()
                resp, err := m.client.Get(t)
                latency := time.Since(start)

                status := 0
                if resp != nil {
                    status = resp.StatusCode
                    resp.Body.Close()
                }
                if err != nil || status >= 500 {
                    msg := fmt.Sprintf(":red_circle: %s DOWN (HTTP %d, %v)", t, status, latency)
                    sendSlackWebhook(m.slack, msg)
                }
            }
        }(target)
    }
}

Watch all regional URLs for TopVideoHub:

monitor := &UptimeMonitor{client: &http.Client{Timeout: 10 * time.Second}, slack: os.Getenv("SLACK_WEBHOOK")}
monitor.Watch([]string{
    "https://topvideohub.com/health",
    "https://topvideohub.com/?region=JP",
    "https://topvideohub.com/?region=KR",
    "https://topvideohub.com/?region=TW",
}, 60*time.Second)

Grafana Dashboard

prometheus.yml:

scrape_configs:
  - job_name: 'topvideohub'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 30s

Key Grafana panels:

Panel	Query
Videos fetched / region	`rate(topvideohub_videos_fetched_total[5m])`
P95 fetch latency	`histogram_quantile(0.95, rate(topvideohub_fetch_duration_seconds_bucket[5m]))`
Region freshness	`topvideohub_region_freshness_seconds` (alert if > 28800)

With these panels, you can spot quota exhaustion before it causes stale data and detect slow region fetches — all without logging into the server.

This article is part of the Building TopVideoHub series. Check out TopVideoHub to see these techniques in action.

Database Sharding Strategies for Growing Video Platforms

ahmet gedik — Sat, 09 May 2026 18:00:01 +0000

When a Single Database Isn't Enough

At DailyWatch, we aggregate trending videos from 8 regions with a cron pipeline running every 2 hours. The videos table grows by thousands of rows daily. At some point, queries slow down, indexes bloat, and backups take too long. That's when you start thinking about sharding.

This article covers three sharding strategies we evaluated, with real PostgreSQL examples.

Strategy 1: Range Partitioning by Date

The most natural fit for time-series video data. PostgreSQL's declarative partitioning handles this natively:

-- Parent table (no data stored directly here)
CREATE TABLE videos (
    id BIGSERIAL,
    video_id VARCHAR(16) NOT NULL,
    title TEXT NOT NULL,
    view_count BIGINT DEFAULT 0,
    region VARCHAR(4) NOT NULL,
    category_id INTEGER,
    fetched_at TIMESTAMP NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, fetched_at)
) PARTITION BY RANGE (fetched_at);

-- Monthly partitions
CREATE TABLE videos_2026_01 PARTITION OF videos
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE videos_2026_02 PARTITION OF videos
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE videos_2026_03 PARTITION OF videos
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- Each partition gets its own indexes automatically
CREATE INDEX ON videos_2026_01 (video_id);
CREATE INDEX ON videos_2026_02 (video_id);
CREATE INDEX ON videos_2026_03 (video_id);

The query planner automatically prunes partitions:

-- Only scans videos_2026_03, skips others entirely
EXPLAIN ANALYZE
SELECT * FROM videos
WHERE fetched_at >= '2026-03-01' AND fetched_at < '2026-03-15';

Automating Partition Creation

Don't create partitions manually. Use a function:

CREATE OR REPLACE FUNCTION create_monthly_partition(target_date DATE)
RETURNS TEXT AS $$
DECLARE
    partition_name TEXT;
    start_date DATE;
    end_date DATE;
BEGIN
    start_date := DATE_TRUNC('month', target_date);
    end_date := start_date + INTERVAL '1 month';
    partition_name := 'videos_' || TO_CHAR(start_date, 'YYYY_MM');

    -- Check if partition already exists
    IF NOT EXISTS (
        SELECT 1 FROM pg_class WHERE relname = partition_name
    ) THEN
        EXECUTE format(
            'CREATE TABLE %I PARTITION OF videos FOR VALUES FROM (%L) TO (%L)',
            partition_name, start_date, end_date
        );
        EXECUTE format(
            'CREATE INDEX ON %I (video_id)',
            partition_name
        );
        RETURN 'Created: ' || partition_name;
    END IF;

    RETURN 'Already exists: ' || partition_name;
END;
$$ LANGUAGE plpgsql;

-- Call before each month
SELECT create_monthly_partition('2026-04-01');

Strategy 2: List Partitioning by Region

When queries almost always filter by region, partition on that dimension:

CREATE TABLE videos_by_region (
    id BIGSERIAL,
    video_id VARCHAR(16) NOT NULL,
    title TEXT NOT NULL,
    view_count BIGINT DEFAULT 0,
    region VARCHAR(4) NOT NULL,
    fetched_at TIMESTAMP NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, region)
) PARTITION BY LIST (region);

CREATE TABLE videos_americas PARTITION OF videos_by_region
    FOR VALUES IN ('US', 'BR', 'CA');

CREATE TABLE videos_europe PARTITION OF videos_by_region
    FOR VALUES IN ('GB', 'DE', 'FR');

CREATE TABLE videos_apac PARTITION OF videos_by_region
    FOR VALUES IN ('IN', 'AU');

This shines when your application logic is already region-aware. At DailyWatch, each region's trending page only queries its own partition.

Strategy 3: Application-Level Sharding

When PostgreSQL partitioning isn't enough (different databases on different servers), shard at the application level:

import hashlib
from typing import Dict
import asyncpg

class VideoShardRouter:
    """Routes video queries to the correct database shard."""

    def __init__(self, shard_configs: Dict[str, str]):
        # shard_configs: {"shard_0": "postgres://...", "shard_1": "postgres://...", ...}
        self.shard_count = len(shard_configs)
        self.pools: Dict[str, asyncpg.Pool] = {}
        self.configs = shard_configs

    async def initialize(self):
        for name, dsn in self.configs.items():
            self.pools[name] = await asyncpg.create_pool(dsn, min_size=2, max_size=10)

    def get_shard(self, video_id: str) -> str:
        """Consistent hash-based routing."""
        hash_val = int(hashlib.md5(video_id.encode()).hexdigest(), 16)
        shard_idx = hash_val % self.shard_count
        return f"shard_{shard_idx}"

    async def insert_video(self, video_id: str, title: str, region: str, views: int):
        shard = self.get_shard(video_id)
        pool = self.pools[shard]
        await pool.execute(
            "INSERT INTO videos (video_id, title, region, view_count) VALUES ($1, $2, $3, $4) "
            "ON CONFLICT (video_id) DO UPDATE SET view_count = $4",
            video_id, title, region, views,
        )

    async def get_video(self, video_id: str) -> dict:
        shard = self.get_shard(video_id)
        pool = self.pools[shard]
        row = await pool.fetchrow(
            "SELECT * FROM videos WHERE video_id = $1", video_id
        )
        return dict(row) if row else None

    async def search_all_shards(self, query: str, limit: int = 20) -> list:
        """Fan-out query across all shards, merge results."""
        import asyncio
        tasks = [
            pool.fetch(
                "SELECT * FROM videos WHERE title ILIKE $1 ORDER BY view_count DESC LIMIT $2",
                f"%{query}%", limit,
            )
            for pool in self.pools.values()
        ]
        results = await asyncio.gather(*tasks)
        merged = [dict(row) for shard_rows in results for row in shard_rows]
        merged.sort(key=lambda v: v["view_count"], reverse=True)
        return merged[:limit]

Which Strategy to Choose?

Strategy	Best When	Complexity	DailyWatch Fit
Date partitioning	Time-series queries dominate	Low	Good for archiving
Region partitioning	Region-scoped queries	Low	Great for trending pages
App-level sharding	Multiple DB servers needed	High	Future-proofing

For most video platforms under 10M rows, PostgreSQL native partitioning is the right call. You get automatic partition pruning, simpler ops, and no application code changes. App-level sharding is the nuclear option for when you've outgrown a single server.

Key Takeaway

Start with date-based partitioning. It's the lowest-effort, highest-impact sharding strategy for content platforms where "recent" is always the most-queried dimension. You can always add region partitioning as a sub-partition later.

This article is part of the Building DailyWatch series. Check out DailyWatch to see these techniques in action.

Database Sharding Strategies for Growing Video Platforms

ahmet gedik — Sat, 09 May 2026 09:00:01 +0000

Sharding Is Not Your First Problem

Most video platforms can serve millions of rows from a single PostgreSQL instance with proper indexing. Design your schema to be shardable from day one, but shard only when necessary.

TrendVidStream serves 8 regions — UAE, Finland, Czech Republic, Denmark, Belgium, UK, Switzerland, US. The natural partition keys are region and fetched_at.

Strategy 1: PostgreSQL List Partitioning by Region

CREATE TABLE videos (
    id           BIGSERIAL,
    video_id     TEXT NOT NULL,
    title        TEXT NOT NULL,
    region       TEXT NOT NULL,
    category_id  INT,
    view_count   BIGINT DEFAULT 0,
    fetched_at   TIMESTAMPTZ DEFAULT NOW(),
    embed_broken BOOLEAN DEFAULT FALSE,
    title_lang   TEXT,
    PRIMARY KEY (id, region)
) PARTITION BY LIST (region);

CREATE TABLE videos_ae PARTITION OF videos FOR VALUES IN ('AE');
CREATE TABLE videos_fi PARTITION OF videos FOR VALUES IN ('FI');
CREATE TABLE videos_cz PARTITION OF videos FOR VALUES IN ('CZ');
CREATE TABLE videos_dk PARTITION OF videos FOR VALUES IN ('DK');
CREATE TABLE videos_be PARTITION OF videos FOR VALUES IN ('BE');
CREATE TABLE videos_ch PARTITION OF videos FOR VALUES IN ('CH');
CREATE TABLE videos_gb PARTITION OF videos FOR VALUES IN ('GB');
CREATE TABLE videos_us PARTITION OF videos FOR VALUES IN ('US');
CREATE TABLE videos_other PARTITION OF videos DEFAULT;

-- Indexes propagate to all partitions automatically
CREATE INDEX idx_videos_views ON videos (view_count DESC);
CREATE INDEX idx_videos_fetched ON videos (fetched_at DESC);

Strategy 2: Sub-Partition by Date for Event Data

CREATE TABLE video_events (
    id       BIGSERIAL,
    video_id TEXT NOT NULL,
    region   TEXT NOT NULL,
    event    TEXT NOT NULL,
    occurred TIMESTAMPTZ DEFAULT NOW()
) PARTITION BY LIST (region);

CREATE TABLE video_events_ae PARTITION OF video_events
    FOR VALUES IN ('AE')
    PARTITION BY RANGE (occurred);

CREATE TABLE video_events_ae_2026q1 PARTITION OF video_events_ae
    FOR VALUES FROM ('2026-01-01') TO ('2026-04-01');

CREATE TABLE video_events_ae_2026q2 PARTITION OF video_events_ae
    FOR VALUES FROM ('2026-04-01') TO ('2026-07-01');

Strategy 3: Application-Level Shard Routing

import asyncpg

SHARD_MAP = {
    'AE': 'postgresql://user:pass@db-mena/trendvidstream',
    'GB': 'postgresql://user:pass@db-eu-west/trendvidstream',
    'BE': 'postgresql://user:pass@db-eu-west/trendvidstream',
    'CH': 'postgresql://user:pass@db-eu-central/trendvidstream',
    'CZ': 'postgresql://user:pass@db-eu-central/trendvidstream',
    'DK': 'postgresql://user:pass@db-eu-north/trendvidstream',
    'FI': 'postgresql://user:pass@db-eu-north/trendvidstream',
    'US': 'postgresql://user:pass@db-us/trendvidstream',
}

_pools: dict[str, asyncpg.Pool] = {}

async def shard_pool(region: str) -> asyncpg.Pool:
    dsn = SHARD_MAP.get(region, SHARD_MAP['US'])
    if dsn not in _pools:
        _pools[dsn] = await asyncpg.create_pool(dsn, min_size=2, max_size=10)
    return _pools[dsn]

async def trending(region: str, limit: int = 50) -> list[dict]:
    pool = await shard_pool(region)
    async with pool.acquire() as conn:
        rows = await conn.fetch(
            'SELECT video_id, title, thumbnail_url, view_count FROM videos '
            'WHERE region = $1 ORDER BY view_count DESC LIMIT $2', region, limit,
        )
    return [dict(r) for r in rows]

Cross-Shard Global Trending

import asyncio

async def global_trending(limit: int = 20) -> list[dict]:
    regions = list(dict.fromkeys(SHARD_MAP.keys()))
    per_shard = await asyncio.gather(*[trending(r, limit) for r in regions],
                                    return_exceptions=True)
    merged = []
    for result in per_shard:
        if isinstance(result, list):
            merged.extend(result)
    merged.sort(key=lambda v: v['view_count'], reverse=True)
    return merged[:limit]

For TrendVidStream, the global trending page completes in 12ms at current scale. The shard routing layer is ready for when that grows.

Migration Checklist

Add region to every table's primary key.
Add PostgreSQL list partitions (ATTACH PARTITION is zero-downtime).
Move hot partitions to read replicas as read load grows.
Introduce shard routing when write throughput demands separate hosts.
Never shard prematurely — each step adds operational complexity.

This article is part of the Building TrendVidStream series. Check out TrendVidStream to see these techniques in action.

Database Sharding Strategies for Growing Video Platforms

ahmet gedik — Sat, 09 May 2026 01:00:01 +0000

Scaling a European Video Database

ViralVidVault stores viral video data from 7 European regions. As the database grows, queries that scan the entire videos table get slower. Sharding -- splitting data across multiple tables or databases -- is how you keep query performance flat as data grows linearly.

Here are three strategies, with concrete PostgreSQL implementations.

Strategy 1: List Partitioning by Region

The most natural fit for a multi-region video vault. Each European region gets its own partition:

CREATE TABLE videos (
    id BIGSERIAL,
    video_id VARCHAR(16) NOT NULL,
    title TEXT NOT NULL,
    view_count BIGINT DEFAULT 0,
    virality_score FLOAT DEFAULT 0,
    region VARCHAR(4) NOT NULL,
    category_id INTEGER,
    fetched_at TIMESTAMP NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, region)
) PARTITION BY LIST (region);

-- One partition per tracked region
CREATE TABLE videos_pl PARTITION OF videos FOR VALUES IN ('PL');
CREATE TABLE videos_nl PARTITION OF videos FOR VALUES IN ('NL');
CREATE TABLE videos_se PARTITION OF videos FOR VALUES IN ('SE');
CREATE TABLE videos_no PARTITION OF videos FOR VALUES IN ('NO');
CREATE TABLE videos_at PARTITION OF videos FOR VALUES IN ('AT');
CREATE TABLE videos_gb PARTITION OF videos FOR VALUES IN ('GB');
CREATE TABLE videos_us PARTITION OF videos FOR VALUES IN ('US');

-- Default partition for any unexpected regions
CREATE TABLE videos_other PARTITION OF videos DEFAULT;

-- Each partition gets its own index
CREATE INDEX ON videos_pl (virality_score DESC);
CREATE INDEX ON videos_nl (virality_score DESC);
CREATE INDEX ON videos_se (virality_score DESC);
CREATE INDEX ON videos_no (virality_score DESC);
CREATE INDEX ON videos_at (virality_score DESC);
CREATE INDEX ON videos_gb (virality_score DESC);
CREATE INDEX ON videos_us (virality_score DESC);

Now region-scoped queries are lightning fast:

-- Only scans videos_pl, not the other 6 partitions
SELECT * FROM videos
WHERE region = 'PL' AND virality_score >= 85
ORDER BY virality_score DESC
LIMIT 20;

Strategy 2: Composite Partitioning (Region + Date)

For even finer granularity, partition by region first, then by date:

CREATE TABLE videos_partitioned (
    id BIGSERIAL,
    video_id VARCHAR(16) NOT NULL,
    title TEXT NOT NULL,
    virality_score FLOAT DEFAULT 0,
    region VARCHAR(4) NOT NULL,
    fetched_at TIMESTAMP NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, region, fetched_at)
) PARTITION BY LIST (region);

-- Region partition, further sub-partitioned by month
CREATE TABLE videos_gb_part PARTITION OF videos_partitioned
    FOR VALUES IN ('GB')
    PARTITION BY RANGE (fetched_at);

CREATE TABLE videos_gb_2026_02 PARTITION OF videos_gb_part
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');
CREATE TABLE videos_gb_2026_03 PARTITION OF videos_gb_part
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

This is ideal for queries like "show me viral UK videos from this week."

Strategy 3: Application-Level Sharding

When you outgrow a single PostgreSQL instance, shard at the application layer:

import hashlib
from typing import Dict, List
import asyncpg

class RegionShardRouter:
    """Route queries to region-specific database instances."""

    SHARD_MAP = {
        "western": ["GB", "NL", "AT"],   # Western Europe shard
        "nordic": ["SE", "NO"],           # Nordic shard
        "eastern": ["PL"],                # Eastern Europe shard
        "global": ["US"],                 # Global shard
    }

    def __init__(self, shard_dsns: Dict[str, str]):
        self.dsns = shard_dsns
        self.pools: Dict[str, asyncpg.Pool] = {}
        # Build reverse lookup: region -> shard name
        self.region_to_shard = {}
        for shard, regions in self.SHARD_MAP.items():
            for r in regions:
                self.region_to_shard[r] = shard

    async def initialize(self):
        for name, dsn in self.dsns.items():
            self.pools[name] = await asyncpg.create_pool(dsn, min_size=2, max_size=10)

    def _get_pool(self, region: str) -> asyncpg.Pool:
        shard = self.region_to_shard.get(region, "western")
        return self.pools[shard]

    async def get_viral_videos(self, region: str, min_score: float = 85.0, limit: int = 20) -> list:
        pool = self._get_pool(region)
        return await pool.fetch(
            "SELECT * FROM videos WHERE region = $1 AND virality_score >= $2 "
            "ORDER BY virality_score DESC LIMIT $3",
            region, min_score, limit,
        )

    async def search_all_regions(self, query: str, limit: int = 20) -> list:
        """Fan-out search across all shards, merge by virality score."""
        import asyncio
        tasks = [
            pool.fetch(
                "SELECT * FROM videos WHERE title ILIKE $1 ORDER BY virality_score DESC LIMIT $2",
                f"%{query}%", limit,
            )
            for pool in self.pools.values()
        ]
        results = await asyncio.gather(*tasks)
        merged = [dict(r) for shard_rows in results for r in shard_rows]
        merged.sort(key=lambda v: v["virality_score"], reverse=True)
        return merged[:limit]

Choosing the Right Strategy

Strategy	Best For	Complexity	ViralVidVault Fit
Region list partition	Region-scoped queries	Low	Ideal for trending pages
Region + date composite	Time-bounded + region queries	Medium	Great for analytics
App-level shard	Multiple DB servers	High	Future scaling

For ViralVidVault, region-based list partitioning is the clear winner. Every page on the site is implicitly region-scoped -- "viral in Poland," "trending in Sweden" -- so partition pruning eliminates 80%+ of rows from every query automatically.

Partition Maintenance

Automate monthly partition creation:

CREATE OR REPLACE FUNCTION ensure_monthly_partitions(base_region TEXT, months_ahead INT DEFAULT 3)
RETURNS VOID AS $$
DECLARE
    m INT;
    part_name TEXT;
    start_d DATE;
    end_d DATE;
BEGIN
    FOR m IN 0..months_ahead LOOP
        start_d := DATE_TRUNC('month', NOW() + (m || ' months')::INTERVAL);
        end_d := start_d + INTERVAL '1 month';
        part_name := format('videos_%s_%s', lower(base_region), TO_CHAR(start_d, 'YYYY_MM'));
        IF NOT EXISTS (SELECT 1 FROM pg_class WHERE relname = part_name) THEN
            EXECUTE format(
                'CREATE TABLE %I PARTITION OF videos_%s_part FOR VALUES FROM (%L) TO (%L)',
                part_name, lower(base_region), start_d, end_d
            );
        END IF;
    END LOOP;
END;
$$ LANGUAGE plpgsql;

Start simple, partition early. It's far easier to add partitioning from day one than to migrate a bloated table later.

This article is part of the Building ViralVidVault series. Check out ViralVidVault to see these techniques in action.