Forem: nareshipme

Dev Journal: 2026-04-15

nareshipme — Wed, 15 Apr 2026 05:30:02 +0000

env: node: No such file or directory

Whisper Hallucination on Silence: Why Your Transcript Loops the Same Phrase

nareshipme — Tue, 14 Apr 2026 16:17:27 +0000

Whisper Hallucination on Silence: Why Your Transcript Loops the Same Phrase

The Problem

Have you ever transcribed audio with Whisper (or any ASR model) and gotten something like this in the output:

"Hello, welcome to our show. The weather is nice today."
"Hello, welcome to our show. The weather is nice today."
"Hello, welcome to our show. The weather is nice today."
"Hello, welcome to our show. The weather is nice today."
...

Repeating the same phrase 50+ times, even though the audio was clearly different?

What's Happening

This is called hallucination, and it happens when the model encounters silence, background noise, or low-confidence audio segments. Here's why:

1. Silence Gets Interpreted as Speech

When there's silence (or very quiet audio), the model's audio embeddings are near-zero. The model, trained on speech, tries to "fill in" what it thinks should be there. It loops the most recent phrase because that's the closest match to the empty audio.

2. Background Music Triggers Loops

Live streams, podcasts, or interviews often have background music or ambient noise. Whisper (and other models) struggle with non-speech audio. The model sees "something happening" and tries to match it to its training data — which is speech — so it loops phrases.

3. No Confidence Threshold

Whisper outputs text for every audio segment, even if the model is barely confident. There's no built-in VAD (voice activity detection) to skip silent/background-only segments.

The Workaround: Use Indian-Language-First Models

For content in Indian languages (Telugu, Tamil, Hindi, etc.), Sarvam AI's Saarika/Saaras models are purpose-built for Indian audio patterns. They handle:

Indian accents and code-switching
Background noise better
Less hallucination on silence

We migrated from Whisper to Sarvam for transcription, and the hallucination issue disappeared.

Long-Term Fix: Add VAD

The proper solution is to add voice activity detection before transcription:

// Pseudo-code for VAD preprocessing
const vad = new VAD();
const speechSegments = await vad.detect(audioBuffer);

// Only transcribe speech segments
for (const segment of speechSegments) {
  const transcript = await whisper.transcribe(segment);
  fullTranscript += transcript;
}

This skips silence and background-only segments entirely, preventing hallucinations.

The Lesson

Don't trust ASR models with silence or background noise. Always preprocess with VAD, or use models trained for your specific audio domain (like Sarvam for Indian languages).

Sources

Have you experienced ASR hallucinations? What worked for you?

Dev Journal: 2026-04-14

nareshipme — Tue, 14 Apr 2026 06:30:01 +0000

env: node: No such file or directory

Dev Journal: 2026-04-13

nareshipme — Mon, 13 Apr 2026 06:30:03 +0000

env: node: No such file or directory

Dev Journal: 2026-04-12

nareshipme — Sun, 12 Apr 2026 06:30:02 +0000

env: node: No such file or directory

Building a Robust Real-Time Transcription Pipeline in Next.js: STT, Streaming, and Error Recovery

nareshipme — Sat, 11 Apr 2026 17:19:24 +0000

Building a Robust Real-Time Transcription Pipeline in Next.js: STT, Streaming, and Error Recovery

Transcribing video and audio at scale is deceptively complex. You're not just calling an STT API and returning the result — you're managing file chunking, concurrent requests, partial failures, and real-time updates. Here's how we built a production-grade transcription pipeline in Next.js.

The Problem

When processing user-uploaded videos, we needed:

Streaming results as transcription completes (not waiting for the whole file)
Automatic retry on API failures without losing progress
Chunk-based processing to avoid timeout and memory limits
Multi-language support with fallback providers
Real-time UI updates as captions are generated

Architecture

We use a three-layer approach:

Client: Stream audio blobs in 30-second chunks
API Route: Queue chunks, handle retries, emit progress events
Background Worker: Heavy lifting with Inngest (idempotent steps, automatic retries)

Layer 1: Client-Side Streaming

// Send 30-second chunks to the API
async function transcribeChunked(file: File) {
  const CHUNK_SIZE = 30 * 1000; // 30 seconds
  const duration = await getAudioDuration(file);

  for (let start = 0; start < duration; start += CHUNK_SIZE) {
    const blob = await extractAudioChunk(file, start, start + CHUNK_SIZE);
    const formData = new FormData();
    formData.append('audio', blob);
    formData.append('startTime', start.toString());

    const response = await fetch('/api/transcribe', {
      method: 'POST',
      body: formData,
    });

    // Stream events back
    const reader = response.body!.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const text = decoder.decode(value);
      const lines = text.split('\n').filter(Boolean);

      for (const line of lines) {
        const event = JSON.parse(line);
        if (event.type === 'caption') {
          updateCaption(event.text, event.startTime);
        }
      }
    }
  }
}

Layer 2: API Route with Queueing

// app/api/transcribe/route.ts
import { inngest } from '@/lib/inngest';

export async function POST(req: Request) {
  const formData = await req.formData();
  const blob = formData.get('audio') as Blob;
  const startTime = parseInt(formData.get('startTime') as string);
  const videoId = formData.get('videoId') as string;

  // Trigger background job, return streaming response
  const runId = await inngest.send({
    name: 'video/transcribe-chunk',
    data: { videoId, audioBuffer: await blob.arrayBuffer(), startTime },
  });

  // Stream progress back to client
  return new Response(
    streamTranscriptionProgress(videoId, runId),
    { headers: { 'Content-Type': 'application/x-ndjson' } }
  );
}

Layer 3: Inngest Worker with Resilience

// lib/inngest.ts
inngest.createFunction(
  { id: 'transcribe-chunk', priority: { run: 10 } },
  { event: 'video/transcribe-chunk' },
  async ({ event, step }) => {
    const { videoId, audioBuffer, startTime } = event.data;

    // Step 1: Try primary STT provider (Sarvam for Indian languages)
    let transcript = await step.run('sarvam-stt', async () => {
      try {
        return await callSarvamSTT(audioBuffer);
      } catch (err) {
        if (err.status === 429) throw err; // Retry
        return null; // Fallback
      }
    });

    // Step 2: Fallback to Gemini if primary fails
    if (!transcript) {
      transcript = await step.run('gemini-stt', async () => {
        return await callGeminiSTT(audioBuffer);
      });
    }

    // Step 3: Store result (idempotent)
    await step.run('store-caption', async () => {
      await db.captions.upsert({
        videoId,
        startTime,
        text: transcript,
      });
    });

    // Step 4: Emit to client
    await step.run('emit-progress', async () => {
      await publishEvent(`transcribe:${videoId}`, {
        type: 'caption',
        text: transcript,
        startTime,
      });
    });
  }
);

Key Patterns

1. Retry Logic Without Infinite Loops

Inngest handles retries automatically, but we distinguish:

429 (rate limit): Throw to trigger exponential backoff
Network error: Auto-retry (Inngest)
Invalid audio: Fail gracefully, skip chunk

if (err.message.includes('invalid audio format')) {
  await db.captions.create({ videoId, startTime, text: '[SILENT]' });
  return; // Don't retry
}

throw err; // Let Inngest retry

2. Streaming Results Early

Don't wait for all chunks. Emit captions as they complete:

await publishEvent(`transcribe:${videoId}`, {
  type: 'progress',
  processed: Math.floor((startTime / duration) * 100),
});

3. Idempotent Storage

Always use upsert for captions, keyed by (videoId, startTime):

db.captions.upsert({
  videoId,
  startTime, // Natural composite key
  text: transcript,
});

This way, retries don't create duplicates.

4. Detect Language Dynamically

For multi-language content, detect before selecting STT:

const language = await step.run('detect-language', async () => {
  const sample = audioBuffer.slice(0, 10000); // First 100ms
  return await detectLanguage(sample);
});

if (language === 'hi') return callSarvamSTT(audioBuffer);
return callGeminiSTT(audioBuffer);

Performance Tuning

Chunk size: 30s balances API limits vs. latency
Concurrency: Process up to 3 chunks in parallel, then rate-limit
Timeout: Set Inngest step timeout to 60s (STT can take 20-40s)
Storage: Cache transcript in Redis during processing, write to DB once

Monitoring

Track:

Transcription latency per chunk (should be <2x audio duration)
Retry rate by provider (>5% means reconsider fallback)
Cost per minute of audio (to catch runaway bugs)

posthog.capture('transcribe_chunk', {
  duration: 30,
  latency: Date.now() - start,
  provider: 'sarvam',
  success: true,
});

Summary

A robust transcription pipeline is:

Chunked at the network boundary (30s chunks)
Queued with automatic retries (Inngest)
Streaming to avoid client timeouts
Redundant with fallback providers
Idempotent to survive failures gracefully

Once you have this foundation, adding features (speaker diarization, emotion detection, keyword highlighting) becomes straightforward — they're just additional steps in the pipeline.

Have you hit transcription limits? Drop a comment with your approach.

Dev Journal: 2026-04-10

nareshipme — Fri, 10 Apr 2026 09:30:04 +0000

env: node: No such file or directory

Fixing yt-dlp in Docker: n-challenge EJS scripts, Deno 2.x, and the player_client=ios cookie trap

nareshipme — Thu, 09 Apr 2026 19:08:29 +0000

If you run a video processing worker that downloads YouTube content with yt-dlp, you've probably hit the n-challenge wall: downloads stall or fail with a Sign in to confirm you're not a bot error, or the n-parameter isn't being decoded correctly, leaving you with throttled or broken streams. After a few rounds of debugging our Railway-hosted worker, here's everything we fixed and why.

Background: what is the n-challenge?

YouTube applies a per-request throttling parameter called n to video URLs. yt-dlp has to solve a JavaScript challenge (embedded in the YouTube player) to decode this parameter into a valid token. If the challenge isn't solved, requests are throttled to ~50 KB/s or rejected outright.

The solver is written in JavaScript, and yt-dlp ships with multiple backends for running it: a pure-Python fallback, and native JS runtimes (Node.js, Deno, PhantomJS). The JS path is dramatically faster and more reliable — but only if the runtime is actually available and the yt-dlp installation includes the EJS scripts that invoke it.

We had three separate bugs causing failures. Here's each one.

Bug 1: Installing yt-dlp as a curl binary skips the EJS scripts

The most common Docker pattern for yt-dlp is to grab the standalone binary from GitHub releases:

RUN curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp \
    -o /usr/local/bin/yt-dlp \
    && chmod a+rx /usr/local/bin/yt-dlp

This works for basic downloads, but the standalone binary bundles only a subset of yt-dlp's assets. The EJS (Embedded JavaScript) scripts that yt-dlp uses to run the n-challenge solver in an external JS runtime are not included in the single-file binary. They live in the yt_dlp/ package directory and are only present when you install via pip.

The fix is to install with pip using the [default] extra, which pulls in all optional dependencies including the JS solver scripts:

RUN pip3 install --break-system-packages "yt-dlp[default]"

The --break-system-packages flag is required on Debian/Ubuntu images that use externally-managed Python environments (PEP 668). Without it, pip refuses to install into the system Python on newer base images. This flag is safe here because we're in a container — there's no system package manager conflict to worry about.

Why [default] and not just yt-dlp? The [default] extra includes:

brotli — for brotli-compressed responses
certifi — updated CA bundle
mutagen — audio metadata writing
pycryptodomex — AES decryption for some streams
websockets — live stream support

None of these are strictly required for n-challenge solving, but they prevent silent fallbacks to slower code paths.

Bug 2: No JS runtime available for the n-challenge solver

Once the EJS scripts are present, yt-dlp needs a JS runtime to execute them. Our base image (node:20-slim) has Node.js, but we wanted a second option that's more isolated and doesn't require the full Node module resolution chain.

We added Deno 2.x as the JS runtime:

RUN curl -fsSL https://deno.land/install.sh \
    | DENO_INSTALL=/usr/local sh \
    && deno --version

Setting DENO_INSTALL=/usr/local puts the deno binary at /usr/local/bin/deno, which is already on PATH. No .deno/bin in PATH needed.

yt-dlp auto-detects available JS runtimes at runtime in this priority order: deno, node, phantomjs. With Deno present, it takes priority. Deno's V8 sandbox also means the n-challenge JS executes with no filesystem or network access by default — a minor but real security improvement over running arbitrary YouTube JS in Node.

The full updated system deps block:

RUN apt-get update && apt-get install -y \
    ffmpeg python3 python3-pip curl ca-certificates unzip \
    --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* \
    && pip3 install --break-system-packages "yt-dlp[default]" \
    && yt-dlp --version \
    && curl -fsSL https://deno.land/install.sh | DENO_INSTALL=/usr/local sh \
    && deno --version

Note unzip is required by the Deno installer.

Bug 3: player_client=ios silently drops cookies

This one was subtle. Our yt-dlp invocation was passing --extractor-args youtube:player_client=ios to use the iOS player client, which historically bypassed some bot detection. But it has a critical flaw: the iOS client ignores cookies.

YouTube's iOS API endpoint doesn't use browser cookie authentication — it uses OAuth tokens. When you pass --cookies with a Netscape-format cookies file (the kind you export from a browser), the iOS client simply ignores it. For age-restricted videos, private videos, or accounts that have verified status, this means your cookies are silently dropped and you get a public-access response (or a rejection).

The fix is to switch to clients that actually honor cookies:

// Before
"--extractor-args",
"youtube:player_client=ios",

// After
"--extractor-args",
"youtube:player_client=web,mweb,android",

The web client is the standard browser client — fully compatible with cookie authentication. mweb is the mobile web client, useful as a fallback. android is the Android app client, which is good for bypassing some rate limits while still respecting cookies.

yt-dlp tries them in order and uses the first one that succeeds. In practice, web handles the vast majority of cases, and mweb/android serve as automatic fallbacks without any extra code.

Keeping yt-dlp current without rebuilding

YouTube frequently updates the player JS, which changes the n-challenge algorithm. A yt-dlp binary that was current two weeks ago may fail today. We handle this by self-updating on every container start:

CMD ["sh", "-c", "yt-dlp -U && node --max-old-space-size=200 dist/server.js"]

yt-dlp -U checks for a newer pip release and upgrades in place if one exists. This adds ~2-3 seconds to cold start but means you never need to rebuild the image just to fix a YouTube extractor bug.

Summary of changes

What	Before	After
yt-dlp install	`curl` binary from GitHub	`pip install yt-dlp[default]`
JS runtime	None (Python fallback)	Deno 2.x + Node.js
player_client	`ios`	`web,mweb,android`
Cookie support	Silently broken	Working
EJS scripts	Missing	Included via pip package

If you're running yt-dlp in Docker and seeing mysterious throttling, bot-detection failures, or cookies that seem to be ignored, this combination of fixes is likely what you need. The pip install + Deno runtime is especially easy to overlook when the standalone binary appears to work fine for public videos.

Debugging a Silent Failure: Presigned R2 Uploads and the Inngest Job That Never Ran

nareshipme — Thu, 09 Apr 2026 18:34:45 +0000

We shipped a file upload feature for ClipCrafter — paste a video URL or upload a file — and it looked like everything was working. The S3-compatible PUT request to Cloudflare R2 returned a 200. No errors in the console. No red Vercel logs.

Yet every uploaded video sat in limbo, never processed. Here's the bug, the fix, and what we learned.

The Architecture in 30 Seconds

ClipCrafter processes videos using Inngest background functions. The flow looks like this:

User uploads a file → we generate a presigned R2 URL via a /api/upload route
Client does a direct PUT to that presigned URL (R2 / Cloudflare)
Client calls /api/projects/[id] to update the project with metadata
A separate action triggers inngest.send({ name: "video/process", data: { projectId } })
Inngest picks up the event and runs the process-video function

Simple enough. So what broke?

The Bug: A Missing `PATCH`

After a successful PUT to R2, our frontend was calling the project PATCH route — but only to save the video title. We never sent the r2_key back.

// ❌ BEFORE — only patching the title
await fetch(`/api/projects/${projectId}`, {
  method: "PATCH",
  body: JSON.stringify({ title: fileName }),
});

Meanwhile, in our Inngest process-video function:

const project = await supabase
  .from("projects")
  .select("r2_key, source_url")
  .eq("id", event.data.projectId)
  .single();

if (!project.data?.r2_key && !project.data?.source_url) {
  throw new Error("No video source found"); // 💥 always hit for uploads
}

The job failed on the very first step, every single time. And because Inngest retries silently in the background, nothing surfaced as an obvious user-facing error — we just never saw the video get processed.

The Fix: Extend the PATCH Route and Send the Key

Two small changes:

1. Extend /api/projects/[id]/route.ts to accept r2_key:

// ✅ AFTER
const { title, r2_key } = await req.json();
const updatePayload: Record<string, string> = {};
if (title) updatePayload.title = title;
if (r2_key) updatePayload.r2_key = r2_key;

await supabase.from("projects").update(updatePayload).eq("id", projectId);

2. Send r2_key from the client after the PUT succeeds:

// ✅ AFTER — send both title and r2_key
const r2Key = `uploads/${projectId}/${fileName}`;

// Direct PUT to presigned URL
await fetch(presignedUrl, {
  method: "PUT",
  body: file,
  headers: { "Content-Type": file.type },
});

// Save the key back to the project row
await fetch(`/api/projects/${projectId}`, {
  method: "PATCH",
  body: JSON.stringify({ title: fileName, r2_key: r2Key }),
});

That's it. One missing field. One extra line in the payload. Everything downstream just works.

The Lesson: Presigned URLs Create Invisible Gaps

With direct-upload patterns (presigned S3/R2 URLs), your backend is deliberately out of the loop during the actual transfer. That's the point — it offloads bandwidth from your API server. But it also means:

Your API never sees the bytes
Your API never confirms the upload succeeded
Your API's database row stays stale until you explicitly update it

If any step between "PUT succeeds" and "database updated" fails silently, you end up with orphaned uploads. The file is in R2. The DB says it isn't. Downstream jobs trust the DB.

A defensive pattern we're now adopting: after the presigned PUT, always PATCH back a upload_completed_at timestamp alongside the key. If that timestamp is null when the Inngest job runs, skip or surface an error with a meaningful message instead of throwing cryptically.

What Else Landed: Billing Scaffolding (Not Live Yet)

Alongside this fix, we scaffolded the billing layer for ClipCrafter — Stripe + Razorpay support, Supabase subscriptions and usage tables, plan limits enforced inside the Inngest pipeline itself:

// Inside process-video Inngest function
await step.run("check-usage", async () => {
  const allowed = await isUsageAllowed(projectOwnerId);
  if (!allowed) throw new NonRetriableError("Usage limit reached");
});

// ... process video ...

await step.run("increment-usage", async () => {
  await incrementUsage(projectOwnerId);
});

Putting the usage gate inside Inngest rather than at the API layer means even background retries respect plan limits. More on the full billing design in a future post.

If you're building a video tool and want to see this pipeline in action, give ClipCrafter a try — paste a YouTube link or upload a file and get shareable clips in minutes.

Have you been burned by a presigned URL silent-failure bug? Drop a comment — I'd love to compare notes.

How We Made Waiting Fun: Whimsical Loading States and Stepped Progress in Next.js

nareshipme — Wed, 08 Apr 2026 15:43:20 +0000

Long-running background jobs are a fact of life in video processing. When a user uploads a 30-minute video, your AI pipeline might need two or three minutes to download, transcribe, and clip it. The worst thing you can do is show a spinner and leave people wondering if something broke.

In ClipCrafter — an AI-powered tool that turns long videos into short, shareable clips — we recently overhauled our processing UX. Here's how we did it with React, TypeScript, and a little personality.

The problem

Our original processing screen was a single progress bar with a generic "Processing…" label. Users had no idea which stage the pipeline was in, how long it would take, or whether the page was even still working. Support messages spiked every time processing took longer than 90 seconds.

Solution 1 — Rotating whimsical messages

We created a dedicated module — loadingMessages.ts — that maps each pipeline stage to an array of light-hearted messages:

// src/lib/loadingMessages.ts
const stageMessages: Record<ProcessingStage, string[]> = {
  downloading: [
    "Summoning your video from the internet…",
    "Convincing the server to share…",
    "Downloading at the speed of vibes…",
  ],
  transcribing: [
    "Eavesdropping scientifically…",
    "Teaching the AI to listen…",
    "Turning sound waves into words…",
  ],
  analyzing: [
    "Asking the AI what slaps…",
    "Finding the best moments…",
    "Separating signal from filler…",
  ],
  // ... more stages
};

A useEffect hook cycles through the array for the current stage every three seconds, giving the interface a living, breathing feel:

useEffect(() => {
  const messages = stageMessages[currentStage];
  const interval = setInterval(() => {
    setMessageIndex((i) => (i + 1) % messages.length);
  }, 3000);
  return () => clearInterval(interval);
}, [currentStage]);

The psychological impact is real — users see movement and humour, which makes the wait feel shorter.

Solution 2 — A stepped progress indicator

Instead of one bar, we render five dots connected by lines — one per pipeline stage. The active dot pulses; completed dots turn green. The component is pure CSS and state:

const stages = ["download", "transcribe", "analyze", "clip", "export"];

{stages.map((stage, i) => (
  <div key={stage} className="flex items-center">
    <div className={cn(
      "h-3 w-3 rounded-full",
      i < currentIndex ? "bg-green-500" :
      i === currentIndex ? "bg-blue-500 animate-pulse" :
      "bg-zinc-700"
    )} />
    {i < stages.length - 1 && (
      <div className={cn(
        "h-0.5 w-8",
        i < currentIndex ? "bg-green-500" : "bg-zinc-700"
      )} />
    )}
  </div>
))}

Paired with a time estimate — "Usually 2–3 min for a 30 min video" — users know exactly where they are and roughly when to expect results.

Bonus — Fixing the clip drag glitch

While we were in the code, we fixed a nasty drag interaction bug. Our clip timeline let users drag handles to adjust start and end times, but every pixel of movement triggered a React state update and a database PATCH. The video player jumped erratically and the UI felt laggy.

The fix was ref-based tracking:

const dragValueRef = useRef(currentTime);

const onMouseMove = (e: MouseEvent) => {
  dragValueRef.current = pixelToTime(e.clientX);
  debouncedSeek(dragValueRef.current);
};

const onMouseUp = () => {
  setClipTime(dragValueRef.current);
  updateClipOnServer(dragValueRef.current);
};

By keeping the drag value in a ref and only committing on mouseup, we eliminated re-renders during the drag and reduced API calls from dozens to exactly one.

Takeaways

Loading copy matters. Whimsical, rotating messages humanize the wait and reduce perceived latency.
Show progress structure, not just a bar. Stepped indicators give users a mental model of where they are in the pipeline.
Use refs for drag interactions. React state during mousemove is a performance trap — keep the hot loop in a ref and sync state on commit.

If you're building a video or media processing tool and want to skip straight to the fun part, give ClipCrafter a try — drop in a YouTube link and get clips in minutes.

Built with Next.js, TypeScript, Inngest, and Tailwind CSS.

Teaching an AI Agent to Do TDD (And Actually Mean It)

nareshipme — Tue, 07 Apr 2026 16:32:50 +0000

One of the more interesting problems I've been sitting with lately is this: how do you make an AI coding agent disciplined?

Not capable — that part is mostly solved. But disciplined. The kind of discipline that stops a developer from writing five tests at once before any of them are green. The kind that insists every acceptance criterion has a named test before you call the story done. That's the hard part.

The Problem With "Just Tell It"

My first instinct was to write a rule: "follow TDD." Simple enough. But vague instructions produce vague compliance. The agent would write a test, write all the implementation, then nod along when you asked if it did TDD. Technically yes. Practically no.

The fix was to get specific about the atomic unit. Not "write tests first" but "write exactly one failing test, make it pass, commit — then and only then write the next test." That single constraint changes the whole rhythm. It forces small commits, it surfaces design issues early, and crucially, it's checkable. Either the commit history shows alternating test/impl commits or it doesn't.

Vertical Slices as Test Scenarios

The other shift was in how stories are broken down into tasks. Instead of horizontal layers — "write the DB migration, then the service layer, then the controller, then the UI" — I moved to vertical slices expressed as BDD scenarios. Each task looks like:

Given [starting state]
When [user or system action]
Then [observable outcome]
Hint: [which layer to touch and roughly how]

This matters because it keeps the agent (or any developer, really) anchored to user-observable behavior rather than implementation shape. The hint gives just enough technical direction without over-constraining the solution. And because the scenario doubles as a test case, there's no gap between "task complete" and "test written."

Traceability Before You Merge

The last piece was AC traceability. Before a PR gets created, the agent now has to produce an explicit mapping: Acceptance Criterion → test(s) that cover it. If a criterion has no test, the story isn't done. Full stop.

This sounds obvious but it's surprisingly easy to skip when you're moving fast. Making it a gate — not a suggestion — changes the default. The PR description ends up with a little table that reviewers can actually use.

Takeaway

The pattern here applies beyond AI agents. Codifying your team's development discipline as explicit, checkable constraints — rather than cultural norms — makes it stick. Norms drift. Checklists don't.

If you're working with AI coding assistants, the investment in tight agent configuration pays back fast. The more precise the rules, the less you spend reviewing slop.

How We Replaced Polling With Server-Sent Events for Real-Time Video Processing Updates

nareshipme — Mon, 06 Apr 2026 15:42:22 +0000

When you are building a video editing SaaS, one of the trickiest UX problems is keeping users informed about long-running server-side tasks. Transcoding a clip, generating thumbnails, running AI-based scene detection — these jobs can take anywhere from a few seconds to several minutes. Early on at ClipCrafter, we leaned on the simplest approach: polling.

The Polling Problem

Our first implementation looked something like this:

// hooks/useJobStatus.ts — the naive version
export function useJobStatus(jobId: string) {
  const [status, setStatus] = useState<JobStatus>("pending");

  useEffect(() => {
    const interval = setInterval(async () => {
      const res = await fetch(`/api/jobs/${jobId}`);
      const data = await res.json();
      setStatus(data.status);
      if (data.status === "complete" || data.status === "failed") {
        clearInterval(interval);
      }
    }, 2000);
    return () => clearInterval(interval);
  }, [jobId]);

  return status;
}

This worked, but it came with real costs. Every two seconds, every user with an active job was hammering our API. During peak hours we were seeing thousands of unnecessary requests per minute. Our database was fielding repetitive SELECT queries for jobs that had not changed since the last poll. Latency was also mediocre — users could wait up to two seconds after a job finished before the UI updated.

Why We Chose Server-Sent Events Over WebSockets

We evaluated two alternatives: WebSockets and Server-Sent Events (SSE). WebSockets are powerful, but they introduce bidirectional complexity we did not need. Our updates flow in one direction: server to client. SSE gave us exactly that with a simpler protocol, automatic reconnection built into the browser EventSource API, and straightforward compatibility with our Next.js API routes.

The SSE Implementation

On the server side, we created a Next.js Route Handler that streams job updates:

// app/api/jobs/[jobId]/stream/route.ts
import { NextRequest } from "next/server";
import { getJobUpdates } from "@/lib/job-events";

export async function GET(
  req: NextRequest,
  { params }: { params: { jobId: string } }
) {
  const encoder = new TextEncoder();
  const stream = new ReadableStream({
    async start(controller) {
      const send = (data: Record<string, unknown>) => {
        controller.enqueue(
          encoder.encode(`data: ${JSON.stringify(data)}\n\n`)
        );
      };

      // Send current state immediately
      const current = await getJobUpdates(params.jobId);
      send(current);

      // Subscribe to future updates
      const unsubscribe = onJobUpdate(params.jobId, (update) => {
        send(update);
        if (update.status === "complete" || update.status === "failed") {
          controller.close();
        }
      });

      req.signal.addEventListener("abort", () => {
        unsubscribe();
        controller.close();
      });
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      Connection: "keep-alive",
    },
  });
}

The onJobUpdate function subscribes to a lightweight Redis Pub/Sub channel. When our background worker finishes a processing step, it publishes an event and every connected client receives the update within milliseconds.

On the client side, the hook became cleaner and more responsive:

// hooks/useJobStatus.ts — SSE version
export function useJobStatus(jobId: string) {
  const [status, setStatus] = useState<JobStatus>("pending");
  const [progress, setProgress] = useState(0);

  useEffect(() => {
    const source = new EventSource(`/api/jobs/${jobId}/stream`);

    source.onmessage = (event) => {
      const data = JSON.parse(event.data);
      setStatus(data.status);
      setProgress(data.progress ?? 0);
      if (data.status === "complete" || data.status === "failed") {
        source.close();
      }
    };

    source.onerror = () => {
      // EventSource reconnects automatically on transient errors.
      // We only close on terminal failures.
      console.warn("SSE connection interrupted — retrying");
    };

    return () => source.close();
  }, [jobId]);

  return { status, progress };
}

Results After the Switch

The numbers spoke for themselves. API requests related to job status dropped by over 90 percent. Average time-to-UI-update went from around one second down to under 100 milliseconds. Server CPU utilization on our API tier dropped noticeably during peak processing windows, and users started commenting that processing felt faster even though the actual transcoding time had not changed.

Lessons Learned

A few things we picked up along the way. First, always send the current state as the first event in the stream. If a client connects after a job has already progressed, they should not have to wait for the next update to see where things stand. Second, set a reasonable server-side timeout so you do not hold connections open indefinitely for abandoned tabs. We close idle streams after five minutes. Third, use Redis Pub/Sub or a similar lightweight broker rather than polling the database from inside your SSE handler — otherwise you have just moved the polling problem from client to server.

Try It Yourself

If you are building any kind of async processing pipeline in a Next.js app, SSE is a pragmatic middle ground between polling and full WebSocket infrastructure. It took us about a day to migrate and the payoff was immediate.

Want to see this in action? ClipCrafter uses this pattern across all of our video processing workflows. Give it a spin and watch your progress bars update in real time.

Have questions about SSE patterns in Next.js? Drop a comment — happy to dig into edge cases.

Forem: nareshipme

Dev Journal: 2026-04-15

Whisper Hallucination on Silence: Why Your Transcript Loops the Same Phrase

Whisper Hallucination on Silence: Why Your Transcript Loops the Same Phrase

The Problem

What's Happening

1. Silence Gets Interpreted as Speech

2. Background Music Triggers Loops

3. No Confidence Threshold

The Workaround: Use Indian-Language-First Models

Long-Term Fix: Add VAD

The Lesson

Sources

Dev Journal: 2026-04-14

Dev Journal: 2026-04-13

Dev Journal: 2026-04-12

Building a Robust Real-Time Transcription Pipeline in Next.js: STT, Streaming, and Error Recovery

Building a Robust Real-Time Transcription Pipeline in Next.js: STT, Streaming, and Error Recovery

The Problem

Architecture

Layer 1: Client-Side Streaming

Layer 2: API Route with Queueing

Layer 3: Inngest Worker with Resilience

Key Patterns

1. Retry Logic Without Infinite Loops

2. Streaming Results Early

3. Idempotent Storage

4. Detect Language Dynamically

Performance Tuning

Monitoring

Summary

Dev Journal: 2026-04-10

Fixing yt-dlp in Docker: n-challenge EJS scripts, Deno 2.x, and the player_client=ios cookie trap

Background: what is the n-challenge?

Bug 1: Installing yt-dlp as a curl binary skips the EJS scripts

Bug 2: No JS runtime available for the n-challenge solver

Bug 3: player_client=ios silently drops cookies

Keeping yt-dlp current without rebuilding

Summary of changes

Debugging a Silent Failure: Presigned R2 Uploads and the Inngest Job That Never Ran

The Architecture in 30 Seconds

The Bug: A Missing PATCH

The Fix: Extend the PATCH Route and Send the Key

The Lesson: Presigned URLs Create Invisible Gaps

What Else Landed: Billing Scaffolding (Not Live Yet)

How We Made Waiting Fun: Whimsical Loading States and Stepped Progress in Next.js

The problem

Solution 1 — Rotating whimsical messages

Solution 2 — A stepped progress indicator

Bonus — Fixing the clip drag glitch

Takeaways

Teaching an AI Agent to Do TDD (And Actually Mean It)

The Problem With "Just Tell It"

Vertical Slices as Test Scenarios

Traceability Before You Merge

Takeaway

How We Replaced Polling With Server-Sent Events for Real-Time Video Processing Updates

The Polling Problem

Why We Chose Server-Sent Events Over WebSockets

The SSE Implementation

Results After the Switch

Lessons Learned

Try It Yourself

The Bug: A Missing `PATCH`