Forem: Aakash Gour

I Tested OpenAI, Anthropic, and Cohere for Bulk Content Generation. Here's What the Data Actually Shows.

Aakash Gour — Thu, 16 Apr 2026 06:30:05 +0000

My content pipeline needed to process 10,000 articles a month.
I had three serious API options: OpenAI, Anthropic, and Cohere.

Every comparison article I found online was either two years old, benchmarked on toy examples, or written by someone with a vendor relationship. So I ran my own.

Three weeks, 4,200 test requests, one specific use case: bulk content generation at production scale. Here's what happened.

My Evaluation Criteria (And Why These Specific Metrics)
Before I get to the numbers, let me be clear about what I was optimizing for. "Best LLM API" is meaningless. Best for my use case is what I cared about:

Output quality on structured content — I needed articles with consistent heading structure, tone, and word count. Not just fluent text.

Cost per 1,000 words — At 10K articles/month, a $0.002 difference per article is $20/month. A $0.02 difference is $200/month.
Latency (p50 and p95) — The p95 matters more than p50 for bulk work. One slow request holds up a queue.

Instruction adherence — If I say "use h2 headers, not h3," does it actually do that across 1,000 requests? Or does it drift?
Error rate over volume — Rate limit errors, context errors, malformed responses. What breaks at scale?

I didn't test: coding tasks, reasoning, math, or anything multimodal. Those benchmarks exist everywhere. This one doesn't.

The Test Setup
Same prompt, same word count target, same structural requirements, across all three providers. I wrote a simple Node.js harness to run the tests and log results to a SQLite database.

import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";
import { CohereClient } from "cohere-ai";
import Database from "better-sqlite3";

const db = new Database("benchmark.db");

db.exec(`
  CREATE TABLE IF NOT EXISTS results (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    provider TEXT,
    model TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    latency_ms INTEGER,
    cost_usd REAL,
    heading_count INTEGER,
    word_count INTEGER,
    error TEXT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
  )
`);

async function runBenchmark(provider, generateFn, prompt) {
  const start = Date.now();

  try {
    const result = await generateFn(prompt);
    const latency = Date.now() - start;

    // Count h2 headings to measure instruction adherence
    const headings = (result.text.match(/^## /gm) || []).length;
    const words = result.text.split(/\s+/).length;

    db.prepare(`
      INSERT INTO results (provider, model, prompt_tokens, completion_tokens, 
                           latency_ms, cost_usd, heading_count, word_count)
      VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    `).run(
      provider,
      result.model,
      result.promptTokens,
      result.completionTokens,
      latency,
      result.cost,
      headings,
      words
    );

    return { success: true, latency };
  } catch (err) {
    db.prepare(`
      INSERT INTO results (provider, model, error) VALUES (?, ?, ?)
    `).run(provider, "unknown", err.message);

    return { success: false, error: err.message };
  }
}

I ran 1,400 requests against each provider — enough to get stable percentiles and surface intermittent errors. The prompt asked for a 600-word article with exactly 3 ## sections and a specific tone. Straightforward structural requirements. The kind of thing you'd run 10,000 times.

The Numbers

At 10,000 articles a month (averaging 600 words each), the cost difference between gpt-4o and gpt-4o-mini is roughly $167/month. That's not nothing.

The p95 numbers are where things get interesting. Cohere's command-r-plus had the highest p95 latency by a significant margin — nearly 2x OpenAI's gpt-4o and almost 4x Anthropic's claude-haiku-4-5. For synchronous use cases this would be painful. For queued bulk generation it's manageable, but you need to account for it in your timeout settings.

Claude Haiku had the best p95 of any capable model. If latency matters more than cost in your use case, that's worth noting.

Instruction Adherence
This is the metric nobody else was measuring. I asked for exactly 3 ## sections in every request.

Claude Sonnet followed structural instructions the most consistently. This surprised me — I expected the output quality gap between Sonnet and Haiku to be larger than the instruction adherence gap. It wasn't. Haiku drifted noticeably more on structure.

Cohere's models had the most drift. command-r would frequently add extra sections or collapse two sections into one. For casual content this is fine. For template-driven content pipelines where downstream parsing depends on consistent structure, it's a problem.

At 10,000 requests/month, a 2.8% error rate means 280 failed generations that need retries. That's not catastrophic, but it's a cost: retry logic, queue overhead, and the occasional job that fails three times and needs manual intervention.

OpenAI and Anthropic both had error rates under 1% in my test. Cohere's error rate was high enough that I'd budget for retry infrastructure before relying on it at scale.

Output Quality: The Part That's Hard to Put in a Table
I spot-checked 150 outputs across providers — 50 per provider, sampled across model tiers. I evaluated them on:

Tone consistency with the prompt
Logical flow between sections
Avoidance of filler phrases ("In conclusion...", "It's important to note...")
Whether the content was actually useful or just plausible-sounding

The honest assessment:
Claude Sonnet produced the most editable drafts. The structure was clean, the tone held throughout, and it was less prone to the kind of filler-heavy conclusions that make AI content feel generic. If I was generating content that humans would lightly edit before publishing, Sonnet gave editors the least work.

GPT-4o was close behind. Slightly more verbose, occasionally padded, but strong structural instincts and good default tone. If you're already in the OpenAI ecosystem and using the Assistants API, there's no compelling reason to switch just for content generation.

Claude Haiku surprised me on quality given its cost. The outputs weren't Sonnet-level, but they were significantly better than I expected from a model at that price point. For high-volume, lower-stakes content (product tags, meta descriptions, brief blurbs), Haiku is underrated.

Cohere command-r-plus had the most inconsistent quality. Some outputs were excellent. Others had structural problems or tonal drift mid-article. For human-reviewed content pipelines this is manageable. For automated pipelines where content goes straight to a CMS, the variance is a real issue.

GPT-4o-mini was fine. Not inspiring. Solid enough for use cases where you're generating high volumes of content that gets human reviewed anyway. At its price point, the quality-per-dollar ratio is hard to beat.

The Retry Handler I Ended Up Writing
Every provider needs retry logic. Here's the one I landed on after testing various approaches:

async function generateWithRetry(generateFn, prompt, options = {}) {
  const {
    maxRetries = 3,
    baseDelay = 1000,
    maxDelay = 30000,
  } = options;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await generateFn(prompt);
    } catch (err) {
      const isRateLimit =
        err.status === 429 ||
        err.message?.includes("rate limit") ||
        err.message?.includes("too many requests");

      const isRetryable = isRateLimit || err.status >= 500;

      if (!isRetryable || attempt === maxRetries) {
        throw err;
      }

      // Exponential backoff with jitter
      // Without jitter, retrying clients hit the API in waves and cause more rate limits
      const exponential = baseDelay * Math.pow(2, attempt);
      const jitter = Math.random() * 0.3 * exponential;
      const delay = Math.min(exponential + jitter, maxDelay);

      console.log(
        `Attempt ${attempt + 1} failed (${err.status}). Retrying in ${Math.round(delay)}ms...`
      );
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

The jitter is important. I learned this the hard way: without it, rate-limited requests all retry at the same interval, which creates another burst that triggers another rate limit. Jitter spreads the retry load.

What Can Go Wrong (And Did)
Rate limits hit differently at scale than in testing. My test harness ran requests at a controlled rate. In production, queue drains aren't that clean — you get bursts when a lot of jobs land at once. I hit OpenAI rate limits in production that I never hit in testing because of this. Solution: implement a token bucket limiter, not just a fixed delay between requests.

Instruction adherence degrades with longer prompts. The 3-section test used a clean, short prompt. When I added more context (brand guidelines, examples, negative constraints), adherence dropped across all models. Claude Sonnet held up best under prompt complexity. GPT-4o-mini degraded the most.

Cohere's context window handling is different. A few of my requests hit a content filtering response that wasn't a standard API error — it returned a 200 with a specific response body structure. My generic error handler missed it and logged a successful request with garbled output. Read Cohere's error documentation more carefully than I did.

Cold starts are real with Anthropic's API. A small percentage of Haiku requests (roughly 2-3% in my data) had latencies 3-5x higher than normal. Not errors — just slow. I don't know if this is model loading, infrastructure, or something else, but it showed up consistently enough to affect the p95.

My Recommendation (Conditional, As It Should Be)
For high-volume, cost-sensitive generation where content gets human review: gpt-4o-mini or command-r. The cost savings are significant. The quality gap is real but acceptable if humans are in the loop.

For high-volume, automated pipelines where structure consistency matters: claude-haiku-4-5. Best p95 latency, solid instruction adherence, reasonable cost. The quality is better than the price suggests.

For lower-volume, higher-quality generation that feeds into editorial workflows: claude-sonnet-4-5. The instruction adherence and output editability are worth the cost premium when you're generating content that humans will touch.

For Cohere: If you have a specific reason to use it (enterprise contract, data residency requirements, a use case where Command-R performs unusually well for your specific domain), fine. For general content generation benchmarked against my criteria, it didn't compete with the OpenAI and Anthropic options.

One thing I'd do differently: I didn't test Anthropic's prompt caching for repeated system prompts. For bulk content generation where you're sending the same lengthy system prompt with each request, caching can significantly reduce input token costs. That's the next benchmark I'm running.

What's your use case? I'm curious whether the instruction adherence numbers match what others are seeing — especially if you're doing high-volume structured generation. Different domains might surface different failure modes than content generation did.

How I Built a Keyword-to-Blog-Post Pipeline in Python (Under 50 Lines)

Aakash Gour — Mon, 13 Apr 2026 11:24:34 +0000

I had a list of 40 keywords and needed a blog post for each one.
Writing them manually would take two weeks.
Writing a script to generate them took one afternoon — and 47 lines of Python.
Here's exactly how I built it.

This isn't a tutorial about AI being magic. It's a tutorial about the specific, unsexy plumbing you need to turn a keyword into a structured, usable blog post — with retry logic, output formatting, and a folder of .md files you can actually work with.

By the end of this, you'll have a working script that:

Takes a list of keywords from a .txt file
Sends a structured prompt to the OpenAI API
Parses and saves each response as a Markdown file
Handles rate limit errors without crashing

Why this is harder than it looks
The naive version is 5 lines:

That works exactly once, in a Jupyter notebook, for a demo.
In practice, you hit three problems immediately:

Rate limits. OpenAI's default tier for gpt-4o is 3 requests per minute. Try to fire 40 at once and you'll get a RateLimitError on request 4.
Unstructured output. "Write a blog post" gets you anything from a 200-word paragraph to a 3,000-word essay with inconsistent headers. If you're using this content anywhere, you need predictable structure.
No persistence. If the script crashes on keyword 22, you've lost the first 21. You need to write each output to disk as it completes.

The 47-line version handles all three.

Prerequisites

Python 3.9+
An OpenAI API key
openai library: pip install openai

That's it. No frameworks, no databases, no Docker.

The setup
Create this file structure:

Your keywords.txt should look like this — one keyword per line:

The script
Here's generate.py in full:
import os
import time
from pathlib import Path
from openai import OpenAI, RateLimitError

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

OUTPUT_DIR = Path("output")
OUTPUT_DIR.mkdir(exist_ok=True)

PROMPT_TEMPLATE = """Write a blog post about: "{keyword}"

Use this exact structure:

[Title]

Introduction

[2-3 sentences introducing the topic]

[Section 1 heading]

[3-4 sentences]

[Section 2 heading]

[3-4 sentences]

[Section 3 heading]

[3-4 sentences]

Conclusion

[2-3 sentences wrapping up with a practical takeaway]

Tone: conversational and practical. Avoid fluff. Total length: ~400 words."""

def slugify(keyword: str) -> str:
return keyword.lower().replace(" ", "-").replace("/", "-")

def generate_post(keyword: str, retries: int = 3) -> str:
for attempt in range(retries):
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(keyword=keyword)}],
max_tokens=700,
temperature=0.7,
)
return response.choices[0].message.content
except RateLimitError:
wait = 20 * (attempt + 1) # back off: 20s, 40s, 60s
print(f" Rate limit hit. Waiting {wait}s before retry {attempt + 1}/{retries}...")
time.sleep(wait)
raise RuntimeError(f"Failed to generate post for '{keyword}' after {retries} retries.")

def run(keywords_file: str = "keywords.txt"):
keywords = Path(keywords_file).read_text().strip().splitlines()
print(f"Processing {len(keywords)} keywords...\n")

for i, keyword in enumerate(keywords, 1):
    output_path = OUTPUT_DIR / f"{slugify(keyword)}.md"

    if output_path.exists():
        print(f"[{i}/{len(keywords)}] Skipping '{keyword}' — already generated.")
        continue

    print(f"[{i}/{len(keywords)}] Generating: '{keyword}'")
    content = generate_post(keyword)
    output_path.write_text(content, encoding="utf-8")

    # OpenAI free tier: ~3 requests/minute. This keeps us just under.
    time.sleep(22)

print("\nDone. Check the output/ folder.")

if name == "main":
run()

What each part is actually doing
The prompt template is doing the real work here. "Write a blog post" is too open-ended — the model will vary wildly in length and structure. The template locks in a specific H2 structure, a word count target, and a tone instruction. Your output becomes predictable enough to actually use.

The retries loop in _generate_post _uses exponential-ish backoff — 20s, 40s, 60s — because OpenAI's rate limit errors are temporary. Most of the time, waiting 20 seconds is enough. The retry loop means the script keeps running instead of crashing and forcing you to restart.

The if output_path.exists(): continue check is the most important line for long runs. If you're processing 100 keywords and the script dies at #73, you don't want to regenerate the first 72. This check skips already-completed files and resumes from where you left off.

The time.sleep(22) at the bottom of the loop is tuned for the free tier rate limit. If you're on a paid OpenAI tier with higher limits, you can drop this to time.sleep(2) or remove it entirely and let the retry logic handle any occasional errors.

Running it

Output:

Your output/ folder now has three .md files, each with consistent structure:

What can go wrong

RateLimitError even with the sleep. This happens if you've been running the script multiple times in the same minute. The rate limit is per-minute across all your requests, not just this script. Fix: increase time.sleep(22) to time.sleep(30) if you're hitting it consistently.

The model ignores your structure prompt. This happens more with gpt-3.5-turbo than gpt-4o. If you switch models to reduce cost, test with 5 keywords first and inspect the output structure before running it on your full list.

You get back an empty string. Rare, but it happens. The generate_post function returns the raw content string — add a check after the call: if not content.strip(): raise ValueError(...) to catch and flag empty responses before they get written to disk as empty files.

Special characters in keywords break the filename. The slugify function handles spaces and forward slashes, but if your keywords have apostrophes, colons, or question marks, you'll get OS-level errors. Add .replace("'", "").replace(":", "").replace("?", "") to the slugify function if your keyword list is user-generated.

What I'd add next

This script is intentionally minimal — 47 lines, no external dependencies beyond openai. But if you're running this in production for more than a few hundred keywords, here's what breaks next:

Cost tracking. Add a token counter so you know what each run costs before you've spent $40 without noticing.
Quality validation. A second API call that checks whether the output meets a minimum quality bar (does it have all the required sections? Is it close to the target word count?). This sounds expensive but catching bad outputs early is cheaper than rewriting them manually.

Concurrent requests. The serial approach (one keyword at a time, sleep 22 seconds) is slow. With asyncio and a proper rate limiter, you can process 3 keywords simultaneously and cut wall-clock time by ~60%.

I'll cover the async version in the next post.

The full script on GitHub

The complete source, with a requirements.txt and a sample keywords.txt to test with, is at:

Drop a star if it saved you time.

Have you hit rate limits running keyword pipelines at scale? What's your retry strategy — and are you running requests serially or concurrently? Curious what others have landed on.