The latest articles on Forem by zy j (@zy_j_bc7535dad2d7952ef5e1). https://forem.com/zy_j_bc7535dad2d7952ef5e1 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3773359%2Fcb97336d-e283-495b-8957-e5d653ee1a2e.png https://forem.com/zy_j_bc7535dad2d7952ef5e1 en zy j Tue, 17 Feb 2026 13:04:29 +0000 https://forem.com/zy_j_bc7535dad2d7952ef5e1/i-hit-the-llm-quota-wall-so-i-built-a-router-that-cut-costs-without-killing-ux-4n0f https://forem.com/zy_j_bc7535dad2d7952ef5e1/i-hit-the-llm-quota-wall-so-i-built-a-router-that-cut-costs-without-killing-ux-4n0f <h1> The day my LLM app hit the quota wall (and the boring fix that saved it) </h1> <p>I had one of those "this is fine" moments.</p> <p>My little app was working. People were using it. Then one evening it started failing in the dumbest way:<br> <strong>random errors + slow replies + a bill that didn’t match my mental math.</strong></p> <p>Nothing “mystical” happened. I just hit the quota wall.</p> <p>Here’s what I did next (it’s not fancy, but it’s the first thing I’d ship again).</p> <h2> What the quota wall looks like (in a real product) </h2> <p>It usually shows up as one of these:</p> <p>1) <strong>Hard stops</strong> — requests fail, users bounce.<br> 2) <strong>Soft stops</strong> — latency spikes, retries pile up, UX feels laggy.<br> 3) <strong>Silent drift</strong> — outputs get worse or inconsistent, and you spend time “re-doing” work.</p> <p>If you’re trying to make money, the killer isn’t “the model isn’t smart.”<br> It’s <strong>you can’t predict delivery or cost</strong>.</p> <h2> The boring fix: cheap-by-default + upgrade only when it matters </h2> <p>I stopped calling the expensive model for everything.</p> <p>Instead I used a simple rule:</p> <ul> <li> <strong>Default:</strong> cheap model</li> <li> <strong>Upgrade:</strong> only for user‑facing, high‑stakes, or irreversible steps</li> <li> <strong>Fallback:</strong> if it errors / times out / quota caps, try another provider/model</li> </ul> <p>That’s it.</p> <p>It’s basically how you’d run any service with reliability requirements.</p> <h2> The “if/then” rules I actually use </h2> <p>If I had to explain it to a friend, I’d write it like this:</p> <ul> <li>If the task is <strong>drafting</strong> → cheap model</li> <li>If the task is <strong>final copy users will see</strong> → stronger model</li> <li>If the output must follow a strict format → cheap model first, <strong>upgrade only if it fails validation</strong> </li> <li>If it’s taking longer than X seconds → switch to the fast fallback</li> <li>If the request fails with quota/rate-limit → switch provider immediately</li> </ul> <p>Validation can be dumb:</p> <ul> <li>did it include required fields?</li> <li>did it follow the JSON/schema?</li> <li>is it empty/too short?</li> </ul> <p>You don’t need a PhD. You need a gate.</p> <h2> The only 3 numbers worth tracking (or you’re guessing) </h2> <p>I log these, otherwise I’m just vibes:</p> <ul> <li> <strong>cost per request</strong> (and per user/day)</li> <li> <strong>latency</strong> (p50/p95)</li> <li> <strong>redo rate</strong> (how often I have to rerun / fix / user complains)</li> </ul> <p>If you want a money metric:<br> <strong>cost per useful outcome</strong> (total cost / successful completions).</p> <h2> Quick question (I want replies) </h2> <p>If you’ve shipped anything with LLMs:</p> <ul> <li>Where did you hit the wall first — <strong>cost</strong>, <strong>rate limits</strong>, or <strong>quality drift</strong>?</li> <li>What kind of app is it (writing tool / support / automation / something else)?</li> </ul> <p>Drop your constraints (budget + latency). I’ll reply with the exact routing rules I’d start with.</p> <p>Context that pushed me to write this:<br> <a href="https://dev.to/dor_amir_dbb52baafff7ca5b/i-kept-hitting-the-quota-wall-with-ai-coding-tools-so-i-built-a-router-5lj">https://dev.to/dor_amir_dbb52baafff7ca5b/i-kept-hitting-the-quota-wall-with-ai-coding-tools-so-i-built-a-router-5lj</a></p> ai productivity saas startup