The latest articles on Forem by Rich Robertson (@rich_robertson). https://forem.com/rich_robertson https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3839458%2Fd65f70e9-3030-41ef-a774-e248437d62bb.jpeg https://forem.com/rich_robertson en Rich Robertson Thu, 02 Apr 2026 17:50:48 +0000 https://forem.com/rich_robertson/how-i-built-a-retrieval-backed-chatbot-to-replace-my-resume-screening-step-131n https://forem.com/rich_robertson/how-i-built-a-retrieval-backed-chatbot-to-replace-my-resume-screening-step-131n <blockquote> <p><em>This is a condensed cross-post. <a href="https://www.myrobertson.com/blog/how-i-built-the-askrich-chatbot-for-technical-screening" rel="noopener noreferrer">Read the full version on my site</a>.</em></p> </blockquote> <p>Recruiter conversations have a consistent problem: a resume gives breadth, but not the system details behind the impact statements. I built <strong>AskRich</strong> to close that gap — a chatbot that lets hiring teams ask specific technical questions and get citation-backed answers grounded in my actual portfolio and writing.</p> <h2> What it does </h2> <p>AskRich is optimized for realistic recruiter workflows:</p> <ul> <li>One-click prompt chips for common questions (architecture trade-offs, delivery scope, measurable outcomes)</li> <li>A freeform chat input for custom questions</li> <li> <strong>Citations attached to every answer</strong> — so a recruiter can verify the source instead of taking generated output at face value</li> <li>A lightweight, dependency-free web UI in plain JavaScript</li> </ul> <p>The citation-backed output is the core product decision. Once someone can see <em>where</em> an answer comes from, they tend to ask sharper follow-ups — and the conversation gets more useful faster.</p> <h2> Architecture overview </h2> <p>At a high level: a thin web client over a retrieval-backed chat API.<br> <a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fr6r42efpgpap15k9lema.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fr6r42efpgpap15k9lema.png" alt=" " width="800" height="1200"></a></p> <p>The Worker supports three runtime modes: <code>upstream</code> (proxy to retrieval API), <code>local</code> (built-in corpus), and <code>openai</code> (direct model path with retrieval-aware constraints). This lets me test and route independently without redeploying the client.</p> <h2> The part that actually made it better: a feedback loop </h2> <p>The first version was anecdotal. I'd notice a weak answer, edit something, and hope for the best.</p> <p>The current version records structured events for every question, answer, and thumbs-up/thumbs-down interaction — all linked by stable event IDs. That lets me triage a specific low-rated answer with its exact question text, citation count, latency, and backend mode instead of debugging in the abstract.</p> <p>Triage classifies failures into four buckets:</p> <ul> <li> <strong>Corpus gap</strong> — the content just isn't there</li> <li> <strong>Retrieval/ranking issue</strong> — the right content exists but isn't surfaced</li> <li> <strong>Prompt/format issue</strong> — generation quality or response clarity</li> <li> <strong>Out-of-scope</strong> — the question type needs routing or a guardrail</li> </ul> <p>Changes are tested, compared against a baseline, and only promoted when they improve answer quality without regressing citation clarity.</p> <h2> Rate limiting at the edge </h2> <p>Rate limiting is enforced in the Cloudflare Worker before any chat execution. Client identity is derived from a one-way hash of request context (IP + origin + user-agent) — raw IPs aren't stored as persistent identifiers.</p> <p>Two guards run in sequence: an hourly quota and a burst interval. If either is exceeded, the API returns <code>429</code> with <code>Retry-After</code>. If KV storage is unavailable, the limiter degrades gracefully (fail-open) to preserve availability.</p> <h2> What I'd do next </h2> <ul> <li>Tighten citation quality metrics and regression gating for high-frequency questions</li> <li>Promote successful A/B retrieval/prompt variants into default behavior</li> <li>Expand corpus gap-closing using the weekly triage workflow</li> </ul> <p><strong><a href="https://www.myrobertson.com/ask-rich.html" rel="noopener noreferrer">Try AskRich →</a></strong></p> <p>Ask it about architecture decisions, migration strategy, or platform delivery outcomes.</p> <p><em><a href="https://www.myrobertson.com/blog/how-i-built-the-askrich-chatbot-for-technical-screening" rel="noopener noreferrer">Full write-up with architecture diagrams and implementation detail on my site.</a></em></p> distributedsystems architecture webdev career Rich Robertson Mon, 23 Mar 2026 07:00:01 +0000 https://forem.com/rich_robertson/backpressure-in-distributed-systems-stability-correctness-and-graceful-degradation-2b4f https://forem.com/rich_robertson/backpressure-in-distributed-systems-stability-correctness-and-graceful-degradation-2b4f <h2> Backpressure in Distributed Systems: Stability, Correctness, and Graceful Degradation </h2> <p>Most distributed systems do not fail because average traffic is a little too high. They fail when work arrives faster than it can be completed, queues grow faster than they can drain, and retries or fan-out make the original problem worse.</p> <p>That is why backpressure matters. It is not just a throughput tweak or a queue setting. It is the control mechanism that lets a system say “not now” before it collapses.</p> <p>Distributed systems rarely fail because average demand is slightly above average capacity. They fail when arrival rates outrun service rates, queues accumulate faster than they can drain, and retries or fan-out amplify the initial disturbance. In that regime, backpressure is not a minor throughput optimization; it is the control mechanism that keeps throughput, latency, and resource usage inside a stable operating envelope (IBM, 2026; Reactive Streams, 2022).</p> <h2> Backpressure as Flow-Control Feedback </h2> <p>Backpressure is best modeled as feedback that propagates downstream capacity limits upstream. Producers are not permitted to emit unbounded work; instead, the system signals when demand must be delayed, reduced, or rejected. Reactive Streams formalizes this requirement for asynchronous boundaries, where unchecked producers would otherwise force receivers to buffer unbounded data (Reactive Streams, 2022).</p> <p>The same control principle applies to RPC chains, event pipelines, brokered messaging, and service meshes.</p> <h2> Queueing Stability, Not Just Performance Tuning </h2> <p>Queueing fundamentals explain why this matters: when offered load persistently exceeds effective service capacity, queue length and response time become nonlinear near saturation. IBM guidance highlights the practical implication: as utilization approaches limits, latency rises sharply, and backlog itself becomes a failure source through memory pressure, timeout cascades, and degraded dependency behavior (IBM, 2026; Amazon Web Services, 2022).</p> <p>This is why unbounded queues are hazardous. They often postpone visible failure while silently increasing tail latency and resource debt. AWS reliability guidance recommends failing fast and limiting queues specifically to avoid insurmountable backlog states (Amazon Web Services, 2022). A bounded queue is therefore not a concession; it is an explicit overload signal that enables timely corrective action.</p> <h2> Autoscaling Cannot Replace Immediate Overload Control </h2> <p>Autoscaling is necessary but temporally mismatched with burst-driven overload. Requirements-driven studies of microservice autoscaling show that generic threshold-based policies can react slowly and allocate capacity suboptimally under dynamic workloads (Nunes et al., 2024). Scale-out decisions require observation windows, control decisions, scheduling, and startup latency. Overload can materialize much faster through retries, correlated bursts, and dependency regressions.</p> <p>Stable systems therefore combine medium-timescale capacity adaptation with millisecond-timescale flow control.</p> <h2> Operational Backpressure Primitives </h2> <p>Four mechanisms recur across robust architectures.</p> <ul> <li> <strong>Admission control</strong> limits accepted work to protect critical resources under contention.</li> <li> <strong>Bounded buffering</strong> exposes pressure instead of concealing it.</li> <li> <strong>Load shedding</strong> discards lower-priority work so critical paths stay within service objectives.</li> <li> <strong>Adaptive concurrency</strong> adjusts in-flight work based on observed latency, making admission limits responsive to current system stress rather than static guesses (Amazon Web Services, n.d.-a; Amazon Web Services, n.d.-b; Netflix, 2025).</li> </ul> <p>Recent practice adds policy awareness to these primitives. Netflix reports service-level-prioritized shedding that preserves high-value requests while trimming less critical traffic only when needed. The objective is not indiscriminate reduction; it is selective reduction aligned with user and business criticality (Gancarz, 2024).</p> <h2> Asynchronous Pipelines and Backlog Liability </h2> <p>Event-driven topologies improve decoupling and failure isolation, but they also enable producers to outpace consumers for prolonged intervals. Reactive Streams treats this as a first-order correctness constraint for asynchronous processing (Reactive Streams, 2022). AWS operational guidance reaches the same conclusion in queue-backed systems: durability benefits collapse if backlog growth is unmanaged during spikes or partial outages (Amazon Web Services, n.d.-b).</p> <h2> Fairness and Isolation in Shared Infrastructure </h2> <p>Backpressure is also an isolation mechanism. In multi-tenant systems, aggregate stability is insufficient if one tenant or request class can consume disproportionate shared capacity. AWS links admission control and rate limiting directly to fairness and predictable performance in shared environments (Amazon Web Services, n.d.-a). Effective overload control therefore enforces both platform protection and workload isolation.</p> <h2> From Thresholds to Closed-Loop Regulation </h2> <p>A consistent industry direction is visible: fixed thresholds are yielding to closed-loop regulation driven by observed latency, queue depth, concurrency, and service objectives. Autoscaling research, adaptive concurrency control, and prioritized shedding all converge on the same systems insight: stability under load must be actively governed (Nunes et al., 2024; Netflix, 2025; Gancarz, 2024).</p> <p>The engineering conclusion is straightforward. A system is not meaningfully scalable unless it can refuse or defer work when required. Designs that can only accept additional load are not resilient; they are temporarily lucky. Backpressure is what converts capacity uncertainty into disciplined behavior and graceful degradation (IBM, 2026; Reactive Streams, 2022; Nunes et al., 2024).</p> <p>I write more about distributed systems, platform architecture, and production engineering at my site:<br> <a href="https://www.myrobertson.com" rel="noopener noreferrer">https://www.myrobertson.com</a></p> <h2> References </h2> <ul> <li>Amazon Web Services. (2022). REL05-BP04: Fail fast and limit queues. AWS Well-Architected Framework.</li> <li>Amazon Web Services. (n.d.-a). Fairness in multi-tenant systems. Amazon Builders’ Library.</li> <li>Amazon Web Services. (n.d.-b). Avoiding insurmountable queue backlogs. Amazon Builders’ Library.</li> <li>Gancarz, R. (2024, November 23). Netflix rolls out service-level prioritized load shedding to improve resiliency. InfoQ.</li> <li>IBM. (2026). WebSphere Application Server performance cookbook: Statistics. IBM.</li> <li>Netflix. (2025). concurrency-limits [Software repository]. GitHub.</li> <li>Nunes, J. P. K. S., Nejati, S., Sabetzadeh, M., &amp; Nakagawa, E. Y. (2024). Self-adaptive, requirements-driven autoscaling of microservices. ACM/ArXiv.</li> <li>Reactive Streams. (2022). Reactive Streams 1.0.4.``</li> </ul> distributedsystems backend architecture reliability