The latest articles on Forem by Kinan Nasri (@kinannasri). https://forem.com/kinannasri https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3685112%2F24483039-dd41-41c9-a587-b75e984849e4.png https://forem.com/kinannasri en Kinan Nasri Tue, 30 Dec 2025 00:04:11 +0000 https://forem.com/kinannasri/why-averages-lie-p99-latency-is-what-users-actually-feel-4l4p https://forem.com/kinannasri/why-averages-lie-p99-latency-is-what-users-actually-feel-4l4p <p>Most performance dashboards look fine.</p> <p>Average latency: low<br><br> CPU: stable<br><br> Memory: healthy </p> <p>And yet users complain that “the app feels slow”.</p> <p>This usually isn’t a mystery. It’s a metrics problem.</p> <h2> Averages hide pain </h2> <p>Imagine this system:</p> <ul> <li>99% of requests complete in <strong>10ms</strong> </li> <li>1% of requests take <strong>1000ms</strong> </li> </ul> <p>The average latency is ~20ms.<br><br> Looks great on a chart.</p> <p>But 1 out of every 100 users experiences a <strong>full second</strong> pause.</p> <p>That’s not an edge case — that’s a real user.</p> <h2> Tail latency is the real signal </h2> <p>This is why percentiles matter:</p> <ul> <li> <strong>p50</strong> tells you what’s typical </li> <li> <strong>p90 / p95</strong> show rising variability </li> <li> <strong>p99</strong> shows where systems actually break down </li> </ul> <p>If your p99 is bad, your system feels bad — even if averages look perfect.</p> <h2> Jitter makes it worse </h2> <p>High variance is often more damaging than slow performance.</p> <p>A stable 40ms system feels faster than one that jumps between 5ms and 200ms.</p> <p>That variability — <strong>jitter</strong> — is what makes UIs stutter, audio glitch, and frames drop.</p> <h2> A small tool to see this clearly </h2> <p>I wanted a way to analyze raw timing data without dashboards, heavy dependencies, or full observability stacks.</p> <p>So I built <strong>Latency Lens</strong> — a small CLI tool focused on:</p> <ul> <li>p50 / p90 / p95 / p99</li> <li>jitter (standard deviation and MAD)</li> <li>spike detection</li> <li>worst-case time windows</li> </ul> <p>No averages-first thinking. Just tail behavior.</p> <p>GitHub:<br><br> <a href="https://github.com/KinanNasri/latency-lens" rel="noopener noreferrer">https://github.com/KinanNasri/latency-lens</a></p> <p>PyPI:<br><br> <a href="https://pypi.org/project/latency-lens/" rel="noopener noreferrer">https://pypi.org/project/latency-lens/</a></p> <h2> The takeaway </h2> <p>If users say your system feels slow:</p> <ul> <li>Don’t look at averages </li> <li>Look at p99 </li> <li>Look at variance </li> <li>Look at spikes over time </li> </ul> <p>That’s usually where the truth is.</p> performance python systems opensource