Forem: Andrew

Why We Built Proactive Briefings Instead of Another Dashboard

Andrew — Thu, 16 Apr 2026 00:49:44 +0000

Dashboards are a pull medium. You have to remember to check them, find time to open them, and then interpret what you see. For engineering leaders who are already managing incident queues, planning meetings, and code reviews, that pull rarely happens until something is already wrong. We built AI briefings because we wanted risk visibility to be push.

The dashboard problem

The engineering metrics dashboard has become the default answer to a real problem: how do you give engineering leaders visibility into risk without adding meetings to their calendar? The dashboard promises visibility on demand. The practical reality is that demand rarely materializes until after an incident.

We have talked to dozens of engineering managers who have Koalr, LinearB, or Jellyfish dashboards open in a pinned tab. Most of them check it reactively — after a bad deploy, during a retrospective, when a VP asks why MTTR spiked last week. The dashboard is excellent for those conversations. It is not where risk gets caught before it becomes an incident.

The pattern we kept seeing was this: the information was in the system. The high-risk PR had been scored. The CODEOWNERS gap had been flagged. The SLO burn rate was elevated. But nobody was looking at the dashboard that Monday morning when the deploy queue was filling up.

The pull vs. push problem

Pull (Dashboard)

Requires intent to check
Competes with every other tab
Raw data requires interpretation
No context about what changed since yesterday
Gets checked reactively after incidents

Push (Briefing)

Arrives where the team already is (Slack)
Narrative summary, not raw metrics
Delta-focused — what changed this week
Actionable recommendations, not alerts
Gets read before the deploy queue fills

The design constraint: no alert fatigue

The obvious answer to "the dashboard doesn't get checked" is more alerts. Add a PagerDuty rule for high-risk PRs. Slack-notify on every score above 70. This is the wrong answer. Alert fatigue is already endemic in engineering teams, and adding more low-signal notifications makes engineers trust the channel less, not more.

The design constraint for the briefing was: one message per week, per engineering manager, surfacing only the signals that changed materially. Not every high-risk PR — only the pattern shift. Not every CODEOWNERS gap — only when coverage has dropped enough to matter. Not raw scores — a narrative that tells you what to do with them.

This forced a different architecture than a notification system. A notification system fires on threshold breaches. A briefing synthesizes a week of data into a coherent picture of what the risk landscape looks like now versus what it looked like last week.

What goes into the briefing

The weekly risk briefing is generated by Claude from a structured data payload containing the week's deploy activity. The inputs to the synthesis are:

→ Risk score distribution. How many deploys scored in the safe, moderate, high, and critical ranges this week versus last week. The absolute numbers matter less than the direction.
→ High-risk concentrations. Which services are contributing disproportionately to high-risk scores. A spike in payments-service risk is more actionable than a diffuse increase across 20 services.
→ Signal-level drivers. Which of the 33 signals are contributing most to elevated scores this week. Change entropy up? CODEOWNERS coverage down? Coverage delta deteriorating? Each has a different remediation path.
→ MTTR and incident context. Whether MTTR improved or deteriorated this week, and whether any incidents co-occurred with high-risk deploys — which feeds the model's accuracy signal.
→ Positive signals. Teams or services that had notably low risk scores this week. Surfacing what is working creates a reinforcement mechanism, not just a problem log.

Why LLM synthesis, not templates

The briefing could have been a templated report. Pull the top 3 highest-risk services, list the most common signal contributors, format as bullet points. This would have been faster to build and easier to predict.

We chose LLM synthesis because the value of the briefing comes from narrative coherence — the ability to say "payments-service and auth-service are both elevated this week, and both have CODEOWNERS gaps as the primary driver, which suggests a governance issue rather than a change volume issue." A template cannot make that connection. It can surface the two data points separately, but it cannot synthesize the pattern.

The synthesis also allows the briefing to be appropriately calibrated to context. A week where MTTR improved and risk scores are down is a different briefing than a week where three high-risk deploys shipped on a Friday before a bank holiday weekend. The LLM generates the right emphasis for the actual situation.

Severity classification: critical, warning, info

Each briefing card is classified as critical, warning, or info. This is not determined by the LLM — it is a deterministic classification based on the underlying metrics before synthesis:

Critical

High-risk score concentration above threshold, or incident co-occurrence with high-risk deploys in the same week. Requires action before the next release window.

Warning

A signal trending in the wrong direction that has not yet produced incidents but warrants monitoring. Coverage drift, emerging CODEOWNERS gaps, MTTR regression.

Info

A clean week, or a positive signal worth reinforcing — a team that has maintained low risk scores for three consecutive weeks, or model accuracy trending above target.

The classification is shown first in the briefing so the reader can triage at a glance. An engineering manager receiving a Slack digest at 9am on Monday should be able to determine within 10 seconds whether this week requires immediate action or a quick scan.

What we learned from use

The most consistent feedback we have received from engineering managers using the briefing is that it changed how they start their Monday. Not dramatically — it takes 90 seconds to read — but it means they arrive at the first standup already knowing whether there is a risk concentration to address.

The second most common feedback is about specificity. The briefing names services, names signals, and names the engineers whose PRs are driving elevated scores. Vague reporting ("risk is elevated this week") does not produce action. Specific reporting ("payments-service has the highest change entropy in 90 days, and three of the five contributors this week had no prior file-level expertise in the modified paths") does.

The briefing does not replace the dashboard. For deep investigation, for quarterly review, for explaining a trend to a VP, the dashboard is still the right tool. What the briefing does is ensure that the information in the system gets to the right people at the right time — before the deploy queue fills up, not after the incident report.

The weekly briefing described here is live in Koalr — it runs every Monday and lands in Slack and email. The risk scoring that powers it is free for teams up to 5 contributors. If you want to see what a score looks like on a real PR before committing to anything: koalr.com/live-risk-demo

We Scored 28 Famous Open Source PRs for Deploy Risk

Andrew — Mon, 13 Apr 2026 02:18:54 +0000

TL;DR
The React Hooks PR that changed every React application on earth? Three words in the commit message. One feature flag removed. It scored 91 out of 100 for deploy risk. The Svelte 5 release scored 99. A 65-line TypeScript change scored 79 and silently broke type inference in codebases worldwide. We ran 28 landmark open source pull requests through Koalr's deploy risk model. Here is what we found — and why it matters for the PRs your team ships every week.

The problem with code review
Modern code review answers one question well: is this code correct?

It answers a different question poorly: how likely is this to cause a production incident?

Those are not the same question. A PR can be clean, well-written, and thoroughly reviewed — and still wreck production because it touches a critical path nobody flagged, because the reviewer had twelve other PRs open, or because it is the fourth consecutive revert of a feature that never landed cleanly.

Most teams have no objective signal for the second question. They have green checkmarks.

What deploy risk scoring is
Koalr scores every pull request from 0 to 100 before it merges. The score is built from 36 signals:

Blast radius signals

How many files changed
What services those files belong to
Whether shared libraries or interfaces were modified
CODEOWNERS compliance — did the right people review the right files

Change quality signals

File churn — how recently and how often these files have been modified
Change entropy — how spread across the codebase the diff is
Lines added vs deleted ratio
Test coverage of changed files

Context signals

Reviewer load — how many open PRs each reviewer currently has
Author's recent incident rate
Time since last deploy to the same service
Revert history on the changed file set

History signals

Consecutive reverts of the same feature
Recent incident correlation with this file set
PR age — how long the branch has been open
A score of 0–39 is Low. 40–69 is Medium. 70–89 is High. 90–100 is Critical.

The score does not replace review. It gives reviewers a number to orient around before they start reading.

The experiment
We pulled 28 of the most consequential pull requests in open source history and ran them through the model. These are PRs the industry knows by name — the ones that shipped features used by millions of developers, or broke them.

Here is what the model said.

The obvious ones scored as expected

Svelte 5 release https://github.com/sveltejs/svelte/pull/13701 — score 99

The full runes rewrite merged to main. Thousands of files changed, the entire reactivity model replaced, years of migration work consolidated into one merge. Of course it scored critical. High blast radius, enormous file count, fundamental architecture change. The model does what you would expect.

TypeScript modules conversion https://github.com/microsoft/TypeScript/pull/51387 — score 98

Microsoft's conversion of the entire TypeScript compiler codebase from namespaces to ES modules. It touched every source file in the compiler, changed the build system, and dropped dependencies. If any PR in history deserved a mandatory all-hands review before merge, it was this one.

The surprising ones — small diffs, enormous blast radius
This is where it gets interesting.

React PR #14679 "Enable hooks!" https://github.com/facebook/react/pull/14679 — score 91

The commit message is three words. The diff is the removal of a single feature flag. You could read the entire change in thirty seconds.

It scored 91.

Why? Because the model does not count lines — it looks at what the changed code controls. A feature flag in a framework used by tens of millions of applications is not a small change. It is a detonation switch. The blast radius is every React application on earth. The model flagged it correctly.

Signals fired:
  blast_radius_score: 0.97
  feature_flag_detected: true
  downstream_consumers: critical
  reviewer_load: 0.2 (core team — low load)

Final score: 91 / Critical

Node.js PR #41749 "lib: add fetch" https://github.com/nodejs/node/pull/41749 — score 82

One file changed: the bootstrap script that runs inside every Node.js process. Adding the global fetch API touched the most critical execution path in the runtime.

Single-file PR. High score. The file changed is what matters, not how many files changed.

TypeScript PR #57465 "Infer type predicates from function bodies" https://github.com/microsoft/TypeScript/pull/57465 — score 79

65 lines of new code. One function modified.

Those 65 lines changed type inference behavior across the entire checker, producing new type errors in codebases that had compiled cleanly for years. A reviewer looks at 65 lines, sees clean code, approves it. The model sees that those 65 lines live inside the type checker core and have cross-cutting effects on every downstream consumer.

This is the failure mode standard review misses every time.

The revert pattern

Next.js PR #45196 https://github.com/vercel/next.js/pull/45196 — score 88

Title: "Revert 'Revert 'Revert 'Revert 'Initial metadata support''"''

PR body: "Hopefully last time."

Four consecutive reverts of the same feature. The model has a specific signal for this: repeated churn on the same file set with revert commits in recent history. It is one of the strongest predictors of another rollback. The PR scored 88 before anyone read a single line of the diff.

The one that surprised us most

The Jest-to-Vitest migration in tRPC — PR #3688 https://github.com/trpc/trpc/pull/3688 — scored 67. Medium risk.

At first glance, that sounds about right for a test runner swap. But look at what actually changed: every single test file in the repository, plus the root configuration, plus the CI pipeline. The surface area was enormous.

The score was “only” 67 because the risk model correctly identified that none of the changed files were production code paths — only test infrastructure. A test runner change cannot break a production deployment directly. What it can do is make future regressions invisible, which is a subtler and harder-to-measure risk.

The model is honest about what it can and cannot see. Broken test infrastructure does not score as a deploy risk — it scores as a coverage risk. Different signal, different response.

The score table
Here are eight of the 28 PRs we scored, with the risk level and the primary reason for the score:

What this means for your PRs
The open source examples are useful because they are public and well-documented. But none of those teams needed a risk model — the React core team was reviewing the hooks PR. It still would have scored 91.

The real value is the ordinary PR your team ships on a Thursday afternoon, reviewed by one person in fifteen minutes, that quietly introduces a breaking change nobody caught. That team does not have the React core team. They have two engineers, a Monday morning deadline, and a PR that looks fine.

That is who Koalr is built for.

Try it
The live risk demo at koalr.com/live-risk-demo scores any public GitHub PR in seconds. No account, no install. Paste a URL, get a score.

If you want to score your own team's PRs — every PR, automatically, as part of your GitHub workflow — there is a free trial at app.koalr.com/signup .

We Scored 28 Famous Open Source PRs for Deploy Risk — Here's What We Found

Andrew — Mon, 13 Apr 2026 01:35:54 +0000

The React Hooks PR that changed every React application on earth? Three words in the commit message. One feature flag removed. It scored 91 out of 100 for deploy risk. The Svelte 5 release scored 99. A 65-line TypeScript change scored 79 and silently broke type inference in codebases worldwide.

We ran 28 landmark open source pull requests through Koalr's deploy risk model. Here is what we found — and why it matters for the PRs your team ships every week.

The problem with code review

Modern code review answers one question well: is this code correct?

It answers a different question poorly: how likely is this to cause a production incident?

Most teams have no objective signal for the second question. They have green checkmarks.

What deploy risk scoring is

Koalr scores every pull request from 0 to 100 before it merges, built from 36 signals across four categories:

Blast radius — files changed, services affected, CODEOWNERS compliance, shared library modifications

Change quality — file churn, change entropy, lines added vs deleted, test coverage of changed files

Context — reviewer load, author's recent incident rate, time since last deploy, revert history on the changed file set

History — consecutive reverts of the same feature, recent incident correlation, PR age

0–39 → Low
40–69 → Medium
70–89 → High
90–100 → Critical

The score does not replace review. It gives reviewers a number to orient around before they start reading.

The obvious ones scored as expected

Svelte 5 release — score 99

The full runes rewrite merged to main. Thousands of files changed, the entire reactivity model replaced, years of migration work consolidated into one merge. High blast radius, enormous file count, fundamental architecture change. The model does what you would expect.

TypeScript modules conversion — score 98

Microsoft's conversion of the entire TypeScript compiler from namespaces to ES modules. Touched every source file in the compiler, changed the build system, dropped dependencies. If any PR in history deserved a mandatory all-hands review before merge, it was this one.

The surprising ones — small diffs, enormous blast radius

This is where it gets interesting.

React PR #14679 "Enable hooks!" — score 91

The commit message is three words. The diff is the removal of a single feature flag. You could read the entire change in thirty seconds.

It scored 91.

The model does not count lines — it looks at what the changed code controls. A feature flag in a framework used by tens of millions of applications is not a small change. It is a detonation switch. The blast radius is every React application on earth.


json
{
  "blast_radius_score": 0.97,
  "feature_flag_detected": true,
  "downstream_consumers": "critical",
  "reviewer_load": 0.2
}

Score: 91 / Critical