Forem: CodeRabbit

Show me the prompt: What to know about prompt requests

Arindam Majumder — Thu, 29 Jan 2026 05:34:00 +0000

In the 1996 film Jerry Maguire, Tom Cruise’s famous phone call, where he shouts “Show me the money!” cuts through everything else. It’s the moment accountability enters the room.

In AI-assisted software development, “show me the prompt” should play a similar role.

As more code is generated by large language models (LLMs), accountability does not disappear. It moves upstream. The question facing modern engineering teams is not whether AI-generated code can be reviewed, but where and how review should happen when intent is increasingly expressed before code exists at all.

The Twitter debate: Prompts versus pull requests

Earlier this week, Gergely Orosz of Pragmatic Engineer shared a quote on Twitter (or X, if you prefer) from an upcoming podcast with Peter Steinberger, creator of the self-hosted AI agent Clawdbot.

Steinberger’s point was straightforward but provocative: as more code is produced with LLMs, traditional pull requests may no longer be the best way to review changes. Instead, he suggested, reviewers should be given the prompt that generated the change.

That idea quickly triggered a polarized response.

Supporters argued that reviewing large, AI-generated diffs is becoming increasingly impractical.

From their perspective, the prompt captures intent more directly than the output. It tells reviewers what the developer was trying to accomplish, what constraints they set, and what scope they intended. In addition, a prompt can be re-run or adjusted, which makes it easier to validate the approach without combing through thousands of lines of generated code.

Critics, however, pointed to issues that prompts alone do not solve: determinism, reproducibility, git blame, and legal accountability.

Because LLM outputs can vary across runs, models, and configurations, approving a prompt does not necessarily mean approving the exact code that ultimately ships. For audits, ownership, and downstream liability, that distinction matters. In their view, code review cannot be replaced by “prompt approval” without weakening the guarantees that PR-based workflows were designed to provide.

The core disagreement, then, is not whether prompts should be part of review. It is where accountability should live in an AI-assisted workflow: primarily in the prompt, primarily in the code, or in a deliberately structured combination of both.

What is a prompt request?

A prompt request is exactly what it sounds like: a request by a developer for a peer review of their prompt before feeding it into an LLM to generate code. Or, in the case of multi-shot or conversational prompts, a review of the conversation between the developer and the agent.

Instead of starting review at the diff level, a prompt request asks reviewers to evaluate the instructions given to the LLM so they can sign off on or contribute to the context, intent, constraints, and assumptions that guide the model’s output. A typical prompt request may include:

The system and user prompts
Relevant repository or architectural context
Model selection and configuration
Constraints, invariants, or non-goals
Examples of expected behavior

The goal is to make explicit what the model was asked to do before evaluating how well it did it.

In this sense, a prompt request functions more like a design artifact than a code artifact. It captures intent at the moment of generation and helps ensure the prompt is comprehensive and explicit enough to address the requirements. It can help teams better align around how they prompt and ensure that everyone is using the same context to generate code.

Good news: Prompt requests and pull requests are not in conflict

Much of the debate this week stemmed from treating prompt requests and pull requests as competitors. Either you do a prompt request or a pull request, some commenters suggested.

However, they shouldn’t be.

After all, they address different failure modes at different stages of the development lifecycle. Just like you’re not going to skip testing because you did a code review, you shouldn’t skip a code review because you did a prompt request.

Prompt requests are valuable because they ensure alignment and best practices early, before any code is generated or committed. They help teams align on what should be built, define boundaries, and constrain agent behavior. Because large language models are non-deterministic, capturing intent explicitly becomes even more important upstream, where variability is highest.

A prompt request can also help ensure that a prompt is optimized for the specific model or tool that will be used to generate the code, something essential in ensuring the quality of the output of increasingly divergent models (something we’ve consistently found in our evals).

Pull requests remain essential later, when teams review the exact code that will ship. They preserve determinism, traceability, testing, auditing, and accountability. One captures intent. The other captures execution.

Treating prompt requests as replacements for pull requests creates a false tension. Used together, they complement each other. Doing a prompt request and then skipping a pull request seems reckless and like tempting fate since the actual code produced hasn’t been validated.

Why teams are drawn to prompt requests

When done as part of the regular software development workflow that includes a thorough code review, prompt requests are a way to shift left and catch issues early. It ensures a team is aligned on the goals of the feature, can help optimize the prompt for the model it’s using, and can ensure that the proper context is being supplied to improve the generated output. This can cut down significantly on review and issues later on.

When used alone without doing a pull request after the code is generated, the primary appeal of prompt requests is cognitive efficiency and speed.

AI has dramatically increased the speed at which developers can produce code, but the review process has not kept pace. As AI-authored changes grow larger and more frequent, line-by-line review becomes increasingly difficult and cognitively taxing to complete. Subtle defects slip through not because engineers don’t care, but because reviewing enormous, machine-generated diffs is mentally taxing.

Prompts, by contrast, are typically shorter and more declarative. Reviewing a prompt allows engineers to reason directly about scope, intent, and constraints without getting buried in implementation details produced by the model.

Prompt-first review works particularly well for:

Scaffolding and boilerplate generation
Small changes
Greenfield prototypes
Fast-moving teams optimizing for iteration speed
Hobby projects where defects in prod aren’t that consequential

In these cases, the most important question is often not “is every line correct?” but “is this what we meant to build?”

Where prompt requests fall short

When used in concert with pull requests, there are few downsides since they simply offer another opportunity to review the proposed code change before generation. The biggest one is the time and cognitive effort it takes and how this could become a new bottleneck for code generation if it takes too long to get a review.

When treated as a replacement for pull requests, the biggest limitation of prompt requests is non-determinism.

After all, the same prompt can produce different outputs across runs or models. That makes reviewing prompts a weak substitute for reviewing an auditable record of what actually shipped. From the perspective of git blame, compliance, or legal accountability, prompt reviews alone are insufficient.

There are also real security and correctness risks. You might think you covered everything in your prompt but it may encode unsafe assumptions, omit edge cases, or fail to account for system-specific constraints that would normally be caught during careful code review. Reviewing intent does not guarantee that the generated output is secure, performant, or compliant.

Finally, prompts are highly contextual. A prompt that looks reasonable in isolation can still produce problematic implementations if the reviewer lacks deep familiarity with the codebase, infrastructure, or runtime environment. While prompt reviews are designed to limit this by bringing in additional sets of eyes to improve the prompt, human reviewers make mistakes all the time on actual code. Add in the unpredictability of a model and that’s a recipe for bugs and downtime These risks increase as prompts are reused or gradually modified over time or if you change models.

Prompt requests work best before pull requests

Used together, prompt requests and pull requests offset each other’s weaknesses.

A practical workflow might look like this:

A developer proposes a prompt request describing the intended change, constraints, and assumptions. This can involve just one prompt or a series of prompts for different parts of the code being generated. In the case of conversational prompts, the dev might propose a conversational response or share their transcript with the LLM after the fact. In that case, the review could help reprompt the agent to generate a better result.
The team reviews and aligns on the prompt(s) before code generation.
The code is generated and committed.
A traditional pull request reviews the concrete output for correctness, safety, and fit.

In this model, prompt requests act as an upstream alignment step for AI-generated work. They reduce ambiguity early, potentially shrink downstream diffs, and make pull requests easier to review.

Prompt requests do not replace the later rigor needed in pull requests. They just add more rigor earlier.

Are prompt requests going to replace pull requests?

Let’s be honest, prompt requests are unlikely to fully replace pull requests. No one thinks a large publicly traded company is going to trust AI-generated output so faithfully, they’ll bet their revenue (and future) on it without careful review.

While we are bullish on prompt requests at CodeRabbit, the industry is still in the early stages of their adoption, and today’s LLMs are not capable of fully replacing pull requests.

Will prompt requests work instead of pull requests for smaller open-source or single-maintainer projects? We are likely heading toward that reality sooner rather than later, but pull requests remain an essential part of the current software development lifecycle. This is especially true for production systems, regulated environments, or large teams with shared ownership and long-lived, complex codebases.

Pull requests exist because software development ultimately involves shipping specific, deterministic artifacts into production. As long as that remains true, teams will need a concrete mechanism to review, test, audit, and approve the exact code that runs.

The more realistic future is not prompt requests instead of pull requests. It is prompt requests before pull requests.

What is becoming clear is that the quality of the prompt increasingly determines the quality of the output. Treating prompts as first-class artifacts acknowledges that reality without abandoning the safeguards that traditional code review provides.

In that sense, “show me the prompt” does not remove accountability. It shifts some of it earlier, where it can reduce rework, surface intent, and make the pull request stage easier rather than unnecessary.

Interested in trying CodeRabbit? Get a 14-day free trial.

Why users shouldn’t choose their own LLM models: Choice is not always good

Arindam Majumder — Thu, 29 Jan 2026 01:29:00 +0000

Giving users a dropdown of LLMs to choose from often seems like the right product choice. After all, users might have a favorite model or they might want to try the latest release the moment it drops.

One problem: unless they’re an ML engineer running regular evals and benchmarks to understand where each model actually performs best, that choice is liable to hurt far more than it helps. You end up giving users what they think they want, while quietly degrading the quality of what they produce with your tool with inconsistent results, wasted tokens, and erratic model behavior.

For example, developers may unknowingly pick a model that’s slower, less reliable for their specific task, or tuned for a completely different kind of reasoning pattern. Or they might choose a faster model than they need that won’t comprehensively reason through the task.

Choosing which model to use isn’t a matter of personal taste… It's a systems-level optimization problem. The right model for any task depends on measurable performance across dozens of task dimensions, not just how recently it was released or how smart users perceive it to be. And that decision should belong to engineers armed with eval data, not end users who wrongly believe they’ll get better results with the model they personally prefer.

The myth of ‘preference’ in AI model selection

Many AI platforms love to market model choice as a premium feature. “Choose GPT-4o, Claude, or Gemini” sounds empowering and gives users the impression that they will get the best or latest experience. It taps into the same instinct that makes people want to buy the newest phone the week it launches: the feeling that newer and bigger must mean better.

The reality, though, is that most users have no idea which model actually performs best for their specific use case. And even if they did, that answer would likely shift from one query to another. The “best” model for code generation might not be the “best” for bug detection, documentation, or static analysis. There might also be multiple models that are best at different parts of a code review or other task, depending on what kind of code is being reviewed.

Some tasks require greater creativity and reasoning depth; others need precision and consistency. A developer who blindly defaults to “the biggest model available” for coding help, often ends up with slower, more expensive, and less deterministic results. In some cases, a smaller, domain-tuned model will handily outperform its heavyweight cousin.

Why model selection is an evaluation problem, not preference

Model selection isn’t a matter of taste… it's a data problem. Behind the scenes, engineers run thousands of evaluations across tasks like code correctness, latency, context retention, and tool integration. These aren’t one-time benchmarks; they’re continuous systems designed to measure how models actually perform under specific, reproducible conditions. The results form a kind of performance map which shows which model excels at refactoring versus summarizing code or which one handles long-context reasoning without drifting off-topic.

End users never see that map. While some might read benchmarks or articles about a model’s performance, most are making decisions blind, guided mostly by hunches, Reddit posts, or vague impressions of “smartness.”

Even if they wanted to, users rarely have the time or infrastructure to run their own evals across hundreds of tasks and models. The result is that people often optimize for hype rather than outcomes… choosing the model that feels cleverest or sounds more fluent, not the one that’s objectively better for the job.

And human perception alone is a terrible way to evaluate model competence. A model that seems chatty and confident can be consistently wrong, while one that feels hesitant might actually deliver the most accurate, reproducible results. Without hard data from evaluations, those distinctions disappear.

The prompting paradox

One critical drawback to choosing your own model is that no two LLMs think alike. Each model interprets prompts slightly differently. Some are more literal, others more associative; some favor verbosity, others prefer minimalism. A prompt that works perfectly on GPT-5 might completely derail on Sonnet 4.5, leading to hallucinated code, missing context, or an output that ignores key constraints.

Temperature, context length, and formatting differences only make the problem worse. A model with a higher temperature parameter might produce creative explanations but rewrite variable names, while another with stricter formatting rules could break markdown or indentation. These small mismatches can quietly poison a workflow, especially in environments where consistent structure matters most like with code reviews, diff comments, or documentation summaries.

When users choose their own models, they unknowingly disrupt the prompt-engineering assumptions that keep those workflows stable in systems where the prompts are written for the user. Every prompt is tuned with certain expectations about how the model parses instructions, handles errors, and formats its output. Swap out the model and those assumptions collapse.

It’s even harder to navigate in situations where the user writes the prompt themselves, like with AI coding tools. Users rarely have enough context, knowledge, and experience to write effective prompts for each model. However, over time, they might find a few prompting methods that help them get the best out of a particular model. If they later change to a new model, they often find their old prompts aren’t as effective and need to learn from scratch trying to get the best results from that new model.

That’s why well-designed systems rely on model orchestration, not user preference. In review pipelines or agentic systems, predictability is everything. You need each component to behave consistently so downstream tools and other models can interpret the results. Giving users the freedom to swap models isn’t customization; it’s chaos engineering without the safety net.

The hidden costs of model freedom

Once users can switch models at will, all the invisible consistency that makes AI-assisted workflows dependable begins to crumble. The consequences aren’t abstract; they’re measurable and they multiply fast.

Across teams, the first thing you notice is inconsistency. Two developers can run the same review prompt and get completely different feedback. One gets a precise diff comment, the other might get a philosophical musing on the meaning of clean code. That inconsistency makes it impossible to reproduce results, which is deadly for any process that relies on traceability or QA.

Then there’s cost. Larger models burn through tokens faster and often respond slower, introducing both financial waste and latency drag. And when users unknowingly pick models with shorter context windows, the result is truncated inputs or missing context. It’s like asking someone to summarize a novel after reading only half of it.

The better alternative: Dynamic, data-driven routing

The smarter alternative to user-driven chaos is dynamic, data-driven routing. That means systems that automatically choose the right model for the right task. Instead of asking users to guess which LLM might perform best, auto-routing engines make that choice in real-time based on metrics, evals, and historical performance.

Think of it as orchestration, not selection. A large model might be routed in for creative reasoning, open-ended problem solving, or complex code explanations. A smaller, domain-tuned model might handle deterministic checks, linting, or static analysis where precision and speed matter more than eloquence. The system continuously evaluates the outcomes tracking correctness, latency, and user feedback in order to refine its routing logic over time.

This approach turns what used to be human guesswork into an adaptive, evidence-based process. The routing system learns which models excel at which tasks, under which conditions, and how to balance cost, speed, and quality.

Advanced teams already operate this way. In CodeRabbit, for example, the orchestration layer sits between the user and the models, using structured prompts, eval data, and performance histories to dispatch requests intelligently. Developers don’t have to think about which LLM is behind a particular review comment. The system has already chosen the optimal one, validated against internal benchmarks.

In short, dynamic routing makes model choice invisible. The user gets consistently high-quality results; the engineers get measurable control and efficiency. Everyone wins. Except the dropdown menu.

Expertise is in the system, not the slider

The takeaway here is simple: model selection isn’t a feature, it’s a quality control issue. The best results come from systems that make those choices invisibly and are grounded in data, not gut instinct. When model routing is automatic and performance-based, users get consistent, high-quality outputs without needing to think about which model is doing the work.

Every product that puts a “Choose your LLM” dropdown front and center is outsourcing an engineering decision to the least equipped person to make it.

Or, put another way: the best AI tool UI is no LLM dropdown at all.

Curious what it looks like when an AI pipeline optimizes for LLM fit? Try CodeRabbit for free today!

An (actually useful) framework for evaluating AI code review tools

Arindam Majumder — Wed, 28 Jan 2026 12:24:44 +0000

Benchmarks promise clarity. They’re supposed to reduce a complex system to a score, compare competitors side by side, and let the numbers speak for themselves. But, in practice, they rarely do.

Benchmarks don’t measure “quality” in the abstract. They measure whatever the benchmark designer chose to emphasize, under the specific constraints, assumptions, and incentives of the evaluation.

Change the dataset, the scoring rubric, the prompts, or the evaluation harness, and the results can shift dramatically. That doesn’t make benchmarks useless, but it does make them fragile, manipulable, and easy to misinterpret. Case in point: database benchmarks.

Database benchmarks: A cautionary tale

The history of database performance benchmarks is a useful example. As benchmarks became standardized, vendors learned how to optimize specifically for the test rather than for real workloads. Query plans were hand-tuned, caching behavior was engineered to exploit assumptions, and systems were configured in ways no production team would realistically deploy.

Over time, many engineers stopped trusting benchmark results, treating them as marketing signals rather than reliable indicators of system behavior.

AI code review benchmarks are on the same trajectory

We’re currently seeing AI code review benchmarks go down a similar path. As models are evaluated on curated PR sets, synthetic issues, or narrowly defined correctness criteria, tools increasingly optimize for benchmark performance rather than for the messy, contextual, high‑stakes reality of real code review.

The deeper problem is not just that benchmarks can be misleading, it’s that many “ideal” evaluation designs are difficult to execute correctly in real engineering environments. When an evaluation framework is too detached from real workflows, too easy to game by badly configuring your competitor’s tool, or too complex to run well, the results become hard to trust.

What follows below is a practical framework for effectively evaluating AI code review tools that balances rigor with feasibility, and produces results that are both meaningful and interpretable.

Start from your objectives and make them explicit

Before assembling datasets or choosing metrics, it’s critical to define what you actually care about. “Better code review” means different things to different teams, and an evaluation that doesn’t encode those differences will inevitably optimize for the wrong outcome.

Common objectives include:

Catching real defects and risks before merge
Improving long‑term maintainability and reducing technical debt
Avoiding low‑value noise that degrades review quality
Maintaining developer trust and adoption

It’s also important to distinguish between leading indicators and lagging indicators. Outcomes like fewer production incidents or higher long‑term throughput are real and important, but they often emerge over months, not weeks. Shorter evaluations should focus on signals that correlate strongly with those outcomes, such as the quality of issues caught, whether they are acted on, and how developers respond to the tool.

Explicitly ranking your objectives such as quality impact, precision, developer experience, and throughput, helps ensure that your evaluation answers the questions that actually matter to your organization.

Determine what kind of evaluation is needed

The most reliable evaluation of any tool involves a real-world pilot over a controlled offline benchmark. This allows you to see how it works in day-to-day situations versus just evaluating a tool based on criteria defined by a third party vendor.

In-the-wild pilot

The most reliable signals come from observing how a tool behaves in real, day‑to‑day development.

Real‑time evaluation reflects actual constraints: deadlines, partial context, competing priorities, and human judgment. It shows not just what a tool can detect in theory, but what it surfaces in practice, and whether those issues matter enough for developers to act on them.

For this, select a few teams or projects for each tool and run each tool for a period of time under normal usage.

Measure things like:

Real-world detection of issues.
Severity of issues caught.
Developer satisfaction and perceived utility.

If possible, design A/B style experiments so you can measure using the tool vs no tool on comparable teams or repos or Tool A vs Tool B on similar workloads, perhaps alternating weeks or branches.

Offline benchmark

For teams that want additional confidence, controlled detection comparisons can provide useful insight if you design it yourself using your own pull requests and criteria so it gives you the data you actually need. However, it’s not required in most cases since it doesn’t provide as much useful data as a pilot and can be time intensive to set up.

One practical approach is to use a private evaluation or mirror repository. A small, representative set of pull requests can be replayed, allowing multiple tools to be run on the same diffs without disrupting real workflows.

These comparisons are best used to understand coverage differences by severity and category, and to identify systematic strengths and blind spots across tools.

After that, you just need to compute the metrics you’re looking to track. For example:

Precision/recall by severity and issue type.
Comment volume and distribution.

Why evaluating multiple tools on the same pull request is usually misleading

If you want to do a head-to-head comparison via either a benchmark or a pilot, a common instinct is to run them all on the same exact pull requests rather than mirroring that PR and running each tool you’re comparing separately on it or running them on different but comparable PRs. On the surface, running them all simultaneously feels fair and efficient. In practice, it introduces serious problems.

When multiple AI reviewers comment on the same PR:

Human reviewers are overwhelmed with feedback and cognitive load spikes.

No single tool can be experienced as it was designed to be used in that case. For example, some tools skip comments if they see another tool has already made that comment leading to the perception that that tool hasn’t found the issue.

Review behavior changes—comments are skimmed, bulk‑dismissed, or ignored

This creates interference effects. Tools influence each other’s perceived usefulness, and attention, not correctness, becomes the limiting factor. Precision metrics degrade because even high‑quality comments may be ignored simply due to volume. That makes it harder to know the percentage of comments your team would accept from each individual tool under normal usage.

The result is that you lose the ability to evaluate usability, trust, workflow fit, and real‑world usefulness. You are no longer measuring how a tool performs in practice, but how reviewers cope with noise.

Running multiple tools on the same exact PR can be useful in narrow, controlled contexts, such as offline detection comparisons, but it is a poor way to evaluate the actual experience and value of a code review tool.

To understand whether a tool helps your team, it often best be experienced in isolation within a normal review workflow.

Structuring fair comparisons without complex infrastructure

There are practical ways to compare tools without building elaborate experimentation harnesses.

Parallel evaluation across repos or teams is often the simplest approach. Select repos or teams that are broadly comparable in language, domain, and PR volume, and run different tools in parallel. Keep configuration effort symmetric and analyze results using normalization techniques (discussed below).

Alternatively, time‑sliced evaluation within the same repo or team can work when parallel groups are not available. Run one tool for a defined period, then switch. This approach requires acknowledging temporal effects—release cycles, workload changes, learning effects—but can still produce useful, directional insights when interpreted carefully.

Finally, simply mirroring PRs and running reviews on them with separate tools also works well, if you want to compare comments on the same PRs.

In all these cases, the goal is to preserve a clean developer experience while collecting comparable data.

In practice, these approaches can also be combined if a team feels like that’s helpful to give them a better idea of how a tool works. Teams may start with parallel evaluation across different repositories or teams, then swap tools after a fixed period. This helps balance differences in codebase complexity or workload over time, while still avoiding the disruption and interference that comes from running multiple tools on the same pull request. As with any time-based comparison, results should be normalized and interpreted with awareness of temporal effects, but this hybrid approach often provides a good balance of fairness, practicality, and interpretability.

Metrics that produce interpretable results

Based on successful deployments across thousands of repositories, we've identified a framework of seven metric categories that provide a complete picture of your integration which we suggest as metrics to measure to our customers.

Each category answers a specific question about your AI implementation:

Architectural Metrics – Is the tool appropriately integrated? How many of an org’s repos are connected, how many extensions are they using (git, IDE, CLI).
Adoption Metrics – Are developers actually using it? These metrics include monthly active users (MAU), the percentage of total repositories covered and week-over-week growth.
Engagement Metrics – Are they just ignoring it or actively collaborating with it? These metrics include PRs reviewed versus Chat Sessions initiated. Also track “Learnings used,” how often the AI applies context from previous reviews to new ones.
Impact Metrics – Is it catching bugs that matter to the team? These metrics include number of issues detected, actionable suggestions, and the “acceptance rate” (percentage of AI comments that result in a code change).
Quality & Security Metrics – Is it preventing expensive bugs and security vulnerabilities? These metrics include Linter/SAST findings, security vulnerabilities caught (e.g., Gitleaks), and reduction in pipeline failures.
Governance Metrics – Is it enforcing standards across the team? These metrics include usage of pre-merge checks, warnings vs. errors, and implementation of custom governance rules.
Developer Sentiment – Are the developers happy with their experience and product? These metrics include survey results, qualitative feedback, and “aha” moments.

Accepted issues as a primary quality signal

Not all metrics are equally informative and some are far easier to misread than others. A practical evaluation should focus more attention on signals that are both meaningful and feasible to measure. One of the strongest indicators of value is whether a tool’s feedback leads to real action.

An issue can reasonably be considered accepted when:

A subsequent commit addresses the comment or thread
A reviewer explicitly acknowledges that the issue has been resolved

This behavioral signal captures correctness, relevance, and usefulness in a way that pure scoring metrics cannot.

Accepted issues should be reported by:

Severity (e.g., critical, major, minor, low, nitpick)
Category (security, logic, performance, maintainability, testing, etc.)

Both absolute counts and rates are informative, especially when interpreted together.

Precision and signal‑to‑noise

Acceptance rate (accepted issues relative to total surfaced) is a practical proxy for precision. On its own, it is insufficient; paired with comment volume, it becomes far more meaningful.

High comment volume with low acceptance is a clear signal of noise. Patterns of systematically ignored categories or directories often reveal where configuration or tuning is needed.

It’s also important to avoid the “LGTM trap.” That means a tool that leaves very few comments, all correct, may appear precise while missing large classes of issues. In many cases, broad coverage combined with configurability is preferable to narrow precision that cannot be expanded.

Coverage and issue discovery in real review flows

In typical workflows, the sequence is:

PR opens → AI review → issues fixed → human review

Because humans review after the tool, it is often impossible to say with certainty which issues humans would have caught independently. Instead of trying to infer counterfactuals precisely, focus on practical signals:

Accepted issues that led to substantive code changes
Accepted issues in categories humans historically miss (subtle logic, edge cases, maintainability)
Consistent patterns of issues surfaced across PRs Sampling can help here. Reviewing a subset of PRs and asking, “Would this issue likely have been caught without the tool?” is often more informative than attempting exhaustive labeling.

Normalization: Making comparisons fair

Raw counts are misleading when pull requests vary widely in size and complexity. Normalization is essential for fair comparison.

Useful normalization dimensions include:

PR size (lines changed, files touched)
PR type (bug fix, feature, refactor, infra/config, test‑only)
Domain or risk area (frontend/backend, high‑risk components)

Comparisons should be made within similar buckets, and distributions are often more informative than averages. Small samples at extremes should be interpreted cautiously.

Interpreting throughput and velocity

Throughput metrics like time‑to‑merge are easy to misread. When a tool begins catching real issues that were previously missed, merge times may initially increase. This often reflects improved rigor rather than reduced productivity.

Throughput should therefore be treated as a secondary metric, normalized by PR complexity and evaluated over time alongside quality indicators. Short‑term slowdowns can be a leading indicator of long‑term gains in code health.

Bringing it all together

A reliable evaluation does not require perfect benchmarks or elaborate experimental design. It requires clarity about objectives, careful interpretation of metrics, and an emphasis on real‑world behavior.

Start with normal workflows and behavioral signals. Normalize to make comparisons fair. Use controlled comparisons selectively to deepen understanding. Combine quantitative metrics with concrete examples of impact.

Final takeaway

Benchmarks are useful starting points, not verdicts.

The most trustworthy evaluations of AI code review tools are grounded in real workflows, user behavior‑based signals, and balance rigor with practicality. When done well, they provide confidence not just that a tool performs well on paper, but that it meaningfully improves both the immediate quality of code changes and the long‑term health of the codebase.

Curious how CodeRabbit performs on your codebase? Get a free trial today!

CodeRabbit's AI Code Reviews now support NVIDIA Nemotron

Arindam Majumder — Wed, 28 Jan 2026 12:17:13 +0000

TL;DR: Blend of frontier & open models is more cost efficient and reviews faster. NVIDIA Nemotron is supported for CodeRabbit self-hosted customers.

We are delighted to share that CodeRabbit now supports the NVIDIA Nemotron family of open models among its blend of Large Language Models (LLMs) used for AI code reviews. Support for Nemotron 3 Nano has initially been enabled for CodeRabbit’s self-hosted customers running its container image on their infrastructure. Nemotron is used to power the context gathering and summarization stage of the code review workflow before the frontier models from OpenAI and Anthropic are used for deep reasoning and generating review comments for bug fixes.

How Nemotron helps: Context gathering at scale

This new blend of open and frontier models allows us to improve the overall speed of context gathering and improves cost efficiency by routing different parts of the review workflow to the appropriate model family, while delivering review accuracy that is at par with running frontier models alone.

High quality AI code reviews that can find deep lying and hidden bugs require lots of context gathering related to the code being analyzed. The most frequent (and most token-hungry) work is summarizing and refreshing that context: what changed in the code and does it match developer intent, how do those changes connect with rest of the codebase, what are the repo conventions or custom rules, what external data sources are available to aid the review, etc.

This context building stage is the workhorse of the overall AI code review process and it is run several times iteratively throughout the review workflow. NVIDIA Nemotron 3 Nano was built for high-efficiency tasks and its large context window (1 million tokens) along with fast speed helps to gather a lot of data and run several iterations of context summarization and retrieval.

Blend of frontier and Open Models

When you open a Pull Request (PR), CodeRabbit’s code review workflow is triggered starting with an isolated and secure sandbox environment where CodeRabbit analyzes code from a clone of the repo. In parallel, CodeRabbit pulls in context signals from several sources:

Code and PR index
Linter / Static App Security Tests (SAST)
Code graph
Coding agent rules files
Custom review rules and Learnings
Issue tickets (Jira, Linear, Github issues)
Public MCP servers
Web search

To dive deeper into our context engineering approach you can check out our blog: The art and science of context engineering for AI code reviews.

A lot of this context, along with the code diff being analyzed, is used to generate a PR Summary before any review comments are generated. This is where open models come in. Instead of sending all of the context to frontier models, CodeRabbit now uses Nemotron Nano v3 to gather and summarize the relevant context. Summarization is at the heart of every code review and is the key to delivering high signal-to-noise in the review comments.

After the summarization stage is completed the frontier models (e.g., OpenAI GPT-5.2-Codex and Anthropic Claude-Opus/Sonnet 4.5) perform deep reasoning to generate review comments for bug fixes, and execute agentic steps like review verification, pre-merge checks, and “finishing touches” (including docstrings and unit test suggestions).

What this means for our customers

CodeRabbit is now enabling Nemotron-3-Nano-30B support (initially for its self-hosted customers) for the context summarization part of the review workflow along with the frontier models from OpenAI and Anthropic. This results in faster code reviews without compromising quality.

We are also delighted to support the announcement from NVIDIA today about the expansion of its Nemotron family of open models and are excited to work with the company to help accelerate AI coding adoption across every industry.

Get in touch with our team to access CodeRabbit’s container image if you would like to run AI code reviews on your self-hosted infrastructure.

What's New in CodeRabbit: January 2026 Edition

Arindam Majumder — Wed, 28 Jan 2026 08:17:14 +0000

January kicked off strong with powerful new APIs for user and metrics management, plus streamlined data export capabilities.

Alongside these product updates, CodeRabbit just won a 2026 DEVIES Award at DeveloperWeek! We're honored to be recognized alongside the best in dev tools for helping teams ship cleaner code, faster.

We're also excited to announce that CodeRabbit's AI code reviews now support NVIDIA Nemotron open models! Self-hosted teams get faster, more cost-efficient reviews without sacrificing quality.

// Detect dark theme var iframe = document.getElementById('tweet-2008303896863928826-670'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2008303896863928826&theme=dark" }

With that context, here's everything we shipped in January 2026:

User Management API (Jan 21)

Managing user seats and roles at scale is now possible through our REST API. Whether you're onboarding a new team or adjusting access across departments, you can now automate the entire process.

What you can do:

List all users in your organization with filters for seat status and role
Bulk assign or unassign up to 500 seats per request
Promote or demote users between admin and member roles in bulk
Get detailed success/failure feedback for each operation

All endpoints support partial success; if one user operation fails, the rest still complete.

See the User Management API documentation for authentication and usage details.

Data Export (Jan 14)

Export your pull request review metrics directly from the CodeRabbit Dashboard. Pick your date range, and download a CSV with complexity scores, review times, and comment breakdowns by severity and category.

Perfect for quarterly reviews, team retrospectives, or building custom analytics dashboards. Access it from the Data Export tab in your dashboard.

See the Data Export documentation for the complete list of fields.

Review Metrics API (Jan 14)

Need programmatic access to your review metrics? The new REST API gives you the same data as the dashboard export, but with full query flexibility. Filter by repository, user, or custom date ranges, and get responses in JSON or CSV format.

What you can do:

Query metrics for any date range programmatically
Filter results by specific repositories or users
Choose JSON for integration or CSV for analysis
Build custom reporting dashboards and automation workflows

See the Metrics API documentation for authentication and usage details.

Stay Tuned

We’re continually working to make CodeRabbit smarter, faster, and more collaborative. More updates are on the way, stay tuned!

Got feedback or want early access to what’s next?

Join us on Discord or follow @coderabbitai on X.

Our new report: AI code creates 1.7x more problems

Arindam Majumder — Tue, 06 Jan 2026 09:53:09 +0000

What we learned from analyzing hundreds of open-source pull requests.

Over the past year, AI coding assistants have gone from emerging tools to everyday fixtures in the development workflow. At many organizations, a part of every code change is now machine-generated or machine-assisted.

But while this has been accelerating the speed of development, questions have been quietly circulating:

Why are more defects slipping through into staging?
Why do certain logic or configuration issues keep appearing?
And are these patterns tied to AI-generated code?

It would appear like AI is playing a significant role. A recent report found that while pull requests per author increased by 20% year-over-year, thanks to help from AI, incidents per pull request increased by 23.5%.

This year also brought several high-visibility incidents, postmortems, and anecdotal stories pointing to AI-written changes as a contributing factor. These weren’t fringe cases or misuses. They involved otherwise normal pull requests that simply embedded subtle mistakes. And yet, despite rapid adoption of AI coding tools, there has been surprisingly little concrete data about how AI-authored PRs differ in quality from human-written ones.

So, CodeRabbit set out to answer that question empirically in our State of AI vs Human Code Generation Report.

Our State of AI vs Human Code Generation Report

We analyzed 470 open-source GitHub pull requests, including 320 AI-co-authored PRs and 150 human-only PRs, using CodeRabbit’s structured issue taxonomy. Every finding was normalized to issues per 100 PRs and we used statistical rate ratios to compare how often different types of problems appeared in each group.

The results? Clear, measurable, and consistent with what many developers have been feeling intuitively: AI accelerates output, but it also amplifies certain categories of mistakes.

READ THE FULL REPORT

Limitations of our study

Getting data on issues that are more prevalent in AI-authored PRs is critical for engineering teams but the challenge was determining which PRs were AI-authored vs human authored. Since it was impossible to directly confirm authorship of each PR of a large enough OSS dataset, we checked for signals that a PR was co-authored by AI and assumed that those that didn’t have it were human authored, for the purposes of the study.

This resulted in statistically significant differences in issue patterns between the two datasets, which we are sharing in this study so teams can better know what to look for. However, we cannot guarantee all the PRs we labelled as human authored were actually authored only by humans. Our full methodology is shared at the end of the report.

Top 10 findings from the report

No issue category was uniquely AI but most categories saw significantly more errors in AI-authored PRs. That means, humans and AI make the same kinds of mistakes. AI just makes many of them more often and at a larger scale.

1. AI-generated PRs contained ~1.7× more issues overall.

Across 470 PRs, AI-authored changes produced 10.83 issues per PR, compared to 6.45 for human-only PRs. Even more striking: high-issue outliers were much more common in AI PRs, creating heavy review workloads.

2. Severity escalates with AI: More critical and major issues.

AI PRs show ~1.4–1.7× more critical and major findings.

3. Logic and correctness issues were 75% more common in AI PRs.

These include business logic mistakes, incorrect dependencies, flawed control flow, and misconfigurations. Logic errors are among the most expensive to fix and most likely to cause downstream incidents.

4. Readability issues spiked more than 3× in AI contributions.

The single biggest difference across the entire dataset was in readability. AI-produced code often looks consistent but violates local patterns around naming, clarity, and structure.

5. Error handling and exception-path gaps were nearly 2× more common.

AI-generated code often omits null checks, early returns, guardrails, and comprehensive exception logic, issues tightly tied to real-world outages.

6. Security issues were up to 2.74× higher

The most prominent pattern involved improper password handling and insecure object references. While no vulnerability type was unique to AI, nearly all were amplified.

7. Performance regressions, though small in number, skewed heavily toward AI.

Excessive I/O operations were ~8× more common in AI-authored PRs. This reflects AI’s tendency to favor clarity and simple patterns over resource efficiency.

8. Concurrency and dependency correctness saw ~2× increases.

Incorrect ordering, faulty dependency flow, or misuse of concurrency primitives appeared far more frequently in AI PRs. These were small mistakes with big implication

9. Formatting problems were 2.66× more common in AI PRs.

Even teams with formatters and linters saw elevated noise: spacing, indentation, structural inconsistencies, and style drift were all more prevalent in AI-generated code.

10. AI introduced nearly 2× more naming inconsistencies.

Unclear naming, mismatched terminology, and generic identifiers appeared frequently in AI-generated changes, increasing cognitive load for reviewers.

READ THE FULL REPORT

Why these patterns appear

Why are teams seeing so many issues with AI-generated code? Here’s our analysis:

AI lacks local business logic: Models infer code patterns statistically, not semantically. Without strict constraints, they miss the rules of the system that senior engineers internalize.
AI generates surface-level correctness: It produces code that looks right but may skip control-flow protections or misuse dependency ordering.
AI doesn’t adhere perfectly to repo idioms: Naming patterns, architectural norms, and formatting conventions often drift toward generic defaults.
Security patterns degrade without explicit prompts: Unless guarded, models recreate legacy patterns or outdated practices found in older training data.
AI favors clarity over efficiency: Models often default to simple loops, repeated I/O, or unoptimized data structures.

What engineering teams can do about it

Adopting AI coding tools isn’t simply about speeding up development. It requires rethinking the guardrails that ensure all code entering production is safe, maintainable, and correct.

Based on the patterns in the data, here are the most important takeaways for teams:

1. Give AI the context it needs

AI makes more mistakes when it lacks business rules, configuration patterns, or architectural constraints. Provide prompt snippets, repo-specific instruction capsules, and configuration schemas to reduce misconfigurations and logic drift.

2. Use policy-as-code to enforce style

Readability and formatting were some of the biggest gaps. CI-enforced formatters, linters, and style guides eliminate entire categories of AI-driven issues before review.

3. Add correctness safety rails

Given the rise in logic and error-handling issues:

Require tests for non-trivial control flow
Mandate nullability/type assertions
Standardize exception-handling rules
Explicitly prompt for guardrails where needed

4. Strengthen security defaults

Mitigate elevated vulnerability rates by centralizing credential handling, blocking ad-hoc password usage, and running SAST and security linters automatically.

5. Nudge the model toward efficient patterns

Offer guidelines for batching I/O, choosing appropriate data structures, and using performance hints in prompts.

6. Adopt AI-aware PR checklists

Reviewers should explicitly ask:

Are error paths covered?
Are concurrency primitives correct?
Are configuration values validated?
Are passwords handled via the approved helper?

These questions target the areas where AI is most error-prone.

7. Get help reviewing and testing AI code

Code review pipelines weren’t created to handle the higher volume of code teams are currently shipping with the help of AI. Reviewer fatigue has been found to lead to more issues and missed bugs. An AI code review tool like CodeRabbit helps by standardizing code reviews acts as a third-party source of truth that standardizes quality across different AI tools that teams might use while reducing the time and cognitive labor needed for reviews. That allows developers to concentrate on reviewing the more complex parts of the code changes and reduce the amount of bugs and issues that end up in production.

READ THE FULL REPORT

The bottom line

AI coding tools are powerful accelerators, but acceleration without guardrails increases risk. Our analysis shows that AI-generated code is consistently more variable, more error-prone, and more likely to introduce high-severity issues without the right protections in place.

The future of AI-assisted development isn’t about replacing developers. It’s about building systems, workflows, and safety layers that amplify what AI does well while compensating for what it tends to miss.

For the teams that want the speed of AI without the surprises, the data is clear: Quality isn’t automatic. It requires deliberate engineering. Even when using AI tools.

An AI code review tool could also help. Try CodeRabbit today.

It's harder to read code than to write it (especially when AI writes it)

Arindam Majumder — Tue, 06 Jan 2026 09:43:32 +0000

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

Brian Kernighan (co-creator of Unix and co-author of The C Programming Language)
I've been programming since I was ten. When it became a career, I got obsessed with code quality: clean code, design patterns, all that good stuff. My pull requests were polished like nobody's business: well-thought-out logic, proper error handling, comments, tests, documentation. Everything that makes reviewers nod approvingly.

Then, LLMs came along and changed everything. I don't write that much code anymore since AI does it faster. Developer’s work now mainly consists of two parts: explaining to a model what you need, then verifying what it wrote, right? I’ve become more of a code architect and quality inspector rolled into one.

And here came a problem I knew all too well from my years as a tech lead:

READING CODE IS ACTUALLY HARDER THAN WRITING IT.

As an open-source maintainer and senior developer, I had to review tons of other people's code, and I learned what Kernighan said the hard way. Reading unfamiliar code is exhausting. You have to reverse-engineer someone else's thought process, figure out why they made certain decisions, and consider edge cases they might have missed.

With my own code, reviewing and adjusting were a no-brainer. I designed it, I wrote it, and the whole mental model was still fresh in my head. Now the code is coming from an LLM and suddenly reviewing "my own code" has become reviewing someone else's code. Except this "someone else" writes faster than I can think and doesn't take lunch breaks.

AI is supposed to help, but if I want to ship production-grade software now, I actually have more hard work to do than before. The irony!

And that’s why, for my first blog post since joining CodeRabbit, I wanted to focus on that fact. This is also, incidentally, why I decided to join CodeRabbit. But we’ll get to that part later.

We’re human (unfortunately for code quality)

Here's where things get uncomfortable: we're human beings, not code-reviewing machines. And human brains don't want to do hard work, thoroughly reviewing something that a) already runs fine, b) passes all the tests, and c) someone else will review anyway. It's so much easier to just git commit && git push and go grab that well-deserved coffee. Job is done!

I went from “writing manually and shipping quality code,” to “generating code fast but shipping… bad code!” The quality dropped not because I had less time as I actually had MORE time since I wasn't typing everything myself. I just tend to “shorten” this verification phase, telling myself "it works, the tests pass, the team will catch anything major."

The problem with "Catching it in review"

At this point, I was already using CodeRabbit to review my team's pull requests (as an OSS-focused dev, I was an early adopter), and those reviews were genuinely helpful! CodeRabbit would catch things that slipped through. Security issues, edge cases, some logic bugs. Those problems that are easy to miss when you're moving fast.

But here's the thing: those reviews were coming too late. The code was already pushed. Already in the repository, visible to the entire team. Sure, CodeRabbit would flag the issues and I'd fix them but not before my teammates had seen my AI-generated code with obvious problems that I didn't bother to review properly.

That's not a great look when you've spent decades building a reputation for quality.

Enter: CodeRabbit in an IDE

Then, I discovered CodeRabbit had an IDE extension. The AI code reviewer I was already using for PRs could also review my code locally, before anything hits the repo. This was exactly what I needed.

When I ask CodeRabbit to check or simply stage my changes, CodeRabbit reviews them right in VS Code, catching issues before git push. Now, my team sees only the polished version, just like the old days. Except now, I'm shipping AI-generated code at AI speeds. And I’m doing it with actual quality control. Automatic reviews mean no willpower required: I don't have to remember to run it, I don't have to open a separate tool. It just happens at commit time. Reviewing doesn't feel like plowing in the rain anymore.

This gets critical when you're looking at potential security headaches, like the one on the screenshot. CodeRabbit caught an access token leak that could've been a total disaster! Issues like this needs to be addressed before that code gets pushed to a repository.

More than that, when it finds something, the fixes are committable. The tool doesn’t tell me to "go figure it out" but gives actual suggestions I can apply immediately, in one click.

For more advanced cases that can’t be resolved with a simple fix, CodeRabbit IDE extension writes a prompt that it sends to an AI agent of your choice. Fun fact: CodeRabbit is so good in writing prompts so I got a lot to learn from, improving my Prompt Engineering skills!

Even the free CodeRabbit IDE Review plan offers incredibly helpful feedback and catches numerous issues. However, the Pro plan unlocks its true power, providing the same comprehensive coverage you expect from regular CodeRabbit Pull Request reviews: tool runs, Code Graph analysis, and much more - there is a huge infrastructure behind every check!

The bottom line

Brian Kernighan was right: reading code is harder than writing it. That was true in 1974 and it's even more true now when AI can generate 300 lines while you're still thinking about a variable name.

We thought AI would make our jobs easier. And it does… if you only count the writing. But the reading verifying, reviewing, and understanding what the AI agent actually built? That got harder.

Many of us are doing 10x the volume at 10x the speed, which means 10x more code to read with the same human brain that gets lazy and wants coffee breaks. The solution isn't to slow down or go back to typing everything manually. The solution is to automate the code review process as thoroughly as we automated the code writing process. If your AI writes the code, another AI should be reading it before you get to it.

The quality of the reviews is why I recently transitioned from being a CodeRabbit user to joining the team. And that’s why you should also try CodeRabbit in your IDE. The free tier means there's basically no excuse not to try it. Your reputation will thank you.

Get started today with a 14-day free trial!

Behind the curtain: What it really takes to bring a new model online at CodeRabbit

Arindam Majumder — Tue, 06 Jan 2026 09:38:11 +0000

When we published our earlier article on why users shouldn't choose their own models, we argued that model selection isn't a matter of preference, it's a systems problem. This post explains exactly why.

Bringing a new model online at CodeRabbit isn't a matter of flipping a switch; it's a multi-phase, high-effort operation that demands precision, experimentation, and constant vigilance.

Every few months, a new large-language model drops with headlines promising “next-level reasoning,” “longer context,” or “faster throughput.” For most developers, the temptation is simple: plug it in, flip the switch, and ride the wave of progress.

We know that impulse. But for us, adopting a new model isn’t an act of curiosity, it’s a multi-week engineering campaign.

Our customers don’t see that campaign, and ideally, they never should. The reason CodeRabbit feels seamless is precisely because we do the hard work behind the scenes evaluating, tuning, and validating every model before it touches a single production review. This is what it really looks like.

1. The curiosity phase: Understanding the model’s DNA

Every new model starts with a hypothesis. We begin by digging into what it claims to do differently: is it a reasoning model, a coding model, or something in between? What’s its architectural bias, its supposed improvements, and how might those capabilities map to our existing review system?

We compare those traits against the many model types that power different layers of our context-engineering and review pipeline. The question we ask isn’t, “is this new model better?” but, “where might it fit?” Sometimes it’s a candidate for high-reasoning diff analysis; other times, for summarization or explanation work. Each of those domains has its own expectations for quality, consistency, and tone.

From there, we start generating experiments. Not one or two, but dozens of evaluation configurations across parameters like temperature, context packing, and instruction phrasing. Each experiment feeds into our evaluation harness, which measures both quantitative and qualitative dimensions of review quality.

2. The evaluation phase: Data over impressions

This phase takes time. We run models across our internal evaluation set, collecting hard metrics that span coverage, precision, signal-to-noise, and latency. These are the same metrics that underpin the benchmarks we’ve discussed in earlier posts like Benchmarking GPT-5, Claude Sonnet 4.5: Better Performance, but a Paradox, GPT-5.1: Higher signal at lower volume, and Opus 4.5: Performs like the systems architect.

But numbers only tell part of the story. We also review the generated comments themselves by looking at reasoning traces, accuracy, and stylistic consistency against our current best-in-class reviewers. We use multiple LLM-judge recipes to analyze tone, clarity, and helpfulness, giving us an extra lens on subtle shifts that raw metrics can’t capture.

If you’ve read our earlier blogs, you already know why this is necessary: models aren’t interchangeable. A prompt that performs beautifully on GPT-5 may completely derail on Sonnet 4.5. Each has its own “prompt physics.” Our job is to learn it quickly and then shape it to behave predictably inside our system.

3. The adaptation phase: Taming the differences

Once we understand where a model shines and where it struggles, we begin tuning. Sometimes that means straightforward prompt adjustments such as fixing formatting drift or recalibrating verbosity. Other times, the work is more nuanced: identifying how the model’s internal voice has changed and nudging it back toward the concise, pragmatic tone our users expect.

We don’t do this by guesswork. We’ll often use LLMs themselves to critique their own outputs. For example: “This comment came out too apologetic. Given the original prompt and reasoning trace, what would you change to achieve a more direct result?” This meta-loop helps us generate candidate prompt tweaks far faster than trial and error alone.

During this period, we’re also in constant contact with model providers, sharing detailed feedback about edge-case behavior, bugs, or inconsistencies we uncover. Sometimes those conversations lead to model-level adjustments; other times they inform how we adapt our prompts around a model’s quirks.

4. The rollout phase: From lab to live traffic

When a model starts to perform reliably in offline tests, we move into phased rollout.

First, we test internally. Our own teams see the comments in live environments and provide qualitative feedback. Then, we open an early-access phase with a small cohort of external users. Finally, we expand gradually using a randomized gating mechanism so that traffic is distributed evenly across organization types, repo sizes, and PR complexity.

Throughout this process, we monitor everything:

Comment quality and acceptance rates
Latency, error rates, and timeouts
Changes in developer sentiment or negative reactions to CodeRabbit comments
Precision shifts in suggestion acceptance

If we see degradation in any of these signals, we roll back immediately or limit exposure while we triage. Sometimes it’s a small prompt-level regression; other times, it’s a subtle style drift that affects readability. Either way, we treat rollout as a living experiment, not a switch-flip.

5. The steady-state phase: Continuous vigilance

Once a model is stable, the work doesn’t stop. We monitor it constantly through automated alerts and daily evaluation runs that detect regressions long before users do. We also listen, both to our own experience (we use CodeRabbit internally) and to customer feedback.

That feedback loop keeps us grounded. If users report confusion, verbosity, or tonal mismatch, we investigate immediately. Every day, we manually review random comment samples from public repots that use us to ensure that quality hasn’t quietly slipped as the model evolves or traffic scales.

6. Why we do all this & why you shouldn’t have to

Each new model we test forces us to rediscover what “good” means under new constraints. Every one comes with its own learning curve, its own failure modes, its own surprises. That’s the reality behind the promise of progress.

Could an engineering team replicate this process themselves? Technically, yes. But it would mean building a full evaluation harness, collecting diverse PR datasets, writing and maintaining LLM-judge systems, defining a style rubric, tuning prompts, managing rollouts, and maintaining continuous regression checks. All of this before your first production review!

That’s weeks of work just to reach baseline reliability. And you’d need to do it again every time a new model launches.

We do this work so you don’t have to. Our goal isn’t to let you pick a model; it’s to make sure you never have to think about it. When you use CodeRabbit, you’re already getting the best available model for each task, tuned, tested, and proven under production conditions.

Because “choosing your own model” sounds empowering until you realize it means inheriting all this complexity yourself.

Takeaway
Model adoption at CodeRabbit isn’t glamorous. It’s slow, meticulous, and deeply technical. But it’s also what makes our reviews consistent, trustworthy, and quietly invisible. Every diff you open, every comment you read, is backed by this machinery. Weeks of evaluation, thousands of metrics, and countless prompt refinements all in service of one thing:

Delivering the best possible review, every time, without you needing to think about which model is behind it.

Try out CodeRabbit today. Get a free 14-day trial!

CodeRabbit's AI Code Reviews now support NVIDIA Nemotron

Arindam Majumder — Tue, 06 Jan 2026 05:14:16 +0000

TL;DR: Blend of frontier & open models is more cost efficient and reviews faster. NVIDIA Nemotron is supported for CodeRabbit self-hosted customers.

How Nemotron helps: Context gathering at scale

CodeRabbit architecture with Nemotron support

Blend of frontier and Open Models

Code and PR index
Linter / Static App Security Tests (SAST)
Code graph
Coding agent rules files
Custom review rules and Learnings
Issue tickets (Jira, Linear, Github issues)
Public MCP servers
Web search

To dive deeper into our context engineering approach you can check out our blog: The art and science of context engineering for AI code reviews.

What this means for our customers

Get in touch with our team to access CodeRabbit’s container image if you would like to run AI code reviews on your self-hosted infrastructure.

Our 10 best posts of the year: A 2025 CodeRabbit blog roundup

Arindam Majumder — Fri, 02 Jan 2026 13:30:49 +0000

This year, we dove deep into all kinds of topics, from the philosophical shift toward “Slow AI” to the practical realities of building with increasingly sophisticated LLM models to why you shouldn’t trust threads with 🚀on vibe coding for code you intend to ship to prod.

Here’s a look back at our most impactful posts from the past year in case you missed them:

1. The end of one-size-fits-all prompts: Why LLM models are no longer interchangeable

Arindam Majumder for CodeRabbit

Oct 24 '25

The end of one-sized-fits-all prompts: Why LLM models are no longer interchangeable

#ai #webdev #programming #productivity

Comments

7 min read

For years, developers could swap LLM models like interchangeable parts but now those days are over. This piece explores how modern AI models have separated in fundamental ways, from reasoning approaches to output formats, making LLM choice a critical product decision rather than a simple configuration change. We break down what this means for developers and why the “one prompt fits all” era is now over.

2. The rise of 'Slow AI': Why devs should stop speedrunning stupid

Arindam Majumder for CodeRabbit

Nov 6 '25

The rise of ‘Slow AI’: Why devs should stop speedrunning stupid

#ai #webdev #programming #productivity

Comments 1

6 min read

Fast isn’t always the way to go. While AI coding tools promise lightning-speed development, this article makes the case for slowing down. We explore why AI tools that take time to reason through problems produce better, more maintainable code than those optimized purely for speed. Drawing on data from a number of studies, we examine the paradox of developer confidence versus actual trust in AI-generated code and why “Slow AI” might be an antidote to technical debt.

3. AI code metrics: What percentage of your code should be AI-generated?

The title is clickbait (we admit it), but the question remains: how do you measure the impact of AI on your codebase? This post challenges the notion that “percentage of AI-generated code” is a meaningful metric. Instead, we explore what engineering teams should actually measure when evaluating AI’s role in their development process, and why focusing on the wrong metrics can lead to dangerous blind spots in code quality.

4. Handling ballooning context in the MCP era: Context engineering on steroids

The Model Context Protocol (MCP) promised easy integration between LLMs and external tools. But in reality, it created a context overload problem. This article tackles the issue of ballooning context windows and how to engineer your way out of them. We explore why MCP’s elegance can become a liability without deliberate context engineering and share strategies for keeping your AI tools sharp and focused rather than drawing in a black hole of data.

5. 2025: The year of the AI dev tool tech stack

When Microsoft and Google both announced that AI generates 30% of their code, it became clear: we’re not talking about single tools anymore, we're talking about stacks. This post explores the emerging ecosystem of layered AI dev tools across the software development lifecycle. From foundational coding assistants to essential code review layers, we map out what a modern AI dev tool stack looks like and share sample configurations teams are using.

6. Why emojis suck for reinforcement learning

👍feels good, but is it teaching your AI reviewer anything? This article explores why emoji-based feedback, while universal, falls short at improving AI performance over time. We break down the simplicity trap and explain which nuanced feedback works to build better AI code reviews. Spoiler: it’s not as simple as a thumbs up or thumbs down.

7. Vibe coding: Because who doesn't love surprise technical debt!?

“Vibe coding,” the practice of prompting AI tools with vibes and hoping for the best is everywhere. And it’s creating technical debt at an unprecedented scale. What happens when developers rely heavily on AI assistants like Claude Code, ChatGPT, and GitHub Copilot without proper processes in place? We dive into the hidden costs of moving fast and breaking things when your entire codebase depends on it.

8. Good code review advice doesn't come from threads with 🚀 in them

Twitter threads promising “10 vibe coding and review tips every dev should know” are everywhere. But here’s the truth: practical code review advice requires full context, nuance, and experience. This blog questions the idea that code review wisdom is distilled into a tweet, from fresh eyes to AI-assisted review layers that understand your specific context.

9. CodeRabbit's Tone Customizations: Why it will be your favorite feature

Ever wish your code reviewer could channel Gordon Ramsay? Or maybe your disappointed mom? We talk about CodeRabbit’s tone customization feature, which lets you adjust how your AI code reviewer communicates, from encouraging and gentle to bgenerated code), share setup instructions, and celebrate the creative ways developers are cusrutally honest. We dive into why tone matters in code review (especially when dealing with AI-tomizing their review experience.

10. CodeRabbit commits $1 million to open source

Open source is the foundation of modern software development, from package managers to frameworks to the infrastructure we all depend on. This post announced CodeRabbit’s $1 million USD commitment to open-source software sponsorships, reflecting our gratitude for what open source enables and our ongoing support for the developers and projects that power the ecosystem we all build on.

The bottom line: Our blog rocks, you should read it weekly in 2026
Each of these blogs represents a piece of the larger conversation about how AI is reshaping software development. We hope these insights will help you ship better code, refine your AI development setup, tackle context engineering challenges, or simply avoid technical debt from "vibe-coding."

Try out CodeRabbit today with a 14-day free trial.

Measuring what matters in the age of AI-assisted development

Arindam Majumder — Fri, 02 Jan 2026 13:13:06 +0000

Every engineering leader I talk to is asking the same question: "Is AI actually making us better?"

Not "are we using AI" (everyone is). Not "is AI generating code" (it clearly is). And not even, “What percentage of our code is AI generating?” (unless you’re Google or Microsoft and announce this publicly). The real question is whether AI adoption is translating into shipping faster, better quality code and making for more productive and happier engineering teams.

The problem is that most tooling gives you vanity metrics. Lines of code generated. Number of AI completions accepted. These tell you nothing about what happens after the AI writes code. Does it survive review? Does it ship? Does it break production?

CodeRabbit sits at a unique vantage point in the development lifecycle. We review both human-written and AI-generated code. We see what gets flagged, what gets accepted, and what makes it to merge. We watch how teams iterate, how reviewers respond, and where friction accumulates. We’ve been able to see all those things and knew there was value to that for teams, as well.

So, today, we are releasing a new analytics dashboard that puts this visibility directly into the hands of engineering leaders.

The 3 questions every engineering leader asks

When teams adopt AI tooling, three questions dominate every conversation with directors, VPs, and platform leads:

Is our review process faster or slower?

AI-generated code often produces more PRs, larger diffs, and different kinds of issues. If your review process cannot keep up, you have not gained velocity. You have created a bottleneck.

Is code quality improving or degrading?

More code is not better code. The question is whether AI-assisted development is catching bugs earlier, reducing security issues, and maintaining the standards your team has set.

How do we prove ROI to the business?

Engineering leaders need to justify tooling spend. Saying "developers like it" is not sufficient. You need numbers that connect to business outcomes: time saved, defects prevented, throughput gained.

The CodeRabbit Dashboard answers all three.

What CodeRabbit’s Dashboard shows

The dashboard is organized into five views, each designed to answer a different class of question. Let me walk through what engineering leaders care about most in each section.

Summary: The Executive View

The Summary tab gives you the numbers that matter for a leadership update. In the screenshot above, you can see the core metrics at a glance:

In the screenshot there were 145 merged PRs from 86 active users over the selected period. This is your throughput baseline.

Median Time to Last Commit: This measures how long it takes developers to finalize their changes after a PR becomes ready for review. Short times indicate tight feedback loops and clear reviewer expectations. Spikes here often signal bottlenecks.

Reviewer Time Saved: This metric answers the ROI question. CodeRabbit models the effort of a senior reviewer and estimates how much human review time the AI has offset. For budget conversations, this number translates directly into saved engineering hours.

CodeRabbit Review Comments: A low acceptance rate would indicate noise. A high rate indicates trusted, actionable feedback. Acceptance rate is the quality signal. If reviewers and authors are acting on CodeRabbit feedback at least half the time, the tool is surfacing relevant issues.

The donut charts break down comments by severity (Critical, Major, Minor) and category (Functional Correctness, Maintainability, Security, Data Integrity, Stability). This tells you what kinds of problems CodeRabbit is catching. If most comments are Minor/Maintainability, that is a different story than Critical/Security.

Average Review Iterations per PR: Always know how many cycles a typical PR goes through before merge. High iteration counts can indicate unclear requirements, poor PR quality, or overloaded reviewers. Tracking this over time shows whether your process is tightening or loosening.

Tool Findings: CodeRabbit surfaces findings from your existing static analysis tools. This consolidates your quality signals into one view.

Quality Metrics: Where Are the Real Problems?

The Quality Metrics tab answers: "Is CodeRabbit catching the right things?"

Acceptance Rate by Severity: How often developers act on CodeRabbit comments at each severity level? Consistent acceptance across severity levels suggests CodeRabbit is calibrated well to your team's standards.

Acceptance Rate by Category: This breaks it down further:

Data Integrity and Integration
Functional Correctness
Maintainability and Code Quality
Security and Privacy
Stability and Availability

These numbers help you understand where CodeRabbit adds the most value. If Security acceptance is low, it might indicate false positives in that category. If Maintainability acceptance is high, developers trust CodeRabbit for code quality guidance.

Bar charts: These show raw counts. How many comments were posted versus accepted in each category. This gives you more info about what kinds of comments you’re finding.

Tool Findings: This breakdown shows which static analysis tools contributed findings so you’re aware which are providing more findings for your codebase.

Time Metrics: Where does work get stuck?

The Time Metrics tab tracks velocity through the review process. This is the data you need to find bottlenecks so you can fix them.

Time to Merge: We measure the full duration from review-ready to merged, these include looking at various metrics including these figures as shown in the above example:

Average: 1.2 days
Median: 1.4 hours
P75: 14 hours
P90: 4 days

The gap between median and P90 is revealing in the example. Most PRs merge in 1.4 hours, but the slowest 10% take nearly 4 days. That tail is worth investigating.

Time to Last Commit: This focuses on how long it takes developers to complete their final changes. Here’s the data in the above example:

Average: 2.4 days
Median: 4.5 hours
P75: 2 hours
P90: 5 days

Compare this to Time to Merge. If the last commit happens quickly but merge takes much longer, PRs are sitting idle after code is done. That delay often comes from approval bottlenecks, release gates, or unclear ownership.

Time to First Human Review: How long do PRs wait before a human looks at them? Here’s the example in the screenshot:

Average: 3.4 days
Median: 1.9 hours
P75: 3 hours
P90: 2 days

The median here is under 2 hours, but the average is dragged up by outliers. The weekly trend charts on the right side of the dashboard let you track whether these metrics are improving or regressing.

Organizational Trends: The macro view

The Organizational Trends tab shows patterns over time.

Weekly Pull Requests: Created and Merged PRs plots your team's throughput. In the screenshot, both created and merged PRs trend downward from mid-November toward December. This could reflect end-of-year slowdown, a shift in project priorities, or an emerging backlog.

Weekly Active Users: Is where you look for engagement. The chart shows fluctuation between weekly active users, with a dip around late October.

Weekly Pipeline Failures: Here you can track CI/CD health. Here the decrease in CodeRabbit users correlates with additional pipeline failures.

Most Active PR Authors and Reviewers: Here’s where you can identify contribution patterns. In this data, multiple authors are tied for first place on both creating and reviewing PRs. This could indicate that these engineers are all at risk of being overwhelmed which could lead to a backlog.

Data Metrics: The audit trail

The Data Metrics tab provides per-user and per-PR detail for teams that need auditability, coaching insights, or root cause analysis.

Active User Details table: This shows each developer's activity including PRs created and merged, time to last commit, total comments posted, and acceptance rates broken down by severity. You can see at a glance who is shipping frequently, who has long review cycles, and whose code generates more critical feedback.

Pull Request Details table: This looks at individual PRs with info about their repository, author, creation time, first human review time, merge time, estimated complexity, reviewer count, and comment breakdown. For any PR that took unusually long or generated unusual feedback patterns, you can dig into the specifics.

Tool Finding Details table: Here you’ll find a list of every static analysis finding by tool, category, severity, and count. This is useful for identifying which rules generate the most noise and which surface the most value.

Why this data matters more now

We are in a transition period for software development. AI is generating more code than ever. Developers are reviewing code they did not write. Engineering managers are being asked to prove that AI investments are paying off.

The organizations that navigate this transition well will be the ones with visibility into their own processes. Not just "are we using AI," but "is AI helping us ship better software faster."

CodeRabbit is one of the few tools positioned to answer that question. We see the code. We see the reviews. We see what ships. And now, with these dashboards, engineering leaders can see it too.

The dashboards are available now for all CodeRabbit users. Filter by repository, user, team, or timeframe to analyze performance in the context that matters most to your organization.

If you are an engineering leader trying to measure AI impact, this is where you start.

Curious? Try CodeRabbit today with a 14-day free trial.

2025 was the year of AI speed. 2026 will be the year of AI quality.

Arindam Majumder — Fri, 02 Jan 2026 13:06:51 +0000

The year 2025 will be remembered as the moment AI-assisted software development entered its acceleration era. Improvements in the capabilities of coding agents, copilots, and automated workflows allowed teams to move faster than ever.

But alongside that acceleration came a growing tension. Teams were shipping code at unprecedented velocity, yet trust in AI-generated changes didn’t grow at the same rate. Developers reported feeling both empowered and uneasy: they could produce more output, but they couldn’t always be certain that the output was correct.

Postmortems, operational incidents, and late-stage defects increasingly pointed to subtle logic errors, configuration oversights, and design misunderstandings introduced by AI. We recently wrote about how 2025 had an unprecedented number of incidents. And our recent State of AI vs. Human Code Generation Report found that AI code has 1.7x more issues and bugs in it.

That trust gap is now impossible to ignore and it sets the stage for what comes next. If 2025 was the year of speed, then 2026 will be the year of quality, the moment when engineering organizations shift their focus from just “how fast can we generate code?” to an equal focus on “how confident can we be in the code we ship?”

The industry is moving into a new phase, one defined not just by acceleration, but also by accountability, reliability, and correctness. We’ll share how we got here and the 4 shifts that companies should make to how they use AI in 2026.

2025: The year of speed

2025 was the year when “ship faster” crystallized into a core performance metric for engineering organizations. Leaders often emphasized velocity, tracking PR throughput, diff volume, cycle time, and the raw number of AI-assisted changes as measures of progress. Many companies positioned AI-generated code as a symbol of innovation and sometimes even as a badge of competitiveness.

Major players like Microsoft and Google highlighted how much of their code was now produced or assisted by AI, framing volume as the signal to watch. The focus was on scale: how much code AI could help generate, how quickly, and with how little human intervention.

Quality, consistency, and maintainability became secondary concerns in the conversation.

The hidden costs: Operational incidents and quality regressions

But the speed came with a cost. As teams pushed more AI-authored code into production, a surge of subtle defects began surfacing later in the release cycle. Issues that were once caught through careful review or design deliberation now slipped through.

SRE and operations teams bore much of the impact. Incident reports revealed misaligned assumptions between human-written components and AI-generated logic. Infrastructure configurations created by AI introduced fragility that wasn’t always immediately visible. Our recent report found that AI generated code had up to 75% more logic and correctness issues in areas that were more likely to contribute to downstream incidents.

As 2025 progressed, more production incidents and postmortems pointed to AI-generated code as a contributing factor.

Developers felt empowered by AI in 2025, but uneasy about the code produced

For developers, 2025 was both liberating and unsettling. Many described feeling genuinely empowered: able to build more, experiment more, and clear more tasks in less time.

Yet, alongside that empowerment came growing discomfort about the reliability of the code being produced. Developers increasingly reported moments where the AI-generated solution “looked right” but didn’t feel trustworthy. Reviewing AI-authored code often proved more cognitively demanding than writing it from scratch (something we wrote about here), and subtle errors could be easy to miss in large, machine-generated diffs.

Why quality became the pain point no one could ignore

By the end of 2025, the industry-wide trust gap in AI-generated code had become too large to ignore.

We heard this firsthand when we themed our booth at re:Invent around the Vibe Code Cleanup Specialist meme. That generated conversations with CTOs and other senior engineering leaders about how they felt like their jobs had become, in large part, focused on cleaning up AI mistakes. These conversations showed a pretty widespread consensus across industries and companies: it was time for a return to quality code.

AI had made coding faster, but it had not made correctness automatic. And without correctness, speed loses its value.

The economic reality set in

The final catalyst for the shift toward quality was financial. As more organizations embraced AI-first development, the downstream cost of defects became increasingly visible. Things like code reviews and testing took more time. Outages became more frequent, rollback rates increased, and teams were forced into unplanned refactoring cycles to correct issues introduced by generative tools.

Executives and finance leaders started to quantify the impact: operational incidents, missed SLAs, reliability regressions, and customer churn all carry a price. The cost savings promised by AI-generated code began eroding as teams spent more time debugging and recovering from AI-introduced errors.

Organizations started asking a different set of questions, not “how much code can AI produce?” but “what is the true cost of code that hasn’t been properly validated?”

2026: The year of quality

Organizations are entering 2026 with a different set of priorities. Speed is no longer the only metric that separates high-performing teams from struggling ones; quality has become the true competitive differentiator. Engineering leaders are beginning to shift their KPIs away from raw throughput and toward indicators of correctness and maintainability.

Defect density, review load, merge confidence scores, test coverage, and long-term maintainability metrics are likely to replace cycle time as the numbers that matter most this year. Teams are starting to optimize, not for how quickly code could be generated, but for how reliably it could be trusted. In this new environment, “correct code” will become the new definition of productivity.

Predictions: What 2026 will look like & how to adapt

The shift toward quality will reshape how engineering teams operate, evaluate tools, and measure success. By the end of 2026, several trends will become unmistakably clear.

Shift 1: Companies will track different AI-related metrics

First, companies will begin formally tracking AI-related defect metrics. Instead of treating AI-generated bugs as anecdotal, organizations will measure them with the same rigor used for security incidents or system reliability. Metrics such as AI-attributed regression rates, incident severity linked to AI-generated changes, and review confidence scores will become standard engineering dashboards.

Shift 2: Third party tools will be used to validate AI-code

Second, organizations will adopt more third-party tools designed specifically to validate their coding agents and protect production systems. These tools will act as independent safeguards, offering objective assessments of code quality and catching issues that the generating agent cannot reliably detect since they introduced them in the first place. Enterprises will increasingly view external third party tools for validation as essential risk mitigation rather than optional tooling.

Shift 3: Multi-agent workflows will be used to validate code

Multi-agent workflows will normalize continuous review and validation. Instead of a single agent generating code and hoping for correctness, multi-agent systems will create a layered workflow: one agent writes, another critiques, another tests, and another validates compliance or architectural alignment. These chains will reduce the cognitive burden on developers and raise the certainty that the code entering production is safe, stable, and coherent.

Shift 4: Companies will develop governance around how to use AI

As quality becomes the defining engineering priority, teams start building structured governance around how AI is used. Organizations introduce explicit policies on acceptable AI usage, documentation requirements, and review expectations.

Taken together, these shifts will signal a broader evolution: AI development is moving from experimentation to discipline, from speed to stability, and from novelty to operational maturity.

Conclusion: AI use will finally grow up this year

The story of 2025 was a story of speed. But it also revealed a harder truth: when speed is easy, quality is the real challenge.

In the coming year, the industry will grow up when it comes to their AI use. Engineering organizations that thrive will be the ones that design workflows around reliability, maintainability, and architectural clarity. They will be the companies that treat AI not as a shortcut, but as a system that demands robust validation, thoughtful oversight, and careful integration into existing processes.

The next wave of AI innovation will not be defined by how fast we can generate code. It will be defined by how confidently we can ship it. The future belongs to teams that prioritize correctness, trustworthiness, and long-term stability.

Make your reviews easier in 2026 and catch more defects. Try CodeRabbit today for free!