Forem: Amit Ben-Ari

Why git log --oneline is Killing Your AI-Generated PRs

Amit Ben-Ari — Thu, 09 Apr 2026 06:30:00 +0000

Originally published at hivetrail.com

I build HiveTrail Mesh, a context assembly tool for LLMs. I use Claude Code, as well as multiple other LLMs, daily. And recentry, I asked it to write a pull request description for a feature I'd just finished - 27 commits across 32 files, several days of real work.

The output was competent. It had a title, a summary, a list of changed files. But reading it back, something felt off. I knew what was in those commits. The architectural decisions, the edge cases I'd hunted down, the bug fixes that weren't obvious from the file names. None of it was there. The PR read like someone had skimmed the index of a book and written a summary without reading the chapters.

So I did what any developer building a context tool probably would: I went looking for why.

What Claude Code Actually Sends to the Model

When you ask Claude Code to write a PR description, it runs two git commands:

git log main..HEAD --oneline
git diff main...HEAD --stat

The --oneline flag returns one line per commit: the abbreviated SHA hash and the subject line. That's it. No commit body, no co-author notes, no extended description you carefully wrote.

The --stat flag returns a diffstat - a summary of which files changed and how many lines were added or removed. Again, no actual content. No diffs, no file contents, no context about what changed or why.

So the model is working from something like:

7c22302 fix(git-tools): harden subprocess wrapper against local gitconfig pollution
6b5fd96 feat(git-tools): replace base branch input with auto-populated select
1246c27 fix(git-tools): resolve validation, state drift, and arch leaks
...

Plus a file change summary. For a 27-commit feature branch, that's the equivalent of asking someone to explain a film by reading the chapter titles on a DVD menu.

The model is good enough to produce something coherent from this - but coherent isn't the same as accurate, and it certainly isn't the same as useful.

What a Good PR Description Actually Needs

Before getting to the fix, it's worth being specific about what's missing.

A PR description serves two audiences with different needs. Developers reviewing the code want to know which layers of the codebase were touched, what edge cases were handled, and why certain decisions were made the way they were. Product managers and QA engineers want to understand user impact, workflow changes, and how to verify the feature works.

When the model only has commit subject lines and a file list, it can infer what changed from the file names. It cannot infer:

The why behind architectural decisions
Edge cases that were discovered and handled mid-implementation
Bug fixes that are buried in commits whose subject lines don't make them obvious
The distinction between new features and hardening work
Testing specifics - what's covered, how, and what a reviewer should manually verify

These are exactly the things that separate a PR description that's useful from one that's just technically accurate.

The Prompt Fix: Better Claude Code Instructions

Before reaching for a different tool, most developers will try the obvious thing first: write a better prompt. And it's a fair instinct - you can absolutely instruct Claude Code to run more thorough git commands, request full diff content, and follow a specific PR structure. Something like "run git log with full commit bodies, fetch the complete diff for each changed file, then write a PR description organized by architectural layer with a key design decisions section" will produce meaningfully better output than the default.

But there are real costs to this approach worth understanding. The most immediate is token burn - asking Claude Code to fetch full diffs and structured commit logs for a large branch will consume significantly more context than the default --oneline summary approach, which adds up quickly if you're on a metered plan. The less obvious problem is consistency. Claude Code operates within a conversation context that degrades over a long session: early instructions get compressed, memory gets summarised, and the careful prompt you wrote at the start of a session may not be fully honoured three hours later when you finally hit merge. You're also now maintaining a prompt, not just a workflow.

For this test, I deliberately used the simplest possible prompt - "based on the staged changes / recent commits, write me a PR title and description" - across all three methods. The goal was to measure what each approach produces from its own capabilities, not what it produces when coached. Prompt engineering can close some of the gap, but it can't change what data the model is actually working from.

The Manual Fix: Give the Model the Full Context

The underlying problem is simple: the model is summarising a summary. The fix is to give it the actual data.

Here's what that looks like manually. Before asking your LLM to write the PR, run this yourself and include the output in your prompt:

Full commit log with bodies:

git log main..HEAD --pretty=format:"%H%n%an%n%ad%n%s%n%b%n---"

Actual file diffs:

git diff main...HEAD

Or per-commit diffs if the full diff is too large:

git log main..HEAD --patch

A word of warning on token volume: for a large feature branch, git diff main...HEAD on a 27-commit, 32-file branch produced around 106,000 tokens in my test - roughly 379KB of XML-structured content. That's well beyond what you'd paste into a chat window, and it approaches or exceeds the context limits of many models.

This is where you need to be selective. For smaller branches - a few commits, a handful of files - pasting the full diff directly works fine. For larger branches, you have options:

Feed it to a model with a large context window (Gemini Pro handles this comfortably)
Trim to the files and commits most relevant to the PR's purpose
Use the structured approach described in the next section

Either way, the quality difference is immediate. When I ran the same prompt - "based on the staged changes / recent commits, write me a PR title and description" - with the full structured context versus Claude Code's summary approach, the outputs were not comparable. The full-context version knew about BOM-aware file encoding, NiceGUI deleted-slot errors, the decision to use @computed_field to eliminate state drift. The summary version knew that git_service.py was a new file.

What the Results Actually Showed

I ran three versions of the same PR description for the same feature branch:

Version 1 - Claude Code (Sonnet 4.6)
Working from --oneline and --stat only. Produced a competent, file-oriented description with a good "Key design decisions" section. Flat formatting, no inline code styling, read like a wall of text.

Version 2 - Claude web chat (Sonnet 4.6) + full structured context
The same model, but fed the complete PR Brief XML (106k tokens of structured diffs, commit metadata, and file content). Layered by architectural section, included product context, named specific edge cases and why they were handled the way they were, referenced exact test counts.

Version 3 - Claude web chat (Haiku 4.5) + full structured context
The cheapest Claude model, same full context. Produced a description nearly as strong as Version 2, with better structured sections for testing guidance and explicit "Key Design Decisions."

I asked Gemini 3 Pro to evaluate all three as a neutral third party, framed as a senior developer and product manager. The ranking: Version 2 first, Version 3 second, Version 1 third.

The conclusion that stood out: Haiku 4.5 with full context outperformed Sonnet 4.6 with shallow context. The model tier mattered less than the context quality.

Gemini's summary of the gap was pointed: Version 1 "forces the reviewer to do the heavy lifting." Versions 2 and 3 "treat the PR description as living documentation."

Google Gemini 3 Pro's comparison table:

PR Text	Primary Strength	Primary Weakness	Best Audience
PR 1 (Claude Code)	Exceptional, step-by-step test plan.	Flat formatting makes it hard to scan. Lacks inline code styling.	QA and Product Managers.
PR 2 (Sonnet + Mesh)	Strongest structural hierarchy and includes "Key Design Decisions."	Testing section is a status report, not an actionable guide.	Senior Developers and Architects.
PR 3 (Haiku + Mesh)	Excellent markdown formatting (inline backticks) and explicit bug-fix callouts.	Lacks the deeper architectural "why" found in PR 2.	General Code Reviewers.

Version 1: Claude Code with Sonnet 4.6:

Title:
feat(#282): add Git Tools source type (Commit Brief + PR Brief)

---

Description:

Summary
Introduces Git Tools as a new source type in the Stack, enabling users to generate token-optimized LLM context from local git repositories.

Two item types are supported:

- Commit Brief - scans a single commit and produces a structured XML summary of changed files and diff content
- PR Brief - scans a branch against a base branch, producing a structured XML summary with per-commit metadata and a selectable file/commit
checklist

What's included
- services/git_service.py - new service with full subprocess wrapper...
- models/stack/git_tools.py - Pydantic models for CommitBriefStackItem...
- core/state/sources/git_tools.py + core/state/stack/handlers/git_tools.py...
- components/common/forms/git_tools/ - CommitBriefForm, PRBriefForm...
- components/sections/stack/dialogs/edit_commit_brief.py + edit_pr_brief.py - edit dialogs with...
- components/sections/source/panels/git_tools/ - source panel with auto-populated base branch select ...
- Full pytest coverage in tests/services/test_git_service.py

Key design decisions
- Base branch is auto-populated from the repo rather than free-text input...
- Partial generation failures surface as a warning state...
- Commit log is stored as structured...
- subprocess calls are wrapped to suppress local...
- Follows the existing async stack lifecycle...

Version 2: Claude Sonnet 4.6 chat with Mesh:

**feat(#282): Add Git Tools source type (Commit Brief + PR Brief)**

---

## Summary

Introduces Git Tools as a fourth source type in HiveTrail Mesh, alongside Notion, Local Files, and Context Blocks. Provides two workflows for assembling LLM context from a git repository:

- **Commit Brief** - scans uncommitted changes (staged/unstaged/untracked), assembles file content + unified diffs against HEAD for commit message generation
- **PR Brief** - scans branch-wide changes vs a base branch, includes file content, diffs, structured commit log, and uncommitted change warnings for PR description generation

---

## What's Changed

**Models & Architecture**

- New `GitChangedFile`, `GitCommitEntry`, ... models with Pydantic...
- `PRBriefStackItem.commit_count` as a `@computed_field` to eliminate...
- `warning` status added to `StackItemBaseModel`...
- Git item types registered in `StackItemUnion`...

**Service Layer**

- New `git_service.py` - stateless subprocess service covering...
- Hardened `_run_git` with... to prevent ANSI corruption and pager hangs
- BOM-aware file reading (UTF-16 LE/BE) in...
- `GitResult` dataclass for structured content...
- `generate_git_content` produces structured... XML nodes with ... CDATA string

**State & Stack**

- `GitToolsSource` bridge in... wrapping all git service calls
- `GitToolsHandler` integrated into the standard async stack lifecycle...
- `StackManager.update_git_item`, ... extended for git types
- Stack persistence (`StackPersistence`) handles ... in save/load
- Full facade coverage on...

**UI - Source Panel**

- `GitToolsPanel` with ..., following ... conventions
- `CommitBriefForm` and ... forms with path selector, scan button...
- `FileChecklist` and ... reusable components with select/deselect all, .... with local timezone formatting
- Base Branch field auto-populated from ... disables immediately on path change to prevent stale state
- Uncommitted changes warning banner in...
- Files/Commits sections hidden until...

**UI - Stack**

- `GitToolsStackCard` with type-specific... confirmed Reload action
- `EditCommitBriefDialog` and ... with Settings/Content tabs, ... banner for partial failures
- Insert action enabled for ...
- Edit routing wired in ...

**Bug Fixes & Hardening**

- `notify` called before ... to prevent NiceGUI ... errors
- Stale scan ... on path change
- `commit_log` fallback ... prevent Pydantic ...
- Output manager processing logs ...

**Tests**

- 41 new tests in ... covering parse, pre-checks, scan, ... and PR XML generation (199 total passing)

Version 3: Claude Haiku 4.5 chat with Mesh:

**feat(#282): complete Git Tools integration-forms, dialogs, stack cards, and content generation**

---

# PR Description

## Overview

This PR completes the Git Tools feature (Commit Brief and PR Brief) end-to-end, integrating UI components, edit dialogs, stack management, and async content generation. The feature is now wired into the source panel, stack lifecycle, and preset persistence, with full test coverage for the git_service module.

## What's Included

### Forms & Components

- **CommitBriefForm** ... forms with path selection, branch auto-population, file/commit checklists, and rescan workflows
- **FileChecklist** ...Reusable list components with ... dialogs
- **GitToolsStackCard**: Stack ... confirmation
- **EditCommitBriefDialog** ... dialogs with Settings/Content tabs, ... display

### State & Architecture

- **GitToolsSource** (`core/.../git_tools.py`): Domain bridge wrapping ... asyncio threading
- **GitToolsHandler** (`core/.../git_tools.py`): Pre-generated ... (no JIT reads)
- **git_service.py**: Stateless subprocess service with ... merge logic
- **Stack persistence** updated to serialize/deserialize ...
- **StateManager facade** ... git-specific methods ...

### UI Integration

- Git Tools added to source panel...
- Stack card factory routes ...
- Stack component instantiates both ...
- BaseStackCard updated to handle ... enabled Insert button
- Output manager log messages ...

### Content Generation

- **generate_git_content()**: Async XML generator with BOM-aware file reading...
- Returns **GitResult** dataclass with ...
- Partial success mode: items transition to `warning` state ...

### Data Models

- **GitChangedFile**, ...
- **ScanResult**, ... dataclasses
- FileStatus ... enums
- Pydantic `@computed_field` for ...

### Testing

- 41 new git_service tests covering:
    - _parse_name_status with ...
    - Pre-checks (is_repo, ...)
    - scan_commit_changes with ... states
    - scan_pr_changes with ... flag
    - apply_default_checks (...)
    - merge_scan_with_saved (preserve user ...)
    - parse_commit_log and ...
    - generate_git_content integration with ...

## Key Design Decisions

1. **Pre-generated content model**: Git items store content at ...
2. **Graceful partial success**: When some files fail ... insert partial context.
3. **Clean Architecture enforcement**: GitToolsSource wraps ... No direct service imports.
4. **Concurrent diff fetching**: Diffs fetched in parallel with ... avoid resource exhaustion.
5. **Merge logic for rescan**: When users rescan, ... across repository state changes.
6. **BOM-aware encoding detection**: UTF-16 files (with BOM) and UTF-8 with BOM are decoded correctly; Windows cp1252 default avoided.

## Testing Guidance

- All 199 tests pass (including 41 new ... tests)
- git_service tests use ...
- Integration tested via forms/dialogs in the app
- Warning state rendering tested in ...

---

**Closes #282**

The Easier Path: Structured Context Assembly

Constructing that full context manually - running the right git commands, handling encoding issues, structuring the output so the model can navigate the 106k-token PR Brief without losing the thread - is non-trivial. For a one-off experiment it's fine. As a repeatable workflow before every PR, it's friction most developers won't sustain.

This is exactly the problem I built HiveTrail Mesh to solve. The PR Brief source type runs the git commands, structures the output as navigable XML with per-commit nodes, handles BOM-aware encoding, and lets you select which files and commits to include before the context gets assembled. The output goes to your clipboard, ready to paste into whichever LLM you want to use.

If you want to try it, Mesh is in limited beta and free during the beta period. But honestly, if you want to test the manual approach first on your next branch, the git commands above will get you there.

Quick Reference: The Git Commands That Actually Feed the Model

If you want to try the full-context approach on your next branch before merging, these are the three commands worth knowing:

Full commit log with bodies (not just subject lines):


git log main..HEAD --pretty=format:"%H%n%s%n%b%n---"

Complete diff across the branch:


git diff main...HEAD

Per-commit diffs with context (useful for smaller branches):


git log main..HEAD --patch

A practical note on scope: for a large feature branch, the full diff will be large - potentially 100k+ tokens. Before feeding it to a model, skim the file list and drop binaries, generated files, and lockfile changes. What remains is usually 20-40% smaller and significantly more useful to the model.

If you'd rather not run and filter these manually every time, this is the workflow HiveTrail Mesh automates - structured XML output, file selection, token count before you export.

The Broader Point

Claude Code isn't doing something wrong. It's making a reasonable tradeoff - fast, low-cost, good enough for most cases. The --oneline approach keeps the token cost down and the response time fast. For a quick commit message or a small fix, it's fine.

But for complex feature branches where the PR description is going to be read by your team, reviewed by senior engineers, and live in your repository history for years - it's worth spending an extra 30 seconds to give the model the full picture.

The quality of your AI output is constrained by the quality of the context you provide. For PR descriptions, the full diff is the context. Everything else is a summary of a summary.

Amit builds HiveTrail Mesh, a context assembly tool for developers working with LLMs. If you found this useful, join our beta.

We Ran the Same Experiment Twice. Different Feature, Different Models, Same Winner.

Amit Ben-Ari — Tue, 07 Apr 2026 07:00:00 +0000

Originally published at hivetrail.com

How two independent PR generation benchmarks pointed to the same conclusion about context quality - and why your model choice matters less than you think.

Here's a finding that should change how you think about AI tooling: in two independent experiments using real production code, a "budget" model fed rich context consistently outperformed flagship models operating on shallow git summaries. The budget model didn't just win. It won by a landslide, unanimously, against models that cost significantly more per token.

This isn't a post about which model is best. It's about why the question itself might be the wrong one to ask.

The setup

HiveTrail Mesh is a context assembly tool. One of its core features is PR Brief - it scans a git branch against a base branch, reads every changed file in full, assembles all diffs and commit metadata into a structured XML document, and hands it to an LLM. The output is typically a 100K–380K token document containing everything an LLM needs to write a comprehensive PR description.

We used this workflow as the basis for both experiments. The prompt in each case was deliberately simple:

Based on the staged changes / recent commits, write me a PR title and description.

No elaborate prompting. No chain-of-thought instructions. Just the raw context and a task.

Experiment 1: The budget model vs. the flagship agent

The first experiment ran on the Git Tools feature - a substantial new addition to HiveTrail Mesh covering 27 commits across 32 files, with async XML generation, state management, UI components, and 41 new tests.

We ran three conditions:

Condition A - Claude Code (Sonnet 4.6), native git context. Claude Code ran git log main..HEAD --oneline and git diff main...HEAD --stat - the standard abbreviated approach. Generated in about 25 seconds.

Condition B - Haiku 4.5, Mesh context. Mesh assembled a 380KB XML file (~106K tokens) covering every changed file, diff, and commit. Haiku 4.5 received this in full.

Condition C - Sonnet 4.6, Mesh context. Same Mesh XML, same prompt, given to Sonnet 4.6.

Gemini 3 Pro evaluated all three as a senior software developer and product manager.

The verdict was unambiguous. The Mesh-fed PRs were called "significantly stronger" across every dimension: product context, workflow clarity, architectural structure, technical depth, and testing visibility. The Claude Code version was characterised as reading like "a rough draft or a quick brain dump before hitting Create Pull Request."

This wasn't a knock on Sonnet 4.6. It was a knock on what Sonnet 4.6 was given to work with.

Claude Code - like most agentic coding tools - acts like a developer who skims the commit titles and says "looks good to me." It reads summaries: which files changed, roughly how many lines, what the commit subjects say. HiveTrail Mesh acts like the reviewer who actually pulls down the branch and reads every single file. The difference in output reflects that difference in reading.

Haiku 4.5 with full context outperformed Sonnet 4.6 with shallow context. A cheaper, faster model given the complete picture wrote a better PR than a more capable model working from a summary.

But here's the part that should really give you pause: Haiku 4.5 didn't just beat Sonnet 4.6's native shallow context - it beat Sonnet 4.6 when both were fed the exact same Mesh XML. The budget model outperformed the flagship on a level playing field.

Final ranking:

Haiku 4.5 + Mesh - best overall structure, key design decisions, quantified test coverage
Sonnet 4.6 + Mesh - excellent markdown, clear bug-fix callouts, strong architecture section
Sonnet 4.6 native (Claude Code) - good test plan, but flat structure and shallow context throughout

Experiment 2: Can Gemini CLI beat its own model family?

Several months later, we ran a second experiment on a completely different feature - the GitHub API integration for HiveTrail Mesh, covering 24 files and 22 commits.

The framing this time was sharper. The question wasn't "which model is best" - it was "can an agentic tool using native git context compete with the same model family when context is properly assembled?"

Gemini CLI was the subject under test. It has its own git tooling, can run shell commands, and is built by the same team behind the models it would be competing against. If any tool could close the context gap through smart native tool use, Gemini CLI was the candidate.

We set it against seven Gemini models - ranging from Gemini 3 Fast to Gemini 3.1 Pro with high thinking - all fed via HiveTrail Mesh. We also added Haiku 4.5 via Mesh as an external reference point, since it had won Experiment 1.

Three independent judges evaluated all nine PR texts blind, without knowing which model produced which:

Google Gemini 3 Pro
Anthropic Claude Opus 4.6
OpenAI ChatGPT

Scoring: 9 points for 1st place, 1 point for last. Maximum possible: 27.

Rank	Model	Gemini Pro	Opus 4.6	ChatGPT	Total
1	Haiku 4.5 + Mesh	9	9	9	27
2	Gemini Flash 3 preview (Thinking Low) + Mesh	8	7	8	23
3	Gemini 3 Fast + Mesh	7	6	4	17
4	Gemini 3.1 Pro preview (Thinking High) + Mesh	2	8	6	16
Tied 5	ChatGPT + Mesh	6	1	7	14
Tied 5	Gemini Flash 3 preview (Thinking High) + Mesh	5	4	5	14
7	Gemini 3.1 Flash Light preview (Thinking High) + Mesh	3	5	3	11
8	Gemini 3 Pro + Mesh	4	3	2	9
9	Gemini CLI (native context)	1	2	1	4

Two results stand out.

First, Haiku 4.5 received a perfect score - 9 from every judge, unanimously, with a 4-point gap over second place. All three judges independently placed it first for the same reasons: dedicated test coverage sections, specific method names and API behaviors called out by name, explicit reasoning behind architectural decisions, and reviewer notes that no other entry included. Opus 4.6 called it "the most complete and production-grade PR description" of the nine.

Second, and more telling: Gemini CLI finished last. Not second to last - last, with 4 points, behind every Mesh-fed entry including smaller, cheaper Gemini variants. Its own model family, given better context by a different tool, beat it at every position in the table.

The reason is the same as Experiment 1. Gemini CLI ran git log -n 10 --stat and a few shell commands. Fast, low-cost, reasonable for most tasks - but it produced the same shallow picture. The resulting PR covered the surface of the changes without the architectural reasoning, edge case handling, or quantified test results that the Mesh-fed models could draw on because they had actually read the code.

It's worth noting that the Mesh PR Brief isn't just raw file content dumped into a prompt. It's structured XML - commits organized chronologically, files grouped by change type, diffs nested within their commit context. That structure helps LLMs navigate 100K+ token documents more efficiently than a flat wall of text would. So "full context" here means both more information and better-organized information. Both matter.

After the main competition, we ran Claude Code on the same feature - not as a competitor, but as a consistency check. Same pattern as Experiment 1: a short, surface-level PR based on abbreviated git output. The shallow-context behavior isn't specific to any one tool or vendor. It's structural - it's what happens when speed is optimized over depth of reading.

The pattern

Context quality sets the ceiling. Model choice determines where within that ceiling you land.

Run both experiments side by side and the picture is hard to argue with.

Experiment 1 tested context delivery method with the same model family. Mesh-assembled context won over native git context regardless of model tier - and the budget model beat the flagship even on a level playing field.

Experiment 2 tested whether a sophisticated agentic tool could close that gap through smart native tool use. It couldn't - and it finished last against its own model family.

Different features. Different PR Briefs. Different competitive sets. Different judges. The only constant was the relationship between context quality and output quality.

When an AI tool reads a few lines of git log to write a PR, it isn't producing a poor result because it's a bad model. It's producing a poor result because it has been given a poor picture of what changed and why. Give any capable model the full picture - every file, every diff, every commit, structured and organized - and the output improves dramatically.

The implication runs both ways. A "budget" model with rich context outperforms a flagship with shallow context. And a flagship with shallow context produces flagship-priced shallow output.

What this means for your workflow

If you're using AI tools for PR descriptions today, the most impactful change probably isn't switching models.

Agentic coding tools are optimized for speed and low token cost - they read summaries, not full file content. That's the right tradeoff for interactive coding tasks, where you want fast feedback and low latency. For a PR covering 20+ files and weeks of work, summary-level context produces summary-level output.

The alternative is deliberate context assembly before you prompt: read every changed file in full, preserve the diff structure, organize commits chronologically, package everything in a format the LLM can navigate. You could build a script to do this - pull every changed file, run the diffs, format it into structured XML. It's achievable engineering. It's also a few days of work to do properly, and more to maintain as your codebase evolves.

That's exactly why we built HiveTrail Mesh's PR Brief. Point it at a branch and within seconds it has scanned every changed file, assembled the diffs, and produced a structured 100,000+ token XML document - faster than most agentic tools complete their own context gathering. The remaining time in the workflow is just the LLM responding, which varies by model (a few seconds for smaller models, up to ~30 seconds for the larger ones). The total end-to-end time is competitive with agentic coding tools - with dramatically better output to show for it. Use any LLM you prefer: Claude, Gemini, ChatGPT, whatever fits your workflow. The model choice, as these experiments suggest, matters less than you might expect.

For teams where PRs serve as living documentation, get reviewed by multiple people, or feed downstream into release notes - the tradeoff is straightforward. For a solo developer pushing a two-file fix, probably not worth it.

What we didn't test

In the spirit of intellectual honesty:

Prompt engineering. Both experiments used a minimal prompt. A carefully crafted prompt might narrow the gap somewhat - though we'd expect the ceiling to remain lower without full file content. It's also worth noting that the Mesh PR Brief's structured XML format is itself a form of context organization: commits are sequenced chronologically, files are grouped by change type, and diffs are nested within their commit context. That structure likely helps LLMs parse large documents more efficiently than flat CLI output would.

Other writing tasks. Both experiments focused on PR descriptions. Commit messages, technical documentation, and code review summaries likely follow the same pattern, but we haven't tested them.

Newer model releases. These experiments used models current at the time of testing. Rankings will shift as new models release - though the underlying dynamic (context quality determines ceiling) should hold.

Cost efficiency. Haiku 4.5 is significantly cheaper per token than most of the models it beat. The cost-per-quality-point story is compelling but token pricing changes frequently enough that any number we published here would be stale quickly.

Closing thought

The most useful takeaway from two experiments isn't a model recommendation. It's a workflow question worth asking before you prompt: what does the model actually see?

If the answer is "a handful of commit subject lines and a diffstat," you've already constrained the output - regardless of which model is on the other end.

The models are good enough. The context is usually the bottleneck.

HiveTrail Mesh is a context assembly tool for developers and product teams. PR Brief assembles a token-optimized, structured XML document from your git branch - ready to paste into any LLM. Try the beta →