Forem: Maxim Saplin

Long-Horizon Agents Are Here. Full Autopilot Isn't

Maxim Saplin — Mon, 30 Mar 2026 06:21:06 +0000

A good sanity check for long-horizon agents is not a benchmark. It is a task that is easy to verify and hard to fake.

That is why I still like my small hyperlink_button experiment so much. On paper, it sounds trivial: a Streamlit control that looks like a text link but behaves like a button. In reality, it is exactly the kind of task that exposes whether an agent can actually work.

The task is small enough that you can tell if it succeeded. But it is also awkward enough to matter: Python on the Streamlit side, React/TypeScript on the frontend side, packaging, integration, docs, testing, and all the usual places where “looks plausible” is not the same as “works.”

That is why I think this kind of project is a better test than a flashy benchmark. The real question is not whether a model can emit code. The real question is whether the workflow around it can keep it honest: make it read the right docs, implement the actual requirement, and prove it did not cheat.

That question feels especially relevant right now, because early 2026 has been full of confident claims that long-horizon agents crossed a real threshold.

METR has been tracking AI progress in terms of how long a task an agent can complete, not just how well it performs on narrow benchmarks. Sequoia’s “2026: This is AGI” proposed a deliberately practical definition: AGI is the ability to “figure things out.” And Anthropic’s “Measuring AI agent autonomy in practice” added real deployment data: longer Claude Code runs, more strategic auto-approval, and a shift from step-by-step approval toward active monitoring and interruption.

At the same time, the major product teams all published their own frontier stories:

If you only read the headlines, you land in one of two lazy positions.

Either developers are cooked.
Or the whole thing is smoke and mirrors.

I think both reactions miss what is actually changing.

The real breakthrough is operational

The most important shift is not that models suddenly became autonomous software teams. The more interesting shift is that they can now operate inside real environments.

They can use a CLI. They can inspect files and logs. They can run code. They can read docs. They can check whether a change actually worked. They can keep iterating inside a feedback loop instead of handing a blob of code back to a human and hoping for the best.

That is a much bigger change than “better autocomplete” or “bigger context.”

It also explains why software is the natural first home for long-horizon agents. Software is unusually legible, testable, and reversible. You can run something, compare outputs, inspect logs, and decide whether the result is acceptable. In many other domains, verification is just as hard as doing the work in the first place.

That is one reason Anthropic’s autonomy data is so interesting. The pattern is not “experienced users blindly trust agents more.” It is subtler than that. They approve more automatically, but they also interrupt more strategically. The oversight style changes.

That matches my own experience almost exactly.

The mature workflow is not “approve every action forever.”

It is “let the system move, but stay close enough to redirect it when it starts drifting.”

The flagship demos were real. They were also unusually favorable.

I do think the big public demos matter. But I also think they are easy to misread.

The interesting part of Cursor’s post is not that a swarm of agents can brute-force software into existence. The interesting part is that coordination turned out to be hard, flat self-coordination was brittle, and simpler planner/worker structure worked better than more clever schemes.

The interesting part of Anthropic’s C compiler experiment is not just “an LLM built a compiler.” It is that the agents were operating in a world with unusually strong feedback: serious tests, known-good oracles, structured tasks, and a domain with decades of prior art. Chris Lattner’s review and Pushpendre Rastogi’s analysis are valuable precisely because they make that visible.

And OpenAI’s harness engineering post may be the clearest articulation of the new role split: humans steer, agents execute. The environment, observability, repository docs, architecture rules, and feedback loops become first-class engineering artifacts.

That does not make these demos fake.

It does make them easier to interpret correctly.

They are not proofs that software teams can be replaced by autonomous agent swarms. They are proofs that strong harnesses, rich feedback, and explicit structure can now unlock a surprising amount of useful work.

That is a big deal. It is just a different deal than the headlines suggest.

There is also a simpler reason these demos were unusually favorable: they were not blank-slate tasks. Browsers sit on top of standards, reference implementations, and mountains of prior art. Compilers sit on top of decades of specifications, tests, literature, and engineering patterns. Even when the outcome is new, the terrain is already heavily mapped.

That matters.

Two orchestration patterns, neither of them magic

After the talk, I found it useful to separate two broad ways people currently try to orchestrate long-running agent work.

The first is the Ralph pattern: fresh agent instances in a loop, with memory externalized into git history, progress files, and task state. It is crude, but honest. Each run starts with clean context.

The second is LLM-native orchestration, where a lead agent manages subagents or teammates inside a shared workflow. Claude Code agent teams are a good example: separate contexts, shared tasks, direct inter-agent messaging, and an explicit lead.

In theory, the second model should feel much smarter.

In practice, my own experiments did not convince me that prompt-level orchestration is the real unlock.

What I saw was much messier. The manager often wanted to become an executor. It would stop and ask for confirmation. It would ignore the delegation policy. In some runs it violated the brief completely and fell back to the exact CSS or JS workaround I had explicitly ruled out.

That does not mean subagents are useless.

It means orchestration is still fragile.

Right now it feels more like a product and training problem than something you can solve by writing a sufficiently stern prompt.

What actually worked better

The patterns that helped were much less romantic.

Give the model a CLI.
Give it docs within reach.
Run a preflight check before it writes code.
Make verification cheap.
Prefer headless checks over fragile visual wandering.
Use parallelism only when tasks are truly independent.
Add a QA-style handoff before the real human handoff.
Observer, watch out for drift.
Interrupt and intervene.
Brace for impact - 100% there will be bugs and deficiencies.

That changed the economics of the work.

Once the agent could run code, inspect outputs, and verify behavior directly, it stopped acting like a pure code generator and started acting more like an operator. Not an autonomous engineer. Not a magical coworker. More like a very fast worker inside a good harness.

That distinction matters.

The value is not just “the model got smarter.”

The value is that the model can now participate in a loop.

Why I still don't buy the full autopilot story

At the far end of the spectrum sits the software-factory vision, or what Simon Willison described in his write-up of StrongDM as the Dark Factory: agents writing code, agents testing code, agents reviewing code, with humans mostly stepping out of the implementation loop.

I find that direction fascinating.

I also think it clarifies how much infrastructure is required before “no human review” sounds remotely plausible.

In my own work, fully unattended runs still tend to produce something functionally OK but awkward, sloppy, or strangely overcomplicated. They may satisfy a narrow verifier while violating the spirit of the task. They may finish the easy 95% and quietly give up on the hard 5%. They may pass checks and still feel wrong.

That is not a theoretical objection.

That is what I keep seeing.

And honestly, it also matches the broader pattern in public demos. The output can be impressive, useful, and real while still being rough, unstable, or harder to trust than the headline implies.

That is why I think the most useful conclusion is narrower than the hype, but stronger than the skepticism.

The real state of long-horizon agents

Long-horizon agents are real. They already change how software gets built.

But the practical value today comes less from autonomous software teams and more from supervised software operations: strong specs, strong harnesses, cheap verification, explicit context, and active steering.

The fully autonomous rocket-to-Mars version still disappoints me.

The version where I launch five agents in parallel, let them work on bounded tasks, and then challenge the result like a tough lead or QA engineer is already genuinely useful.

That, to me, is the real state of agentic engineering in early 2026.

Ran out of Cursor tokens and switched to GitHub Copilot: Side-by-Side

Maxim Saplin — Wed, 18 Feb 2026 17:38:27 +0000

Update, April 1 (and this is not a joke). Insider Preview version is way more usable and capable as of now. Throughout February and March I have seen a flow updates and most of the below concerns I've brought up are now fixed. Noticed a few Microsoft employee views in my LinkedIn in Feb, could it be this blog post turned into a backlog? :)

DISCLAIMER! The best AI coding tool is the one available to you, that gives you the best model and reasonable token limits. From the text below it might look like GitHub Copilot is a horrible product - it's not. I use Copilot and I'm productive. It's just an irritating experience when I switch from Cursor.

The banner is a screenshot from my Cursor 2025 retrospective with almost 1T tokens used - I guess one might call me a heavy user. I've been using it since 2023 and it happens to be my favourite VSCode fork. I tried different AI assisted IDEs: Kiro, Antigravity, Windsurf, Project IDX; used VSCode extensions such as Continue, Cody.

When my monthly token limit in Cursor ran out last December, I've been spending more time with GH Copilot (the Insider Preview version with the newest features). Before that I occasionally used Copilot and mostly followed its progress from media/posts and my colleagues' discussions. It's hard to miss the major AI Coding assistant which Copilot is. Since 2023 I have formed an opinion that GH Copilot is an inferior product compared to Cursor which lagged by ~6 months. Recently the gap in new feature releases in Copilot has narrowed yet the execution is not great.

What I don't like about Copilot

Plan Mode is a gray piece of misery compared to Cursor's implementation. I use it a lot in Cursor but see no reason to use it in Copilot. When I tried it for the first time in GH I didn't even understand that the plan was provided - it was just a few paragraphs of text produced by a subagent and clicking the 'Proceed' button just switched the mode to 'Agent' and pasted 'Proceed' text into chat. All of that seemed like a waste of tokens on subagent that did many tool calls and provided a very generic response. In Cursor you get a detailed and structured .MD plan; there's a 'Build' button allowing you to spawn a new agent in a new dialog (with a different model of choice and a clean context); or you can proceed implementing it in the same thread.

Dialog features are poor (and it's the core of UX). For example, you can't clone dialogs or branch out from certain messages in the middle - something I used a lot in Cursor to manage the ever growing threads and context overflows. There are a few more conveniences around overall UX that are missing in GH and keep the experience irritating (e.g., jumpy prompt input, adding a selected piece of a file to the dialog was not instantly apparent due to a faint animation, etc.)

There's no manual dialog summarisation, only automatic. Here's how I got trapped by this "feature"... In the middle of a chat (and I had no idea how big the chat was, since there was no token counter; otherwise I'd have branched it into a new thread) I typed "Proceed". After the implementation started and I saw a few tool calls summarisation kicked in and the agent got lost and "What do you want me to proceed with?".

Token counter missing for too long. Insider preview has added this feature at the end of January.
- The issue requesting the feature in Copilot has been sitting since April 2025 and collected many reactions. Cursor had the context window usage indicator since I can't remember when.
Shorter context windows. For example, GPT-5 family has 272K input limit and Anthropic's Claude models by default allow for 200K total context size. I had this perception that in Copilot my dialogs hit the summarisation threshold sooner than in Cursor - turns out there's a reason for that. Why have these low defaults?

Gemini 3 Pro instability. My favourite model of November randomly threw errors in longer dialogs - trying Again didn't help; I had to drop those dialogs or switch models. Never noticed this instability in Cursor.
GitHub instructions look inferior to Cursor's rules. For example, there are no semantic rules - where an agent pulls relevant instructions automatically. I even had to do a small workaround for that handy feature. Recently Insider Preview added support of Agent Skills which does exactly that, yet
Piling-up legacy in prompts management. There are instructions, chat modes, different approaches to prompts - recently when doing a cleanup in our teams repo where GH Copilot was used there were a lot of questions around "how do I do my guardrails properly". A good example in my opinion is how Cursor dropped its Rules discipline making Agent Skills the default choice and instantly provided a migration path for existing Cursor rules/commands.
- This also gives another example of a half-baked feature in Copilot. Agent Skills in Copilot are automatic only - the model decides when the skill is pulled into the thread. And for some reason there's no way to explicitly reference the skill. We used /spec and /task slash commands for Spec-Driven development, and those are called explicitly. When introducing Agent Skill Cursor added both option to use those - automatic or via slash commands.
Missing Multi-model parallel agents - Cursor allows you to pick several models to process a single prompt; each one creates a Git worktree and you can proceed working in the worktree you liked the most. Copilot has a Background agent feature allowing you to spin up a new GH Copilot CLI agent - while it also relies on a worktree it doesn't give the same convenience.

Getting newer models can be slow. GH announcements of model availability in Copilot come the same day the model is introduced. Yet it's often opt-in when Copilot subscription admins enable new models manually. In the case of Cursor I learn about new model releases from its model picker
No choice of reasoning effort for models. For example, for GPT-5.2 there's only a single line in the picker, while in Cursor there are 8 options ( low, medium, high, xhigh, and then the same four with the -fast suffix, which is twice as expensive but faster). Technically, one can switch reasoning effort to "High" for OpenAI models, though only under experimental setting "Chat: Responses Api Reasoning Effort", which is a bit awkward and hard-to-reach feature.

Restoring checkpoints can be unreliable. I ended up with a broken solution a few times when going back in chat history. Frankly, it is not always reliable in Cursor either; sometimes agents tend to make changes bypassing standard edit tools. It just seems GH checkpoint restoring was less reliable.
System prompts seem awkward and less effective. For instance, in Copilot I often get the agent responding with a "Plan" section after it completes a long thread. Essentially it fills the top of its report with a scroll of what the plan was. Who cares when job is done? Very confusing after switching from Cursor. Besides, when using Copilot in CLI it often gets the intent wrong and doesn't produce the right command, requiring further interaction.

The recent Cursor release of subagents is yet to be matched by Copilot. The UX is better; the whole orchestration seems more polished. See below how in Cursor I kicked off parallel agents in their own worktrees which in turn kicked off subagents - all in one click. Compare to the very simplistic GH variant:

Models in Copilot can't view image files - you can only paste an image into chat; this way they do see images, otherwise they are blind. Use case? Using ADB to take screenshots and saving them in PNG for further inspection - it took me hours running failing verification loops before I realized Copilot lacked that trivial ability. Cursor does this well.

What I Like about Copilot

(Long awaited) Token counter gives a breakdown. It's curious to observe how agentic coding has recently leaped forward due to verification - you can easily check how much tool call results occupy in the dialog.

You can inspect prompts - under "Output > GitHub Copilot Cha"t you can view very detailed LLM traces. For example, you can see what sort of prompts are used to wrap your interactions, might be useful, especially if you like tinkering.

Open about standard tools - there's no UI in Cursor to control standard tool selection, only MCP ones. If you are up for tinkering you can configure tool bundles, can see their exact names. For example, I often explicitly ask GH to use the runSubagent tool to delegate to subagents - works like a charm for bigger tasks.

Kinda open-source - while the back-end part has not been open-sourced, the extension has been. Besides, many AI coding assistant features have been merged into vscode directly, making the creation of third-party extensions much easier. Though it's a pity that GH Copilot always requires a sign-in locking out of true local LLM use - the ticket for that is very popular and has been sitting for almost a year.
Easier installation of MCP - I found the integration in GH easier (button click); with Cursor I had to update config files.
Ecosystem and integration with GitHub - you have Copilot integrated in GH web app; you can easily assign issues to Cloud agents via you phone while browsing GitHub; the extension is accessible in plenty of IDEs (though people say non-VSCode IDEs struggle with feature parity). They have recently added support for Claude Code and Codex allowing you to run other major coding agents through a GH subscription. The breadth and outreach of Copilot is great.

More tokens - it feels like GH's premium requests model allows for more usage compared to Cursor's token-based pricing. Unfortunately there's no user-facing dashboard in Copilot to draw a clear comparison.

From the Creators of SharePoint...

Pun intended. Corporate touch adds a certain flavour making software disgusting. SharePoint or Dynamics CRM are in my view classical examples - ugly UI, slow. The ".aspx" extensions in URLs remind of decades-old ASP.NET Web Forms used to build them.

Somehow GitHub Copilot follows in the steps of other corporate products... It often feels like software that is created by people who (a) don't use it and (b) don't care. A product built by a slideware company.

Just recently this "don't care" approach has surfaced when a user discovered an exploit to bypass billing. That was hilarious! A vulnerability report was submitted privately to Microsoft Security Response Center; the folks there told that billing wasn't their responsibility and advised to create a ticket on a public GitHub repo - where everyone could see the exploit and free-ride Microsoft on tokens. And even after that the GH issue got closed automatically by some AI bot. A few days later it was re-opened after the exploit received public attention and media coverage.

Copilot vs Others might be a yet another Harvard Business School case study on how a large established company turns slow and loses touch with the market, while more nimble and energetic startups build better products.

Cursor's Apple Magic

"It just works" often comes to my mind when I use Cursor. There aren't that many options and toggles. They like building minimalist and refined UI (one of the reasons I don't like GitHub - because it's often ugly to my eye). A small example, Copilot in CLI:

Vs. Cursor:

There's a bit of closedness and secrecy at AnySphere. Take for example their Composer release where they compare their model to an unnamed best-on-the-market model and vaguely describe what they did - not even mentioning what the context window size for the new model is. Or how they implemented the "use your own API key" feature when they process all LLM requests on their back-end making use within a closed perimeter impossible.

Apple vs. Microsoft, iOS vs. Android, startup vs. enterprise - all those analogies sum up my impressions when comparing Cursor to Copilot.

Long-horizon agents: OpenCode + GPT-5.2 Codex Experiment

Maxim Saplin — Thu, 22 Jan 2026 16:13:07 +0000

Sequoia Capital has recently published a blog post arguing that AGI has been achieved because "Long-horizon agents are functionally AGI". About the same time Cursor team has published their experiments with long-running agents that coded a web browser from scratch.

And my recent reflections of the past year made me realize what a huge stride has AI coding made over the course of just one year.

Along the lines of agentic coding and long-horizon execution, here's my recent experiment using OpenCode and GPT-5.2 Codex (predominantly at high reasoning level, sometimes switching to medium and xhigh)...

Approach: the main dialogue (or session in terms of OpenCOde) is the an orchestrator agent; you explicitly ask it to delegate individual tasks to sub-agents (OpenCode uses task built in tool for that ), verify them, and integrate the results. Why? Cause we don't want to hit the context window limit of the model. Though it could be an interesting experiment, relying on one single long thread with compaction happening from time to time.

Task: rewrite a previously vibe-coded provider for litellm which implements a cascade of requests to several LLMs (implementing strategies, such Mixture-of-Agents or LLM Council strategies) before returning a final response.

Results:

About 4 hours of pure agent work time
Orchestrator session — $4.13, 157k tokens of dialogue length by the end of the task
16 sub-agent sessions — $9.73
Total spent $13.86, about 2M tokens
26 files changed in Git
Only 5 tests written (some Kiro+Sonnet/Opus would probably have gone wild and generated a hundred test doing no real work) — all green
The app works — the provider executes multiple llm queries aggregating the final respond, the Streamlit dashboard shows the recorded traces.

While doing he work agents did plenty of tools calls, scrawl the code-base, made file edits and most importantly tested the changes being made (often the changes didn't work and the agents had to fix what was broken):

For these ~4 hours of agent time, it took about half an hour of human effort and ~10 user messages. 6 major human-in-the-loop touchpoints:

Discuss the scope, formulate a requirements .MD
Kick off the work by explicitly asking to delegate to sub-agents and make sure the tests are green
Ask to run a real case with actual LLM interaction
At xhigh resoning level, ask to analyze real LLM interaction test case failure and give a fix plan
Run the fix loop with a real LLM interactions
Finishing touches asking to fix the failing tests and tidy up the docs

The orchestrator/subagents approach has effectively allowed to fit in 2 million tokens worth of work into 157K token long main thread with the orchestrator - there's still room given that GPT-5.2 Codex has a 400K context window.

P.S> I liked OpenCode a lot, more that I liked Codex.

Cursor-like Semantic Rules in GitHub Copilot

Maxim Saplin — Thu, 08 Jan 2026 21:22:58 +0000

Both GitHub Copilot and Cursor offer ways to define guardrails for agents in the form of Instructions and Rules respectively. On the surface they look the same - just different names for a feature for customizing how AI assistants adapt to your project, be it unit test creation, documentation, or maintaining certain parts of the codebase.

Yet when I turned to GitHub Copilot, I discovered that Instructions are very different conceptually - you define a single file that gets applied to a given repo, folder, or file extensions. In other words, the idea is that you are supposed to (a) have a large .MD file covering lots of topics and (b) rely on relevancy determined by file locations/names.

This approach seems problematic in many ways:

It's an LLM anti-pattern, bloating the model's context with huge blocks of text without the ability to organize instructions into smaller, targeted documents
It's not convenient, instruction relevance is determined by file name pattern matching

Cursor's approach seems much better. The official docs propose breaking down Rules into files no longer than 500 lines. Besides, each Rule has a header section (frontmatter metadata) describing the scope of the rule:

---
description: "Standards for code quality, linting, and modern API usage in Flutter."
globs: lib/**/*.dart, test/**/*.dart
---
# Flutter Code Quality & Modernization
## 1. Run the Analyzer
After making substantive changes to Dart code, **ALWAYS** run `flutter analyze` to catch errors, warnings, and deprecations.
...

These targeted, small, semantic Rules were something I lacked when switching to GitHub Copilot. I liked how Cursor can match rules based on task in the dialog, not file location. Yet I quickly found an easy workaround - use copilot-instructions.md as a registry of smaller instructions/rules. Besides, it can serve as a shim for existing Cursor rules, making it easier for the coexistence of guardrails used by both AI assistants:

# Nothingness - GitHub Copilot Instructions
This is a Flutter media controller application. Consult the relevant rule files in `.cursor/rules/` when working in their domains.

## Rules Index
| Rule File | When to Consult |
|-----------|-----------------|
| `flutter-best-practices.mdc` | Writing/modifying Dart code. Covers linting, modern APIs, deprecations. |
| `testing-standards.mdc` | Adding features, models, services, widgets, screens. Covers test organization & mocking. |
| `documentation.mdc` | Adding architecture components or complex logic. Covers doc structure. |
| `flutter-commands.mdc` | Running Flutter CLI commands. Covers sandbox permissions. |
| `github-actions-polling.mdc` | Working with CI/CD workflows. Covers polling strategies & failure handling. |
| `rule-creation.mdc` | Creating/modifying rules in `.cursor/rules/`. Covers format & best practices. |

## Agent Behavior
1. **Context efficiency**: Don't load all rules—consult only those relevant to the current task
2. **Run validation**: Always run `flutter analyze` after Dart changes
3. **Reference docs**: Point to existing documentation rather than re-explaining

It turns out modern models fine-tuned for agentic flows are quite curious and tend to follow up on relevant leads they find in the context:

AI Dev: Plan Mode vs. SDD — A Weekend Experiment

Maxim Saplin — Thu, 04 Dec 2025 17:13:48 +0000

Three months ago, I tested Kiro's Spec-Driven Development (SDD) workflow and walked away impressed but frustrated. The AI built 13,000 lines of Rust code with 246 tests... that took 30 minutes to run, checked God-knows-what, left CI/CD broken beyond repair, and produced a codebase I couldn't maintain. Fast-forward to this weekend: I built a complete mobile app using Cursor + Gemini 3 Pro + Flutter—structured, maintainable, and shipped in one evening plus half a day.

The difference? Let's unpack...

What I Have Done

Built a Flutter app targeting Android and macOS (mainly for UI debugging) from scratch -> https://github.com/maxim-saplin/nothingness:

It shows currently playing media, provide media controls (pause, next, etc.), displays spectrum analyzer using mic
Used Cursor + Gemini 3 (and some GPT 5.1 and Opus 4.5), mostly Plan and Agent modes
Added 6 Cursor rules acting as Guardrails and Guidelines for agents
26 Unit/integration tests
Focus on Docs:
- I didn't save the MDs produced by plan mode
- Yet I asked to follow a simple discipline adding important tech decisions to the docs/ folder
- Had a separate Cursor rule for docs
Set-up and validated GH MCP is working, agents can autonomously build CI/CD
Working CI/CD with GitHub Actions - build/test on commit, release by request
Saturday evening and Sunday (~ 8h effort)
Spent ~$50 in tokens

Model	Tokens	Cost
gemini-3-pro-preview	42757369	$32,02
gpt-5.1-high	9721834	$5,79
claude-4.5-opus-high-thinking	9065436	$8,66
gpt-5.1-codex-high	276380	$0,20
composer-1	10999	$0,01
Grand Total	61832018	$46,68

Why even do this app in the first place? Well, I've been driving an "analog" VW Polo for a week while my EV was in a workshop. I had a serious withdrawal during this time missing plenty of "conveniences" my Zeekr has provided: watching/listening-in on YouTube videos, highway autopilot allowed me to doom-scroll, 15 inch OLED infotainment screen always loaded with info (nav, videos).

During the 2nd week of digital withdrawal I felt a sudden relief.. That was a 90-s vibe, a nice song coming through car audio, pixelated LCD screen showing the name of a popular at the time artist, no urges to pick up the phone and scroll while staying at the traffic lights. That reminded me of a video touching on the subject how gadgets and constant connectivity steals from our lives... Why not create a simple app that darkens the infotainment in my EV?

SDD Sidenote

After my Kiro experiment in September I moved on to scaling SDD approach to actual production work.

First, I tried GitHub SpecKit with my Cursor enterprise subscription (I couldn't use Kiro free tier with commercial code base) - and I didn't like what I saw. After Kiro it felt bloated, too many artifacts loaded with text, extra steps etc.

Turned out, there were Kiro prompts circling around the internet. By tweaking those ones a bit and putting into the right place I've recreated Kiro experience in Cursor - check out this gist for details.

Over that week I successfully shipped 4 features in Python/Dart codebase - merged and rolled to prod. All of that while multi-tasking and occasionally switching to check the results OR untangle roadblocks. I had mixed feelings, losing grip of the codebase, being lost in a flux.

Some of the lessons learned:

Feasibility Checks are Mandatory: Models often propose impossible or broken solutions (e.g., bad data flows, unworkable stacks). Always verify feasibility before implementation to avoid wasting days.
Aggressively Prune "Bloat": AI tends to over-engineer (excessive env vars, extra containers, verbose docs). Reducing scope before code generation saves massive cleanup time later.
Read Specs: Bugs caught during spec review are far cheaper than bugs caught in implementation. Poor doc review compounds AI-generated issues.
The "Shallow" Trap: AI allows you to avoid deep diving into tech, but this backfires during debugging. You are often faster if you understand requirements and the underlying tech/codebase rather than blindly trusting the agent.
Avoid "Time Sinks": Be ruthless about abandoning low-value features (e.g., "Geo in Analytics," complex filters) that the AI suggests but struggles to implement cleanly.

This time I felt in Control

In October Cursor team has introduced their response to ever growing demand for "think before you do" approaches - Plan Mode. Since then I mostly used this mode rarely reverting to SDD. And I never kept the produced Plans/Designs (unlike the specs produced by SDD). I saw Plan Mode more of more structured approach allowing to spend tokens and have an "alignment" ceremony with an agent on a "transaction" - something barely small, a task, a deliverable... Part of this transaction could be a doc put into a dedicated place, to keep traces of important decisions and be used later.

While working on nothingness it felt natural to plan the implementation, argue certain decisions, decide on document creation rule, document, add Cursor rule to create rule, create rule, design testing framework, expand test coverage... The experience was quite different - I felt complete control and confidence what I do. Even if there were any bugs or deficiencies I had no doubts those will be easily fixable.

One could say I vibe coded an app over the weekend - I would argue I exercised a disciplined approach and produced maintainable code that can be built upon. And indeed over the next day I did quite a lot of refactoring and added multiple features.

The "Plan Mode" wasn't just about generating a to-do list; it acted as an alignment ceremony. It was a deliberate pause—spending tokens to "think" and clarify intent before rushing into implementation. In the same dialog I could switch between Plan and Agent modes multiple times, periodically compacting the conversation via /summarise command. When the thread was done - feature delivered, task done - I could nudge the agent to check test coverage (sometimes new tests were added) or if a doc is worth adding.

What about the Structured Approach?

While most of the work flowed naturally and I did not struggle with heavy ceremonies (think BMad or SpecKit), there was software engineering common sense, paying attention to structure (of solution and work execution):

First prompt was a feasibility check of what felt like the most unclear/challenging part:
After some discussion with the agent I outlined the requirements and worked on the plan proposed by the agent:
In further dialogs I asked the model to define documentation discipline and when I decided it was worth making a pause and leaving traces of the docs I prompted the model to make a detour. Those docs were later used by agents when ramping up new features.
When the minimal version of the app was running together with the agent we agreed on a general approach to testing, documented it, added the initial coverage and later on added and modified test harness in accordance with the testing discipline which emerged early on. Again, a best practice that protects against regressions and also is a strong signal to AI agent in terms of how good or bad it does with newer features.
While reviewing the produced code I had several occasions challenging the solution breakdown (i.e. why downstream code must be aware of upstream scaling details) - that led to a few refactors, test updates and new docs being created.
For CI/CD it was a deliberate step validating how MCP tooling works and if the agent can engage. While doing so a number of Cursor rules popped up explaining peculiarities when interacting with GitHub Actions and sandboxed CLI execution when dealing with Flutter commands.

The MCP Stutter

I decided to let the agent autonomously set-up GitHub Actions CI/CD. In order to do this I needed GitHub MCP server working properly. This led to a few hours of "setup tax" that are worth mentioning:

The Auth Trap: My Personal Access Token expired, wasted time browsing and configuring. Classic.
The Tool Bundle Limits: GitHub's MCP server recently changed how it bundles tools. The default configuration exposed a limited set of tools (about 20), missing the critical Actions-related tools I needed. The agent initially couldn't "see" the CI/CD failures because it literally didn't have the tools in its context.
Validating MCP tooling: I explicitly probed agent for MCP connectivity, that helped a lot, yet didn't solve the feedback loop completely (see next point).

Troubleshooting YAML workflows: Recently I've been noticing that LLMs struggle with YAML formatting. For an hour an agent struggled to get CI/CD running due to YAML file syntax error - it pushed the broken file, checked CI/CD job status on the server, saw it errored and then proceed to check job log - which was empty. Turns out, in case of workflow syntax the error should be checked in a dedicated 'annotation' file of workflow run tool call - this rule handles GH Actions feedback loop.

Once I fixed the tool configuration, the payoff was massive - green and easily maintainable CI/CD pipeline.

Tips and Tricks

If you want to replicate this "Plan Mode" flow, here are the non-obvious lessons:

Treat Plans as Disposable: Unlike Kiro or strict SDD, I didn't treat the generated "Plan" as a sacred artifact to be committed to the repo. It's a transient thought process. The result of the plan (code, specific docs) is what matters.
Know the Task: as long as you are confident in what you are building, quite often we don't realise what we're building (feasibility, consistency, why?)
Choose Familiar Tech Stack: it's easier to spot issues by skimming through generated code and docs
Rules as Guardrails: I added 6 specific Cursor rules (.cursor/rules). One was specifically for documentation: "If you change logic, you must update the docs/ folder." This forced the agent to maintain a "Technical Decisions" log alongside the code, which saved me from the "black box" problem later.
Use /summarize Ruthlessly: Long context windows are great, but models get "dumb" and expensive as the chat grows (especially past 20-30k tokens). I frequently used the /summarize command to compress the history. It keeps the agent sharp and the costs down.
Weekend Models: Anecdotally, gemini-3-pro-preview performed significantly better on Saturday/Sunday than during the week. Perhaps less traffic?

Model and Harness Progress

I attribute my satisfaction with the results to significant progress the models have made over the past 3 months - more reliable in agentic settings (multi-turn dialogs with extensive tool use), it feels like the recent GPT 5+, Claude 4.5 and Gemini 3 are models that can be relied upon producing more substantial code and docs - no more shallow verbosity or pointless unit tests.

Same goes about tooling, AI IDE assistants like Cursor do great in terms of context engineering and providing models with efficient tools and environments feeding relevant info and establishing effective feedback loops.

Disclaimer: When to Use What

This experiment convinced me that for greenfield projects, prototypes, or "Solopreneur" work, this Plan Mode + Guardrails approach is superior to heavy SDD. It's agile, keeps you in the driver's seat, and maintains momentum.

However, SDD still has its place. If I were tackling a massive legacy enterprise codebase, or working in a large team where "hidden knowledge" is the enemy, I would likely revert to a stricter Spec-Driven approach (like SpecKit or custom workflows). There, the overhead of generating strict artifacts pays off in alignment and safety.

But for building a bespoke infotainment system for my car in a single weekend? AI coding with discipline is the future.

AI Dev: Testing Kiro

Maxim Saplin — Mon, 25 Aug 2025 12:08:10 +0000

Kiro is a yet another VSCode fork (just like Cursor or Windsurf) that integrates AI coding features. What caught my attention was the "spec-driven development" > it makes total sense proposing a structured approach to dev (as opposed to "vibe coding"). I got my invitation and over the weekend tested Kiro. I decided to re-create a command line cross-platform disk performance benchmark that was built in 2018 using .NET. This time I picked Rust and used AI. My expectations were low, yet I was impressed in a good way, I (or was it Kiro) did build a working app with solid test coverage! At times Kiro was left alone working for extended periods of time following the plan... And it maintained coherence - that impressed me the most. The result is not perfect, there're some things that don't work (i.e. CI/CD is broken and God knows how much time is needed to recover it), nevertheless part of blame is on me, I could have asked for less and be more attentive to the specs. Over the course of my experiment I have extensively documented the process. These notes were used to create the below blog post using Grok 4.

Update, Aug 27: After spending few more days with the app Kiro produced I am less enthusiastic. Kiro still falls for the shortcomings of other AI tools that eagerly produce code and complete the prompt "no matter what" > I poked cpdt2 codebase, using Cursor and Kiro and trying to recover CI/CD making it work, trying to get the app compile and run in Linux (under Dev Containers) - and non of the attempts succeeded under reasonable time. A classic AI SDLC dilemma, getting the result fast, wasting loads of time fixing and making it working. I think Kiro is a powerful tool (staying coherent while working on multiple tasks) yet when left unattended it can easily bloat your solution with loads of scope you, as a human, wouldn't be able to process. Is it the problem of the tool or of a human using it? Part of issue is on me, could have been more thorough and critical when sketching the specs. Anyways, below is a sample of me trying to make the integration tests running fast, launching a "spec > design > task" and eventually discovering that I went the wrong/non-feasible route wasting couple of hours. Btw, in a separate chat Kiro happily acknowledged the issue (and btw, whatever it proposed in this chat was also not feasible):

Hey folks, it's Maxim here—back with another dive into the wild world of AI-assisted coding. If you've read my piece on Continue.dev, you know I'm all about testing these tools in the trenches, warts and all. This time, I spent a lazy Sunday (well, "lazy" if you ignore the occasional CoD: Modern Warfare 3 breaks) experimenting with Kiro, a new AI-native IDE that promises "Spec-Driven Development." Spoiler: It turned a vague prompt into a fully functional cross-platform Rust app, but not without some hilarious detours and existential questions about my role as a developer.

Back in 2018, I built CrossPlatformDiskTest (CPDT), a .NET-based storage speed tester that racked up 500k downloads on Android. It measured sequential/random reads/writes, memory copies, and more—nothing fancy, but it scratched an itch for benchmarking drives across platforms. This GUI app is in turn based on a Command Line Tool. Fast-forward to 2025: I decided to recreate the CLI version in Rust (a language I barely remember from a 2021 LinkedIn course) using Kiro. No hands-on coding from me—just prompts, reviews, and AI orchestration. The result? A repo called cpdt2 with 72 files, 13k lines of code, 246 tests, and even GitHub Actions for CI/CD. But let's break down the journey, because this wasn't just coding—it was coding while AI did the heavy lifting.

The Setup: From Prompt to Plan

Kiro's big hook is its structured workflow: Spec > Design > Tasks, all in Markdown. It's like forcing yourself to think before you code, which is honestly a breath of fresh air compared to the "prompt-and-pray" chaos of other tools.

I kicked things off with this prompt:

I want to create a cross-platform disk speed test utility. It must be compilable as a command line tool for macOS, Windows, and Linux. It must have an isolated library/component that runs the speed tests and that I can later integrate with other non-CLI apps (e.g., GUI). The tests must include sequential and random read and write measurements with block sizes of 4MB for sequential and 4KB for random (default can be overridden), it must create a test file in a given device (CLI must provide a list of devices available in the system, for system drives utilize OS facilities to get writable app folder). The app must mitigate the effects of buffered reads and cached writes (by default disabling those). The stats collected must include min, max, and avg speeds. Additionally, the app must implement a 5th test - memory copy.

Kiro (powered by Claude 3.7 or 4—I stuck with 4) fleshed it out into requirements, added niceties like MB/s units and progress indicators, and even suggested Android/iOS support when I nudged it. It generated a design doc, broke everything into 23 traceable tasks (e.g., core library setup, platform-specific implementations, CLI args, tests), and queued them up.

Kiro UI? Clean and intuitive—rounded corners, tabbed chats, and a content pane that feels like a souped-up VS Code. One quirk: Use # instead of @ for context in chats. I stumbled there once, but overall, it was smooth sailing.

The Build: AI Takes the Wheel, I Play CoD

With tasks queued, Kiro started chugging away. It handled everything from project setup (Cargo.toml, build.rs) to platform-specific code for Windows, macOS, Linux, Android, and iOS. I "supervised" by reviewing diffs in Cursor (using GPT-5 at high reasoning mode) and occasionally fixing linter warnings or slow tests.

Highlights (and lowlights):

Early Wins: Tasks 1-5 flew by—core config, progress tracking, stats. Kiro even added unit tests when I prompted. A quick Cursor review confirmed it was solid, though I had to install Rust manually after a terminal hiccup.
Platform Shenanigans (Tasks 6-8): Implementing non-buffered I/O across OSes? Kiro nailed it, but linter warnings piled up in unrelated files. I copy-pasted errors into the chat; Kiro fixed most, but it sometimes "hallucinated" checks. Still, better than older LLMs that'd just generate BS.
Testing Drama (Tasks 9-17): The first real run was Task 9. Tests took forever (47 seconds initially) because of oversized files like 2GB dummies. I manually timed them in VS Code's Test Explorer and prompted fixes—down to 13 seconds. One test suite hung for 10-20 minutes; Kiro eventually debugged it. I even created Cursor rules for "runtime checks" (build, test, run the app) to double check after Kiro.
The Big Queue (Tasks 18-23): I dumped the rest in one go. Kiro took ~1 hour, pausing twice for CLI approvals. It added 120+ tests, code coverage tracking, docs (like TESTING.md), and even GitHub Actions for CI/CD—plus a release script for crates.io. Mind-reading? I was thinking about CI/CD, and poof, there it was.

Meanwhile, I switched tabs to save Urzikstan in CoD MW3. Vibe-coding at its finest: AI builds while I snipe baddies. But cracks appeared—integration tests felt inconsistent, and I had to revert/restart once due to messy file placements (Rust's idiomatic unit tests in-source files tripped me up, given my rusty Rust knowledge).

I used Cursor and GPT-5 High between the Kiro tasks to review Git diffs - not much value, most of the reveiws where "OK" and the rest of the doc I didn't care to read.

End result? The app runs! Pick a path, run benchmarks, get stats. It even lists devices and handles caching as specified. But oops—one original req (interactive device selection for system drives) got lost in the shuffle. And 35 linter issues lingered, plus failing GitHub Actions. Fixable, but a reminder that AI isn't perfect.

Code Stats: Bloat or Brilliance?

Compare cpdt2 to my 2018 .NET version:

cpdt2 (Rust + AI): 72 files, 13k LOC, 1.9k comments, 3.5k blanks. Includes benches, docs, scripts, and heavy testing/CI.
2018 CPDT (.NET): 23 files, 1.8k LOC. Leaner, but no automation.

AI inflated the codebase (thanks to tests and infra), but it works cross-platform without me writing a line. In 2018, that took a week of my life; this was one Sunday.

Reflections: Is This the Future of Coding?

Kiro enforces discipline—think before coding—which aligns with prompt engineering best practices. It's not just "prompt > code"; it's a harness for coherent, long-horizon work. The agent stayed on-task for hours, breaking down complexity without losing context.

But here's the rub: I coded blindly, barely glancing at the code. Am I even a developer anymore? It felt like pushing buttons while AI steered—fun, but I lost touch with the codebase. Maintainability? No clue. And without my prior CPDT knowledge, I'd be lost prompting effectively. Non-tech folks? Forget it; this still needs domain expertise.

Side thoughts: Are high-level languages becoming assembly? I don't grok Rust tooling, but do I need to? AI rejection of dumb asks (e.g., fixing non-existent code) is a win over older models. Yet, running in a container from the start would've avoided potential disk litter from test files.

Overall, Kiro's a promising tool—like a Swiss Army knife that mostly cuts, but occasionally needs sharpening. It turned my experiment into a working app, honed my "AI orchestration" skills, and left me pondering: If AI builds while I game, what's left for humans? Dive in, try it, and let me know your thoughts in the comments!

If you're curious, check out cpdt2 on GitHub. And yes, I'll fix those linter warnings... eventually.

LLMs are Bad at Math

Maxim Saplin — Fri, 13 Jun 2025 06:20:11 +0000

LLMs are known to struggle with math. Not in those PhD level tasks from AIME eval, where the reasoning models compete and shine... But rather in the everyday math we deal with - additions, multiplications, etc.

Take for example Grok 3's DeepSearch where I prompted it to "... list countries by their GDP per capita in Japanese Yen". As you can see in the screenshot below, the agent did it most reasonably - found a readily available GDP per capita table from IMF, came up with a USD to JP¥ conversion rate, and created a summary table with IMF data converted using the exchange rate.

In its explanation of the approach "... each USD value was multiplied by 146 to get JPY. For example, Luxembourg’s 140,941 USD became 20,577,186 JPY (140,941 × 146)" Grok 3 makes a calculation mistake. My non-AI native calculator gives me 21,891,386 as the result of 140,941 × 146 multiplication. All the cells in the following table were also wrong.

I went further by testing Grok in 3 different modes:

No thinking + Web Search
Thinking + Web Search
DeepSearch

For each of the modes, the approach by Grok was the same: finding source data in USD, pegging to a certain exchange rate, doing the calculation, and outputting the resulting table. If we put aside the questions of why in all 3 cases the exchange rate was different, why pick a certain list of countries (and never use the full list of countries and territories)... I tested how one of the best SOTA models (Grok-3 ?Mini) faired with converting USD to JPY:

No thinking + Web Search: 32 countries, 3 wrong calculations
Thinking + Web Search: 13 countries, all correct
DeepSearch: 11 countries, 11 wrong (deviating at ~0.5% from true values)

The complete calculation verification is available in this spreadsheet.

The example demonstrates a very common pitfall in LLM use. Any prompt and any context dealing with numbers may require the model to do the basic math. Likely it will not resort to using a tool call (i.e. asking a Python interpreter to run calculations) hence the numbers produced by LLM are not trustworthy. And I rarely see that prompts with numbers are followed by a tool call for calculus, models readily return completions with calculations done.

Say you have Office 365 Copilot, Claude, ChatGPT, or any other chatbot doing errands for you. You ask it to look into an invoice and highlight value-for-money outliers. Or you are working on a quote and ask the chatbot to prepare a report. Or as a PM you use the AI assistant to look into sprint stats and evaluate velocity. There are numerous cases requiring basic number crunching. And if your life depends on the accuracy of those numbers I wouldn't trust any digit in the result. No matter what LLM product you use, Perplexity, Glean, Deep Research, Copilot, Gemini - all are based on LLMs that are bad at math.

But how bad are LLMs at this sort of math? Assume you have the correct input (though it is rarely the case, models can easily hallucinate at any step, e.g. while processing a table in a picture). What are the chances LLM will get the math right?

I've created a benchmark testing just that: llm_arithmetic. It prompts a model multiple times to do additions, subtractions, multiplications, and divisions of random numbers - and registers the accuracy.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Model                                      ┃ Trials ┃ Correct % ┃  NaN % ┃  Dev % ┃ Comp. Tok. ┃       Cost ┃      Avg Error ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ o4-mini-2025-04-16-medium                  │    480 │    97.08% │  0.00% │  2.92% │ 1110603.00 │  $4.903872 │         0.002% │
│ o4-mini-2025-04-16-medium-4k               │    480 │    93.54% │  0.00% │  6.46% │ 1083780.00 │  $6.741561 │         0.001% │
│ o4-mini-2025-04-16-low                     │    480 │    88.96% │  0.00% │ 11.04% │  575871.00 │  $2.551050 │         0.959% │
│ deepseek-r1                                │    480 │    84.17% │  0.21% │ 15.62% │ 1462524.00 │  $3.210413 │      2669.789% │
│ claude-sonnet-4-20250514-thinking16000     │    480 │    76.04% │  0.00% │ 23.96% │ 1332908.00 │ $20.085939 │      1740.396% │
│ o3-mini-2025-01-31-medium                  │    480 │    75.21% │  0.00% │ 24.79% │  945716.00 │  $4.178371 │         2.287% │
│ grok-3-mini-beta-high                      │    480 │    71.88% │  1.25% │ 26.88% │    2702.00 │  $0.006156 │       827.580% │
│ deepseek-r1-4k                             │    480 │    70.00% │  0.00% │ 30.00% │  620371.00 │  $0.000000 │       712.913% │
│ qwen3-32b@cerebras-thinking                │    480 │    69.58% │  5.62% │ 24.79% │ 2767460.00 │  $0.000000 │ 840317057.169% │
│ qwen3-14b@q8_0-ctx4k-thinking              │    480 │    66.25% │  0.21% │ 33.54% │ 2338564.00 │  $0.000000 │      9492.622% │
│ o1-mini-2024-09-12                         │    480 │    66.04% │  0.00% │ 33.96% │  572960.00 │  $7.617905 │      6825.446% │
│ claude-opus-4-20250514-thinking16000       │    480 │    65.83% │  0.00% │ 34.17% │  396158.00 │  $0.000000 │      1831.015% │
│ qwen3-14b@iq4_xs-ctx32k-thinking           │    480 │    65.83% │  0.83% │ 33.33% │ 2552276.00 │  $0.000000 │      8152.815% │
│ qwen3-32b@iq4_xs-ctx16k-thinking           │    480 │    65.62% │  3.75% │ 30.63% │ 3499454.00 │  $0.000000 │      5227.605% │
│ o3-mini-2025-01-31-low                     │    480 │    65.21% │  0.00% │ 34.79% │  284738.00 │  $1.270064 │         5.435% │
│ qwen3-14b@iq4_xs-ctx4k-thinking            │    480 │    65.00% │  0.42% │ 34.58% │ 2245910.00 │  $0.000000 │  72213401.589% │
│ qwen3-14b@q4_k_m-ctx4k-thinking            │    480 │    64.79% │  0.00% │ 35.21% │ 2334475.00 │  $0.000000 │      3769.350% │
│ claude-sonnet-3.7-20250219-thinking4096    │    480 │    57.08% │ 18.96% │ 23.96% │ 1214269.00 │ $18.306354 │       889.557% │
│ gemini-2.5-pro-preview-03-25               │    480 │    55.83% │  0.00% │ 44.17% │    5517.00 │  $0.078019 │        20.602% │
│ qwen3-14b@iq4_xs-ctx32k-thinking-4k        │    480 │    55.21% │  0.21% │ 44.58% │  710967.00 │  $0.000000 │       988.474% │
│ claude-sonnet-3.7-20250219-4k              │    480 │    52.50% │  0.00% │ 47.50% │    4213.00 │  $0.000000 │      2217.925% │
│ xai/grok-3-mini-beta                       │    480 │    51.46% │  0.00% │ 48.54% │    2511.00 │  $0.006060 │       913.579% │
│ claude-sonnet-3.7-20250219                 │    480 │    51.04% │  0.00% │ 48.96% │    4147.00 │  $0.114204 │      1302.437% │
│ claude-opus-4-20250514                     │    480 │    50.42% │  0.00% │ 49.58% │    4169.00 │  $0.572685 │      5037.315% │
│ gemini-2.5-flash-preview-04-17-thinking    │    480 │    50.42% │  0.21% │ 49.38% │  521284.00 │  $0.315585 │        27.894% │
│ claude-sonnet-4-20250514                   │    480 │    50.00% │  0.00% │ 50.00% │    4125.00 │  $0.113868 │        20.410% │
│ gemini-2.5-flash-preview-04-17-thinking    │    480 │    49.79% │  0.21% │ 50.00% │  310022.00 │  $1.087891 │       481.693% │
│ claude-3.5-haiku                           │    480 │    49.58% │  0.00% │ 50.42% │    3987.00 │  $0.029816 │      3351.666% │
│ gpt-4.5-preview-2025-02-27                 │    480 │    49.58% │  0.00% │ 50.42% │    2647.00 │  $1.607175 │        24.709% │
│ gpt-4.1-2025-04-14-4k                      │    480 │    48.54% │  0.00% │ 51.46% │    2688.00 │  $5.163010 │        25.919% │
│ gemini-2.5-flash-preview-04-17-no-thinking │    480 │    48.54% │  0.00% │ 51.46% │    5238.00 │  $0.005956 │        30.566% │
│ gpt-4.1-2025-04-14                         │    480 │    48.12% │  0.00% │ 51.88% │    2729.00 │  $0.068629 │      7284.099% │
│ qwen3-32b@cerebras                         │    480 │    46.46% │  0.00% │ 53.54% │    7457.00 │  $0.000000 │        63.979% │
│ qwen3-32b@iq4_xs-ctx16k                    │    480 │    46.04% │  1.04% │ 52.92% │    7132.00 │  $0.000000 │        63.271% │
│ qwen3-14b@iq4_xs-ctx32k                    │    480 │    45.21% │  1.67% │ 53.12% │    7533.00 │  $0.000000 │ 392239118.901% │
│ gpt-4-0613                                 │    480 │    41.04% │  0.00% │ 58.96% │    2450.00 │  $0.631020 │    362466.402% │
│ gpt-4.1-nano-2025-04-14                    │    480 │    38.54% │  0.42% │ 61.04% │    2841.00 │  $0.002749 │    686001.894% │
│ gpt-35-turbo-0125                          │    480 │    35.62% │  0.62% │ 63.75% │    2438.00 │  $0.011725 │        43.177% │
│ gpt-35-turbo-1106                          │    480 │    33.96% │  0.21% │ 65.83% │    2560.00 │  $0.011907 │       409.261% │
│ gpt-4o-mini-2024-07-18                     │    480 │    32.29% │  0.00% │ 67.71% │    2862.00 │  $0.004137 │        64.570% │
│ claude-2.1                                 │    480 │    13.33% │  0.00% │ 86.67% │    2661.00 │  $0.000000 │       174.584% │
│ deepseek-r1-distill-qwen-14b@iq4_xs        │    480 │    10.21% │ 70.21% │ 19.58% │ 1113604.00 │  $0.000000 │       163.793% │
└────────────────────────────────────────────┴────────┴───────────┴────────┴────────┴────────────┴────────────┴────────────────┘

My observations based on testing a range of models:

In general, models are fine with small numbers (2-3 digits)
Performance is worse with multiplication and the worst with division
There's a huge gap in performance between models
o3/o4 models are surprisingly good, I'd trust it with number crunching tasks where accuracy under 1 percent is tolerable

Grok 3 API - Reasoning Tokens are Counted Differently

Maxim Saplin — Thu, 15 May 2025 16:24:21 +0000

I've learned it the hard way... If you use the recently released Grok-3 Mini reasoning model (which is great by the way) you might have your token usage reported wrong...

TLDR;

While both OpenAI and xAI report reasoning usage in usage.completion_tokens_details.reasoning_tokens field:

OpenAI includes reasoning tokens in usage.completion_tokens
xAI doesn't include

Hence for OpenAI (and according to my tests for Deepseek R1) in order to get the total tokens you can use the old good completion_tokens field. With xAI you need to add up the 2 values to get the right totals (and get you cost estimations correct).

Neither litellm nor AG2 (out of recently used LLM libs) adjust the reported usage for that Grok's quirk.

Not fully OpenAI Chat Completions API Compliant

Grok API provides a compatible OpenAI endpoint. For reasoning models the didn't invent the wheel and use the standard reasoning_effort parameter just like OpenAI does with its' o1/o3/o4 models. Yet for some reasons xAI decided to deviate from OpenAI's approach to reasoning tokens accounting.

That's unfortunate this inconsistency got into prod API from xAI.

XYZ% of Code is Now Written by AI... Who Cares?

Maxim Saplin — Thu, 01 May 2025 18:30:30 +0000

Microsoft CEO Satya Nadella said that "as much as 30% of the company’s code is now written by artificial intelligence" (Apr 2025).
Anthropic's CEO made a forecast that "in 12 months, we may be in a world where AI is writing essentially all of the code," (Mar 2025).
Google CEO stated that "more than a quarter of code they've been adding was AI-generated" (Oct 2024).

When I see this sort of title I often have a sense the XYZ figure has the connotation of software engineers replacement rate. Code written by AI is the code not written by humans, we don't need those 30% of humans to type on their keyboards. With the media's focus on sensationalism and competition for the reader's attention, I don't see why they wouldn't optimize for more drama...

While this sort of speculation is curious (how can those CEOs get measurements of the metric beyond making guesstimates based on some clues/heuristics?), I don't see much meaning beyond merely evaluating the rates of adoption of AI coding tools...

100% of Code is Generated, 70% of Code is Deleted After Review

Let me give you a recent example. I have worked on a small project creating a local python interpreter wrapped as MCP Tool, think of Code Interpreter for ChatGPT.

Why even bother, aren't there Python tools already? There are ones, yet it's either Python execution in local env which is dangerous OR relying on Docker or remote environments that need some effort to set up.

The idea was to wrap into an MCP Server the custom-made, sandboxed, local Python interpreter provided with HuggingFace's smolagents library.

After cloning smolagents' repo, investigating the codebase, and creating a small example of isolated use of the interpreter I've instructed Cursor's Agent to create a new MCP Server project. I showed it the example, and the interpreter code, and gave a link to MCP Server docs by Anthropic. The agent created a complete linter-warnings-free code base.

Yet in the next couple of hours, I have iterated on the produced code. I've removed most of the files and lines. Used AI actively, both the autocompletion and chat, i.e. typed not much Python by myself.

Can I state that 100% of code was AI-generated? Probably. Does this imply that:

I was not needed in the process of building software (100% replaced by AI).
Or did I get a 300x productivity boost since as an average human I can type 30 words per minute while SOTA models generate them at ~3000WPM (~150-200 tokens per second)

Here's the stats:

1st version by Claude 3.7/Cursor Agent: 9 files, 1062 lines, 45 comments, 158 blanks
Final modified and published version: 4 files, 318 lines, 9 comments, 79 blanks

While iterating on the code base I used my brain cycles to make sense of what AI had produced, also gaining a better understanding of what actually needed to be built - and that takes effort and time. Sometimes writing code is easier than reading. Besides writing code (or better say low-level modifications) has a very important function of learning the code base and giving you time for the task to sink in and make sense.

After all, I dropped ~70% of AI-generated code. Does it tell much? Does it mean AI code is junk if it had to be thrown away? Generating in minutes reworking/debugging in hours? I don't think so. Yet the rework percentage isn't that telling metric alone, just like the percent generated metric.

One might say that the example is isolated. Creating from scratch some small project is a corner case not met that often in real life. That's true. Yet I think it makes a relevant point and puts some numbers. There's the same tendency to remove/rework a lot of generated code when maintaining a large code base. The larger the scope of the task, the more agentic the workflow, and the more lines/files are touched - the more you have to fix. For some reason, the best AI tools still have a hard time getting the "vibes" of the project - they struggle creating consistent changes that follow the "spirit" of the code base.

Building Software is not About Writing Code

It's about integrating and shipping code. Did you know that at some point Microsoft had a 3-year release cycle of Windows and "on average, a release took about three years from inception to completion but only about six to nine months of that time was spent developing “new” code? The rest of the time was spent in integration, testing, alpha and beta periods" 1, 2.

Writing code is just one very important part, yet it is not the only one. Did you know that (according to a recent Microsoft study) developers spend just 20% of their time coding/refactoring (that's where the XYZ% AI generated metric lands):

Working with teams and customers, building software I see many things where AI can barely help.

What if your stakeholders become unresponsive, play internal politics, and can't make up their minds about the requirements? Will ChatGPT (or some fancy "agent") chase the client, flash out all the contradictions in requirements (7 green lines, 1 must be transparent), communicate with the whole team and mitigate any of the core risks?

Even if you have what seems to be refined requirements... How much time will it take for every individual team member to embrace what is the "thing" he or she is trying to achieve? How much time will it take for the team to find the internal consensus on how to organize around the goal, break down the scope, and bridge business requirements to implementation detail? Will Gen-AI tools accelerate the team dynamics leapfrogging from forming and storing to norming and performance in days, not weeks?

I see it all the time: people are slow thinkers, there are natural constraints on how much info our brain can process, how many social connections we can build and maintain, etc. Generating lots of texts that few care to read (and fewer try to understand) doesn't solve anything.

Given the current state and trajectory of AI tools in software development, I see them as isolated productivity tools where human is the bottleneck. There's little progress with AI agents filling all the gaps a human worker does in a daily routine. Even at a higher level of AI autonomy people would still need time to make up their minds, evolve their perspectives, talk, and agree.

Productivity

Ultimately businesses seek for more work to be done with less effort/money. Adopt AI in dev teams, and cut costs/headcounts by some magic number (for some reason it's always 20-30%) - that doesn't seem to be working that way. There's no definitive demonstration of step changes in developer productivity across the industry. I like these 2 examples, studies into developer productivity with AI from last Autumn:

👍 Microsoft, Accenture (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566) reported a 26% increase in completed tasks.
👎 Uplevel (https://lnkd.in/eHnbrWAQ) found no change in cycle time and a 41% increase in bugs.

P.S> Did you know that as of April 2025, the popular AI coding assistant Aider has ~80% of its code generated with Aider ;)

P.P.S> Until the "P.P.S>" started the article was exactly 1200 words, it took me several days to contemplate and 4 hours to write. GPT 4.1 would have required 12 seconds to generate a similar size blog post :)

GPT 4.1, o3, o4-mini - OpenAI releases through the lens of LLM_Chess

Maxim Saplin — Mon, 21 Apr 2025 17:21:10 +0000

This will be a quick post. I've ran the recent OpenAI models through LLM Chess eval:

o4-mini and o3 demonstrate solid chess performance and instruction following
GPT 4.1 didn't qualify due to multiple model errors
4.1 Mini is a good increment over 4o Mini, 4.1 Nano didn't impress

Below is a matrix view of models' performance with Y-axis showing chess proficiency and X-axis instruction following:

P.S> The "Notes" section of the leaderboard web site dives deeper into model's performance.

Mercury Coder - A Quick Test of Diffusion Language Model

Maxim Saplin — Fri, 18 Apr 2025 14:34:38 +0000

I have recently touched on how diffusion/transformer models come into new domains - specifically the February news on Larage Language Diffusion models (LLaDA, Mercury).

Last weekend, I received an invitation from Inception Labs to take part in beta testing their Mercury Coder Small model - one of a few representatives of the breed of dLLMs.

The model is presented as (a) based on novel non-transformer tech, (b) matching the performance of SOTA SLMs (think OpenAI GPT Mini, Anthropic Haiku, and Google Gemini Flash) and (c) being 5-10x faster.

With performance I assume how smart the model is and how good the answers it produces are. Speed tells how many tokens per second a model can generate.

Speed

The key selling point from the Mercury introduction post was the generation speed - a 5-10x increase over similar-sized models. That's what I decided to test first.

I have used a simple Python UI that supports OpenAI compatible endpoints and can show tokens per second metric after the response is received:

I've obtained a stable 370 token/second generation with almost 0 variation. What's also curious is that the model can only work with the temperature set to 0 and it always produced the same answer:

mercury-coder-small / 1102 tokens
TTF Chunk: 0.85s / TPS: 369.13
TTF Chunk: 0.73s / TPS: 369.86
TTF Chunk: 0.77s / TPS: 368.73
TTF Chunk: 0.77s / TPS: 370.96

On the one hand, it is faster than the 200 tokens we've seen from models like GPT-4o Mini and Gemini 2.0 Flash. On the other hand, it's twice lower than the advertised 737 tok/s. I have also tried a simple curl request with no streaming yet received the same speed.

A side note. Mercury provides OpenAI compatible chat completions API (but who doesn't these days..). It turned out it supported streaming responses. And that was a surprise for me. The key differentiator of dLLM from traditional LLMs is how they produce a large block of text and then gradually change parts in it (the diffusion effect), rather than spill out tokens one by one. I don't see how streaming can be implemented in that case except by completing the full generation in the backend and then simulating the streaming (at a significant slowdown).

For comparison, recently I got my hands on GPT 4.1 Nano. With the same prompt, it produced a reply of similar size (1000 tokens), and the speed fluctuated between 150 and 340 tokens per second:

gpt-4.1-nano-2025-04-14 / 1096-1251 tokens
TTF Chunk: 0.31s / TPS: 341.31
TTF Chunk: 0.45s / TPS: 305.91
TTF Chunk: 0.34s / TPS: 268.98
TTF Chunk: 0.36s / TPS: 200.13
TTF Chunk: 0.45s / TPS: 132.89
TTF Chunk: 0.45s / TPS: 150.13

Performance

I didn't use the model as a daily driver (in IDE or as a chatbot) and would not pretend to have a comprehensive answer to how good it is...

First of all, I ran a few prompts in chat, and it seemed OK, generating code snippets and giving basic answers.

Then I ran it through my own LLM Chess eval. It took place in the middle-bottom of the list, a typical SLM struggling to adhere to game protocol making on average 10 moves before breaking the game loop:

token accounting is wrong, the model is not as verbose

The major conclusion is that the new tech doesn't solve the old problem of LLM hallucinations. The common failure mode is yielding illegal moves. See in the example dialog traces below how the model gets the list of legal moves and still fails to pick one (making up a combination not on the list):

--------------------------------------------------------------------------------
Proxy (to Player_Black):

g8h8,f8e7,f8d6,f8c5,f8b4,f8a3,e8e7,d8e7,d8f6,d8g5,d8h4,c8b7,b8c6,b8a6,e4f6,e4d6,e4g5,e4c5,e4g3,e4c3,e4f2,e4d2,h7h6,g7g6,f7f6,d7d6,c7c6,a7a6,h7h5,g7g5,f7f5,d7d5,c7c5,a7a5

--------------------------------------------------------------------------------
Player_Black (to Proxy):

Thank you for providing the list of legal moves. After analyzing the position and the available moves, I'll make a move to develop my pieces and control the center of the board. I'll move my knight from g8 to h6. Please provide the command 'make_move g8h6'.

--------------------------------------------------------------------------------
Proxy (to Player_Black):

Failed to make move: illegal uci: 'g8h6' in rnbqkbr1/pQpp1ppp/8/4p3/2P1n3/N4P1N/PP1P2PP/R1B1KB1R b KQq - 0 11

I have also run the model through LiveBench (public dataset from November 2024) which I had readily configured for local runs:

category                              average  coding  data_analysis  instruction_following  language  math  reasoning
model
google_gemma-3-27b-it@iq4_xs             50.7    36.9           52.8                   82.1      31.9  53.5       47.3
gpt-4.1-nano-2025-04-14                  42.7    40.6           46.0                   60.0      24.0  46.8       39.1
gemma-2-27b-it@iq4_xs                    39.8    36.6           48.1                   67.6      29.5  25.0       32.0
mercury-coder-small                      35.9    34.4           44.7                   53.2      12.4  35.1       35.8

Any Good?

The model doesn't impress with its smarts. Speed, that's something we've seen already - Groq and Cerebras have been serving popular open-source models such as Llama, Gemma, Owen, etc. for quite a while - using their custom hardware and boasting thousands of tokens per second.

At the same time, I don't think Mercury is a failure. It's a win. They've built a capable model that qualifies as a general-purpose chatbot. And they did it leveraging completely new ideas (in LLMs at least). There's no need for custom hardware to run diffusion models at insane speeds. While Groq/Cerebras might find it difficult to find any applications to their LPUs beyond transformer model inference.

The model is in 2023 in terms of performance. Looking forward to new increments, hope they can match SOTA models of 2024-2025 - this will bring low-cost high-speed LLM inference.

Llama 4 - 10M Context? Coding? Decent Follow-up?

Maxim Saplin — Tue, 08 Apr 2025 19:04:53 +0000

Meta has brought the long-awaited Llama 4 models on Saturday, April 5. Llama 3 came out on April 26, 2024, so this year Meta came earlier, looks like a good sign. The newer models were retrained on (!)40T tokens, an almost 3x bump from the previous version's 15T training dataset. The model brought plenty of changes, including MoE architecture, multi-modality, huge context windows, a giant 2T version behind the scenes, and evaluations showing that the model matches other SOTA models. And yet I didn't find the release exciting.

No Longer "Local LLaMA"

None of the released models are of reasonable sizes. Llama 2 had 7B and 13B variants, and Llama 3 came at 8B the first day. The name "llama" became a synonym for LLM hobbyist tinkering with local execution, large communities were formed around the Llama family of models... There are super popular projects that bear the name:

llama.cpp is probably the most popular runtime for local models (not just Llama). There's a whole ecosystem with llama.cpp's native GGUF model format
Ollama might be the most popular tool to discover, download, and run GGUF models locally.
locall_LLaMa is a subreddit with 426K users

In 2023 Llama models ignited open-source research in LLMs, papers were published, enthusiasts fine-tuned models on their gaming GPUs, and tinkering recipes flooded the internet. Alpacas, Vicunas, Wizards (and plenty of other fine-tunes) were published as open-weight models. At that time there was the famous "We Have No Moat" leaked document questioning Google's ability to succeed in the LLM race while against Open-source community (equipped with Llama)...

With Meta boasting how the smallest 109B Llama 4 Scout can fit into a single $20k H100 GPU I don't see how the new models can be treated as a follow-up Llama 1, 2, 3. As of now the lineage is broken, and there's no clue if true local models are coming.

Long Context

The new models promised super huge context windows at 1M and 10M tokens - something unseen in open models. There's also a claim of nearly perfect retrieval in the "Needle in the Haystack" test:

A few weeks ago I've found an interesting take context window eval, the one testing models' comprehension - if they can truly understand long texts, build mental representations and retaliations between subjects, answer complicated questions - Fiction.liveBench.

For some reason their version of the results is not sorted/colorized, here's the April 6 version I formatted:

The ranking puts Llama 4 to the bottom :(

Coding

Another disappointment came from the AIDER benchmark - here the larger 400B Maverick model occupies the bottom part of the leaderboard right below Qwen2.5-Coder-32B-Instruct. Here are a few items from the leaderboard:

claude-3-7 (no thinking): 60.4%
claude-3-5-sonnet-20241022: 51.6%
Qwen2.5-Coder-32B-Instruct: 16.4%
Llama 4 Maverick: 15.6%

Yet if we look at the coding part of the LiveBench suddenly Maverick is not that bad, it's actually great! The model is above Claude 3.5 and 3.7, Anthropic models that earned the reputation of best ones for programmers.

AI Benchmarks Skepticism

After I created my own LLM Chess benchmark I care about benchmark scores much less :) My experience with LLMs can be summarised with a simple statement "sometimes they work, sometimes they don't". LLM performance is very task specific, the same model used inside IDE can shine with one request and fail miserably with another.

If one took at face value needle-in-the-haystack results for Llama 4, one would be convinced it is the best model long prompts supporting the largest models. The ones coming from Fiction.liveBench would cross out Llama 4 as garbage. The same polar conclusions will come after evaluating coding abilities based on Aider and LiveBench figures.

There has been plenty of criticism towards evals. People complain that they don't tell the full story, they do not reflect their real-life experience. E.g. there is valid criticism that HumanEval could be a good coding benchmark in 2021 at the dawn of chat models yet it is barely relevant to the day-to-day routine of software engineers. Many evals are just fixed questions with well-known answers and LLMs are great at memorizing - there's is a valid point questioning any static "question - right answer" kind of benchmark... And suspecting AI labs in "overfitting" (or simply put cheating) on benchmarks - after all there are huge incentives to score great in those press releases.

LMArena (aka LMSys, Chatbot Arena) is often one of the key metrics that AI shops boast upon releases - this is a rating score testifying human preferences. Yet for quite some time this score is just noise for me... And this study demonstrated how easy it is to trick humans overoptimizing via RLHF.

By the way, there's a sketchy story with a loud intro of Llama 4 as the 2nd best in the arena... Followed by users suspecting something was not playing out and an official clarifications that arena's Llama 4 (a) was neither the Scout or Maverick version released and (b) was specifically tuned for human preferences (to score higher). Not cheating, "optimising for human preferences" as they say :)

P.S>

I have my reservations in regards to fictionLive.bench. While I like the idea there's little info on implementation, there's no code, no tech report... There're many things that make the inner perfectionist unhappy: what is the token-size of 0-token story, how many questions are there, why not rank the models by average? If same attention to detail characterises the actual implementation, I question the findings...

P.P.S>

My LLM Chess Eval has recently brought similar contradictions in evals. Gemma 2 was one of the best open models in the leaderboard, it was consistent in the game loop demonstrating solid instruction following - I would say the best out of smaller models. These results related to my own chatting experience putting Gemma 2 models above Llama 3. Gemma 3 release got me excited, I anticipated a successor that will surpass the previous model. And it didn't happen :)

P.P.P.S>

Context window size interested me for quite a while. Here's my article putting into perspective token sizes of various artifacts. I've been keeping eye on related evals, unfortunately, there're not many (the aforementioned needle in the haystack, Ruler). My own intuition is that even 10% or 20% percent of context can be too much and degrade LLM performance - I am one of those guys who starts clean chats for every small conversation.

When you check the headboards you can notice that there's always never 100% accuracy in retrieval or it drops right away, e.g. at 1k tokens. Take from example Ruler:

When I used tools such as Perplexity, or DeepResearch, or ant RAG based chatbot I always have this at the back of my mind, anticipation that there're plenty inaccuracies and missed bits in the generated responses.

Here's a my story of (unproductive) use of GPT-4.5 to convert the aforementioned fiction.live Bench into a table to colorize and sort in Excel (the screenshot I shared above).