Forem: Tensorlake

Everything you need to know about OpenAI GPT-5.4 ✌️

Shrijal Acharya — Sat, 21 Mar 2026 14:08:05 +0000

OpenAI’s new GPT-5.4 is here, and on paper at least, it looks like one of their strongest all-rounder models so far.

TL;DR

In this article, we take a quick look at OpenAI GPT-5.4, go through its official benchmarks, and then compare it in one small coding task against Anthropic’s general-purpose model, Claude Sonnet 4.6, to see how it actually performs.

We briefly go over what GPT-5.4 is, what OpenAI is claiming with this model, and why it looks like one of their strongest all-rounder releases so far.

We look at the official benchmarks around coding, reasoning, tool use, and computer-use capabilities to get an idea of how strong the model looks on paper.

Instead of relying only on benchmarks, we also compare GPT-5.4 against Claude Sonnet 4.6 in one small, quick coding task (not enough to judge fully, but still...).

Brief on OpenAI GPT-5.4

So, before we jump into the coding test, let me give you a quick brief on GPT-5.4, because this is one of OpenAI’s biggest model releases in a while.

OpenAI released GPT-5.4 on March 5, 2026, and they are positioning it as their most capable and efficient frontier model for professional work.

What makes this model interesting is that OpenAI is not selling it as just a coding model, and not just a reasoning model either. They are basically pitching it as an all-round professional work model that combines strong reasoning, strong coding, better tool use, and much better performance on practical work like spreadsheets, presentations, etc.

Honestly, this part matters more than it sounds. A lot of real AI work is not just prompting or writing code, it is dealing with PDFs, spreadsheets, slides, and all kinds of unstructured data. That is also where something like Tensorlake makes sense, because it helps turn that mess into something models can actually work with.

And the specs are also pretty wild. GPT-5.4 supports a 1.05M token context window with 128K max output tokens, which is pretty good room to work with. All in all, this helps the model remember things better. Also, a thing to note is that the knowledge cutoff for this model is August 31, 2025.

Now, let's talk about the part we mostly care about.

On the official OpenAI benchmarks, GPT-5.4 scores 57.7% on SWE-Bench Pro (Public), which puts it basically side by side with GPT-5.3-Codex, a coding-focused model, at 56.8%. So yes, OpenAI says this general-purpose model is slightly better than GPT-5.3-Codex, a coding-focused model, which I personally have not had the best experience with compared to Claude models, and that is kind of wild to think about.

OpenAI says GPT-5.4 is their first general-purpose model with native computer-use capabilities, which is a pretty big deal. That means it is built not just to generate text or code, but also to operate across software, work from screenshots, and handle more agent-like workflows. On OSWorld-Verified, it scores 75.0%, which OpenAI says is above human performance on that benchmark. 🤯

One thing I also like here is that OpenAI is claiming GPT-5.4 is their most factual model yet. It is said to be 18% less likely to contain any errors compared to GPT-5.2.

For API developers, pricing matters, of course.

The standard GPT-5.4 model is listed at $2.50 per 1M input tokens, $0.25 cached input, and $15 per 1M output tokens. GPT-5.4 Pro is way more expensive at $30 input and $180 output per 1M tokens, and OpenAI says it can take several minutes on hard tasks, so that one is clearly for cases where you really want the best answer and are okay paying for it.

💁 The normal GPT-5.4 model is probably the one most people will actually care about day to day, and that's what I'd prefer.

And as always, benchmarks are benchmarks. But on paper at least, GPT-5.4 looks like one of the strongest all-rounder models OpenAI has shipped so far.

Quick Coding Test

As this is a general-purpose model instead of a coding-tuned model, comparing the model's ability solely on coding is just not fair. But as developers, we mostly care about how good the model is at coding anyway, so just to give you an idea of how this model performs, we will do a quick test.

As you can see, there's not much difference in SWE-Bench between GPT-5.4 and GPT-5.3-Codex:

GPT-5.4: Latency (s): 1,053, Accuracy: 57.7%, Effort: xhigh
GPT-5.3-Codex: Latency (s): 1,114, Accuracy: 57.2%, Effort: xhigh

But to give you an idea of what to expect from this model in coding, I will run one small, quick test.

Let's take two general models, one from Anthropic, Claude Sonnet 4.6, and one from OpenAI, GPT-5.4, not pro, and compare them against each other to show the difference in their coding skills.

For the test, we will use the following CLI coding agents:

Claude Sonnet 4.6: Claude Code (Anthropic’s terminal-based agentic coding tool)
OpenAI GPT-5.4: Codex CLI

As GPT-5.4 is said to be strong in frontend, why not test it on frontend itself?

Test: Figma Design Clone with MCP

In this test, we'll be comparing both models on a Figma design, a complex dashboard with so many things happening in the UI.

Here's the Figma design that I'll ask both models to clone:

Prompt:

Build a **pixel-accurate clone** of the attached Figma design frame using the **provided Next.js project** as the starting point. Do **not** create a new project. Instead, implement the UI inside the existing codebase.

https://www.figma.com/design/8quNKljV0spv67VAGsA75D/Dashboard-Design-Concept--Community---Copy-?node-id=69-123&t=Tvu2UB7UDMqkvPRb-4

Please match the design as closely as possible, with close attention to layout, spacing, alignment, typography, colors, borders, shadows, corner radius, and overall visual balance.

Requirements:

* use the existing **Next.js** setup
* keep the code clean and componentized
* make the page responsive without changing the intended design
* use semantic HTML where appropriate
* avoid adding your own design decisions unless necessary
* if any part of the design is unclear, make the most reasonable choice and stay visually consistent

Prioritize **design accuracy first**, then code quality.

GPT-5.4

GPT-5.4 pretty much one-shotted the entire implementation in one go, which was honestly nice to see. It did not need any follow-up prompt, no fixing, nothing. It just took the Figma frame through MCP and started building the whole thing right away.

The final result actually looked decent. I would not call it pixel-perfect by any means, but compared to Claude Sonnet 4.6, I’d say the implementation looked noticeably better overall. The whole thing feels more like a static picture of the design than an interface you can actually interact with.

Time-wise, it took roughly 5 minutes to get to a working to the working build.

Here’s the demo:

You can find the code it generated here: GPT-5.4 Code

Token usage looked like this:

Total Token Usage: 166,501
Input Token Usage: 151,595
Cached Input Tokens: 1,291,776
Output Token Usage: 14,906
Reasoning Tokens: 1,479

And the following code changes:

Code Changes: 3 files changed, 803 insertions(+), 82 deletions(-)

To be honest, I still would not say this is the kind of code implementation you can just ship straight to production and call it done. But for a one-shot frontend clone from a Figma frame, this was a pretty solid attempt.

Claude Sonnet 4.6

Claude Sonnet 4.6 went straight into the implementation right away. It did run into an issue at first, not really a build error, but more of one of those annoying Next.js image gotchas:

After that, I gave it a quick follow-up prompt, and almost instantly, it fixed the issue and came back with a decent implementation.

As you’d expect, it did manage to clone the project structure and get the UI in place. And again, the same issue, there's just no functionality whatsoever. It just feels like a picture with no interactivity.

Here’s the demo:

You can find the code it generated here: Claude Sonnet 4.6 Code

Time-wise, it took 9 minutes 56 seconds to get to a working result, and the follow-up fix was pretty much instant.

Token usage, based on Claude Code’s model stats, looked like this:

Input Token Usage: 84
Output Token Usage: 35.4K

And the following code changes:

Code Changes: 10 files changed, 1017 insertions(+), 84 deletions(-)

To be honest, I’m not really impressed, but I’m not disappointed either. The result feels pretty neutral overall. It was able to use tools, get fairly close to the UI, and produce something usable for comparison, but the implementation itself feels a bit weird and not all that convincing.

Conclusion

So, after all the benchmarks, claims, and hype, I think the fairest takeaway is this: GPT-5.4 looks very strong on paper, and for a lot of people it works and is an upgrade, but it still doesn’t seem like it is the best model you can get for coding.

So yeah, I’d say GPT-5.4 is probably one of the strongest all-rounder models OpenAI has shipped so far, but whether it beats Claude, be it Sonnet or Opus, for coding in real usage is still something you’ll want to judge from your actual hands-on testing, not just benchmarks.

And honestly, that’s the real takeaway here anyway.

These models keep getting better at a speed that is honestly hard to keep up with. So rather than getting too stuck on who won one benchmark, the better thing to do is probably to keep building, keep testing, and keep learning how to use these models better for your use case.

What do you think, is GPT-5.4 actually that good, or is Claude still your go-to? 👇

Shrijal AcharyaFollow

Full Stack SDE • Open-Source Contributor • Collaborator @Oppia • Mail for collaboration

🔥Claude Opus 4.6 vs. Sonnet 4.6 Coding Comparison ✅

Shrijal Acharya — Thu, 05 Mar 2026 14:04:59 +0000

Anthropic recently dropped the updated Claude 4.6 lineup, and as usual, the two names everyone cares about are Opus 4.6 and Sonnet 4.6.

Opus is the expensive “best possible” model, and Sonnet is the cheaper, more general one that a lot of people actually use day to day. So I wanted to see what the real gap looks like when you ask both to build something serious, not a toy demo.

Benchmark-wise, there’s a difference of course, but it doesn’t look that huge when it comes to SWE and agentic coding.

I kept it super basic: one test (but a big one), same prompt, same workflow. I just compared how close they got without me stepping in.

⚠️ NOTE: Don’t take the result of this test as a hard rule. This is just one real-world coding task, run in my setup, to give you a feel for how these two models performed for me.

TL;DR

If you just want the takeaway, here’s the deal with these models:

First, Opus 4.6 is the peak for coding right now. At the time of writing, it’s basically the OG, and nothing else comes that close.

Claude Opus 4.6 had a cleaner run. It hit a test failure too, but fixed it fast, shipped a working CLI + Tensorlake integration, and did it with way fewer tokens. Rough API-equivalent cost (output only) came out around ~$1.00, which is kind of wild for how big the project is.
Claude Sonnet 4.6 Surprisingly close for a cheaper, more general model. It built most of the project and the CLI was mostly fine, but it ran into the same issue as Opus and couldn’t fully recover. Even after an attempted fix, Tensorlake integration still didn’t work. Output-only cost was about ~$0.87, but it used way more time and tokens overall to get there.

💡 Obviously, this isn’t a test to “compare” the two head-to-head. It’s just to see the difference in code quality. In general, there’s never really been a fair comparison between Opus and Sonnet since their very first launch, Opus has always been on another level.

Test Workflow

ℹ️ NOTE: Before we start this test, I just want to clarify one thing. I'm not doing this test to compare whether Sonnet 4.6 is better than Opus 4.6 for coding, because obviously Opus 4.6 is a lot better. This is to give you an idea of how well Opus 4.6 performs compared to Sonnet.

For the test, we will use everyone's favorite CLI coding agent, Claude Code.

As both models are from Anthropic, it works best for both and is not biased toward either.

We will test both models on one decently complex task:

Task: Build a complete Tensorlake project in Python called research_pack, a “Deep Research Pack” generator that turns a topic into:
a citation-backed Markdown report, and
a machine-readable source library JSON with extracted text, metadata, summaries, you get the idea.

It also has to ship a nice CLI called research-pack with commands like:

research-pack run "<topic>"
research-pack status <run_id>
research-pack open <run_id>

We’ll compare the overall feel, code quality, token usage, cost, and time to complete the build.

💡 NOTE: Just like my previous tests, I’ll share each model’s changes as a .patch file so you can reproduce the exact result locally with git apply <file.patch>.

Why Tensorlake?

Tensorlake is a solid choice for this Opus 4.6 vs Sonnet 4.6 test because it is a real platform with enough complexity to quickly show whether a model can actually build something end to end. It has an agent runtime with durable execution, sandboxed code execution, and built in observability, so the test is not just writing a few functions, it is wiring up a production workflow.

And selfishly, it is also a good dogfood moment. 👀 If a model can spin up a Tensorlake project from scratch and get it working, that is a pretty strong sign for two things: these recent models are getting scary good and how usable Tensorlake is for building serious agent style pipelines.

Coding Tests

Test: Deep Research Agent

For this test, both models had to build the research_pack Tensorlake project in Python. The goal was simple: give it a topic, it crawls stuff, figures out sources, improves them, and spits out:

report.md with [S1] style citations
library.json with the full source library
a clean CLI: research-pack run/status/open
plus Tensorlake deploy support so you can trigger it as an app, not just locally

You can find the prompt I’ve used here: Research Agent Prompt

One thing that went a bit crazy is that both models ran into basically the exact same/similar issue during the run.

That shows how similarly these models can behave, which is kind of creepy. If you give them the exact same task and constraints, they’ll often make similar choices. I wanted to call that out because you might’ve noticed the same pattern too.

Not surprisingly, Opus fixed it much faster and with way fewer tokens. Sonnet took longer, burned a lot more context trying to debug it, and even after the fix pass, it still didn’t fully work.

Claude Opus 4.6

Opus was pretty straightforward.

It did hit a failure while running tests, but it was a quick fix. After that, everything looked clean: CLI worked, offline mode worked, and overall all the feature flags seem to work perfectly.

Here’s the acceptance checklist it generated at the end, I really love it as it created this after making sure all tests pass, and everything is in place, that's how it's done.

Here's the demo of the working CLI:

Note: The API key visible in the below demo videos has been revoked. Please don’t try to use it.

...and how it integrates with Tensorlake:

You can find the code it generated here in a patch file: Opus 4.6 Patch file

Cost: ~$1.001

ℹ️ NOTE: As I'm using a Claude plan and not on API usage, this is roughly calculated based on the input/output tokens.

Duration: 20 minutes 6 seconds + ~1 min 40 sec for the fix

Output Token Usage: 33.2K + ~4K for the fix

Code Changes: 156 files changed, 95013 insertions(+)

ℹ️ You can see the complexity of the project for yourself, and you’ll probably be shocked at how good these models have gotten. It’s no longer just boilerplate or small refactors. They can build a complete, end-to-end project from scratch from a single prompt. We’re officially in the real AI era.

Claude Sonnet 4.6

Sonnet was… close, but not quite as clean as Opus.

Just like Opus, it ran into a test failure during the run. This is one of those things you’ll notice with similar models: same prompt, same codebase, and they sometimes hit the exact similar weird issue.

Here’s the demo of the CLI (you’ll see it mostly working, but there are some rough edges) and not as well implemented as Opus:

...and how it integrates with Tensorlake:

It's not working as you can see. Sonnet did attempt a fix, but still couldn't get to a working state with Tensorlake. But overall, it was super close.

You can find the code it generated here: Sonnet 4.6 Patch

Cost: ~$0.87

ℹ️ Same as Opus 4.6, this is an approximate cost based on the input/output tokens.

Duration: 33 minutes 48 seconds + ~3m 18s for the attempted fix

Output Token Usage: 52.9K + ~5K for the fix (didn't work)

Code Changes: 88 files changed, 23253 insertions(+)

🤷‍♂️ I can’t really complain about Sonnet’s performance, other than this one issue. It still got almost everything working. And to be fair, Sonnet isn’t Anthropic’s flagship coding model like Opus. It’s more of a general-purpose model, and Opus also comes with a pretty big cost difference, so the gap in code quality is kind of expected.

And please don’t try using the API keys shown in the video, as it’s already revoked.

Conclusion

Opus as a lineup is just too good. If you want an end-to-end product that works most of the time with minimal hand-holding, go with Opus. If you want something cheaper, and you’re okay finishing the last bit yourself, Sonnet is still solid.

Even in this one test, you can already see the gap in implementation quality, token usage, and time spent.

And if Anthropic can cut Opus to half its price, or even get it close to Sonnet’s, it’d be over for most other models.

For me, the best way to use these models is still the same: let them build most of it fast, then run it, test it, and clean up the rough parts yourself.

Let me know your thoughts in the comments. ✌️

The End of Database-Backed Workflow Engines: Building GraphRAG on Object Storage

Diptanu Gon Choudhury — Tue, 03 Feb 2026 14:09:26 +0000

GraphRAG sounds elegant in theory: build a knowledge graph from your documents, traverse it intelligently, and get better answers than vanilla RAG.

Then you look at the compute requirements.

To build a GraphRAG system, you need to: parse documents, chunk text, generate embeddings for every chunk, extract concepts from every chunk, compute pairwise similarities, build graph edges, and store everything in a queryable format. For a single 100-page PDF, that’s thousands of API calls, millions of similarity computations, and hours of processing.

Now imagine doing this for 10,000 documents. Or 100,000

What GraphRAG Actually Needs from Infrastructure

The algorithm is straightforward: chunk, embed, extract concepts, build edges, traverse. The infrastructure requirements are not.

Parallel execution

‍Documents are independent. Processing them sequentially wastes time. You need a system that can spin up workers on demand and distribute work across them.

Heterogeneous compute

PDF parsing needs memory. Embedding generation is I/O-bound waiting on API calls. Concept extraction needs CPU for NLP models. Running all of these on the same machine means over-provisioning for the hungriest step.

Durable execution

A 10-hour ingestion job will fail somewhere. Network timeout. Rate limit. OOM. When step 3 fails, it needs to read step 2’s output from somewhere durable. Without checkpointing, you start over from zero.

Job orchestration

You need something that spins up workers, tracks dependencies, retries failures, aggregates partial results, and decides whether to proceed or abort.

The DIY Stack

Building this yourself means assembling:

Kubernetes

‍Kubernetes is used for container orchestration. But Kubernetes doesn’t know anything about your jobs. It manages containers, not computations. It won’t schedule your tasks, track dependencies, or handle fan-out.

Celery + Redis

‍Celery and Redis are typically used for task queuing. Note: queuing, not parallel execution. Celery distributes tasks to workers, but it is fundamentally a message broker with worker processes attached. It doesn’t understand data locality, can’t optimize task placement, and treats every task as independent. When you need real parallelism, map-reduce over ten thousand chunks, aggregating partial results, handling stragglers, Celery gets you partway there. For the rest, you end up writing glue code or reaching for Spark.

Spark

‍Spark is brought in for actual parallel compute. Now you are running a third system. Spark knows how to partition data, schedule parallel tasks, and aggregate results. But Spark wants to own the entire pipeline. Mixing Spark jobs with Celery tasks means shuffling data between systems, managing two job lifecycles, and debugging failures that span both.

Postgres

‍Postgres is used for job metadata and durability. This is the state that workflow engines like Airflow and Temporal manage except now you are building it yourself.

The glue code

‍You have a container orchestrator that doesn’t understand jobs, a task queue that doesn’t understand parallelism, and a compute engine that doesn’t integrate cleanly with either. You end up writing hundreds of lines to bridge these systems, and every bridge is a place where failures hide.

And this assumes you get it right the first time. You won't.

Kubernetes was built for orchestrating long-running microservices, not bursty batch jobs. The Cluster Autoscaler checks for unschedulable pods every 10 seconds, then provisions nodes that take 30-60 seconds to come online. For a GraphRAG pipeline that needs to fan out to 500 workers immediately, that's minutes of latency before work even starts. The autoscaler prioritizes stability over speed a reasonable tradeoff for web services, but painful for batch processing.

This is why most GraphRAG implementations stay as notebooks. The infrastructure tax is too high.

A Different Approach: Object-Store-Native Compute

For the past two years, we've been quietly building a new serverless compute stack for AI workloads at Tensorlake.

It powers our Document Ingestion API, which processes millions of documents every month across a heterogeneous fleet of machines fully distributed, fully managed. Document processing was our testbed: OCR, layout detection, table extraction, entity recognition. Every document touches multiple models, multiple machines, multiple failure modes. If the infrastructure couldn't handle that, it couldn't handle anything.

But the compute stack itself is general purpose. It replaces the entire Kubernetes + Celery + Spark + Postgres stack with a single abstraction:

Write your workflow as if it runs on a single machine. In production, it gets transparently distributed across CPUs and GPUs, and scales to whatever the workload demands.

No queues to configure. No job schedulers to manage. No Spark clusters to provision. No glue code bridging systems that weren't designed to work together.

The key insight: use S3 as the backbone for durable execution instead of databases. AI workloads deal in unstructured data—documents, images, embeddings, model outputs. This data already lives in object storage. By building the execution engine around S3 rather than Postgres or Cassandra, we eliminated an entire class of serialization problems and made checkpointing nearly free.

GraphRAG on Tensorlake

Each step runs as an isolated function with its own compute requirements.

Step-level functions

The magic is in .map(). Fan out to thousands of workers with one line:

Execution Flow

When a function fails, Tensorlake doesn't re-execute successful steps - it reads the checkpointed output from S3 and continues. If the pipeline dies at chunk 847, the retry resumes from the last checkpoint, not from zero.

This isn't a batch job you run manually, it's a live HTTP endpoint. Deploy once, and it's available on-demand whenever someone wants to add a document to the knowledge graph:

No documents in the queue? The system scales to zero. A thousand PDFs arrive at once? Tensorlake spins up workers to handle them in parallel. You're not paying for idle clusters or babysitting Spark jobs. The infrastructure responds to the workload, not the other way around.

The Results

Try It

git clone https://github.com/tensorlakeai/examples/tree/main/graph-rag-pipeline

cd graph-rag-pipeline

tensorlake secrets set OPENAI_API_KEY <your-key>
tensorlake secrets set NEO4J_URI neo4j+s://xxx.databases.neo4j.io
tensorlake secrets set NEO4J_PASSWORD <password>

tensorlake deploy app.py

For a small proof of concept, a notebook is fine. For production GraphRAG with retries, scale, and real users, you need infrastructure that doesn’t become the bottleneck.

Built with Tensorlake and Neo4j. See the GraphRAG paper for the original algorithm.

Diptanu Gon ChoudhuryFollow

/diptanu_gonchoudhury_23e

🔥 Claude Opus 4.5 vs GPT 5.2 High vs Gemini 3 Pro: Production Coding Test ✅

Shrijal Acharya — Sun, 18 Jan 2026 12:41:12 +0000

Okay, so right now the WebDev leaderboard on LMArena is basically owned by the big three: Claude Opus 4.5 from Anthropic, GPT-5.2-codex (high) from OpenAI, and finally everybody's favorite, Gemini 3 Pro from Google.

So, I grabbed these three and put them into the same existing project (over 8K stars and 50K+ LOC) and asked them to build a couple of real features like a normal dev would.

Same repo. Same prompts. Same constraints.

For each task, I took the best result out of three runs per model to keep things fair.

Then I compared what they actually did: code quality, how much hand-holding they needed, and whether the feature even worked in the end.

⚠️ NOTE: Don't take the result of this test as a hard rule. This is just a small set of real-world coding tasks that shows how each model did for me in that exact setup and gives you an overview of the difference in the top 3 models' performance in the same tasks.

TL;DR

If you want a quick take, here’s how the three models performed in our tests:

Claude Opus 4.5 was the most consistent overall. It shipped working results for both tasks, and the UI polish was the best of the three. The main downside is cost. If they find a way to achieve this performance while reducing cost, it will actually be over for most other models.

GPT-5.2-codex (high) was one of the best. But it's obviously slower due to the higher reasoning. When it hit, the code quality and structure were great, but it needed more patience than the other two in this repo.

Gemini 3 Pro was the most efficient. Both tasks worked, but the output often felt like the minimum viable version, especially on the analytics dashboard.

💡 If you want the safest pick for real “ship a feature in a big repo” work, Opus 4.5 felt the most reliable in my runs. If you care about speed and cost and you’re okay polishing UI yourself, Gemini 3 Pro is a solid bet.

Test Workflow

For the test, we will use the following CLI coding agents:

Claude Opus 4.5: Claude Code (Anthropic’s terminal-based agentic coding tool)
Gemini 3 Pro: Gemini CLI
GPT-5.2 High: Codex CLI

Here’s the repo used for the entire test: iib0011/omni-tools

We will check the models on two different tasks:

Task 1: Add a global Action Palette (Ctrl + K)

Each model is asked to create a global action menu that opens with a keyboard shortcut. This feature expands on the current search by adding actions, global state, and keyboard navigation. This task checks how well the model understands current UX patterns and avoids repetition without breaking what's already in place.

Task 2: Tool Usage Analytics + Insights Dashboard

Each model had to add real usage tracking across the app, persist it locally, and then build an analytics dashboard that shows things like the most used tools, recent activity, and basic filters.

We’ll compare code quality, token usage, cost, and time to complete the build.

💡 NOTE: I will share the source code changes for each task by each model in a .patch file. This way, you can easily view them on your local system by cloning the repository and applying the patch file using git apply <path_file_name>. This method makes sharing changes easier.

Real-world Coding Tests

Test 1: Add a global Action Palette (Ctrl + K)

The task is simple: all models start from the same base commit and then follow the same prompt to build what is asked in the prompt.

And obviously, as mentioned, I will evaluate the response from the model from the "Best of 3."

Let's start off the test with something interesting:

Here's the prompt used:

This project already has a search input on the home page that lets users find tools. I want to add an improved, global version of this idea that works as an **Action Palette**, similar to what you see in editors like VS Code.

**What to build**

* Pressing **Ctrl + K** (or Cmd + K on macOS) should open a centered action palette overlay from anywhere in the app.
* The palette should support:
  * Searching and navigating to tools (reuse existing tool metadata)
  * Executing actions, such as:

    * Toggle dark mode
    * Switch language
    * Toggle user type filter (General / Developer)
    * Navigate to Home and Bookmarks
    * Clear recently used tools

* Fully keyboard-driven experience:

  * Type to filter
  * Arrow keys to navigate
  * Enter to execute
  * Escape to close

**Notes**

* This should not replace the existing home page search. Think of it as a more powerful, global version that combines navigation and actions.
* The implementation should follow existing patterns, styling, and state management used in the codebase.

GPT-5.2-Codex (high)

GPT-5.2 handled this surprisingly well. The implementation was solid end to end, and it basically one-shotted the entire feature set, including i18n support, without needing multiple correction passes.

That said, it did take a bit longer than some other models (~20 minutes), which is expected since reasoning was explicitly set to high. You can clearly see the model spending more time thinking through architecture, naming, and edge cases rather than rushing to output code. The trade-off felt worth it here.

The token usage was noticeably higher due to the reasoning set to high, but the output code reflected that.

Here's the demo:

You can find the code it generated here: GPT-5.2 High Code

Cost: ~$0.9-1.0
Duration: ~20 minutes (API time)
Code Changes: +540 lines, minimal removals
Token Usage:
- Total: ~203k
- Input: ~140k (+ cached context)
- Output: ~64k
- Reasoning tokens: ~47k

💡 NOTE: I ran the exact same prompt with the same model using the default (medium) reasoning level. The difference was honestly massive. With reasoning set to high, the quality of the code, structure, and pretty much everything jumps by miles. It’s not even a fair comparison.

Claude Opus 4.5

Claude went all in and prepared a ton of different strategies. At the start, it did run into build issues, but it kept running the build until it was able to fix all the build and lint issues.

The entire run took me about 7 minutes 50 seconds, which is the fastest among the models for this test. The features all worked as asked, and obviously, the UI looked super nice and exactly how I expected.

Here's the demo:

You can find the code it generated here: Claude Opus 4.5 Code

To be honest, this exceeded my expectations; even the i18n texts are added and displayed in the UI just as expected. Absolute cinema!

Cost: $0.94
Duration: 7 min 50 sec (API Time)
Code Changes: +540 lines, -9 lines

Gemini 3 Pro

Gemini 3 got it working, but it's clearly not on the same level as GPT-5.2 High or Claude Opus 4.5. The UI it built is fine and totally usable, but it feels a bit barebones, and you don't get many choices in the palette compared to the other two.

One clear miss is that language switching does not show up inside the action palette at all, which makes the i18n support feel incomplete even though translations technically exist.

Here's the demo:

You can find the code it generated here: Gemini 3 Pro Code

Cost: Low (helped significantly by cache reads)
Duration: ~10 minutes 49 seconds (API Time)
Code Changes: +428 lines, -65 lines
Token Usage:
- Input: ~79k
- Cache Reads: ~536k
- Output: ~10.7k
- Savings: ~87% of input tokens served from cache

Overall, Gemini 3 lands in a very clear third place here. It works, the UI looks fine, and nothing is completely broken, but compared to the depth, completeness, and polish of GPT-5.2 High and Claude Opus 4.5, it feels behind.

Test 2: Tool Usage Analytics + Insights Dashboard

This test is a step up from the action palette.

You can find the prompt I've used here: Prompt

GPT-5.2-Codex (high)

GPT-5.2 absolutely nailed this one.

The final result turned out amazing. Tool usage tracking works exactly as expected, data persists correctly, and the dashboard feels like a real product feature. Most used tools, recent usage, filters, everything just works.

One really nice touch is that it also wired analytics-related actions into the Action Palette from Test 1.

It did take a bit longer than the first test, around 26 minutes, but again, that’s the trade-off with high reasoning. You can tell the model spent time thinking through data modeling, reuse, and avoiding duplicated logic. Totally worth it here.

Here’s the demo:

You can find the code it generated here: GPT-5.2 High Code

Cost: ~$1.1–1.2
Duration: ~26 minutes (API time)
Code Changes: Large multi-file update, cleanly structured
Token Usage:
- Total: ~236k
- Input: ~162k (+ heavy cached context)
- Output: ~75k
- Reasoning tokens: ~57k

GPT-5.2 High continues to be slow but extremely powerful, and for a task like this, that’s a very good trade.

Claude Opus 4.5

Claude Opus 4.5 did great here as well.

The final implementation works end to end, and honestly, from a pure UI and feature standpoint, it’s hard to tell the difference between this and GPT-5.2 High. The dashboard looks clean, the data makes sense, and the filters work as expected.

Here’s the demo:

You can find the code it generated here: Claude Opus 4.5 Code

Cost: $1.78
Duration: ~8 minutes (API Time)
Code Changes: +1,279 lines, -17 lines

Gemini 3 Pro

Gemini 3 Pro gets the job done, but it clearly takes a more minimal approach compared to GPT-5.2 High and Claude Opus 4.5.

That said, the overall experience feels very bare minimum. The UI is functional but plain, and the dashboard lacks the polish and depth you get from the other two models.

Also, it didn't quite add the button to view the analytics right in the action palette, similar to the other two models.

Here’s the demo:

You can find the code it generated here: Gemini 3 Pro Code

Cost: Low, with heavy cache utilization
Duration: ~5 minutes (API Time)
Code Changes: +351 lines, -3 lines
Token Usage:
- Input: ~67k
- Output: ~7.1k
- Savings: ~85%+ input tokens served from cache

Overall, Gemini 3 Pro remains efficient and reliable, but in a comparison like this, efficiency alone is not enough. 🤷‍♂️

Conclusion

At least from this test, I can conclude that the models are now pretty much able to one-shot a decent complex work, at least from what I tested.

Still, there have been times when the models mess up so badly that if I were to go ahead and fix the problems one by one, it would take me nearly the same time as building it from scratch.

If I compare the results across models, Opus 4.5 definitely takes the crown. But I still don’t think we’re anywhere close to relying on it for real, big production projects. The recent improvements are honestly insane, but the results still don’t fully back them up.

For now, I think these models are great for refactoring, planning, and helping you move faster. But if you solely rely on their generated code, the codebase just won’t hold up long term.

I don't see any of these recent models as “use it and ship it” for "production," in a project with millions of lines of code, at least not in the way people hype it up.

Let me know your thoughts in the comments.

Shrijal AcharyaFollow

Full Stack SDE • Open-Source Contributor • Collaborator @Oppia • Mail for collaboration

Introducing Agentic Chart Extraction

Diptanu Gon Choudhury — Wed, 14 Jan 2026 11:31:43 +0000

Unlocking Visual Data: Introducing Agentic Chart Extraction

At Tensorlake, we're excited to announce a powerful new capability in our document parsing pipeline: Agentic Chart Extraction.

Agentic Chart Extraction uses an agentic approach to transform static images into dynamic, usable data, unlocking a new layer of value from your documents. Whether you are processing financial reports, scientific papers, or business presentations, you can now access the data behind the visuals.

In the example below, on the left side we find a scatter plot with a larger of points and to the right, the plotting of this data after being processed by our Agentic Chart Extraction. This is challenging since there is a large number of uncorrelated points. We show that our system can generate a structured output that matches the original chart and that we can use this output to replot the chart.

Key Capabilities

Chart type detection: High accuracy across common chart types (line, bar, scatter, pie).
Data series extraction: Returns structured series (category/value pairs, coordinates where available) ready for plotting or analytics.
Robustness: Handles multi-series charts, varying axis scales, and dense point clouds; retains good fidelity even on 50+ point series.
Deliverables: JSON outputs per-chart, evaluation reports, and plottable visualizations.

Supported Output Schemas

Pie chart: slice-centric schema with label, value and optional percentage, colors, and display flags (good for donut/pie summarization use-cases).
Bar chart: supports vertical/horizontal, named series for grouped/stacked bars, x_axis.categories, optional axis bounds/formatting, and per-bar display flags — ideal for categorical comparisons and time-binned revenue/metrics.
Line chart: x/y axis definitions, explicit values for x-axis (numeric or categorical), multiple series with styling (color, line_style, marker), and plotting hints (legend_position, grid) — suited for trends and dense time-series.
Scatter plot: per-series x_data/y_data arrays, marker styling (size, alpha, edge_color) and axis bounds — used for point-wise analyses and correlation extraction.

Schema-Driven Outputs — Directly Plottable

All extracted predictions conform to the predefined JSON schemas (pie/bar/line/scatter). That means:

Consistent ingestion: you can build a single parser that consumes every chart JSON produced by our system — no per-chart ad-hoc parsing required.
Direct re-plotting: each JSON contains numeric arrays plus rendering hints (axis labels, series names, colors, markers). The JSON can be fed directly into plotting code or BI tools to regenerate visuals.

Availability

Chart extraction is currently available in all OCR models. As shown in the example below, charts are extracted and structured in a consistent JSON format.

Additional Examples

Bar Chart Example

Original:

Plotted Prediction:

Predicted JSON:

{
  "type": "bar_chart",
  "title": "Annual Energy Consumption by Source (TWh)",
  "orientation": "vertical",
  "x_axis": {
    "label": "REGION",
    "categories": ["North America", "Europe", "Asia", "Africa"]
  },
  "y_axis": { "label": "TWh", "min": 0, "max": 1000, "format": "number" },
  "series": [
    {
      "name": "Solar",
      "data": [120, 150, 200, 80],
      "color": "#FFD700",
      "show_values": false
    },
    {
      "name": "Wind",
      "data": [180, 220, 300, 60],
      "color": "#00BFFF",
      "show_values": false
    },
    {
      "name": "Hydro",
      "data": [250, 180, 400, 120],
      "color": "#32CD32",
      "show_values": false
    },
    {
      "name": "Nuclear",
      "data": [300, 450, 280, 20],
      "color": "#FF4500",
      "show_values": false
    },
    {
      "name": "Fossil Fuels",
      "data": [500, 400, 800, 350],
      "color": "#B0B0B0",
      "show_values": false
    }
  ],
  "bar_style": "grouped",
  "grid": true
}

Scatter Plot Example

Original:

Plotted Prediction:

Predicted JSON:

{
  "type": "scatter_plot",
  "title": "Urban vs Rural: Income vs Spending",
  "x_axis": {
    "label": "Annual Income ($k)",
    "min": 10,
    "max": 150,
    "scale": "linear"
  },
  "y_axis": {
    "label": "Annual Spending ($k)",
    "min": 0,
    "max": 100,
    "scale": "linear"
  },
  "series": [
    {
      "name": "Urban",
      "x_data": [
        24, 32, 30, 35, 33, 36, 35, 39, 42, 43, 43, 45, 44, 53, 56, 67, 69, 70,
        72, 75, 80, 80, 82, 91, 91, 91, 93, 94, 96, 96, 110, 111, 123, 125, 128,
        130, 134
      ],
      "y_data": [
        4, 4, 11, 11, 19, 22, 26, 26, 23, 19, 14, 13, 36, 31, 30, 27, 30, 52,
        53, 35, 41, 46, 49, 40, 59, 71, 57, 58, 45, 32, 57, 67, 69, 71, 93, 79,
        70
      ],
      "color": "#5da5da",
      "marker": "o",
      "alpha": 0.75
    },
    {
      "name": "Rural",
      "x_data": [
        25, 27, 28, 27, 29, 34, 38, 41, 42, 45, 47, 47, 49, 61, 63, 68, 71, 71,
        71, 77, 80, 82, 91, 98, 105, 106, 106, 109, 115, 119, 121, 121, 125,
        128, 129, 131, 133, 134
      ],
      "y_data": [
        7, 17, 16, 33, 35, 19, 17, 29, 20, 36, 41, 29, 36, 32, 27, 51, 55, 41,
        44, 47, 44, 63, 48, 63, 62, 51, 70, 71, 78, 77, 78, 73, 91, 90, 89, 81,
        80, 78
      ],
      "color": "#faa43a",
      "marker": "s",
      "alpha": 0.75
    }
  ],
  "legend_position": "upper right",
  "grid": true
}

Linear Chart Example

Original:

Plotted Prediction:

Predicted JSON:

{
  "type": "line_chart",
  "title": "Uncorrelated Remote Sensor Readings",
  "x_axis": {
    "label": "Observation Minute",
    "values": [
      0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38,
      40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74,
      76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108,
      110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138,
      140, 142, 144, 146, 148, 150
    ],
    "scale": "linear"
  },
  "y_axis": { "label": "Value", "min": 15, "max": 90, "scale": "linear" },
  "series": [
    {
      "name": "Room A (Stable)",
      "data": [
        56, 54, 48, 55, 50, 54, 41, 55, 52, 49, 40, 44, 53, 40, 60, 42, 50, 39,
        41, 53, 48, 46, 55, 62, 51, 50, 51, 49, 56, 45, 42, 55, 44, 58, 57, 61,
        43, 48, 40, 51, 52, 51, 46, 55, 44, 46, 50, 48, 45, 50, 55, 45, 48, 54,
        55, 43, 41, 50, 49, 50, 52, 58, 50, 47, 43, 53, 48, 48, 51, 56, 44, 66,
        45, 44, 55, 48
      ],
      "color": "#F472B6",
      "line_style": "-"
    },
    {
      "name": "Room B (Cooling)",
      "data": [
        81, 80, 76, 80, 79, 71, 85, 76, 80, 73, 66, 79, 75, 66, 73, 73, 71, 73,
        79, 71, 66, 70, 67, 66, 71, 68, 66, 63, 73, 69, 66, 72, 65, 64, 66, 63,
        69, 64, 64, 61, 67, 60, 70, 65, 64, 63, 69, 64, 64, 61, 67, 60, 70, 65,
        64, 68, 65, 60, 68, 62, 58, 58, 64, 57, 57, 58, 59, 64, 55, 60, 56, 58,
        52, 59, 60, 57
      ],
      "color": "#9CA3AF",
      "line_style": "-"
    },
    {
      "name": "Room C (Cyclic)",
      "data": [
        31, 32, 34, 33, 38, 39, 37, 39, 40, 38, 38, 35, 36, 31, 31, 31, 27, 27,
        30, 26, 26, 24, 21, 21, 25, 20, 20, 22, 19, 21, 22, 25, 24, 24, 22, 25,
        24, 31, 31, 32, 34, 36, 36, 39, 38, 38, 32, 55, 49, 52, 22, 26, 21, 21,
        19, 19, 24, 19, 21, 22, 20, 25, 27, 25, 29, 32, 32, 36, 32, 39, 39, 64,
        42, 62, 40, 34
      ],
      "color": "#FDE047",
      "line_style": "-"
    },
    {
      "name": "Outdoor (Variable)",
      "data": [
        40, 41, 41, 39, 41, 40, 42, 48, 47, 45, 41, 39, 40, 41, 36, 38, 40, 39,
        41, 41, 40, 41, 42, 40, 40, 44, 41, 41, 39, 38, 39, 37, 39, 36, 36, 38,
        40, 38, 39, 39, 39, 38, 39, 40, 41, 42, 44, 43, 46, 42, 42, 43, 47, 45,
        46, 47, 45, 45, 45, 46, 52, 53, 57, 57, 60, 57, 55, 57, 60, 58, 55, 51,
        53, 52, 53, 53
      ],
      "color": "#9CD9D3",
      "line_style": "-"
    }
  ],
  "legend_position": "upper right",
  "grid": true
}

SDK Usage

Install or update to the latest version of tensorlake.

pip install --upgrade tensorlake

You can enable chart extraction in your parse request by selecting as an enrichment option.

from tensorlake.documentai import DocumentAI
from tensorlake.documentai.models.options import (
  EnrichmentOptions,
)

enrichment_options = EnrichmentOptions(
  chart_extraction=True,
)

doc_ai = DocumentAI(api_key=API_KEY)

parse_id = doc_ai.read(
  file_id="file_XXX",  # Replace with your file ID or URL
  enrichment_options=enrichment_options,
)

API Usage

You can enable chart extraction in your parse request by selecting as an enrichment option.

// POST /api/v2/parse

{
  "enrichment_options": {
    "chart_extraction": true
  }
}

Claude Cowork: Architecture, Capabilities, and Usage Overview

Arindam Majumder — Tue, 13 Jan 2026 20:50:51 +0000

TL;DR

Understand what Cowork is and how it extends Claude Code’s agentic capabilities to non-coding, knowledge-based work.
Learn how Cowork differs from standard chat-based AI interactions and why file access changes the workflow.
Get a high-level view of Cowork’s execution model, safety boundaries, and supported use cases.
Know who Cowork is for, what it can (and cannot) do today, and how to approach it responsibly as a beginner.

Introduction

Over the past year, large language models have moved beyond simple question–answer interactions toward systems that can plan, act, and execute multi-step tasks. One of the earliest practical examples of this shift was Claude Code, which allowed developers to delegate real work, such as editing files, running commands, and managing projects, to an AI agent. While this capability was initially designed for programming workflows, users quickly began applying it to a much broader range of tasks.

// Detect dark theme var iframe = document.getElementById('tweet-2010805682434666759-102'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2010805682434666759&theme=dark" }

This led to the introduction of Claude Cowork, released as a research preview on January 12, 2026. Cowork brings the same agentic architecture that powers Claude Code into the Claude Desktop app, making it accessible for knowledge work that does not involve writing code.

This article explains how Cowork works, how it differs from standard chat, and what practical limitations and safety considerations to understand before using it.

What Is Cowork?

Claude Cowork is an agentic task execution mode in the Claude Desktop app that allows Claude to plan and complete multi-step work on your behalf. Instead of responding to individual prompts, Claude operates on a shared folder, where it can read, edit, and create files to produce complete outputs based on the outcome you describe.

This differs from regular Claude conversations, which are optimized for short, interactive exchanges and require users to manage context and files manually. In Cowork, Claude maintains task context over time, breaks work into subtasks when needed, and progresses toward completion while keeping you informed and asking for approval before significant actions.

Cowork is currently available as a research preview for Max plan subscribers using the Claude Desktop app on macOS. It is not supported on web or mobile, does not persist across sessions, and is still evolving based on user feedback. For early access, you can join the waitlist here.

Core Design Principles

Below are the main principles that guide how Cowork is designed and how tasks are executed:

Outcome-oriented task execution: Cowork focuses on completing a clearly defined outcome rather than responding to individual prompts. Users describe what they want done, and Claude determines the steps needed to reach that result.
Persistent task planning and execution: Once a task starts, Claude maintains context throughout the execution. It creates a plan, breaks work into subtasks when required, and continues working without losing state or requiring repeated instructions.
Reduced back-and-forth interaction model: Cowork minimizes conversational overhead. Users do not need to repeatedly guide each step; instead, Claude progresses independently and allows input or corrections when necessary.
Transparency and user steering during execution: Claude surfaces its plan and progress while working. Users can monitor actions, intervene mid-task, or adjust direction before significant changes are made.

System Architecture Overview

Cowork is built on the same agentic foundations as Claude Code, but designed to operate safely and transparently within a desktop environment. The following components define how Cowork executes tasks and interacts with your system:

Local execution on the user’s machine: Cowork runs directly within the Claude Desktop app on your computer. This allows Claude to work with local files and deliver outputs straight to your file system without requiring manual uploads or downloads.
Virtual Machine (VM) isolation: Task execution happens inside an isolated virtual machine environment. This separation limits the impact of errors or malicious behavior while still allowing Claude to perform real work on files you explicitly share.
File system access model: Claude can read, write, create, modify, and delete files only within the folders you grant access to. All file access is explicit, and Claude requests approval before taking significant or potentially destructive actions.
Internet and connector usage: Cowork can use Claude’s existing connectors to access external information. When paired with Claude in Chrome, it can also perform tasks that require browser access, with network permissions remaining under user control.
Shared agentic architecture with Claude Code: Cowork uses the same planning, task decomposition, and sub-agent coordination model as Claude Code. The key difference is accessibility: Cowork exposes these capabilities through the desktop interface for non-coding, knowledge-based workflows.

How Cowork Executes a Task

Below are the steps Cowork follows once you assign a task, from understanding your request to delivering the final output:

Step 1: Task description and intent analysis

When you describe a task, Claude first analyzes the intended outcome rather than treating it as a single prompt. It identifies the scope of work, required resources, and any constraints based on the files and permissions you’ve provided.

Step 2: Planning and decomposition into subtasks

Claude creates a structured plan and breaks the work into smaller, manageable subtasks when needed. This planning phase allows Claude to sequence actions logically and handle complex workflows without losing context.

Step 3: Sub-agent coordination and parallel execution

For more complex tasks, Claude may coordinate multiple internal workstreams in parallel. This allows different parts of the task to progress simultaneously, improving efficiency while maintaining alignment with the overall plan.

Step 4: Progress visibility and user intervention

As work progresses, Cowork surfaces what Claude is doing and why. You can monitor progress, step in to provide clarification, or course-correct before significant actions are taken.

Step 5: Output delivery to the local file system

Once the task is complete, Claude writes the final outputs directly to your local file system. Files are created, updated, or organized within the shared folders, ready for immediate use without additional formatting or manual handling.

Key Capabilities of Cowork

Cowork is designed to support complex, real-world knowledge work by combining agentic execution with direct access to files and external information. The capabilities below define what Cowork can reliably handle today.

Local File Access

Cowork allows Claude to read, write, create, and modify files within folders you explicitly share. This removes the need for manual uploads or downloads and enables end-to-end task execution. Access is controlled at the folder level, giving you precise control over what Claude can see and change.

Multi-Step and Long-Running Tasks

Cowork can handle workflows that require multiple steps without losing context. Tasks can run for extended periods without conversational timeouts interrupting execution. This makes it suitable for work that would otherwise require many separate chat interactions.

Professional Output Generation

Claude can produce finished deliverables rather than raw text outputs. This includes structured documents, spreadsheets with working formulas, and presentation files that are properly formatted and ready for use. Outputs are written directly to your local file system.

Browser and Connector Integration

Cowork can use your existing connectors to access external information sources. When paired with Claude in Chrome, it can also perform tasks that require browser access, with network permissions remaining under your control.

Typical Use Cases

Cowork is best suited for tasks that involve multiple steps, file access, and sustained execution. Below are common categories of work where its agentic model is especially effective:

File and document management: Organizing folders, renaming large batches of files, and converting collections of receipts or screenshots into structured expense reports.
Research and knowledge synthesis: Aggregating information from multiple sources, combining notes or documents, and extracting key themes or action items from transcripts and written material.
Document and presentation creation: Turning rough or unstructured inputs such as notes, voice memos, or drafts into well-formatted documents and presentation files.
Data processing and analysis: Cleaning and transforming datasets, performing statistical analysis, and generating visualizations directly from local data files.

Getting Started with Cowork

This section outlines what you need to use Cowork and how to start your first task. Since Cowork operates directly on your computer and can take real actions, setup and access are intentionally controlled.

Requirements

To use Cowork, the following conditions must be met:

Claude Desktop app on macOS: Cowork is available only through the Claude Desktop application and is not supported on web or mobile clients.
Max plan subscription: Access to Cowork is currently limited to users on the Max plan as part of a research preview.
An active internet connection: An internet connection is required throughout the session for task execution and coordination.

Accessing Cowork

Once the requirements are met, starting Cowork is straightforward:

Open the Claude Desktop app on macOS.
Use the mode selector to switch from Chat to Cowork, which appears as Tasks.
Describe the task you want Claude to complete and specify the desired outcome.
Review Claude’s proposed approach and approve it before execution begins.

The desktop app must remain open while Cowork is running. Closing the app or allowing the system to sleep will end the session and stop the task.

Usage, Permissions, and Security Considerations

Cowork is designed for heavier, more complex work than standard chat, which affects both usage limits and security responsibilities. Understanding these trade-offs is important for using it effectively and safely.

Cowork tasks consume more usage than regular chat because they involve planning, sub-task coordination, and long-running execution. Complex workflows can reach usage limits faster than conversational interactions.

You can use Cowork for tasks that truly benefit from file access and multi-step execution. Batch related work into a single task when possible, and rely on standard chat for simpler or exploratory requests.

Standard chat is better for quick questions, drafting text, or brainstorming. Cowork is more appropriate when the task requires persistent context, structured outputs, or direct interaction with local files.

Below are the permissions and security model of Cowork:

Folder-level access control: You explicitly choose which folders Claude can access. Claude cannot see or modify files outside of the permissions you grant, and access should be limited to only what the task requires.
Internet access permissions: Claude’s network access is restricted by default. If you extend access, such as through browser integration, you should limit it to trusted sources to reduce risk.
MCP (desktop extension) permissions: Desktop extensions expand Claude’s capabilities but also increase the attack surface. Each extension should be evaluated carefully, and permissions should be granted only when necessary.
User responsibility for granted access: You remain responsible for all actions Claude performs on your behalf, including file changes, data access, and interactions with external systems. Monitoring tasks and reviewing plans before approval is essential.

Current Limitations

Cowork is released as a research preview to better understand how agentic systems perform in real-world, non-coding workflows. Early access allows the team to observe usage patterns, identify failure modes, and improve safety mechanisms before wider adoption.

Cowork currently has the following limitations:

macOS-only availability: Cowork is available only through the Claude Desktop app on macOS and is not supported on web, mobile, or Windows.
No cross-device sync: Tasks and outputs do not sync across devices, even when using the same account.
No session persistence: The Claude Desktop app must remain open during execution. Closing the app or putting the system to sleep ends the session.
No memory across sessions: Cowork does not retain context or task history between sessions. Each task starts with a clean state.
No project or sharing support: Cowork cannot be used within projects, and sessions or artifacts cannot be shared with others.

Conclusion

Cowork represents a practical step beyond conversational AI by enabling Claude to plan and complete multi-step work with direct access to local files. It is best suited for tasks that require sustained context, structured outputs, and minimal manual coordination.

At the same time, its agentic nature makes permissions, oversight, and task scoping essential. Used thoughtfully, Cowork can simplify complex knowledge work while keeping users in control of how and where actions are taken.

Gemini 3 vs GPT-5.2: Detailed Coding Comparison

Arindam Majumder — Tue, 16 Dec 2025 17:46:19 +0000

TL;DR

Gemini 3 Pro has strong multimodal capabilities but produces simpler, less structured coding outputs. GPT-5.2 delivers more reliable reasoning and polished, production-ready code with minimal cleanup needed.

By the end of 2025, two major AI releases had reshaped how developers build and ship software: OpenAI’s GPT-5.2 and Google’s Gemini 3 Pro. Both bring improvements in reasoning, coding assistance, and multimodal capabilities, but they take different approaches to solving developer problems.

Gemini 3 Pro was officially launched on November 18, 2025, as Google’s most advanced multimodal model yet, designed to handle complex text, image, audio, and video tasks alongside code and reasoning.

Long-Term Memory:
Episodic: Stores specific past events or "episodes" of interactions (e.g., "The user mentioned they preferred Python over Java last week").
Semantic: A repository of general facts, world knowledge, and entity relationships (e.g., "Paris is the capital of France").
Procedural: Contains the skills and "how-to" logic for performing tasks, often encoded as system prompts, scripts, or specific tool-calling protocols.

A few weeks later, GPT-5.2 was released on December 11, 2025, with a focus on predictable reasoning, strong code generation, and reliable long-context performance. It’s built to support workflows where accuracy and consistency matter most.

Because each model excels in different areas, the best choice often depends on the type of project you are building. To make that clearer, this article compares both models and evaluates them through a set of practical coding challenges to see how they perform in real development scenarios.

Comparing GPT-5.2 and Gemini 3 Pro

Both models are fully multimodal and both support tool use, reasoning chains, and long-context inputs large enough to handle full repositories or multi-service architectures. They are built with safety controls and enterprise features, but they lean in different directions.

Gemini 3 Pro fits naturally into Google’s ecosystem, with tighter integrations across Search, YouTube, and media-centric workflows. It tends to feel more “creative” out of the box, especially for interactive or multimodal builds.

GPT-5.2, on the other hand, focuses on predictable reasoning and developer-grade code generation. Its step-by-step “Thinking” mode makes debugging, refactoring, and architecture planning straightforward, and its ChatGPT interface remains one of the most polished for technical work.

Benchmark Breakdown

Using numbers reported at launch, GPT-5.2 generally leads in structured reasoning and software engineering, while Gemini 3 Pro maintains an edge in multimodal depth.

Below is a streamlined comparison:

Benchmark	What It Measures	Gemini 3 Pro	GPT-5.2	Winner
GPQA Diamond	Graduate-level science (no tools)	91.9%	92.4%	GPT-5.2
AIME 2025	Advanced math (no tools)	95.0%	100%	GPT-5.2
ARC-AGI-2	Abstract visual puzzles (verified)	31.1%	52.9%	GPT-5.2
SWE-Bench Verified	Agentic software engineering	76.2%	80.0%	GPT-5.2
MMMU-Pro	Multimodal reasoning with Python tools	81.0%	80.4%	Gemini 3 Pro
LiveCodeBench Pro	Competitive coding (Elo rating)	2439	~2500+ (inferred from SWE performance)	Tie

Overall, GPT-5.2 pulls ahead on reasoning-heavy and engineering-focused benchmarks, making it stronger for complex coding workflows. Gemini 3 Pro counters with better multimodal depth, which shows up in tasks involving images, audio, and mixed-media reasoning.

TL;DR

GPT-5.2 consistently produced solutions that felt complete, polished, and ready for real use. Its outputs included stronger UI design, smoother interactions, and thoughtful features that went beyond the basic prompt.
Gemini 3 Pro implemented the core logic across all challenges, but the results were simpler. Interfaces were limited, customization options were minimal, and key usability features were often missing.
The ARC-AGI-2 leaderboard reflects this same pattern at scale, showing GPT-5.2 with a clear advantage in reasoning-heavy tasks and overall reliability.
Based on both the coding challenges and broader benchmarks, GPT-5.2 currently offers a better experience for developers who need high-quality, production-oriented code.

Why Coding Challenges?

Benchmarks show how a model performs in isolation, but coding challenges reveal how it behaves when you are actually building, debugging, structuring logic, handling state, and producing code that runs without fuss. They highlight the practical differences that matter in day-to-day development.

So let’s put both models to work and see how they handle real coding scenarios.

Challenge 1: Build a Music Visualizer Using Audio Frequency Data

Prompt:

Create a browser-based music visualizer that analyzes live audio frequency data using the Web Audio API and renders a real-time visualization on an HTML canvas. The visualizer should react dynamically to changes in amplitude and frequency, run smoothly at 60fps, and use efficient data processing suitable for continuous playback.

Gemini 3 Pro Response:

You can find the generated code here.
Output:

ChatGPT 5.2 Response

You can find the generated code here

Output:

Outcome Summary:

ChatGPT 5.2 delivered a more polished and feature-rich visualizer, with a better UI, customization options, and support for uploading and downloading audio files.

Gemini 3 Pro produced a functional visualizer but with a plain UI, limited interactivity, and no option to upload existing audio. ChatGPT 5.2 clearly provided the stronger solution in both design and functionality.

Challenge 2: Collaborative Markdown Editor With Live Preview

Prompt:

Build a web-based Markdown editor that supports real-time collaboration between multiple users. The interface should include a split-pane layout with a text editor on the left and a live preview on the right. As users type, the preview should update instantly without lag. Use WebSockets or a CRDT-based sync layer to merge edits safely, avoid conflicts, and keep all clients in sync. Add basic formatting shortcuts, keyboard navigation, and a clean UI suitable for embedding into a developer tool.

Gemini 3 Pro Response:

You can find the generated code in this repo.

Output:

ChatGPT 5.2 Response

You can find the generated code here.

Output:

Outcome Summary:

Both models produced functional editors that met the core requirements, but ChatGPT 5.2 handled the collaborative aspect more effectively. It added features such as customizable environment names and shareable invite links, making the collaboration experience feel complete.

Gemini 3 Pro implemented real-time syncing but did not present a clear collaborative environment, resulting in a less cohesive experience. ChatGPT 5.2 ultimately delivered the stronger solution for this challenge.

Challenge 3: WebAssembly Image Filter Engine

Prompt:

Create a browser-friendly image processing engine powered by WebAssembly. Write core filters—grayscale, blur, sharpen, and invert—in C++, compile them to WASM, and expose the functions to JavaScript. The app should allow users to upload an image, apply filters with minimal latency, and preview the results instantly. Focus on efficient memory handling between WASM and JS, and design the engine so additional filters can be added without refactoring the entire pipeline.

Gemini 3 Pro Response:

You can find the generated code here.

Output:

ChatGPT 5.2 Response

You can find the generated code here

Output:

Outcome Summary:

Both models successfully delivered a working WebAssembly image filter engine, but ChatGPT 5.2 provided a noticeably more refined solution. It offered multiple filter controls, adjustable blur strength, an easy way to revert to the original image, and thoughtful handling of edge cases.

Gemini 3 Pro produced functional filters, but the interface was limited, lacked fine-grained controls, and did not support combining multiple filters smoothly. Gemini accomplished the core task, but ChatGPT 5.2 delivered a far more complete, user-friendly, and flexible implementation.

Final Verdict

Across all three challenges, GPT-5.2 consistently delivered more complete, polished, and developer-ready solutions. Its interfaces were cleaner, its interactions smoother, and its features aligned closely with what real users expect. Gemini 3 Pro produced functional results, but they lacked refinement and felt limited in flexibility and usability.

GPT-5.2 did not just finish the tasks. It improved them in ways that showed a deeper understanding of real development work, while Gemini 3 Pro focused mainly on core functionality.

As the tweet sums up perfectly, the gap is not closing. It is widening. At this point, GPT-5.2 stands out as the more reliable and capable choice for practical coding tasks.

I Tried Letting Antigravity Build An Agent For Me. Here’s What Actually Happened

Arindam Majumder — Tue, 16 Dec 2025 09:56:21 +0000

I have used a long list of AI coding tools over the past few years, most of them built around the same pattern, as inline suggestions, chat-style prompts, and occasional refactors.

When Antigravity appeared recently, powered by Gemini 3 Pro, an agent-driven IDE where background processes can read your repo, propose changes, run commands, and interact with your app’s runtime environment.

When I first opened it, I saw those agents spin up automatically, parse the project structure, touch multiple files, run package scripts, and even click through the app in a built-in browser. It behaved less like an autocomplete layer and more like a set of automated workflows acting directly on the codebase.

That made me curious. Could this system actually deliver a complete feature if I stepped back and let it run?

To find out, I assigned Antigravity a real task from my own project and observed how far the agents could get with minimal intervention.

What follows is a breakdown of that experiment, what worked, what didn’t, and where agent-driven development currently stands from a practical engineering perspective.

Why I Did This Experiment

I wanted to understand how well Antigravity’s agent-driven workflow performs on realistic engineering tasks. Most AI tools can handle small changes, but a real feature spans multiple layers of a codebase. That’s why I designed a focused experiment:

Experiment: Guest Checkout + Abandoned Cart Email Recovery using Resend/Nodemailer with a TensorLake analytics hook.

This feature was ideal because it touches several parts of a typical project. It includes backend routes, database updates, UI work, an email flow, and a small analytics integration.

I ran the experiment to answer a few practical questions:

Can autonomous agents complete work that crosses backend, frontend, database, and external services?
Do they keep context when modifying multiple files and systems?
How much oversight does a developer actually need to provide?

The goal was simple. I wanted to see how far the agents could take a meaningful feature with minimal intervention and whether agent-driven development can support real-world workflows.

Here is the flow of the experiment:

The Setup

To keep the experiment grounded, I used a real project from my existing codebase. The stack is fairly standard and represents what many teams use today:

Frontend: React with TypeScript
Backend: Node and Express
Database: PostgreSQL
Auth: Basic JWT authentication
Emails: Resend or Nodemailer
Analytics: Tensorlake

Because the feature involved sending recovery emails, I also needed a Resend API key. Setting that up took about two minutes. It’s just a matter of creating a Resend account, grabbing the API key, and dropping it into your .env file.

Antigravity runs inside a unified environment, and its agents have access to several built-in tools:

Code Agent, which edits and refactors files
Terminal Agent, which runs scripts, migrations, and tests
Browser Agent, which interacts with the running application
Research Agent, which pulls patterns or references when needed
Agent Manager, which coordinates all of them

These agents share context and can act independently. They can open files, update logic, install packages, modify migrations, and verify behavior in the browser.

For the experiment, I placed the project on a clean branch and provided a clear feature description. After that, I stepped back and let the agents decide how to approach the work.

How to Execute This Experiment

This is the exact flow I used so that another developer could reproduce the experiment using the same codebase. The project I worked with is available here.

1. Prepare the Repo

Before letting any agents touch the code, make sure the project is in a stable state.

Clone the repo and install dependencies
- git clone https://github.com/Studio1HQ/Ecommerce-platform
- npm install or pnpm install
Create a new branch from main
- Example: feat/guest-checkout-ag-experiment
Confirm the app runs locally without errors
- npm run dev or your project's start script
Run the existing test suite
- npm test or the equivalent command

The goal here is simple: give Antigravity a clean baseline so any failures that follow belong to the experiment, not leftover issues.

2. Open the Project in Antigravity

Open Antigravity
Load the existing repo for this experiment
Wait for the initial indexing or analysis of the project to finish

Once indexing is complete, the agents have a basic understanding of the codebase's structure.

3. Send the Main Experiment Prompt

This is the exact initial prompt I would use inside Antigravity to kick off the experiment.

You are working inside my existing project.

Feature to implement:
- Add a "Guest checkout" flow for users who are not logged in.
- Implement abandoned cart email recovery using Resend or Nodemailer.
- Add a simple Tensorlake analytics hook for cart events.

Project context:
- Frontend: React with TypeScript.
- Backend: Node with Express.
- Database: PostgreSQL.
- Auth: JWT based.
- Emails can use either Resend or Nodemailer, pick one and wire it cleanly.
- Tensorlake should be used only for a minimal event tracking integration.

Constraints:
- Treat this as a real production feature, not a demo.
- Keep changes scoped and readable.
- Prefer small, focused commits and clear structure.
- You should break this feature into missions and execute them using the Code, Terminal, and Browser tools.
- I want you to handle as much as possible. I will only step in if you get stuck or break something repeatedly.

Deliverables:
- Guest checkout flow end to end.
- Abandoned cart recovery emails with a reasonable trigger condition.
- Tensorlake event tracking for at least "cart created" and "cart abandoned".
- Tests updated or added where appropriate.

First, respond with a clear mission plan. Then start executing it step by step.

This sets expectations, defines the stack, and tells the agent to create a mission plan first instead of editing files immediately.

4. Review and Adjust the Mission Plan

Antigravity should reply with something like a breakdown of tasks. For example:

Design the guest checkout data model
Add database changes for guest carts and orders
Implement backend routes for guest checkout
Implement abandoned cart identification and email sending
Add frontend components for guest checkout
Insert Tensorlake tracking
Add or update tests

You do not need this exact list. You just need it to be coherent.

At this step, you:

Check that it understood the feature
Ask for small adjustments if something is clearly missing

5. Let the Agents Execute Across the Stack

Now you let Antigravity do the heavy lifting. Typical actions you will see:

Code Agent
- Creates or updates Express routes for guest checkout
- Adds controllers or services for cart and order handling
- Writes or updates TypeScript types and interfaces
- Creates React components or pages for guest checkout
Terminal Agent
- Runs migrations for new tables or columns
- Executes test suites
- Installs dependencies such as Resend, Nodemailer, or the Tensorlake SDK
Browser Agent
- Opens the local app
- Walks through the guest checkout UI
- Verifies that the flow works end-to-end

Your job at this stage is to observe.

Only intervene when:

It repeatedly breaks the same thing
It introduces an obviously wrong design decision
It gets stuck in a loop of failing tests and incremental fixes that do not converge

6. Verify the Flow with the Browser Agent

Once the main missions complete, explicitly ask the agents to verify the behavior.

Example prompt:

Use the Browser and Terminal tools to verify the complete flow:
- Start a new cart as a guest user.
- Proceed through the guest checkout UI.
- Leave a cart in an abandoned state and trigger the recovery email logic.
- Confirm that Tensorlake tracking events are being sent where expected.

Report what you tested and what worked or failed.

This forces a structured test pass rather than assuming the feature works.

7. Perform a Manual Review

At the end, you should still review everything as a developer.

Look at the diff for each mission
Check routes, models, and migrations
Check how email sending is wired
Verify the Tensorlake calls do not leak sensitive data
Run tests yourself
Try the flow manually in the browser

If needed, you can ask Antigravity to clean up small issues or style problems with targeted prompts.

You can find the execution of Antigravity in this repo.

Outcome

Here is the entire execution:

Here is the email:

What Antigravity Did Surprisingly Well

A few things stood out once the agents started working through the feature.

The task breakdown actually made sense: The mission plan looked like something a real dev would sketch out before starting. Nothing over-engineered, nothing missing.
The agents handed work off cleanly: Code edits, migrations, test runs, and browser checks happened in a reasonable order. It felt coordinated rather than chaotic.
Boilerplate wasn’t a mess: The generated routes, controllers, and React components were straightforward. I didn’t have to untangle odd patterns or rewrite everything.
Data stayed consistent across the stack: Field names, types, and payload shapes lined up. I didn’t see the usual “backend calls it one thing, frontend calls it another” issue.
The email flow was wired up correctly: Resend/Nodemailer setup usually gets messy, but the structure here was clear and easy to follow.
Tensorlake integration was small and sensible: It added a couple of event hooks without turning the code into an analytics playground.
Quick feedback loops: When something broke, the agents patched it fast without spiraling into nonsense fixes.

Where It Still Stumbled (Minor, but Honest)

The backend work was mostly solid, but the UI side definitely exposed some weak spots. The agents could generate components and wiring fast, but getting everything to actually behave the way I wanted took multiple iterations.

A few things stood out:

Frontend was the hardest part for the agents: They could scaffold React components quickly, but the details were often off. State handling, validation, and edge cases needed several rounds of fixes.
Connecting frontend and backend wasn’t always smooth: Endpoints existed, components existed, but stitching them together required back-and-forth corrections. The agents didn’t always keep both sides in sync on the first pass.
Debugging took way longer than generation: The feature would “finish” in minutes, but debugging the UI and flow took a few hours. The agents helped, but they didn’t magically remove the pain points of front-end troubleshooting.

Even with the debugging overhead, this is a feature that would normally take me two to four days end-to-end. With Antigravity, it landed in a few hours. Not perfect, but undeniably faster.

The Final Verdict

By the end of the experiment, the feature shipped. It needed a few rounds of debugging, mostly on the frontend, but the entire flow was working: guest checkout, abandoned cart recovery emails, and the Tensorlake event hook.

The time savings were real. What would normally take two to four days of manual work ended up condensed into a few hours of guiding and debugging the agents. The system is not flawless, but it moves fast enough that the rough edges still net out in your favor.

Antigravity makes the most sense when you need:

multi-file scaffolding
routine backend wiring
repetitive refactors
quick prototyping of end-to-end flows

It’s less ideal when you need tight UI polish, careful architectural decisions, or detailed business logic that isn’t spelled out clearly.

Overall, it feels less like a replacement for a developer and more like an accelerator for one. When the agents stay within context, the speed boost is dramatic. When they drift, it still takes less time to correct them than to write everything yourself.

Checklist for Your Own Antigravity Experiment

Start on a clean branch and confirm your project runs locally with all tests passing.
Set up required environment variables, including your Resend API key, and install the Antigravity browser extension so the Browser Agent can interact with your app.
Write a clear, scoped feature request that spans multiple areas of the stack but isn’t overly complex.
Have Antigravity generate a mission plan first, then review and adjust it before execution begins.
Let the Code, Terminal, and Browser agents run the workflow end-to-end with minimal intervention.
Step in only when the agents lose context, loop, or consistently misinterpret something.
Use the Browser Agent to verify the complete flow once the missions finish.
Do a final manual review to check migrations, API wiring, UI behavior, and any email or analytics logic.

The best way to understand agent-driven development is to see it operate inside your own project. Try a contained feature, let the agents run, and watch what happens. (Antigravity)

TOON vs JSON: A Token-Optimized Data Format for Reducing LLM Costs

Arindam Majumder — Thu, 11 Dec 2025 15:33:22 +0000

Last month, I watched a production RAG pipeline burn $1940 in one weekend. A single 500-row customer table, encoded the usual way in classic JSON, did the damage. The exact same data would have cost $760 in TOON. Same model. Same answers. Same latency. 61 % fewer tokens.

You might have felt it yourself. You add one extra field to your context payload. The token counter spikes by hundreds. Suddenly, you trim keys or pray the model reads the structure right. We all patch around it because JSON has been the default for twenty years.

Most developers forget one detail. JSON landed in 2001. 5 years before the iPhone. 14 years before GPT-1. Douglas Crockford built JSON for Ajax round-trips between browsers and servers, not for trillion-parameter models that bill you per token. Every quoted key. Every repeated field name in an array. Every curly brace made perfect sense in a world without inference pricing.

In 2025, those symbols cost real money.

TOON kills that cost. It preserves every piece of the JSON data model (objects, arrays, numbers, and nulls), but rewrites the text for the one reader you actually pay for the LLM itself. It replaces multiple key rows with a single header row. Drops unnecessary quotes. Uses indentation instead of braces. Adds explicit length guards to prevent the model from guessing array sizes.

This article shows exactly why JSON became an accidental tax on AI work, how TOON removes that tax at the syntax level, and how you add it to your code today without rewriting your stack.

If you pay for tokens, keep reading. Your next bill depends on it.

JSON’s Legacy: A Web Standard, Not an AI One

JSON remains the gold standard for general-purpose data interchange. Its quoted keys, braces, brackets, and commas guarantee unambiguous parsing across every programming language and make payloads easy to inspect in browser consoles.

When JSON was created, those properties solved real problems. Bandwidth was the primary constraint, and token-based pricing did not exist.

Today, the constraint has changed. Take a single object

{
  "id": 1,
  "name": "Alice"
}

It uses ~26 tokens instead of the 6–8 that a human would count. Quotes, colons, commas, and braces each become separate subwords in modern BPE tokenizers.

When that object appears in a 500-row array, the key strings and surrounding punctuation repeat hundreds of times. Real-world benchmarks record 11,842 tokens for pretty-printed JSON and 4,617 tokens for the minified version. The language model receives no additional information from those repetitions; they exist solely for syntactic correctness in traditional parsers.

JSON remains the best choice for REST APIs, configuration files, and any system where token counting is irrelevant. Inside LLM prompts, however, the same syntax becomes unnecessary overhead, directly increasing costs and reducing available context.

What is TOON?

TOON (Token-Optimized Object Notation) is a drop-in text representation for structured data that preserves the full JSON data model, including objects, arrays, strings, numbers, booleans, and nulls. Still, it removes the punctuation and repetition that inflate token counts inside LLM prompts.

Rather than wrapping every object in braces and repeating keys on every row, TOON:

Uses indentation instead of {} and ,
Declares array structure up front so fields don’t repeat
Preserves ordering and schema explicitly
Streams cleanly in line-based form for RAG pipelines
Round-trip losslessly back to JSON

It is not a new database standard. It is not a compression algorithm. TOON gives models the data they need in the form they prefer: less syntax, more signal, fewer tokens.

How TOON Reduces Token Load Without Changing the Data

When JSON is used as model input, its syntax becomes a tax; the characters required for parsing increase the token count and reduce the available reasoning space.

TOON’s approach is to keep the full expressiveness of JSON while changing how the structure appears on the page. It focuses on the tokenizer as the primary consumer instead of the runtime environment.

Note: TOON optimizes repeated structure extremely well, but it isn’t a universal compressor. Highly nested or schema-less data will see smaller savings.

Below is a closer look at the mechanisms behind that change.

Indentation-Based Hierarchy Instead of Symbol-Based Delimiters

JSON depends on punctuation to express scope. Braces define objects. Brackets define arrays. Commas separate members. Tokenizers break each of these into its own subwords.

TOON moves this structural meaning into whitespace:\

Two spaces represent one nesting level
Each key begins a new line when introducing a child object
Context defines interpretation, not braces

Example translation of nested objects:

{
  "user": {
    "profile": {
      "city": "Paris"
    }
  }
}

becomes

user:
  profile:
    city: Paris

This reduces syntactic characters while preserving deterministic parseability. The parser tracks indentation levels instead of punctuation. This is a simpler signal for models to learn.

Header-Driven Arrays Replace Repetition With Declarative Structure

Uniform arrays are common in real data. JSON must repeat every field name and punctuation for every element. TOON compresses this by extracting shape into a single declaration:

items[<row count>]{<field order>}:

Then comes only the values:

items[3]{sku,qty,price}:
  A12,4,19.99
  B18,1,12.50
  C22,3,9.25

Under the hood:

Keys appear once
Column order is guaranteed
Rows are fixed-width logical tuples

On 500-row datasets, this structure often cuts the token count by more than half. The improvement scales linearly with array length.

Technical detection logic

The encoder collapses an array when:

All elements are objects
They share an identical key set
Order of keys is stable
Null fields remain valid inline values

Otherwise, TOON falls back to object-by-object expansion. No ambiguity or silent corruption.

Schema and Cardinality Propagated Into the Prompt

JSON implies structure. TOON exposes it. Models benefit from clearly defined boundaries.

Two design choices matter:

[N] explicitly sets expected row count

• {field1,field2,…} statically enforces column order

These guide extraction tasks in a way punctuation cannot. A model that invents an extra row contradicts the declared cardinality. A misplaced field becomes visibly misaligned.

This reduces hallucination in:

Table reconstruction
RAG answer grounding
Tool responses requiring valid JSON output

Benchmarks show improvements in exact match metrics and fewer malformed outputs when LLMs decode TOON vs JSON.

Optimized for Tokenizers Rather Than Parsers

BPE and unigram tokenizers do not treat structural characters atomically:

Quotes often tokenize as ", plus the first 1–2 characters of the key
Braces become unique token fragments not reused elsewhere
Repeated key names are repeatedly segmented across the prompt

TOON leverages linguistic token merging:

Alphanumeric keys tend to map to single tokens
Indentation and line breaks fall into low-cost whitespace categories
CSV-like patterns trigger high tokenizer reuse

Example token comparison for a 100-row table:

JSON minified: ~2,540 tokens

TOON equivalent: ~1,020 tokens

Same semantics, radically different tokenization behavior.

Deterministic Round-Trip and Streaming Support

The encoder is a pure transformation layer. It does not compress or interpret values. Decoding restores original JSON byte-for-byte (excluding whitespace variation in numbers and optional quotes).

Two primary APIs matter:

import { encode, decode, encodeLines } from '@toon-format/toon';

const text = encode(data);      // Buffer → TOON text
const obj = decode(text);       // TOON text → JSON structure

for await (const chunk of encodeLines(largeData)) {
  // Suitable for incremental context injection in RAG
}

ˆLarge structured payloads can stream without materializing entire documents in memory. This benefits contexts where prompts change on the fly, such as agent pipelines.

Designed to Fail Loudly, Not Silently

JSON accepts constructs that can become fragile when interpreted by a model, missing commas, out-of-order fields, and trailing structure. Models sometimes look correct while outputting semantically broken JSON.

TOON’s strict format makes deviations more observable:

Misindentation breaks structural parse
Mismatched row counts surface immediately
Field order mismatch is an error, not a tolerated reordering

Instead of debugging the LLM, the format itself catches the drift.

Why These Choices Matter

LLMs are probability engines, not parsers. They work best when the signal is strong and the requirements are explicit. TOON’s encoding strategy reduces the number of possible interpretations at every structural boundary, while reducing the token cost at the same time.

It is not a new data model. It is simply a more model-literate representation of the one we already use.

Benchmarks From Real Data

The most honest way to judge a data format is to see how it performs when real pipelines and real models are involved. The TOON benchmark suite focuses on everyday workloads that developers already push into prompts: employee directories, order histories, analytics logs, configuration objects, and nested product catalogs.

There are 209 structured extraction tasks in total. Testing covers four current model families: GPT 5 Nano, Gemini Flash, Claude Haiku, and Grok 4. Token counts are measured using the o200k base tokenizer so the results match real billing.

Here is the average outcome across mixed data shapes:

Format	Accuracy	Tokens	Score*	Savings vs JSON
TOON	73.9%	2,744	26.9	39.6% fewer
JSON compact	70.7%	3,081	22.9	none
YAML	69.0%	3,719	18.6	N A
JSON	69.7%	4,545	15.3	baseline
XML	67.1%	5,167	13.0	N A

Score shows correct extractions per 1,000 input tokens. It is a direct value for cost metric.

Uniform arrays show the biggest advantage. A 500 row e commerce orders dataset that required 11,842 tokens in JSON needed only 4,617 tokens in TOON. This represents a 61 percent reduction. At 1,000 GPT 4o prompts per day, that single workload saves roughly 1,740 dollars every month.

Accuracy improves as well. GPT 5 Nano reconstruction tests rose from 92.5 percent to 99.4 percent. The explicit field alignment and declared row counts help the model avoid dropped or invented entries. Nothing about the underlying information changes. The model simply has less noise to interpret and more room in the context window for data that matters.

How Teams Use TOON in Production

Adopting TOON rarely requires major changes. JSON remains the source of truth in databases and services. The only difference is that data is converted to TOON at the moment it becomes model input. This removes the token overhead that appears only in prompts, not in storage or APIs.

A typical retrieval augmented workflow looks like this:

from toon import encode
records = db.fetch_customers()
prompt = "Answer using this context:\n" + encode(records)

The model reads TOON as structured text without special instruction. If the response needs to return to typed objects, the same library converts it back into JSON. This keeps the rest of the stack untouched.

Agent systems also gain stability. When a tool returns a list of results, TOON’s explicit row counts and column order help the model avoid misalignment errors that would otherwise break the next step in the loop.

Streaming pipelines benefit, too. Because TOON is line oriented, prompts can be built incrementally without waiting for closing braces or bracket completion. The result is faster handoffs from retrieval to inference.

When TOON Helps and When JSON Still Makes Sense

TOON shows its strengths when models read large collections of records. In those prompts, much of the length comes from formatting rather than data. Removing that formatting gives the model the same information in a smaller space.

Some data does not benefit in the same way. Complex, irregular objects leave little structure that can be simplified, so token totals remain close to JSON. And outside of prompts, JSON continues to be a dependable standard for storage, APIs, and logging where token costs do not apply.

The right approach is to test with your own payloads. Measure how many tokens the model actually sees and how reliably it can reconstruct results. TOON is most helpful where structure repeats predictably and cost pressure is high.

Conclusion

Formats usually reflect the problems they were built to solve. JSON was created when the goal was to move data between browsers and servers with as little friction as possible. Its punctuation and repetition are part of that success story.

When that same format is aimed at a language model, the context changes. Models treat every character as a unit of computation, and punctuation becomes something they must process before they can reason about the information it describes. The result is more tokens consumed and less room for the details that matter.

TOON takes the data we already rely on and presents it in a way that models can read with less effort. It removes structure that exists only for traditional parsers while keeping the meaning intact. That difference shows up quickly in token use, in latency, and in the accuracy of structured extraction.

Better results without changing the data itself. That is the practical opportunity now in front of developers.

Gemini 3 is Now Available as an OCR Model in Tensorlake

Diptanu Gon Choudhury — Mon, 08 Dec 2025 05:57:26 +0000

Gemini 3 is now available within Tensorlake

Google’s Gemini model since 2.5 Flash has been great at Document Parsing. The latest Gemini 3 pushes the envelope even further. It has the lowest edit distance(0.115) on OmniDocBench compared to GPT-5.1(0.147) and Claude Sonnet 4.5.

Starting today, you can start using Gemini as an OCR Engine with Tensorlake’s Document Ingestion API. You can ingest Documents in bulk, and convert them into Markdown, classify pages or extract structured data using JSON schema. Tensorlake will take care of queuing, working with rate limits and sending you webhooks as documents are processed.

We put Gemini 3 to the test inside Tensorlake, and the results on "hostile" document layouts were immediate.

Case Study 1: Table Structure Recognition

Document: Google 2024 Environmental Report

Financial and scientific reports use visual cues, like indentation, floating columns, and symbols, to convey meaning. To test this, we fed the complex "Water Use" table from the Appendix into Gemini 3.

The Challenge

The table is semi wireless - some lines separating some of the rows while the columns have no boundaries. The column on the right is disconnected to the main block.

The Gemini 3 Result: Visual Understanding

Gemini3 does a perfect job on understanding this table. This is a screenshot from the Tensorlake Cloud Dashboard.

Case Study 2: VQA + Structured Output

Document: House Floor Plans

We wanted to test if Gemini 3 could parse visual symbols on construction documents. We fit Gemini3 into Tensorlake’s Structured Extraction pipeline.

The Input: A raw PDF of a house plan and a Pydantic schema defining the exact fields we needed (e.g., kitchen_outlets: int, description: Number of standard and GFI electrical outlets, as noted by the legend icon labeled "outlet", that are found in the kitchen and dining nook.).

For reference, here is the kitchen+dining nook area.

The circle with two lines are the outlets, as per the legend on the same page:

The Challenge

There is no text label saying "Outlet" on the diagram, it is only associated with the symbol in the legend The model must identify the specific circle-and-line icon defined in the legend, spatially constrain its search to the visual boundaries of the "Kitchen," and aggregate the count into our JSON structure.

The Result

Gemini 3 successfully understood the visual diagram. It returned a valid JSON object with 6 outlets, correctly distinguishing them from nearby data ports and switches.

Tensorlake blends specialized OCR models and VLMs into a set of convenient APIs. While you could call the Gemini API directly, you would be rebuilding many undifferentiated aspects of a production pipeline. Gemini 3 is now fully integrated with Tensorlake DocAI APIs to read, classify, and extract information from documents.

Tensorlake solves the two biggest headaches of building Document Ingestion APIs using VLMs:

Bulk Ingestion & Rate Limits: From our observation Gemini3 doesn’t handle spiky traffic very well. Throwing 10,000 documents at it will trigger errors due to strict quotas. Tensorlake manages the queue, handling back-off and retries automatically so you can ingest massive datasets without hitting 429 errors.
Chunking Large Files: Tensorlake automatically chunks large documents into chunks of 25 pages to make sure Gemini is able to extract even the most dense pages. We ensure that the output token limit of 64k is not exceeded.

When to use (and NOT use) Gemini 3

Use Gemini 3 when:

Complex Visual Reasoning is required: You need to correlate a chart's color legend to a data table, or count symbols on a blueprint (as shown in the house plan example).

Do NOT use Gemini 3 when:

You need bounding boxes for citation: Gemini 3 does not perform layout detection of objects in documents. If your application requires strict Bounding Boxes to highlight exactly where a specific paragraph or number came from.
You need strict text style and font detection: Visual nuances like strikethroughs, underlines, or specific font colors are often ignored by VLMs, which focus on the "content" rather than the style.

For these tasks, you should use one of Tensorlake’s specialized models, like Model03.

How to use Gemini 3 with Tensorlake

Playground

Gemini 3 is available today in the Tensorlake Playground for experimentation:

Or you can select it with our HTTP API or SDK:

from tensorlake.documentai import DocumentAI, ParsingOptions

client = DocumentAI()

parse_id = client.read(
  file_url="https://tlake.link/docs/real-estate-agreement",
  parsing_options=ParsingOptions(
    ocr_model="gemini3"
  )
)

result = client.result(parse_id)
)

What's Next

Document Ingestion has a lot of edge cases. We want our users to always have access to state of the art models so that they can solve their use cases fairly quickly by changing various aspects of the OCR pipelines with very minimal code changes.

We will add more Foundation Models as OCR model options in Tensorlake’s Document Ingestion API.

Try Tensorlake free

Want to discuss your specific use case?
Schedule a technical demo with our team.

Questions about the benchmark?
Join our Slack community

Diptanu Gon ChoudhuryFollow

/diptanu_gonchoudhury_23e

Benchmarking the Most Reliable Document Parsing API

Sarah Guthals, PhD — Wed, 05 Nov 2025 17:37:05 +0000

Document parsing is the foundation of enterprise AI applications. Whether you're building RAG pipelines, automating insurance claims, or extracting data from financial reports, everything starts with one question: Can you consistently transform messy, real-world documents into structured, machine-readable data?

Our customers need the best document ingestion API for their use cases. They're comparing Azure, AWS Textract, popular open-source models like Docling and Marker.

We built a benchmark that measures what matters: Can downstream systems actually use this output?

Measuring What Actually Matters

Tensorlake both reads documents and extracts structured data, so when choosing what to measure accuracy with, we wanted to ensure we were measuring both document parsing with structural preservation and structured extraction for downstream usability.

The aspects of Document Parsing that we wanted to measure were:

Tables: Ensuring we can parse and measure accuracy of complex tables with merged cells and multi-row headers
Reading Order: In multi-column documents, and documents with complex layouts, we measure whether the reading order is preserved while parsing.
Structured Extraction Accuracy: Measuring direct downstream usability of extracted data. A small OCR error in parsing a table cell can cause failure in achieving the downstream task, while the overall accuracy of the OCR on the document may be high.
Extraction of footnotes, formulas, figures and other non-textual content.

Our Evaluation Methodology

We employ two metrics that better capture these features with real-world reliability:

TEDS (Tree Edit Distance Similarity)

Compares predicted and ground-truth Markdown/HTML tree structures
Captures structural fidelity in tables and complex layouts
Widely adopted in OCRBench v2 and OmniDocBench evaluations
Measures whether the document's logical structure and textual alignment remains intact

TEDS answers: "Is this table still a table?" Not just "Is the text similar?"

JSON F1 (Field-Level Precision and Recall)

Compares extracted JSON against schema-based ground truth
Precision measures correctness of extracted fields
Recall measures completeness of required field capture
F1 score balances both for overall reliability assessment

JSON F1 answers: "Can downstream automation actually use this data?" Not just "Is some text present?"

Together, these metrics answer the essential question: "Can downstream systems use this output?" rather than simply "Is the text similar?"

Stage 1: Document Reading Ability (OCR and Structural Preservation)

Each parsing model generates Markdown/HTML output. We evaluate using TEDS to measure how well structure is preserved; reading order, table integrity, and layout coherence. You can find our updated dataset published here.

We use the public OCRBench v2 and OmniDocBench datasets. However, upon review, we identified inconsistencies in the published ground truth of OCRBench v2. We conducted a comprehensive audit and correction to ensure evaluation accuracy.

Stage 2: Structured Extraction Accuracy (Downstream Usability)

We pass the Markdown through a standardized LLM (GPT-4o) with predefined JSON schemas, measuring JSON F1. This isolates how OCR quality impacts real extraction workflows, where an LLM interprets the parsed text.

Initial JSON schemas and reference answers are generated using Gemini Pro 2.5, then human reviewers audit and correct them to ensure high-quality gold standards.

This methodology ensures fair, reproducible comparisons by varying only the OCR models (Stage 1) while keeping the extraction model constant (Stage 2).

The Results: Public Dataset Performance

Document Parsing Performance

We evaluated leading open-source and proprietary models:

Key Findings:

Tensorlake achieves the highest TEDS score, indicating superior structural preservation
The gap between Docling and production-grade systems is substantial

Table Parsing Performance

We evaluated Tensorlake’s table parsing accuracy using the OmniDocBench dataset — a CVPR-accepted benchmark for comprehensive document understanding tasks (GitHub link).

Table accuracy in OmniDocBench is quantified using a combination of tree-based and string-based metrics. In particular, we measured TEDS (Tree Edit Distance Similarity), which assesses both the structural and textual alignment between predicted and ground-truth HTML tables.

To reproduce our results, generate Markdown outputs using the models listed below, then run the evaluation method provided in the OmniDocBench repository. We have used 512 document images with tables and v1.5 of the code version. Evaluation outputs are released in Huggingface(link)

¹ Marker's Number is from the officially published OmniDocBench repository.

Key Findings:

On OmniDocBench's challenging tables, Tensorlake leads with 86.79% TEDS
Open-source solutions struggle with table extraction (sub-70% TEDS)
Tensorlake maintains table structure even on complex, multi-page tables

Performance on Real World Enterprise Documents

OCR Models are rarely trained on enterprise documents, because they are not publicly available. We wanted to test how well our model performs and others perform on these documents.

Enterprise Document Performance (100 pages)

We curated 100 document pages spanning banking, retail, and insurance sectors. This represents real production workloads: invoices with water damage, scanned contracts with skewed text, bank statements with multi-level tables.

Key Findings:

Tensorlake achieves 91.7% F1 with standard extraction, beating all competitors
The difference between 91.7% and 68.9% F1 is massive: it’s 5 extra fields correctly extracted out of every 20
In production workflows processing thousands of documents daily, this accuracy gap compounds into significant error reduction

But even comparing the higher F1 scores when parsing a standard form, Azure and Textract jumble the reading order and skip data completely, whereas Tensorlake preserves the complex reading order and groups data correctly and accurately:

Delivering the Best Performance/Price Ratio

Accuracy without affordability isn't practical. Here's how Tensorlake compares to other Document Ingestion APIs:

Tensorlake: $10 per 1k pages
- TEDS Score: 86.79
- F1 Score: 91.7
Azure: $10 per 1k pages
- TEDS Score: 78.14
- F1 Score: 88.1
AWS Textract: $15 per 1k pages
- TEDS Score: 80.75
- F1 Score: 88.4

Tensorlake delivers the highest accuracy than both Azure and AWS Textract, matching Azure's cost while AWS Textract is 50% more expensive.

Take the Next Step

When your business depends on accurate document processing, you can't afford to use anything less.

Try Tensorlake free

Want to discuss your specific use case?

Schedule a technical demo with our team.

Questions about the benchmark?

Join our Slack community

New: Vision Language Models for Document Processing

Sarah Guthals, PhD — Thu, 16 Oct 2025 20:52:08 +0000

We've expanded our use of Vision Language Models (VLMs) across multiple DocumentAI features for faster and more accurate document processing on documents with hundreds of pages:

Page Classification: Identify relevant pages in large documents
Figure and Table Summarization: Extract insights from visual elements
Structured Extraction (with skip_ocr): Direct visual understanding for more accurate extraction on harder to parse documents (e.g. scanned documents, engineering diagrams, or documents with complex reading order)

This post focuses on our enhanced page classification capabilities for demonstration. With VLM support, you can quickly process large documents by identifying and extracting from only relevant pages.

Try it in this Colab Notebook.

Key Improvements

Scale & Performance

Handle Large Documents: Classify documents with hundreds of pages without performance degradation
VLM-Powered Classification: Replaced OCR with Vision Language Models for faster, more accurate classification
Selective Processing: Only parse pages that matter, reducing processing time and costs

Recommended Workflow

Classify First: Use the classify endpoint to identify relevant pages based on your criteria
Parse Selectively: Set page_range to only process the classified relevant pages
Extract Efficiently: Apply structured extraction only to pages containing the information you need

Use Case Example: SEC Filings Analysis

This approach is particularly powerful for extracting specific information from lengthy documents like SEC filings. For example, when analyzing cryptocurrency holdings across multiple companies' 10-K and 10-Q reports:

Challenge: Each filing can be 100-200+ pages, but crypto-related information might only appear on 10-20 pages
Solution: First classify pages containing "digital assets holdings", then extract structured data only from those pages
Result: 80-90% reduction in processing time and more focused, accurate extractions

Code Example

from tensorlake.documentai import DocumentAI, PageClassConfig

doc_ai = DocumentAI()

# Step 1: Classify pages
page_classifications = [
    PageClassConfig(
        name="digital_assets_holdings",
        description="Pages showing cryptocurrency holdings on balance sheet..."
    )
]

parse_id = doc_ai.classify(
    file_url=filing_url,
    page_classifications=page_classifications
)

result = doc_ai.wait_for_completion(parse_id=parse_id)

# Step 2: Parse only relevant pages
relevant_pages = result.page_classes[0].page_numbers
page_range = ",".join(str(i) for i in relevant_pages)

final_result = doc_ai.parse_and_wait(
    file=filing_url,
    page_range=page_range,
    structured_extraction_options=[...]
)

Benefits

Cost Efficiency: Process only what you need
Speed: Reduce processing time by focusing on relevant content
Accuracy: VLM classification provides better understanding of page content
Scalability: Handle large document sets without compromising performance

Try It Out

Check out our example notebook demonstrating how to extract cryptocurrency metrics from SEC filings using the new classification approach.

Getting Started

Update to the latest version of Tensorlake:
pip install --upgrade tensorlake

Then start classifying, summarizing, and extracting with improved efficiency!