Forem: Shir Meir Lador

Agent Factory Recap: Supercharging Agents on GKE with Agent Sandbox and Pod Snapshots

Shir Meir Lador — Tue, 07 Apr 2026 13:04:00 +0000

In the latest episode of the Agent Factory, Mofi Rahman and I had the pleasure of hosting, Brandon Royal, the PM working on agentic workloads on GKE. We dove deep into the critical questions around the nuances of choosing the right agent runtime, the power of GKE for agents, and the essential security measures needed for intelligent agents to run code.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

Why GKE for Agents?

Timestamp: 01:49

We kicked off our discussion by tackling a fundamental question: why choose GKE as your agent runtime when serverless options like Cloud Run or fully managed solutions like Agent Engine exist?

Brandon explained that the decision often boils down to control versus convenience. While serverless options are perfectly adequate for basic agents, the flexibility and governance capabilities of Kubernetes and GKE become indispensable in high-scale scenarios involving hundreds or thousands of agents. GKE truly shines when you need granular control over your agent deployments.

ADK on GKE

Timestamp: 06:58

We've discussed the Agent Development Kit (ADK) in previous episodes, and Mofi highlighted to us how seamlessly it integrates with GKE and even showed a demo with the agent he built. ADK provides the framework for building the agent's logic, traces, and tools, while GKE provides the robust hosting environment. You can containerize your ADK agent, push it to Google Artifact Registry, and deploy it to GKE in minutes, transforming a local prototype into a globally accessible service.

The Sandbox problem

Timestamp: 15:20

As agents become more sophisticated and capable of writing and executing code, a critical security concern emerges: the risk of untrusted, LLM-generated code. Brandon emphasized that while code execution is vital for high-performance agents and deterministic behavior, it also introduces significant risks in multi-tenant systems. This led us to the concept of a "sandbox."

What is a Sandbox?

Timestamp: 19:18

For those less familiar with security engineering, Brandon clarified that a sandbox provides kernel and network isolation. Mofi further elaborated, explaining that agents often need to execute scripts (e.g., Python for data analysis). Without a sandbox, a hallucinating or prompt-injected model could potentially delete databases or steal secrets if allowed to run code directly on the main server. A sandbox creates a safe, isolated environment where such code can run without harming other systems.

Agent Sandbox on GKE Demo

Timestamp: 20:25

So, how do we build this "high fence" on Kubernetes? Brandon introduced the Agent Sandbox on Kubernetes, which leverages technologies like gVisor, an application kernel sandbox. When an agent needs to execute code, GKE dynamically provisions a completely isolated pod. This pod operates with its own kernel, network, and file system, effectively trapping any malicious code within the gVisor bubble.

Mofi walked us through a compelling demo of the Agent Sandbox in action.We observed an ADK agent being given a task requiring code execution. As the agent initiated code execution, GKE dynamically provisioned a new pod, visibly labeled as "sandbox-executor," demonstrating the real-time isolation. Brandon highlighted that this pod is configured with strict network policies, further enhancing security.

The Future: Pod Snapshots

Timestamp: 29:39

While the Agent Sandbox offers incredible security, the latency of spinning up a new pod for every task is a concern. Mofi demoed the game-changing solution: Pod Snapshots. This technology allows us to save their state of running sandboxes and then near-instantly restore them when an agent needs them. Brandon noted that this reduces startup times from minutes to seconds, revolutionizing real-time agentic workflows on GKE.

Conclusion

It's incredible to see how GKE isn't just hosting agents; it's actively protecting them and making them faster.

Your turn to build

Ready to put these concepts into practice? Dive into the full episode to see the demos in action and explore how GKE can supercharge your agentic workloads.

Learn how to deploy an ADK agent to Google Kubernetes Engine and how to get your run agent to run code safely using the GKE agent Sandbox.

Connect with us

Shir Meir Lador → LinkedIn, X
Mofi Rahman → LinkedIn
Brandon Royal → LinkedIn

Agent Factory Recap: Reinforcement Learning and Fine-Tuning on TPUs

Shir Meir Lador — Tue, 31 Mar 2026 18:56:42 +0000

In our agent factory holiday special, Don McCasland and I were joined by Kyle Meggs, Senior Product Manager on the TPU Training Team at Google, to dive deep into the world of model fine tuning. We focused specifically on reinforcement learning (RL), and how Google's own infrastructure of TPUs are designed to power these massive workloads at scale.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

When to Consider Fine-Tuning

Timestamp: 3:13

We started with a fundamental question: with foundational models like Gemini becoming so powerful out of the box, and customization through the prompt can often be good enough, when should you consider fine-tuning?

Fine tuning your own model is relevant when you need high specialization for unique datasets where a generalist model might not excel (such as in the medical domain), or when you have strict privacy restrictions that require hosting your own models trained on your data.

The Model Lifecycle: Pre-training and Post-training (SFT and RL)

Timestamp: 3:52

Kyle used a great analogy inspired by Andrej Karpathy to break down the stages of training. He described pre-training as "knowledge acquisition," similar to reading a chemistry textbook to learn how things work. Post-training is further split into Supervised Fine-Tuning (SFT), which is analogous to reading already-solved practice problems within the textbook chapter, and Reinforcement Learning (RL), which is like solving new practice problems without help and then checking your answers in the back of the book to measure yourself against an optimal approach and correct answers.

Why Reinforcement Learning (RL) is Essential

Timestamp: 5:50

We explored why RL is currently so important for building modern LLMs. Kyle explained that unlike SFT, which is about imitation, RL is about grading actions to drive "alignment." It’s crucial for teaching a model safety (penalizing what not to do), enabling the model to use tools like search and interact with the physical world through trial and error, and for performing verifiable tasks like math or coding by rewarding the entire chain of thought that leads to a correct answer.

The Agent Industry Pulse: Why 2025 is the year of RL

Timestamp: 8:33

In this segment, we looked at the rapidly evolving landscape of RL. Kyle noted that it is fair to call 2025 the "year of RL," highlighting the massive increase in investment and launches across the industry:

January: DeepSeek-R1 launched, making a huge splash with open-source GRPO.
Summer: xAI launched Grok 4, reportedly running a 200k GPU cluster for RL at "pre-training scale."
October: A slew of new tooling launches across Google, Meta, and TML.
November: Gemini 3 launched as a premier thinking model.
Recent: Google launched MaxText 2.0 for fine-tuning on TPUs.

The Hurdles of Implementing RL

Timestamp: 10:46

Following the industry trends, we discussed why RL is so difficult to implement. Kyle explained that RL combines the complexities of both training and inference into a single process. He outlined three primary challenges: managing infrastructure at the right balance and scale to avoid bottlenecks; choosing the right code, models, algorithms (like GRPO vs. DPO), and data; and finally, the difficulty of integrating disparate components for training, inference, orchestration, and weight synchronization.

To provide a solution across these dimensions of complexity, Google offers MaxText, a vertically integrated solution to help you perform RL in a highly scalable and performant fashion. MaxText provides highly optimized models, the latest post-training algorithms, high performance inference via LLM, and powerful scalability/flexibility via Pathways.

In contrast to DIY approaches where users assemble their own stack of disparate components from many different providers, Google’s approach offers a single integrated stack of co-designed components, from silicon to software to solutions.

The Factory Floor

The Factory Floor is our segment for getting hands-on. Here, we moved from high-level concepts to practical code with a live demo.

Why TPUs Shine for RL

Timestamp: 12:52

Before diving into the demo, Kyle explained why TPUs are uniquely suited for complex AI workloads like RL. Unlike other hardware, TPUs were designed system-first. A TPU Pod can connect up to 9,216 chips over low-latency interconnects, allowing for massive scale without relying on standard data center networks. This is a huge advantage for overcoming RL bottlenecks like weight synchronization. Furthermore, because they are purpose-built for AI, they offer superior price-performance and thermal efficiency.

Demo: Reinforcement Learning (GRPO) with TPU

Timestamp: 15:53

Don led a hands-on demonstration showing what RL looks like in action using Google's infrastructure. The demo showcased:

Using MaxText 2.0 as an integrated solution for the workload.
Leveraging models from MaxText and algorithms from Tunix.
Handling inference using vLLM.
Utilizing Pathways for orchestration and scaling to run GRPO (Group Relative Policy Optimization).

Conclusion

This holiday special was a great deep dive into the cutting edge of model fine tuning. While foundational models are getting better every day, the future of highly specialized, capable agents relies on mastering post-training techniques like RL, and having the right vertically integrated infrastructure, like TPUs, to run them efficiently.

Your turn to build

We hope this episode gave you valuable tools and perspectives to think about fine-tuning your own specialized agents. Be sure to check out the resources below to explore MaxText 2.0 and start experimenting with TPUs for your workloads. We'll see you next year for a revamped season of The Agent Factory!

Resources

Post-Training Docs https://maxtext.readthedocs.io/en/latest/tutorials/post_training_index.html

Google Cloud TPU (Ironwood) Documentation: https://docs.cloud.google.com/tpu/docs/tpu7x
Google Cloud open source code:
- MaxText - https://github.com/AI-Hypercomputer/maxtext
- GPU recipes - https://github.com/AI-Hypercomputer/gpu-recipes
- TPU recipes - https://github.com/AI-Hypercomputer/tpu-recipes
Andrej Karpathy - Chemistry Analogy: Deep Dive into LLMs like ChatGPT
Paper: "Small Language Models are the Future of Agentic AI" (Nvidia): https://arxiv.org/abs/2506.02153
Fine tuning blog: https://cloud.google.com/blog/topics/developers-practitioners/a-step-by-step-guide-to-fine-tuning-medgemma-for-breast-tumor-classification?e=48754805

Connect with us

Shir Meir Lador → https://www.linkedin.com/in/shirmeirlador/, X
Don McCasland → https://www.linkedin.com/in/donald-mccasland/
Kyle Meggs → https://www.linkedin.com/in/kyle-meggs/

My First Experience Creating Antigravity Skills

Shir Meir Lador — Fri, 20 Mar 2026 15:23:02 +0000

Experimenting with Agent skills for the first time, feeling empowered!

Last week, I was at an event where we taught developers how to build MCP servers, agents, and deploy open models to Google Cloud Run. After the session, one of the developers shared something that really stuck with me: he was already using our content to create specialized Skills to share with his entire team.

I got inspired and decided it was time to dive into Agent Skills. During my last project, the dev-signal agent, I had a lot of learnings about how to bring agents and AI applications to production in a robust and scalable manner. I thought, this is a great opportunity to give my favorite coding agent, Google’s Antigravity (Google’s “agent-first” IDE), those skills so that going forward, it will just do it for me!

In this post, I’ll walk through how I built the 13 production skills in this repository and the patterns behind them.

What are Agent Skills?

As Romin Irani explains in “Getting Started with Google Antigravity Skills”, skills represent a shift from monolithic context loading to Progressive Disclosure.

Agents get “overwhelmed” when providing them too many tools all at once (a phenomenon known as “Tool Bloat”), to solve for that, Skills allow the agent to “load” specialist knowledge only when needed. When you ask an agent to “evaluate a shadow revision,” it will figure out it will need to leverage the Shadow Deployer skill as context for this operation.

Workspace vs. Global Scope

In Antigravity, you can manage these skills in two distinct ways depending on how you want to use them:

Workspace Scope: Located in .agent/skills/ within your project root. These are specific to your project and can be committed to GitHub so your entire team can benefit from the same production patterns.
Global Scope: Located in ~/.gemini/antigravity/skills/. These are your personal utilities that stay with you across every project you work on.

How I built the skills

Following the principles in Daniela Petruzalek’s “Building Agent Skills with skill-creator”, I took a “methodology-first” approach. I used the existing dev-signal blog series I’ve been working on and the codebase itself as core context, asking Antigravity to identify and codify the unique skills needed to build a production agent on Google Cloud.

For some of the more specialized areas, I provided additional context with patterns I’d like to follow, such as the agent evaluation codelab and blog and the agent security codelab, both written by my awesome team.

These 13 skills provide Antigravity (or any developer using them) the crucial toolkit of a Google Cloud Production Engineer. I’m currently finalizing a detailed, step-by-step walkthrough of the dev-signal agent which will be published on the Google Cloud Blog very soon! (follow me for future updates)

In the meantime, you don’t have to wait — the full repository and the skills are available for you to explore and leverage in your own projects today.

Here is the full inventory of the skills:

🏗️ Production Agent

adk-memory-bank-initializer: Long-term state logic with Vertex AI Memory Bank.
agent-containerizer: Mixed-runtime Dockerfiles (Python + Node.js).
cloud-run-agent-architect: Least-privilege Terraform for Cloud Run.
gcp-production-secret-handler: In-memory secret fetching pattern (Secret Manager).
mcp-connector-generator: Standardized MCP connection logic.

📊 Evaluation

gcp-agent-eval-engine-runner: Parallel inference and reasoning trace capture.
gcp-agent-eval-metric-configurator: Setup for Grounding and Tool Use rubrics.
gcp-agent-golden-dataset-builder: Tools for building datasets with reference trajectories.
gcp-agent-shadow-deployer: “Dark Canary” deployment scripts with revision tagging.
gcp-agent-tool-trajectory-evaluator: Custom Python metrics for Precision and Recall.

🛡️ Security

gcp-agent-model-armor-shield: Intelligent firewall (Prompt Injection, RAI, Malicious URL filters).
gcp-agent-safety-gatekeeper: Python integration pattern (safety_util.py) for sanitizing user inputs.
gcp-agent-sdp-template-factory: Terraform for Sensitive Data Protection (PII/Secret redaction).

By codifying these patterns to production skills, Antigravity can now leverage these automatically in my day to day development. I hope you find these as helpful as I do!

Pro tip - self improving skills!

Because these skills were AI-generated, they might not work perfectly for your specific environment on the first try. But that’s actually the best part of working with an agentic IDE. If a skill doesn’t work well for you, don’t just manually fix the code, let the coding agent figure it out. Once it finds the solution, you can ask it to update the corresponding SKILL.md with the learned workflow. This will capture the corrected workflow for the future, ensuring the agent doesn’t repeat the mistake while saving you tokens and time on the next run. Think of these as living documents that actively improve as you build.

Ready to get started? Clone the repository and add these skills to your Workspace or Global Scope to start building your own production-ready agents. Learn more about Agent skills.

Follow me on LinkedIn and X for updates on my next blogs and videos.

How I Turned an Ugly Spreadsheet into an AI Assisted App with Antigravity

Shir Meir Lador — Wed, 18 Feb 2026 17:39:12 +0000

I have a confession to make.

Up until now, I wasn’t that much into “vibe coding.” I used AI all the time for Python coding, but I never really built a whole app from scratch in a language I knew nothing about.

That changed today. I encountered a really annoying problem: I had to review a massive amount of talk submissions for a conference. We’re talking about a massive spreadsheet. Staring at those tiny cells was literally making my eyes hurt.

My initial thought was, “Hey, let’s create a really sharp UI for the submission review.” But then I thought, why stop there? Why not let AI provide me valuable inputs from social media to help me with the review itself?

So, I decided to build TalkScout. And since I wanted to test drive Google Antigravity (Google’s new AI-powered coding agent), I figured this was the perfect opportunity.

Talkscout Dashboard (synthetic data)

Here is how I went from a painful CSV to a fully deployed Cloud Run app-without writing a single line of React code myself.

Step 1: The “Meta-Prompt” (Asking Gemini to Talk to Antigravity)

I didn’t start by coding; I started by chatting. I used meta-prompting to get started.

So, what is meta-prompting, you may ask? It’s actually when you go to Gemini 3 and ask it to write the prompt for the coding agent.

I explained my problem to Gemini 3 in simple words. Gemini 3 acted as my architect. It turned my “brain dump” requirements into a technical spec, defining the component structure and data model. I didn’t have to guess the right words, I just pasted that polished spec into Antigravity.

Step 2: Ditching the Spreadsheet for a Dashboard

With that prompt, Antigravity built the app of my dreams. It allowed me to:

Upload the CSV with all the conference talks.
Get a dashboard showing the status of each talk.
See a beautiful, high-contrast UI to review abstracts and demo plans without squinting at cells.

TalkScout submission review page with high contrast UI

The “Vibe” Fix: It wasn’t all smooth sailing — I actually hit a nasty React hydration error. This can take hours to debug, especially if you’re not a frontend developer… But I simply provided the error message to Antigravity and the coding agent pinpointed the mismatch in the DOM and fixed it in minutes.

Step 3: Integrating Grounded Intelligence

I didn’t just want a UI; I wanted to overcome my own bias. How do I know if a niche topic is actually hot?

I added a button to get an AI Assessment. But I didn’t want hallucinations. I used Google Search Grounding so the AI could search through Reddit, X (Twitter), and LinkedIn for real-world developer signals. That provided me inputs based on the current developer audience mindshare.

TalkScout submission review page with AI social media analysis

Step 4: Calibrating the “Strict” Reviewer

Initially, the AI was way too nice. It was giving high scores to anything with trendy keywords.

I used what’s called few-shot prompting to calibrate it. I gave examples of my scores vs. its scores and introduced what I call the “Marketing Fluff Penalty”.

If a submission reads like a documentation/marketing page? Points docked.
If the submission was way too short? We capped the score at a hard 2.
If it includes war stories and actual learnings — increase rating. After a few examples, it became more calibrated to my taste.

Step 5: The Pivot to Batch Mode

I realized it was taking me too long to ask the AI to evaluate each talk individually while I reviewed it.

So, I asked Antigravity to refactor the backend for Batch Mode. Now, TalkScout processes the entire submission pool in the background. By the time I grab a coffee, the “AI Draft” column is full of insights, allowing me to focus only on the final decisions.

Step 6: Sharing the Goodness (Deploy to Cloud Run)

TalkScout was working great for me, but I thought, “It would be great to share this with the other reviewers.”

This is where Antigravity really showed off. I simply asked it to deploy the app. It automatically recognized my Google Cloud Project ID, handled the containerization, generated the exact deployment commands, and deployed it to Cloud Run.

One simple ask, and minutes later, I had a URL to share with the team.

It Was Pretty Fun!

It was pretty fun to actually solve a real problem I had using Antigravity and vibe coding. I built a tool that handles ingestion, provides a distraction-free rating interface, and provides valuable inputs for my reviews.

I would love to hear from you all - have you recently solved a problem using vibe coding?

If you haven’t already - try playing around with Antigravity and easily deploy your apps to Cloud Run.

Decoding high-bandwidth memory: A practical guide to GPU memory for fine-tuning AI models

Shir Meir Lador — Thu, 15 Jan 2026 15:27:00 +0000

We've all been there. You've meticulously prepared your dataset and written your training script. You hit run, and your excitement builds, only to be crushed by the infamous error: CUDA out of memory.

This is one of the most common roadblocks in AI development. Your GPU's High Bandwidth Memory (HBM), is the high-speed memory that holds everything that's needed for computation, and running out of it is a hard stop. But how do you know how much you need?

To build a clear foundation, we'll start by breaking down the HBM consumers on a single GPU and we'll present key strategies to reduce HBM consumption on a single GPU. Later, we'll explore advanced multi-GPU strategies like data and model parallelism that can help relieve memory pressure and scale your training in the cloud.

Understanding HBM: What's using all the memory?

When you fine-tune a model, your HBM is primarily consumed by three things:

Model Weights: This is the most straightforward. It's the storage space required for the model's parameters—the "brain" that it uses to make predictions. A 7-billion parameter model loaded in 16-bit precision will take up roughly 14 GB before you even process a single piece of data.
Optimizer States and Gradients: This is the overhead that's required for learning. To update the model's weights, the training process needs to calculate gradients (the direction of learning) and the optimizer (like the popular AdamW) needs to store its own data to guide the training. In full fine-tuning, this can be the largest consumer of HBM.
Activations and Batch Data: This is the most dynamic part. When your data (images, text, etc.) flows through the model's layers, the intermediate calculations, or activations, are stored in HBM. The memory needed here is directly proportional to your batch size. A larger batch size means more activations are stored simultaneously, which leads to faster training but much higher memory usage.

Note: These calculations are theoretical minimums. Real-world frameworks add up to 30% overhead due to temporary buffers, kernel launches, and memory fragmentation.

Although it's impossible to get a perfect number without experimentation, you can estimate your HBM needs with this general formula:

Total HBM ≈ (Model Size) + (Optimizer States) + (Gradients) + (Activations)

Further reading: See this excellent JAX e-book that covers these topics in great detail and even has some "try it out yourself" test questions.

Example: Why full fine-tuning is so demanding

To see why running out of memory is such a common problem, let's walk through a real-world example that I recently worked on: fine-tuning the medgemma-4b-it model, which has 4 billion parameters. Our script loads it in bfloat16 precision (2 bytes per parameter).

First, let's calculate the static HBM footprint. This is the memory that's required just to load the model and prepare it for training, before you've even processed a single piece of data.

1. Model Size: The memory that's needed to simply hold the model on the GPU.

4 billion parameters × 2 bytes/parameter = 8 GB

2. Gradients and Optimizer States: The overhead for training every parameter with the AdamW optimizer.

Gradients: 4 billion parameters × 2 bytes/parameter = 8 GB Optimizer States (AdamW): 2 × 4 billion parameters × 2 bytes/parameter = 16 GB

Note: While AdamW is a popular optimizer, other optimizers, such as Adafactor and Lion, have different memory footprints.

Adding these together gives us our baseline HBM cost for a full fine-tuning attempt:

8 GB (Model) + 8 GB (Gradients) + 16 GB (Optimizer) = 32 GB

This 32 GB is the baseline just to start the training process. On top of this, the GPU needs additional memory for activations, which is a dynamic cost that grows with your batch size and input data size. This is why full fine-tuning of large models is so demanding and often reserved for the most powerful hardware.

Key strategies to reduce HBM consumption

The HBM requirement for a full fine-tune can seem impossibly high. But several powerful techniques can reduce memory consumption, making it feasible to train large models on consumer-grade or entry-level professional GPUs.

Parameter-Efficient Fine-Tuning (PEFT) with LoRA

Instead of training all the billions of parameters in a model, Parameter-Efficient Fine-Tuning (PEFT) methods focus on training only a small subset of parameters. The most popular of these is LoRA (Low-Rank Adaptation).

LoRA works by freezing the original model's weights and injecting a tiny number of new, trainable adapter layers into the model architecture. This means the memory-hungry gradients and optimizer states are only needed for these few million new parameters, not the full 4 billion.

The math behind LoRA's memory savings

LoRA doesn't remove the base model from your GPU. The full 8 GB of the original model's weights are still loaded and taking up HBM. They're just frozen, which means that the GPU isn't training them. All of the memory savings come from the fact that you no longer need to store the huge gradients and optimizer states for that massive, frozen part of the model.

Let's recalculate the static HBM footprint with LoRA, assuming it adds 20 million trainable parameters:

1. Model Size (unchanged): The base model is still loaded.

4 billion parameters × 2 bytes/parameter = 8 GB

2. LoRA Gradients & Optimizer States: We now only need overhead for the tiny set of new parameters.

Gradients: 20 million parameters × 2 bytes/parameter = 40 MB Optimizer States: 2 × 20 million parameters × 2 bytes/parameter = 80 MB

The new static HBM footprint is now:

8 GB (Model) + 40 MB (Gradients) + 80 MB (Optimizer) ≈ 8.12 GB

The training overhead has shrunk from 24 GB to just 120 MB. Your new baseline memory requirement is now just over 8 GB. This lower baseline memory requirement leaves much more room for the dynamic memory that's needed for activations, which lets you use a reasonable batch size on a common 16 GB or 24 GB GPU without running out of memory.

Model quantization

Besides training fewer parameters, we can also shrink the ones that we have by using quantization, which involves reducing the numerical precision of the model's weights. The standard precision for modern training is bfloat16 because it offers the dynamic range of float32 with half the memory footprint. But we can reduce HBM usage further by converting weights to lower-precision integer formats like int8 or int4.

Using lower-precision integer formats has a significant impact on HBM when compared to the standard bfloat16 baseline:

bfloat16 (standard): The baseline size (e.g., a 7B model requires ~14 GB).
8-bit precision: Halves the model size (e.g., 14 GB becomes ~7 GB).
4-bit precision: Reduces the model size by a factor of 4 (e.g., 14 GB becomes ~3.5 GB).

The reduction in size lets you fit much larger models into memory with minimal degradation in performance.

A word of warning from experience:
When I started experimenting in this area, my first attempt to load the model using the common float16 data type failed spectacularly. The model's outputs were NaN, and a quick check revealed that every internal value had collapsed into NaN (Not a Number) .

The culprit was a classic numerical overflow. The float16 data type has a tiny numerical range and it can't represent any number larger than 65,504. During training, intermediate values can easily exceed this limit, causing an overflow that creates a NaN. The fix was a simple one-line change to bfloat16, which has a massive numerical range that prevents these overflows and keeps training stable. For fine-tuning large models, always prefer bfloat16 for stability.

Combining LoRA and Quantization: These techniques work best together. Quantized LoRA (QLoRA) is a method that stores the massive base model in a highly efficient 4-bit format (specifically NF4 or NormalFloat 4), while adding small, trainable LoRA adapters in bfloat16. During the training process, the 4-bit weights are dequantized to bfloat16 for computation. Dequantizing in process lets you fine-tune very large models on a single GPU with the memory savings of 4-bit storage and the mathematical stability of 16-bit training.

FlashAttention: An algorithmic speed boost

Finally, FlashAttention is a foundational algorithmic optimization that significantly reduces HBM usage and speeds up training on both single and multi-GPU setups. The attention mechanism in transformers is a primary memory bottleneck because it requires storing a large, intermediate attention matrix. FlashAttention cleverly reorders the computation to avoid storing this full matrix in memory, leading to substantial memory savings and faster execution.

Best of all, enabling FlashAttention is often as simple as a one-line change. In the MedGemma fine-tuning script, this was done by setting the value attn_implementation="sdpa", which can automatically use more efficient backends like FlashAttention if the hardware supports it.

Scaling beyond a single GPU: Advanced strategies

Techniques like LoRA and quantization are useful for lowering HBM needs on a single GPU. But to train truly massive models or to really speed up the process, you'll eventually need to scale out to multiple GPUs. Here are some of the key strategies that can be used to distribute the load and overcome memory limitations.

Data parallelism

Data parallelism is the most common and intuitive approach to scaling. In a Distributed Data Parallel (DDP) setup, the entire model is replicated on each GPU. The key is that the global batch of training data is split, with each GPU processing its own mini-batch concurrently. After each forward and backward pass, the gradients from each GPU are averaged together to ensure that all of the model replicas learn from the entire dataset and they stay in sync. This method is excellent for speeding up training but it doesn't reduce the HBM that's required to hold the model itself, because every GPU needs a full copy.

Model parallelism

When a model is too large to fit into the memory of a single GPU, you must use model parallelism. Instead of replicating the model, this strategy splits the model across multiple GPUs. There are two primary ways to do this:

Tensor parallelism: This method splits a single large operation (like a massive weight matrix in a transformer layer) across several GPUs. Each GPU computes its part of the operation, and the results are combined.
Pipeline parallelism: This technique places different layers of the model onto different GPUs in a sequence. The data flows through the first set of layers on GPU 1, then the output is passed to GPU 2 for the next set of layers, and so on, like an assembly line.

These strategies are more complex to implement than data parallelism, but they're essential for models that are simply too big for one device.

Fully Sharded Data Parallelism (FSDP)

FSDP is a powerful and efficient hybrid strategy that combines the ideas of data parallelism and model parallelism. Unlike standard data parallelism where each GPU holds a full copy of the model, optimizer states, and gradients, FSDP shards (or splits) all of these components across the GPUs. Each GPU only materializes the full parameters for the specific layer that it's computing at that moment, dramatically reducing the peak HBM usage per device. FSDP makes it possible to train enormous models on a cluster of smaller GPUs.

By combining these hardware and software strategies, you can scale your fine-tuning jobs from a single GPU to a powerful, distributed cluster capable of handling even the most demanding AI models.

HBM sizing guide

HBM	Use case and explanation
16 GB	Sufficient for basic inference or fine-tuning with techniques like LoRA using a very small batch size (e.g., 1-2). Expect slower training times at this level.
24 GB	The recommended starting point for a good experience with 4-7 B parameter models. This capacity allows for a more effective batch size (e.g., 8-16) when using LoRA, providing a great balance of training speed and cost.
40+ GB	Necessary for maximizing training speed with large batch sizes or for working with larger models (in the 20+ B parameter range) now or in the future.

Encountering the CUDA out of memory error provides an important lesson in the trade-offs between model size, training techniques, and batch size. By understanding what consumes your HBM, you can make smarter decisions and keep your projects running smoothly.

I hope that this guide has helped clarify the CUDA out of memory error and that it's given you the tools to keep your projects running smoothly. When you're ready to take the next step, Google Cloud has the tools to accelerate your AI development.

Explore GPU configurations for your Cloud Run services and best practices for running Cloud Run jobs with GPU.
For maximum control: Spin up a Compute Engine instance with the latest NVIDIA H100 or A100 Tensor Core GPUs and take full control of your environment.
Looking to optimize your model hosting infrastructure? Take a look at The Ultimate Guide to xPU Inference Configuration.
For a deeper dive into scaling your model, check out How to Scale Your Model.
New to Google Cloud? Get started with the $300 free credit to find the perfect solution for your next project.

Special thanks to Jason Monden and Sayce Falk from the AI compute team for their helpful review and feedback on this post.

Agent Factory Recap: Can you do my shopping?

Shir Meir Lador — Fri, 19 Dec 2025 19:44:58 +0000

In episode #8 of The Agent Factory, Ivan Nardini and I are joined by Prateek Dudeja, product manager from the Agent Payment Protocol Team, to dive into one of the biggest hurdles for AI agents in eccomerce: trust, especially when it comes to money.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

Introducing Agent Payment Protocol

Timestamp: [01:43]

What if an agent could buy concert tickets for you at a specific time that the tickets go on sale. You don't want to miss out! Maybe you want two tickets, and you don't want to spend more than $200. You definitely want to sit in a section with a great view of the stage. To have an agent act as your ticket buyer, you would have to trust that agent with all facets of your request and your credit card. How can you be sure that the agent won't buy 200 tickets or that it won't charge you for a lifetime supply of rubber duckies?

The potential for a messy outcome with this concert ticket request provides insight into a "Crisis of Trust" that can hold back agentic commerce. The good news is there's a way to move forward and build trust.

To solve the "Crisis of Trust," Google introduced the Agent Payment Protocol (AP2), a new open standard. It's not a new payment system; it’s a "trust layer" that sits on top of existing infrastructure. AP2 is designed to create a common, secure language for agents to conduct commerce, using role-based architecture and verifiable credentials.

Agent Payments and the Current Payment System

Timestamp: [02:29]

The current payment system was built for humans using trusted interfaces like browsers, not for autonomous agents, resulting in three main challenges for agents: authorization, agent error, and accountability.

The Agent Payment Protocol addresses these challenges by helping agents communicate securely with merchants and payment partners. The Agent Payment Protocol is available today as an extension for the A2A (Agent2Agent) protocol and relies on agents using the Model Context Protocol (MCP).

Deep Dive into the Agent Payment Protocol

Learn more about how this protocol works, including concepts and flow.

A Role-Based Ecosystem

Timestamp: [04:33]

The protocol is built on a "separation of concerns." Your agent doesn't have to do everything. There are specialized roles:

Shopping Agent: The AI agent you build, great at finding products.
Merchant Endpoint: The seller's API.
Credential Provider: A secure digital wallet (like PayPal, Google Pay, etc.) that manages payment details.
Merchant Payment Processor: The entity that constructs the final authorization message for the payment networks.

_Critical: Your shopping agent never touches the raw credit card number. It doesn't need to be PCI compliant because it delegates the payment to the specialized, secure providers._

Verifiable Credentials (VCs)

Timestamp: [06:15]

The "handshakes" between these roles in the Agent Payment Protocol ecosystem are secured by Verifiable Credentials (VCs). Think of credentials as protocolized, cryptographically signed digital receipts that prove what was agreed upon.

There are three types of verifiable credentials:

Cart Mandate: For "human-present" scenarios. The user reviews a final cart and cryptographically signs it as proof of approval.
Intent Mandate: For "human-not-present" scenarios (like the concert ticket example). The user signs an intent (e.g., "buy tickets under $200"), giving the agent authority to act within those guardrails.
Payment Mandate: Provides clear visibility to payment networks and banks that an AI agent was involved in the transaction.

A Contractual Conversational Model

Timestamp: [08:03]

The Agent Payment Protocol process creates a "Contractual Conversational Model," moving beyond simple API calls to a flow built on verifiable proof.

To understand this flow, we'll walk through a human-present scenario:

Delegation: You tell your agent, "Buy two concert tickets."
Discovery & Negotiation: The agent contacts the merchant's endpoint to prepare the cart.
Finalize Cart: The agent reaches out to your Credential Provider (e.g., your digital wallet). You select the payment method. The agent only gets a reference (like the last 4 digits), never the full credential.
Authorization with Mandates: The agent shows you the final, finalized cart.
You cryptographically sign the Cart Mandate. This is the non-repudiable proof, the "contract."
Purchase: The agent sends this signed mandate to the merchant. The merchant can now trust the purchase mandate is from you. The merchant's payment processor uses the mandate to securely get the payment token from the credential provider and complete the transaction.

This flow all hinges on trust. In the short term, this trust is built using manual allow lists of approved agents and merchants. In the long term, the plan is to use open web standards like HTTPS and DNS ownership to verify identities.

Q&A with Prateek Dudeja

Timestamp: [13:07]

With the concepts explained, the discussion moved to a Q&A with Prateek.

Why a New Protocol for Payments?

_Timestamp: [13:30]
_
Prateek gave a great analogy: HTTPS is a baseline protocol for browsing. Signing in requires stronger authentication. Making a payment requires an even higher level of trust. AP2 provides that "payments-grade security" on top of baseline protocols like A2A and MCP, ensuring the transaction is high-trust and truly from a human.

How Will Agents Find Trusted Partners?

Timestamp: [14:42]

In the short term, agents will use "decentralized registries of trust" (or allow lists) to find merchants they can interact with. Prateek noted that all the roles (merchant, credential provider, etc.) already exist in the payments industry today. The only new role is the Shopping Agent itself.

Accountability: What Happens When Things Go Wrong?

Timestamp: [16:03]

This is the big question. What if your agent shows you blue shoes, you wanted teal, but you click "approve" anyway?

Prateek explained that the signed Cart Mandate solves this. Because you biometrically signed a tamper-proof credential showing the blue shoes, the responsibility is on you. The merchant has cryptographic evidence that you saw and approved the exact product. This protects merchants from fraudulent chargebacks and users from unauthorized agent actions.

Demo: Reference Implementation

Timestamp: [18:04]

Prateek walked through a demo showing the human-present flow. It showed the user prompting the agent, the agent discovering products, and then the Credential Provider (PayPal) getting involved. The user selected their shipping and payment info from PayPal, and the agent only saw a reference. The user then signed the Cart Mandate, and the purchase was completed.

Compatibility and Getting Started

Timestamp: [19:43]

A key question was - Is this compatible with frameworks like LangGraph or CrewAI? Yes. Prateek confirmed the protocol is compatible with any framework. As long as your agent can communicate over A2A or MCP, you can use AP2.

To get started, Prateek directed developers to the GitHub repo. The first step is to see which role you want to play (merchant, credentials provider, etc.) and explore the sample code for that role.

The Future: Dynamic Negotiation

Timestamp: [21:13]

Looking ahead, Prateek shared an exciting vision for "dynamic negotiation." Imagine telling your agent: "I want that red dress that's out of stock. I need it by tomorrow... and I'm willing to pay 30% more".

A merchant's agent could see this "intent" and, if the dress becomes available, automatically complete the sale. What was a lost sale for the merchant becomes a completed order at a markup, and the user gets the exact item they desperately wanted.

Your turn to build

This conversation made it clear that building a secure payment infrastructure is a foundational step toward creating agents that can perform truly useful tasks in the real world. We're moving from a simple, programmatic web to a conversational, contractual one, and this protocol provides the framework for it.

We encourage you to check out the Agent Payment Protocol GitHub repo, think about which role you could play in this new ecosystem, and start building today!

Connect with us

Shir Meir Lador → LinkedIn, X
Ivan Nardini → LinkedIn, X
Prateek Dudeja → Linkedin

The Agent Factory podcast: 5 Episodes to Kickstart Your Journey to Production AI

Shir Meir Lador — Tue, 25 Nov 2025 21:22:16 +0000

We are so proud to announce that a project we're incredibly passionate about has grown into a full-blown resource for developers: The Agent Factory video podcast.

We started this show with a simple mission: to have the conversations developers need to be having about AI agents development. We wanted to move past the hype and focus on what really matters—building production-ready AI agents.

Fast forward to today, and we have 14 episodes published covering everything from architecture patterns to end to end vibe coding of advanced AI applications. To celebrate, we’re sharing our first 5 foundational episodes with the Dev.to community. If you are just starting to build agents or looking to harden your existing systems, this is the perfect place to start.

What to Expect:
We pack every episode with three core segments designed for developers:

🎙️ Agent Industry Pulse: We filter the noise and bring you the latest news you actually need to know.

🛠️ The Factory Floor: A technical deep-dive where we get our hands dirty with code, architectures, and patterns.

❓ Developer Q&A: We answer real questions from the community to help us learn together.

📺 The Starter Pack: Our First 5 Episodes
Here is the chronological journey to get you up to speed, starting from the very beginning.

1. Agents, their frameworks and when to use them (ft. Julia Wiesinger)
We kicked things off by tackling the big questions: What exactly is an agent? How do you choose between frameworks like LangChain, CrewAI, or the Agent Development Kit (ADK)? We were joined by Julia Wiesinger from the ADK team to guide us through building for production.

2. Multi-Agent Systems: Concepts & Patterns
Single agent or multi-agent? In this episode, we break down the architectural patterns that matter - from Supervisors to Swarms. We discuss exactly when you should transition from a single agent to a team of agents to handle complexity and improve reliability.

3. Building Custom Tools for Agents
Agents are only as good as the tools they can use. We dive into Model Context Protocol (MCP), function calling, and how to build secure, authenticated tools that let your agents interact with the real world safely.

4. Memory in Agents (ft. Kimberly Milam)
How do you stop your agent from acting like a goldfish? We chat with Kimberly Milam about implementing long-term memory, managing state, and the "Memory Bank" concept to create personalized experiences that persist across sessions.

5. Tackling the Hardest Questions (ft. Philipp Schmidt)
We sat down with Philipp Schmidt from Google DeepMind for a masterclass on the agent development workflow. We cover context engineering, evaluation strategies, and pro-tips for using the Gemini CLI to speed up your development cycle.

💬 Join the Conversation
We’re truly excited to continue building this community with you. Whether you're stuck on a specific bug or wondering about a new architecture, we want to hear from you.

What are you struggling with right now? Drop your questions in the comments below with #TheAgentFactory, and we might answer them in our next Q&A segment!

➡️ Listen & Subscribe: Google Cloud Tech on YouTube