Forem: Google Developer Experts

Monitoring Sheet Changes with SHEET and SHEETS Functions on Google Sheets

Tanaike — Wed, 08 Apr 2026 05:38:35 +0000

Abstract

Google Sheets recently introduced the SHEET and SHEETS functions. Because they automatically recalculate upon structural changes, developers can utilize them as custom triggers. This article demonstrates how to leverage these functions to detect sheet insertions, deletions, renames, and movements without requiring cumbersome installable triggers in Google Apps Script.

Introduction

On February 23, 2026, Google introduced two pivotal built-in functions to Google Sheets: SHEET and SHEETS Ref. The SHEET function returns the index (sheet number) of a specified sheet or reference Ref. Meanwhile, the SHEETS function provides the total count of sheets within a spreadsheet Ref.

A critical technical characteristic of these functions is their volatility and automatic recalculation based on the spreadsheet's structural metadata. Specifically:

SHEET triggers a recalculation when a sheet is renamed or its position is changed via drag-and-drop.
SHEETS triggers a recalculation whenever a sheet is inserted, duplicated, or removed.

Historically, detecting such structural changes necessitated the use of an installable OnChange trigger in Google Apps Script (GAS). This posed a significant barrier for template distribution, as installable triggers require manual authorization by each user and do not persist through simple file copies.

By leveraging these new functions as "custom triggers," we can effectively bypass the need for installable triggers. When a custom function or formula containing SHEET or SHEETS recalculates, it serves as a catalyst for GAS execution. This enables the creation of self-contained spreadsheets where advanced sheet monitoring logic is activated immediately upon copying the file, significantly improving the user experience and portability of GAS-based solutions.

Implementation Guide

1. Spreadsheet Initialization

Create a new Google Sheets file to serve as your testing environment.

2. Google Apps Script Configuration

Open the Google Apps Script editor bound to your new spreadsheet. Copy and paste the following script, replacing any default code, and save your project.

/**
 * Global constant defining the Script Properties storage key.
 */
const STORAGE_KEY = "SHEET_STATUS_CACHE";

/**
 * Simple trigger to initialize the cached sheet metadata upon opening the spreadsheet.
 */
function onOpen() {
  updateStoredSheetData_();
}

/**
 * Custom function wrapper to retrieve an array of all current sheet names.
 */
function getAllSheetNames() {
  return SpreadsheetApp.getActiveSpreadsheet()
    .getSheets()
    .map((s) => s.getName());
}

/**
 * Core custom function for detecting structural alterations.
 * Priority: If any structural change (Add/Remove/Rename) occurs,
 * subsequent "move" reports are suppressed to avoid noise from index shifting.
 */
function getSheetStatus(_trigger1, _trigger2, returnRawObject = false) {
  const propService = PropertiesService.getScriptProperties();
  const oldDataJson = propService.getProperty(STORAGE_KEY);

  const ss = SpreadsheetApp.getActiveSpreadsheet();
  const currentSheets = ss.getSheets();
  const newData = currentSheets.map((s, i) => ({
    name: s.getName(),
    id: s.getSheetId(),
    index: i + 1,
  }));

  if (!oldDataJson) {
    updateStoredSheetData_();
    return "Initial state recorded.";
  }

  const oldData = JSON.parse(oldDataJson).map((s, i) => ({
    ...s,
    index: i + 1,
  }));

  // Categorize detected modifications into structural changes or simple index moves.
  const { structural, moves } = detectCategorizedChanges_(oldData, newData);

  // Persist the newly fetched metadata to Script Properties.
  propService.setProperty(
    STORAGE_KEY,
    JSON.stringify(newData.map(({ name, id }) => ({ name, id }))),
  );

  // Evaluate which results to return based on change priority.
  // Ignore index movement noise if a primary structural change (addition, removal, rename) occurred.
  const finalDiffs = structural.length > 0 ? structural : moves;

  if (finalDiffs.length === 0) return "No change";
  if (returnRawObject) return JSON.stringify(finalDiffs);

  return finalDiffs.map((d) => d.message).join("\n");
}

/**
 * Retrieves current spreadsheet metadata and persists it to the properties cache.
 */
function updateStoredSheetData_() {
  const ss = SpreadsheetApp.getActiveSpreadsheet();
  const sheetData = ss.getSheets().map((s) => ({
    name: s.getName(),
    id: s.getSheetId(),
  }));
  PropertiesService.getScriptProperties().setProperty(
    STORAGE_KEY,
    JSON.stringify(sheetData),
  );
  return sheetData;
}

/**
 * Differentiates between core structural changes and simple index shifts.
 */
function detectCategorizedChanges_(oldData, newData) {
  const structural = [];
  const moves = [];

  const oldMap = new Map(oldData.map((s) => [s.id, s]));
  const newMap = new Map(newData.map((s) => [s.id, s]));

  // 1. Identify removed sheets (Structural change).
  oldData.forEach((oldSheet) => {
    if (!newMap.has(oldSheet.id)) {
      structural.push({
        type: "removed",
        message: `"${oldSheet.name}" was removed.`,
      });
    }
  });

  // 2. Identify added, renamed, or repositioned sheets.
  newData.forEach((newSheet) => {
    const oldSheet = oldMap.get(newSheet.id);

    if (!oldSheet) {
      // Handle added sheets (Structural change).
      structural.push({
        type: "added",
        message: `"${newSheet.name}" was inserted at tab ${newSheet.index}.`,
      });
    } else {
      const isRenamed = oldSheet.name !== newSheet.name;
      const isMoved = oldSheet.index !== newSheet.index;

      if (isRenamed) {
        // Handle renamed sheets (Structural change).
        structural.push({
          type: "renamed",
          message: `"${oldSheet.name}" was renamed to "${newSheet.name}".`,
        });
      } else if (isMoved) {
        // Handle pure index movements (Non-structural change).
        moves.push({
          type: "moved",
          message: `"${newSheet.name}" was moved from tab ${oldSheet.index} to ${newSheet.index}.`,
        });
      }
    }
  });

  return { structural, moves };
}

/**
 * Optional callback for installable OnChange triggers.
 * If utilized, do not use the custom formula in the sheet.
 */
function onChange(e = {}) {
  const { changeType } = e;
  if (["OTHER", "INSERT_GRID", "REMOVE_GRID"].includes(changeType)) {
    const res = getSheetStatus();
    Browser.msgBox(res);
  }
}

3. Applying the Custom Formula

Return to your Google Sheets interface. In the first sheet, copy and paste the following formula into cell A1:

=getSheetStatus(SHEETS(),MAP(getAllSheetNames(),lambda(sheetname,SHEET(sheetname))))

4. Testing the Workflow

Once the formula is in place, the cell will actively monitor the document's structure. The demonstration below illustrates the expected behavior:

During the demonstration, the following actions trigger a status update in the cell:

Renaming an existing sheet.
Adding a new sheet to the workbook.
Changing the order (moving) of a sheet.
Deleting a sheet.

Appendix: Using Traditional Triggers

If you prefer to use the traditional installable onChange trigger method, you can utilize the onChange function provided at the bottom of the script. In this scenario, you must manually bind the trigger to the onChange function via the Apps Script dashboard and remove the custom formula (=getSheetStatus(...)) from cell A1. The underlying logic handles the detection identically, routing the output to a browser message box instead of a cell.

Summary

The introduction of SHEET and SHEETS functions enables automatic formula recalculation based on structural metadata changes.
These built-in functions can be passed into custom Google Apps Script functions to act as volatile execution triggers.
This methodology eliminates the traditional dependency on installable onChange triggers for monitoring tab modifications.
Removing installable triggers drastically improves script portability and the user experience when distributing spreadsheet templates.
The provided script successfully categorizes and logs structural alterations versus simple index shifts for comprehensive monitoring.

Running Agentic AI at Scale on Google Kubernetes Engine

Saurabh Mishra — Wed, 08 Apr 2026 04:15:15 +0000

The AI industry crossed an inflection point. We stopped asking "can the model answer my question?" and started asking "can the system complete my goal?" That shift from inference to agency changes everything about how we build, deploy, and scale AI in the cloud.

Google Kubernetes Engine (GKE) has quietly become the platform of choice for teams running production AI workloads. Its elastic compute, GPU node pools, and rich ecosystem of observability tools make it uniquely suited not just for model serving but for the orchestration challenges that agentic AI introduces.

This blog walks through the full landscape: what kinds of AI systems exist today, how agentic architectures differ, and what it actually looks like to run them reliably on GKE.

The AI Taxonomy: From Reactive to Autonomous

Before diving into infrastructure, it's worth establishing what we mean by the different modes of AI deployment. Not all AI is "agentic," and the architecture you choose should match the behavior you need

Reactive / Inference

Stateless prompt-response. One request, one LLM call, one answer. The model has no memory between turns. Examples: text classifiers, summarizers, one-shot code generators.

Conversational AI

Multi-turn dialog with session state. The model remembers context within a conversation window. Examples: customer support bots, document Q&A, coding assistants.

Retrieval-Augmented (RAG)

The model can query external knowledge at runtime before generating a response. Introduces a retrieval step vector DBs, semantic search, tool calls to databases.

Agentic AI

The model plans, takes actions, observes results, and loops until a goal is reached. It can call tools, spawn subagents, and make decisions across many steps autonomously.

Multi-Agent Systems

A network of specialized agents collaborating: an orchestrator decomposes a task and delegates to researcher, writer, executor agents that work in parallel or sequence.
Each mode up the stack introduces new infrastructure requirements: more state to manage, longer-lived processes, more concurrent workloads, harder failure modes, and deeper observability needs.

Why GKE for AI Workloads?

Kubernetes is table stakes for any modern distributed system. But GKE specifically brings several features that make it exceptional for AI:

GKE Capabilities for AI

GPU and TPU Node Pools

To handle the heavy lifting of Agentic AI, GKE offers specialized Accelerator Node Pools. This infrastructure allows you to dynamically attach high-end compute resources such as NVIDIA A100, H100, or L4 GPUs and Google TPUs exactly when your agents need them.

Workload Identity & Secret Management

Agentic systems touch many external APIs (databases, external services, third-party tools). Workload Identity Federation lets pods authenticate to Google Cloud services without storing long-lived credentials.

Horizontal Pod Autoscaling with Custom Metrics

Scale agent runner replicas based on queue depth (Pub/Sub backlog, Redis list length) rather than CPU. This allows demand-driven scaling that matches agent workload patterns precisely.

GKE Autopilot & Standard Modes

Autopilot mode handles node management entirely, ideal for teams wanting to focus on agent logic. Standard mode gives full control when you need custom kernel modules or specialized hardware affinity rules.

Cloud Run on GKE for Burst Workloads

Short-lived tool execution steps in an agent pipeline can be offloaded to Cloud Run, which scales to zero between invocations avoiding the overhead of always-on Kubernetes pods for infrequent task

Anatomy of an Agentic AI System

An agentic AI system isn't a single process ,it's a distributed workflow. Understanding its components is essential before mapping it onto Kubernetes primitives.
"An agent is an LLM that can observe the world, decide what to do next, and take actions - in a loop, until a goal is satisfied."

Popular Agentic Frameworks on GKE

Several frameworks have emerged to help teams build agentic systems without reinventing the orchestration wheel. Each has a different philosophy and maps to GKE differently.

Agent Development Kit (ADK)

Google's native framework for building multi-agent systems on Vertex AI. First-class GKE support, tight Gemini integration, built-in evaluation tools. Best choice for teams already on Google Cloud.

LangGraph

Graph-based agent orchestration with explicit state machines. Excellent for complex branching workflows. Containerizes cleanly. LangSmith provides tracing that integrates with GKE logging pipelines

CrewAI

Defines agents as role-playing entities (Researcher, Writer, Editor) with goals and backstories. Simple to model complex human workflows. Ideal for content, analysis, and research pipelines.

Google ADK on GKE >> Native Fit

The Google Agent Development Kit (ADK) is architected to treat Kubernetes as its primary "home," creating a seamless integration where the framework and the platform operate as one. Because ADK is built with a Kubernetes-native philosophy, it transforms GKE from a simple hosting environment into a specialized runtime for autonomous systems.

Observability: The Hard Part

Agentic systems fail in non-obvious ways. An agent might produce a response - but the response could be hallucinated, based on a failed tool call, or the result of an unintended plan branch. Standard HTTP error monitoring doesn't catch this.

The recommended observability stack for GKE-based agentic systems:

Observability Stack

OpenTelemetry Instrumentation

Instrument each agent with OpenTelemetry. Emit spans for every LLM call, tool invocation, and planning step. Export to Google Cloud Trace for full distributed trace visualization.

Structured Logging to Cloud Logging

Log each reasoning step as a structured JSON event: task ID, agent ID, step number, prompt hash, tool name, tool result summary, token counts. Query across traces in BigQuery for post-hoc analysis.
Custom Metrics via Cloud Monitoring

Track agent-specific metrics: tasks completed per minute, average steps per task, tool call success rate, LLM latency P50/P95/P99, and hallucination rate from your eval pipeline.

LLM-specific Tracing (LangSmith / Vertex AI Eval)

Leverage LangSmith or Vertex AI's built-in evaluation capabilities to capture complete prompt–response interactions along with semantic quality metrics. These insights can then be fed back into your continuous improvement cycle.

Security Considerations for Agentic AI on GKE

Agents with tool use are a new attack surface. An agent that can execute code, send emails, or write to a database is a powerful actor - and must be treated like one.

Prompt Injection

Malicious content in retrieved documents can instruct the agent to deviate from its goal. Sanitize all retrieved content before insertion into prompts. Use system-level guardrails in your LLM configuration.

Privilege Escalation

Each agent should operate with the minimum IAM permissions needed for its specific tools. Use Workload Identity with role-specific service accounts never a single all-powerful SA for all agents.

Human-in-the-Loop Gates

For irreversible actions (sending emails, deploying code, database writes), require a human approval step before execution. Implement approval workflows via Pub/Sub pause + Cloud Tasks callback.

Network Policies

Use GKE Network Policies to restrict which agent pods can talk to which services. A researcher agent has no reason to reach the database writer service directly - enforce this in the cluster, not just in code.

What's Next: The Agentic Platform

The direction of travel is clear. GKE is evolving from an application runtime into an agentic platform - a place where autonomous AI systems can be deployed, composed, monitored, and governed with the same rigor we apply to microservices today.
Several emerging capabilities are worth tracking:

Agent-to-Agent Communication (A2A Protocol) - Google's emerging standard for cross-agent RPC, allowing agents built with different frameworks to interoperate. GKE provides the network fabric for this via internal load balancers and service mesh.

Model Context Protocol (MCP) on Kubernetes - MCP is becoming the standard way for agents to discover and call tools. Running MCP servers as sidecar containers or standalone Deployments in GKE makes tool registries cluster-native.

Vertex AI Agent Engine - Google's fully managed orchestration layer for agents that sits above GKE, handling session management, tool routing, and evaluation out of the box. The boundary between GKE and managed agent infrastructure will continue to blur.

"Kubernetes wasn't built for AI. But it turns out the problems of distributed systems - scale, failure, state, observability - are exactly the problems agentic AI inherits."

Core Reference Documentation

https://docs.cloud.google.com/kubernetes-engine/docs/integrations/ai-infra

https://github.com/GoogleCloudPlatform/accelerated-platforms/blob/main/docs/platforms/gke/base/use-cases/inference-ref-arch/README.md

https://docs.cloud.google.com/agent-builder/agent-development-kit/overview

Hands-on Tutorials

https://codelabs.developers.google.com/devsite/codelabs/build-agents-with-adk-foundation

https://cloud.google.com/blog/topics/developers-practitioners/build-a-multi-agent-system-for-expert-content-with-google-adk-mcp-and-cloud-run-part-1

MCP Development with Gemini CLI, and Amazon AWS EKS

xbill — Tue, 07 Apr 2026 19:57:35 +0000

Leveraging Gemini CLI and the underlying Gemini LLM to build Model Context Protocol (MCP) AI applications with Python with a local development environment hosted on AWS EKS.

Aren’t There a Billion Python MCP Demos?

Yes there are.

Python has traditionally been the main coding language for ML and AI tools. The goal of this article is to provide a minimal viable basic working MCP stdio server that can be run locally without any unneeded extra code or extensions.

What Is Python?

Python is an interpreted language that allows for rapid development and testing and has deep libraries for working with ML and AI:

Welcome to Python.org

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-stdio-python-aws$ python --version
Python 3.13.12

Amazon EKS

Amazon Elastic Kubernetes Service (EKS) is a fully managed service from Amazon Web Services (AWS) that makes it easy to run Kubernetes on AWS without needing to install, operate, or maintain your own Kubernetes control plane. It automates cluster management, security, and scaling, supporting applications on both Amazon EC2 and AWS Fargate.

More information is available here:

What is Amazon EKS?

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

gemini

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-stdio-python-aws$ gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

? for shortcuts 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 shift+tab to accept edits 3 GEMINI.md files | 1 MCP server
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 > Type your message or @path/to/file
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 ~/.../mcp-stdio-python-aws (main*) no sandbox (see /docs) /model Auto (Gemini 3) | 239.8 MB

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

The most common MCP Python deployment path uses the FASTMCP library:

Welcome to FastMCP - FastMCP

Where do I start?

The strategy for starting MCP development is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal Hello World Style Python MCP Server is built with stdio transport. This server is validated with Gemini CLI in the local environment.

This setup validates the connection from Gemini CLI to the local process via MCP. The MCP client (Gemini CLI) and the Python MCP server both run in the same local environment.

Next- the basic MCP server is extended with Gemini CLI to add several new tools in standard Python code.

Setup the Basic Environment

At this point you should have a working Python interpreter and a working Gemini CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemini-cli-aws

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gemini-cli-aws
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gemini-cli-aws
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Hello World with HTTP Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.

The HTTP transport allows the MCP Client and Server to be in the same environment or distributed over the Internet.

The connection over HTTP will look similar to this:

mcp.run(
        transport="http",
        host="0.0.0.0",
        port=port,
    )

Running the Python Code

First- switch the directory with the Python MCP sample code:

cd ~/gemini-cli-aws/mcp-https-python-eks

Run the release version on the local system:

cd ~/gemini-cli-aws/mcp-https-python-eks
make release

You can validate the final result by checking the messages:

✦ The release target has been successfully executed, and the server is running locally on port 8080.

  The health check confirms that it is responding:

   1 {"status":"healthy","service":"mcp-server"}

  The process is running in the background with PID 29456.

Deploy to EKS

Now that the server has been tested locally- start the remote deployment to EKS.

Use the deploy target:

 > make deploy
✦ I will now execute the make deploy target to rebuild the Docker image, push it to ECR, and update the Kubernetes
  deployment on Amazon EKS.

Then check the status:

✦ The EKS deployment is fully operational, with the pod READY and the LoadBalancer service accessible via the
  external endpoint.

  Status Summary:
   - Deployment: mcp-server-image (1/1 pods available)
   - External Endpoint: http://a8acf5c3ad24348c89537b575c29b32f-1627118381.us-east-1.elb.amazonaws.com
   - Port: 80

And the endpoint:

✦ The EKS LoadBalancer public endpoint is:
  a8acf5c3ad24348c89537b575c29b32f-1627118381.us-east-1.elb.amazonaws.com

Gemini CLI settings.json

The default Gemini CLI settings.json has an entry for the Python source:

   1 {
   2 "mcpServers": {
   3 "mcp-eks-python-aws": {
   4 "httpUrl": "http://a8acf5c3ad24348c89537b575c29b32f-1627118381.us-east-1.elb.amazonaws.com/mcp"
   5 }
   6 }
   7 }

Validation with Gemini CLI

Leave the HTTP server running and open a new terminal.

Gemini CLI is restarted and the MCP connection over HTTP to the Python Code is validated. The full Gemini CLI Session will start:

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-https-python-aws$ gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

 > /mcp list
Configured MCP servers:

🟢 mcp-eks-python-aws - Ready (1 tool)
  Tools:
  - mcp_mcp-eks-python-aws_greet

And you can then connect to the MCP Server over HTTP:

                                                                    │
 > greet eks
✦ Calling the greet tool with the argument "eks".

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ greet (mcp-eks-python-aws MCP Server) {"param":"eks"} │
│ │
│ eks │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
✦ The greet tool returned: eks

Project Review

The full deployment was reviewed with Gemini CLI:

✦ I have completed my review of the Amazon EKS project setup. The repository is well-structured and follows
  industry best practices for containerized Python applications on Kubernetes.

  Key Strengths
   - Infrastructure as Code: The use of eksctl with eks-cluster.yaml provides a clear and reproducible way to
     manage the EKS cluster.
   - Automation: The Makefile is comprehensive, covering the entire development lifecycle from local testing and
     linting to Docker builds and EKS deployment.
   - Observability: Structured JSON logging is implemented correctly, which is essential for effective log
     analysis in CloudWatch.
   - Reliability: The inclusion of Kubernetes liveness and readiness probes linked to a dedicated /health endpoint
     ensures the service remains stable in production.
   - Security: The eks-cluster.yaml is pre-configured for IAM Roles for Service Accounts (IRSA) via OIDC, allowing
     for fine-grained permissions if needed in the future.

Summary

The strategy for using Python for MCP development with Gemini CLI and AWS was validated with a incremental step by step approach.

A minimal HTTP transport MCP Server was started from Python source code and validated with Gemini CLI running as a MCP client in the same local environment.

The HTTP server was then tested remotely from the AWS EKS deployment.

This approach can be extended to more complex deployments using other MCP transports and Cloud based options.

Implementing a RAG system: Run

Glen Yu — Tue, 07 Apr 2026 17:00:00 +0000

In the "Crawl" and "Walk" phases, I introduced the basics of RAG and explored ways to optimize the pipeline to increase efficiency and accuracy. Armed with this knowledge, it's time to productionize our learnings.

Run

In the "Crawl" and "Walk" phases, we explored RAG fundamentals using local tools, proving how much document processing and re-ranking impact performance. While you could certainly scale those manual workflows into production, do you really wan to manage the infrastructure, data pipelines and scaling hurdles yourself?

Welcome to the "Run" phase. Here we leverage Google Cloud's Vertex AI RAG Engine - a fully managed solution that automates the entire pipeline so you can focus on building, not maintenance.

Vertex AI RAG Engine

Vertex AI RAG Engine is a low-code, fully managed solution for building AI applications on private data. It handles the ingestion, document processing, embedding, retrieval, ranking, and grounding to ensure that the response is highly accurate and relevant.

Optional document pre-processing with Docling

Though RAG Engine comes with its own parser options, I still opted to pre-process my documents using Docling first and upload them to a Google Cloud Storage bucket for ingestion:

if __name__ == "__main__":
    DATA_URL = "https://storage.googleapis.com/public-file-server/genai-downloads/bc_hr_policies.tgz"
    DATA_DIR = download_and_extract(DATA_URL)

    docling_docs = Path("./docling_docs")
    docling_docs.mkdir(parents=True, exist_ok=True)

    print("> Processing PDFs...")
    data_path = Path(DATA_DIR)
    pdf_files = list(data_path.glob("*.pdf"))

    for file in pdf_files:
        try:
            result = converter.convert(file)
            markdown_content = result.document.export_to_markdown()

            docling_doc_name = Path(file).stem + ".md"
            with open(f"docling_docs/{docling_doc_name}", "w", encoding="utf-8") as f:
                f.write(markdown_content)
        except Exception as e:
            print(f"Error on {file}: {e}")

    print("> Uploading Docling docs to GCS...")
    upload_folder_to_gcs("./docling_docs", GCS_BUCKET, GCS_BUCKET_PATH)

One benefit of doing it this way is I get more visibility and control over the document processing step and can validate the contents from the original PDFs with the Docling documents (Markdown). This allows me to use the Default parsing libraries option, which is also free compared to the LLM parser and Document AI layout parser options which have an additional cost and setup component to them.

I do lose out on the benefits of the hybrid chunking strategy that Docling would have provided (as seen in the "Walk" phase), because that is determined by the layout parser that I choose here. If I wasn't using Docling, I think the LLM parser would be the parser option that I'd be gravitating towards.

Vector database options

When it comes to vector database options, you'll see several choices in the RAG Engine menu (including regional "Preview" features). I chose the RagManaged Cloud Spanner (also referred to as the RagManagedDb) because it offers the fastest path from data to insights with the least amount of infrastructure management. While Spanner is typically an enterprise-grade database, the RAG Engine allows you to spin it up on the Basic tier. This allocates 100 processing units, which is 10% of a Spanner node, making it perfect for smaller datasets while still giving you the reliability of a managed service without the enterprise-grade cost.

IMPORTANT: Even on basic tier, this will still run you about $65 USD/month, so please remember to delete and clean up this RAG corpus once you're done experimenting with it.

For those prioritizing flexibility, the RAG Engine also supports third-party options like Pinecone and Weaviate. These are excellent choices if portability is a requirement, allowing you to maintain a consistent vector store even if you decide to shift parts of your RAG stack to a different cloud provider or platform later on.

Ranking & grounding included

Once the RAG corpus is created, you can perform some manual testing to validate. When you ask RAG Engine for search results, re-ranking and grounding is done automatically to ensure relevance and correctness:

Model Armor

In a production setting (especially if it's going to be public facing), you will want guardrails. I've written about Guardrails with Agent Development Kit in the past and is implemented through callbacks within ADK. It works the same way here and can be used to inspect text as it flows into and out of the LLM/agent. Key capabilities include:

Prompt injection & jailbreak detection: Attempts to trick the AI into ignoring instructions
Sensitive Data Protection: Natively integrates with Google's Data Loss Prevention (DLP) to scale for various types of sensitive information (PII)
Malicious URL detection
Responsible AI (RAI) filters: Hate speech, harassment, dangerous, and sexually explicit content

I configured my Model Armor policy template which I then invoked via the Python SDK library to determine whether the text was safe.

Updated example

You can find the code for "Run" phase → here

Querying the vector database (RAG Engine in this case) is a lot less involved as I don't have to write a many of the logic to pass the semantic search results to a re-ranker, because RAG Engine takes care of all of that for me!

I once again ran the same two benchmark questions as I did in the "Crawl" and "Walk" phases:

I threw in a couple of extra questions to make sure Model Armor wasn't sleeping on the job, but overall I liked the detail and accuracy of the answers I was provided.

NOTE: In my updated example, I only added a before_model_callback, meaning I'm only checking the input prompts and not the response. An after_model_callback should be implemented to ensure the generated response is also scanned and preventing the AI from accidentally leaking sensitive internal data it might have retrieved from the RAG corpus (I omitted the output check here simply because I know there's no sensitive data in this particular dataset).

Summary

The purpose of this "Crawl, Walk, Run" series was to take you on a journey from managing code to delivering value. In the earlier phases, we deconstructed the mechanics of RAG works to understand the roles that chunking, embedding, re-ranking play in the overall system. In this final phase, we see how Vertex AI RAG Engine and Model Armor streamline those manual components. By offloading infrastructure management and safety logic to Google Cloud's managed services, you can ensure your system is scalable, accurate, and secure from day one.

Next Steps

Currently in private preview is Vector Search 2.0 with RAG, but reading through its documentation and features, it looks pretty interesting, so once it becomes GA, I will definitely give it a try.
I'm also looking forward to all the new AI-related announcements that is sure to happen at Google Next!

Additional learning

Interested in finding out more about how to secure your agent by sanitizing input and output? Try out this Model Armor Codelab!
Vertex AI RAG Engine isn't your only option for a managed RAG, if you'd like to try a different option that uses Vector AI Search, might I suggest the Building Agents with Retrieval-Augmented Generation Codelab?

Happy learning!

Google Workspace Studio Tutorial: Turn Google Forms into a Full CRM - Auto Lead Capture, Personalized Emails & Team Notifications

Aryan Irani — Tue, 07 Apr 2026 16:53:56 +0000

As a consultant, I've learned one universal truth, The speed of your response determines the quality of your client relationships.

When a potential lead fills out a Get in Touch form on your website, they are at their peak level of interest. If you wait 24 hours to email them back, that interest cools down. If you don't log their details in a CRM immediately, they become a lost lead in your inbox.

In this tutorial, we are going to build a Lead Capture CRM Workflow in Google Workspace Studio. This agent doesn't just collect data - it manages the relationship, logs the lead, drafts the perfect welcome email, and alerts your team, all while you're doing something else.

The Workflow Logic: The Lead-to-Call Pipeline

Using Google Workspace Studio, the agent that we will build will do the following:

Automatically adds every new form submission to a professional Google Sheet
Uses Gemini to write a warm, personalized welcome email
Draft a personalized welcome email
Send you an instant alert in Google Chat.

If you missed the previous AI Email Labeller tutorial, start there for the basics. Prefer video? Click here.

Prefer to watch instead of read? Check out the video version of the guide here.

Prerequisites

Google Workspace Business, Enterprise, or Education plan with Gemini enabled
A Google Form ready (I used Consulting Business Inquiry Form)

A Google Sheet set up as your CRM (we'll map it live)

Step 1: Create the Flow & Add the Starter

Go to studio.workspace.google.com
Click + New flow and name it Lead Capture Agent: Forms to CRM, Email, and Chat
Click Choose a starter → Select When a form response comes in

Choose your form (in my case: Consulting Business Inquiry Form)

Step 2: Building the Ledger (Add a Row)

We need a single source of truth for our leads. Use the Add a row action to point to your Lead CRM Google Sheet.

Set Add row to After last data row. This instantly turns every submission into a clean, searchable CRM row. Map every form field to the correct column:

Timestamp → Step 1: Response time
Full Name → Step 1: Full Name
Work Email → Step 1: Work Email
Company → Step 1: Company (and so on for all fields)

Step 3: Ask Gemini to Write the Personalized Welcome Email

Click + Add step → Ask Gemini
Use this prompt:

Write a warm, professional, and personalized welcome email for a new lead based on the following form response. Make it friendly and mention that we've received their message and will get back to them soon. Use any available details like their name or company to make it feel unique.
Form Response: {{Step 1: Form response}}

Use the + Variables button to insert the specific Form response chip into your prompt. This is what grounds Gemini in your lead's actual data. This is where we go beyond simple data entry. Gemini turns raw form data into a human-sounding email.

Step 4: The Handoff (Gmail & Google Chat)

Finally, we put the agent to work:

Draft an Email: Use the Draft an email action to create a Gmail draft using the content Gemini just generated in Step 3.

Update Rows: Use the Update rows action to change the Lead Status to Draft Created and Welcome Email Sent to Yes. This gives you a bird's-eye view of your sales pipeline right inside your Sheet.

Pro-Tip: Ensure these columns exist in your Sheet first

Notify in Chat: Use the Notify me in Chat action to send an instant alert:

🚀 New Lead Captured:
Details: [Step 1: Form Response]
Email Draft: [Step 4: Email Body]
ID: [Step 4: Email ID]

Testing & Activation

Before you hand the keys of your lead intake over to an AI agent, you need to verify that it's connecting all the dots. Workspace Studio's Test Run feature is your best friend here. I tested with real leads (including the Project Quantum style inquiry and a general business inquiry). Once selected, go ahead and click on Start.

The agent:

Added the row instantly to the Sheet
Drafted a natural, personalized welcome email
Sent the email
Updated the status
Posted a clean notification in Chat

Once your test confirms that every step is green and the data is flowing correctly, it's time to go live. Go ahead and click the blue Turn on button. The agent is now live and will run on every new form submission.

Why this changes everything

By the time you open your laptop, your new lead is already in your CRM, a personalized email is drafted and waiting for your review, and your team is already celebrating the new opportunity in Google Chat.

You haven't just saved time - you've created a scalable sales process.

Conclusion

By the time you open your laptop, your new lead is already logged in your CRM, a personalized welcome email is drafted and waiting in Gmail, and your team has been notified in Chat.

You haven't just saved time - you've created a scalable, professional first-impression system that makes every potential client feel valued from the very first second.

This is the true power of Google Workspace Studio: turning simple forms into intelligent business processes that run 24/7 while you focus on what you do best.

What's Next?

This is the fourth tutorial of the Google Workspace Studio tutorial series. We've built Meeting Assistants, Inbox Gatekeepers, and now CRM Agents. But the possibilities are endless.

📺 Watch the full build: If you want to see exactly how I configured the logic gates for the CRM updates, check out the video in my Google Workspace Studio YouTube Playlist.

🚀 Join the Journey: Subscribe to my channel to see how we build more agents to solve real-world business problems.

💼 Consultation: Looking to deploy these AI agents across your entire team? Reach out via my website for a custom consultation.

Feel free to reach out if you have any issues/feedback at aryanirani123@gmail.com.

Building a Multimodal Cross Cloud Live Agent with ADK, Azure AKS, and Gemini CLI

xbill — Tue, 07 Apr 2026 13:16:14 +0000

Leveraging the Google Agent Development Kit (ADK) and the underlying Gemini LLM to build cross cloud apps with the Python programming language deployed to the Azure app service.

Aren’t There a Billion Python MCP Demos?

Yes there are.

What Is Python?

Python is an interpreted language that allows for rapid development and testing and has deep libraries for working with ML and AI:

Welcome to Python.org

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

admin@ip-172-31-70-211:~/gemini-cli-azure$ python --version
Python 3.13.12

Azure Kubernates Service

Azure Kubernetes Service (AKS) is a fully managed, serverless Kubernetes service on Microsoft Azure that simplifies deploying, scaling, and managing containerized applications. It handles critical tasks like health monitoring, maintenance, and automated upgrades, reducing operational complexity. AKS is used for microservices, DevOps, and cloud-native app development.

More details are available here:

https://azure.microsoft.com/en-us/products/kubernetes-service

Isn’t that Overkill? A whole Cluster Just for some Agents?!

An entire cluster is a large deployment for just a basic ADK server. The goal was to validate that ADK servers can be deployed — and that opens the door for more complex deployments that can take advantage of the full services in the cluster.

Why would I want Gemini CLI with Azure? Isn’t that a Google Thing?

Yes- Gemini CLI leverages the Google Cloud console and Gemini models but it is also open source and platform agnostic. Many applications are already cross-cloud so this enables familiar tools to be run natively on Microsoft Azure.

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

gemini

admin@ip-172-31-70-211:~/gemini-cli-azure$ gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

? for shortcuts 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 shift+tab to accept edits 3 GEMINI.md files | 1 MCP server
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 > Type your message or @path/to/file
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 ~/.../gemini-cli-azure (main*) no sandbox (see /docs) /model Auto (Gemini 3) | 239.8 MB

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

Docker Version Management

The Azure CLI tools need current version of Docker. If your environment does not provide a recent docker tool- the Docker Version Manager can be used to downlaod the latest supported Docker:

Install

Azure CLI

The Azure CLI provides a command line tool to directly access Azure services from your current environment. Full details on the CLI are available here:

Azure Command-Line Interface (CLI) documentation

Agent Development Kit

The Google Agent Development Kit (ADK) is an open-source, Python-based framework designed to streamline the creation, deployment, and orchestration of sophisticated, multi-agent AI systems. It treats agent development like software engineering, offering modularity, state management, and built-in tools (like Google Search) to build autonomous agents.

The ADK can be installed from here:

Agent Development Kit (ADK)

This seems like a lot of Configuration!

Getting the key tools in place is the first step to working across Cloud environments. For a deeper dive- a project with a similar setup can be found here:

MCP Development with Python, and the Azure App Service

Where do I start?

The strategy for starting multimodal real time cross cloud agent development is a incremental step by step approach.

The agents in the demo are based on the original code lab:

Way Back Home - Building an ADK Bi-Directional Streaming Agent | Google Codelabs

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal ADK Agent is built with the visual builder. Next — the entire solution is deployed to Google Cloud Run.

Setup the Basic Environment

At this point you should have a working Python environment and a working Gemini CLI installation. All of the relevant code examples and documentation is available in GitHub. This repo has a wide variety of samples- but this lab will focus on the ‘level_3-lightsail’ setup.

The next step is to clone the GitHub repository to your local environment:

cd ~
git clone https://github.com/xbill9/gemini-cli-azure
cd level_3-aks

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Verify The ADK Installation

To verify the setup, run the ADK CLI locally with Agent1:

xbill@penguin:~/gemini-cli-azure/level_3-aks/backend/app$ adk run biometric_agent
Log setup complete: /tmp/agents_log/agent.20260403_091821.log
To access latest log: tail -F /tmp/agents_log/agent.latest.log
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/cli.py:204: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super(). __init__ ()
Running agent biometric_agent, type exit to exit.



  
  
  Deploying to Azure Kubernates Service


The first step is to refresh the Azure credentials in the current build environment:



xbill@penguin:~/gemini-cli-azure/level_3-appservice$ az login






Run the deploy version on the local system:




xbill@penguin:~/gemini-cli-azure/level_3-aks$ make deploy
./deploy-aks.sh
Ensuring Resource Group exists... 0.0s 0.0s






You can validate the final result by checking the messages:




xbill@penguin:~/gemini-cli-azure/level_3-aks$ make status
Checking AKS status for biometric-scout-aks...
Name State K8sVersion
------------------- --------- ------------
biometric-scout-aks Succeeded 1.33
Kubernetes Services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
biometric-scout-service ClusterIP 10.0.252.23 <none> 80/TCP 68m
Kubernetes Pods:
NAME READY STATUS RESTARTS AGE
biometric-scout-app-5bb549f585-dpwrx 1/1 Running 0 68m
xbill@penguin:~/gemini-cli-azure/level_3-aks$ 






The service will be visible in the Azure console:




  
  
  Running the Web Interface


Start a connection to the deployed app:




  https://biometric-scout-app.orangebeach-e5cba679.canadaeast.azurecontainerapps.io






Then connect tothe app :



Then use the Live model to process audio and video:



Finally — complete the sequence:




  
  
  Summary


The Agent Development Kit was used to enable a multi-modal agent using the Gemini Live Model. This Agent was tested locally with the CLI and then deployed to Azure Kubernates services. This approach validates that cross cloud tools can be used — even with more complex agents.

MCP Development with Python, Gemini CLI, and Amazon AWS ECS Express

xbill — Mon, 06 Apr 2026 17:18:50 +0000

Leveraging Gemini CLI and the underlying Gemini LLM to build Model Context Protocol (MCP) AI applications with Python with a local development environment hosted on AWS ECS Express.

Aren’t There a Billion Python MCP Demos?

Yes there are.

What Is Python?

Python is an interpreted language that allows for rapid development and testing and has deep libraries for working with ML and AI:

Welcome to Python.org

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-stdio-python-aws$ python --version
Python 3.13.12

Amazon ECS Express Configuration

Amazon ECS Express Mode (announced Nov 2025) is a simplified deployment feature for Amazon Elastic Container Service (ECS) designed to rapidly launch containerized applications, APIs, and web services on AWS Fargate. It automates infrastructure setup — including load balancing, networking, scaling, and HTTPS endpoints — allowing developers to deploy from container image to production in a single step.

More details are available here:

Amazon ECS Express Mode

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

gemini

admin@ip-172-31-70-211:~/gemini-cli-aws/mcp-stdio-python-aws$ gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

? for shortcuts 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 shift+tab to accept edits 3 GEMINI.md files | 1 MCP server
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 > Type your message or @path/to/file
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 ~/.../mcp-stdio-python-aws (main*) no sandbox (see /docs) /model Auto (Gemini 3) | 239.8 MB

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

The most common MCP Python deployment path uses the FASTMCP library:

Welcome to FastMCP - FastMCP

Where do I start?

The strategy for starting MCP development is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal Hello World Style Python MCP Server is built with stdio transport. This server is validated with Gemini CLI in the local environment.

This setup validates the connection from Gemini CLI to the local process via MCP. The MCP client (Gemini CLI) and the Python MCP server both run in the same local environment.

Next- the basic MCP server is extended with Gemini CLI to add several new tools in standard Python code.

Setup the Basic Environment

At this point you should have a working Python interpreter and a working Gemini CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemini-cli-aws

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gemini-cli-aws
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gemini-cli-aws
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Hello World with HTTP Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The HTTP transport allows the MCP Client and Server to be in the same environment or distributed over the Internet.

The connection over HTTP will look similar to this:

mcp.run(
        transport="http",
        host="0.0.0.0",
        port=port,
    )

Running the Python Code

First- switch the directory with the Python MCP sample code. Then, run the release version on the local system:

cd ~/gemini-cli-aws/mcp-https-python-ecs-express
make release

If everything is running correctly — you will see the FASTMCP banner:


                 ╭──────────────────────────────────────────────────────────────────────────────╮                 
                 │ │                 
                 │ │                 
                 │ ▄▀▀ ▄▀█ █▀▀ ▀█▀ █▀▄▀█ █▀▀ █▀█ │                 
                 │ █▀ █▀█ ▄▄█ █ █ ▀ █ █▄▄ █▀▀ │                 
                 │ │                 
                 │ │                 
                 │ FastMCP 3.2.0 │                 
                 │ https://gofastmcp.com │                 
                 │ │                 
                 │ 🖥 Server: hello-world-server, 3.2.0 │                 
                 │ 🚀 Deploy free: https://horizon.prefect.io │                 
                 │ │                 
                 ╰──────────────────────────────────────────────────────────────────────────────╯                 

[04/04/26 19:10:50] INFO Starting MCP server 'hello-world-server' with transport 'http' on transport.py:299
                             http://0.0.0.0:8080/mcp                                                              
INFO: Started server process [27502]
INFO: Waiting for application startup.
{"message": "StreamableHTTP session manager started"}
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

At this point — hit Cntrl-C to exit the local HTTP server.

Deploy to ECS Express

Now that the server has been tested locally- start the remote deployment to ECS Express.

Use the deploy target:

 > make deploy
✦ I will execute the full deployment cycle, which includes building the Docker image, pushing it to ECR, and
  deploying to ECS Express Mode.

Then check the status:

 > make status
✦ I will check the status of the ECS Express Mode service.

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ Shell make status [in /home/xbill/gemini-cli-aws/mcp-ecsexpress-python-aws] │
│ │
│ Checking AWS ECS service status for mcp-express-python-aws... │
│ Service: mcp-express-python-aws │
│ Status: ACTIVE │
│ Endpoint: mc-8d69e7dfa87344be98fdb3f7a8fbbbba.ecs.us-east-1.on.aws │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
✦ The ECS Express Mode service mcp-express-python-aws is ACTIVE and available at:
  mc-8d69e7dfa87344be98fdb3f7a8fbbbba.ecs.us-east-1.on.aws

And the endpoint:

  Based on the make status output earlier, the actual endpoint is:
  mc-8d69e7dfa87344be98fdb3f7a8fbbbba.ecs.us-east-1.on.aws

Gemini CLI settings.json

The default Gemini CLI settings.json has an entry for the Python source:

{
  "mcpServers": {
    "aws-ecs-express-python": {
      "httpUrl": "https://mc-8d69e7dfa87344be98fdb3f7a8fbbbba.ecs.us-east-1.on.aws/mcp"
    }
  }
}

Validation with Gemini CLI

Leave the HTTP server running and open a new terminal.

Gemini CLI is restarted and the MCP connection over HTTP to the Python Code is validated. The full Gemini CLI Session will start:


xbill@penguin:~/gemini-cli-aws/mcp-ecsexpress-python-aws$ gemini

 ▝▜▄ Gemini CLI v0.36.0
   ▝▜▄
  ▗▟▀ Signed in with Google: xbill@glitnir.com /auth
 ▝▀ Plan: Gemini Code Assist Standard /upgrade

 > /mcp list
Configured MCP servers:

🟢 aws-ecs-express-python - Ready (1 tool)
  Tools:
  - mcp_aws-ecs-express-python_greet

And you can then connect to the MCP Server over HTTP:

 > greet ECS Express!
✦ I will call the greet tool with "ECS Express!".

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ✓ greet (aws-ecs-express-python MCP Server) {"param":"ECS Express!"} │
│ │
│ ECS Express! │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
✦ The greet tool returned: ECS Express!

Validate Project Setup with Gemini CLI

Gemini CLI was used to summarize the project:

 > summarize the project deployment to Amazon ECS Express
✦ The deployment of the mcp-express-python-aws project to Amazon ECS Express Mode is a streamlined process that
  automates the container lifecycle from build to public endpoint.

  Deployment Workflow (make deploy)
   1. Containerization: Builds a Python 3.13-slim Docker image and pushes it to an Amazon ECR repository
      (mcp-server-repo).
   2. Identity & Access: Automatically provisions necessary IAM roles (ecsTaskExecutionRole and
      ecsInfrastructureRoleForExpressServices) if they do not already exist.
   3. Orchestration: Deploys the container to ECS Express Mode, which simplifies AWS Fargate deployments by
      automatically managing the Application Load Balancer, Security Groups, and Networking.

  Current Infrastructure State
   * Service Name: mcp-express-python-aws
   * Status: ACTIVE
   * Compute: AWS Fargate (Task size: 1 vCPU / 2GB RAM)
   * Scaling: Configured for 1 to 20 tasks, triggered at 60% average CPU usage.
   * Public Access: mc-8d69e7dfa87344be98fdb3f7a8fbbbba.ecs.us-east-1.on.aws (Port 8080)

  The service is currently running and reachable via its assigned ECS Express Gateway endpoint.

Summary

The strategy for using Python for MCP development with Gemini CLI and AWS was validated with a incremental step by step approach.

A minimal HTTP transport MCP Server was started from Python source code and validated with Gemini CLI running as a MCP client in the same local environment.

The HTTP server was then tested remotely from the AWS ECS Express instance.

This approach can be extended to more complex deployments using other MCP transports and Cloud based options.

Observability at Scale: Mastering ADK Callbacks for Cost, Latency, and Auditability [GDE]

Connie Leung — Mon, 06 Apr 2026 16:02:04 +0000

AI orchestrators receive significant attention; however, when deployments become latent and costly, developers often overlook a critical capability: ADK callback hooks. The design patterns and best practices of callback hooks enable developers to refactor logic from agents to callback hooks to add observability, reduce cost and latency, and modify session state dynamically.

This post explores how to create callback hooks at various stages of an ADK agent to demonstrate the following design patterns:

Logging and monitoring performance
Managing dynamic state
Modifying requests and responses
Skipping steps conditionally

Demo Overview

The orchestrator routes the project description to sequentialEvaluationAgent. The sequentialEvaluationAgent consists of project, anti-patterns, decision, recommendation, audit, upload, merger, and email subagents.

In the project, anti-patterns, decision, recommendation, and merger subagents, I implemented callback hooks to demonstrate their capabilities and practicality.

Architecture

Google ADK stands for Agent Development Kit. It is an open source agent framework that enables developers to build and deploy AI agents in a convenient way.

The project, anti-patterns, decision, recommendation, and synthesis agents are LLM agents. These agents require Gemini to reason and generate text responses.

The Audit Trail, Cloud Storage, and Email agents integrate with external APIs or resources that trigger deterministic actions.

Callback Types

ADK provides six types of callback hooks that are called before and after an agent is executed, before and after a model is executed, and before and after a tool is executed.

Callback Type	Description
beforeAgentCallback	Call before the new cycle of an agent
afterAgentCallback	Call after the agent cycle completes
beforeModelCallback	Call before the LLM is called
afterModelCallback	Call after the LLM returns a response
beforeToolCallback	Call before the tool is called
afterToolCallback	Call after the tool is called

When and why I started using ADK callback hooks

When I used ADK web for local testing and debugging, my first priority was correctness. Performance and token usage had lower priorities until the agents were deployed to a QA environment.

When product managers and the QA team recognized apparent slowness and high cost, they would inform me. Next, I examined the agent to identify performance bottlenecks and deterministic steps that could be handled in callback hooks or application code.

The Benefits of Callback

Using callback hooks offers several critical advantages in my subagents:

Type	Benefit
Short Circuit	Call `beforeModelCallback` to validate session data and skip LLM flow when data is invalid
Separation of Concerns	Reset session data in the `beforeAgentCallback` so that tool remains lean and focuses on business logic
Observability	Add logging in the `beforeAgentCallback` and the `afterAgentCallback` to log performance metrics
Dynamic State Management	Create a reusable `afterToolCallback` to increment validation attempts and modify response status to FATAL_ERROR

Prerequisites

Requirements

TypeScript 5.9.3 or later
Node.js 24.13.0 or later
npm 11.8.0 or later
Docker (For running MailHog locally)
Google ADK (For building the custom agent)
Gemini in Vertex AI (to call the model in LLM agents, although not required for the custom agent)

Note: I used Gemini in Vertex AI for authentication due to regional availability. Gemini is currently unavailable in Hong Kong; therefore, I utilized Gemini in Vertex AI instead.

Install npm dependencies

npm i --save-exact @google/adk
npm i --save-dev --save-exact @google/adk-devtools
npm i --save-exact nodemailer
npm i --save-dev --save-exact @types/nodemailer rimraf
npm i --save-exact marked
npm i --save-exact zod

I installed the required dependencies for building ADK agents, converting Markdown strings to HTML, and sending emails to MailHog during local testing.

Pinned dependencies ensure the versioning is the same in development and production environments for enterprise-level applications.

Environment variables

Copy .env.example to .env and fill in the credentials:

GEMINI_MODEL_NAME="gemini-3.1-flash-lite-preview"
GOOGLE_CLOUD_PROJECT="<Google Cloud Project ID>"
GOOGLE_CLOUD_LOCATION="global" 
GOOGLE_GENAI_USE_VERTEXAI=TRUE

# SMTP Settings (MailHog)
SMTP_HOST="localhost"
SMTP_PORT=1025
SMTP_USER=""
SMTP_PASS=""
SMTP_FROM="no-reply@test.local"
ADMIN_EMAIL="admin@test.local"

SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, are required to set up MailHog for local email testing.

SMTP_FROM is a from email address that can be any string in local testing.

ADMIN_EMAIL is an administrator email address receiving emails that the EmailAgent sends. It is an environment variable in my use case because it is the only recipient. If another scenario requires sending emails to customers, the environment variable should be removed.

ADK Callback Patterns in Action

Here is how I implemented these four callback design patterns in practice.

Logging and Monitoring

Callback used: beforeAgentCallback and afterAgentCallback

The agentStartCallback stores the current time in milliseconds in start_time variable in the session state.

import { SingleAgentCallback } from '@google/adk';

export const START_TIME_KEY = 'start_time';

export const agentStartCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  context.state.set(START_TIME_KEY, Date.now());
  return undefined;
};

The agentEndCallback obtains the start time from the session state and calculates the duration. Then, it uses console.log to log the performance metrics. The callback returns undefined so that any subagent always flows to the next one in the sequential workflow.

export const agentEndCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const now = Date.now();
  const startTime = context.state.get<number>(START_TIME_KEY) || now;
  console.log(
    `Performance Metrics for Agent "${context.agentName}": Total Elapsed Time: ${(now - startTime) / 1000} seconds.`,
  );
  return undefined;
};

Next, I call both callback hooks in the subagents to know how long they take to execute. The following example shows how I log the performance metric of a project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterAgentCallback: agentEndCallback,
    ... 
  });

  return projectAgent;
}

Reset Session State

Callback used: beforeAgentCallback

The orchestrator resets variables in the session state before the agent lifecycle begins.

export const AUDIT_TRAIL_KEY = 'auditTrail';
export const RECOMMENDATION_KEY = 'recommendation';
export const CLOUD_STORAGE_KEY = 'cloudStorage';
export const DECISION_KEY = 'decision';
export const PROJECT_KEY = 'project';
export const ANTI_PATTERNS_KEY = 'antiPatterns';
export const MERGED_RESULTS_KEY = 'mergedResults';
export const PROJECT_DESCRIPTION_KEY = 'project_description';
export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

const resetNewEvaluationCallback: SingleAgentCallback = (context) => {
  if (!context || !context.state) {
    return undefined;
  }

  const state = context.state;

  // Clear all previous evaluation data
  state.set(PROJECT_KEY, null);
  state.set(ANTI_PATTERNS_KEY, null);
  state.set(DECISION_KEY, null);
  state.set(RECOMMENDATION_KEY, null);
  state.set(AUDIT_TRAIL_KEY, null);
  state.set(CLOUD_STORAGE_KEY, null);
  state.set(MERGED_RESULTS_KEY, null);
  state.set(VALIDATION_ATTEMPTS_KEY, 0);

  console.log(
    `beforeAgentCallback: Agent ${context.agentName} has reset the session state for a new evaluation cycle.`,
  );

  return undefined;
};

The orchestrator sets the variables to null in the beforeAgentCallback. In the prepareEvaluationTool, the orchestrator only replaces the description with the new project description.

const prepareEvaluationTool = new FunctionTool({
  name: 'prepare_evaluation',
  description: 'Stores the new project description to prepare for a fresh evaluation.',
  parameters: z.object({
    description: z.string().describe('The validated project description from the user.'),
  }),
  execute: ({ description }, context) => {
    if (!context || !context.state) {
      return { status: 'ERROR', message: 'No session state found.' };
    }

    // Set the new description for the ProjectAgent to find
    context.state.set(PROJECT_DESCRIPTION_KEY, description);

    return { status: 'SUCCESS', message: 'Description updated.' };
  },
});

The tool logic is more efficient and the token usage is reduced.

export const rootAgent = new LlmAgent({
  name: 'ProjectEvaluationAgent',
  model: 'gemini-3.1-flash-lite-preview',
  beforeAgentCallback: resetNewEvaluationCallback,
  instruction: `
    ... LLM instruction ....
  `,
  tools: [prepareEvaluationTool],
  subAgents: [sequentialEvaluationAgent],
});

The orchestrator uses the resetNewEvaluationCallback to reset the session variables, and the prepareEvaluationTool to replace the project description. Finally, the sequentialEvaluationAgent starts the agentic workflow to derive a decision and generate the recommendation.

Dynamic State Management

Prerequisite: The subagent uses tool calling to perform action
Callback used: afterToolCallback

The use case is to increment the value of validation_attempts in the session state. After consulting AI, the afterToolCallback stage of project and decision subagents is the perfect location to increment it. Therefore, I define a meta-function that creates a afterToolCallback to increment the validation attempts.

export const VALIDATION_ATTEMPTS_KEY = 'validation_attempts';

export const MAX_ITERATIONS = 3;

import { AfterToolCallback } from '@google/adk';
import { VALIDATION_ATTEMPTS_KEY } from '../output-keys.const.js';
import { MAX_ITERATIONS } from '../validation.const.js';

export function createAfterToolCallback(fatalErrorMessage: string, maxAttempts = MAX_ITERATIONS): AfterToolCallback {
  return ({ tool, context, response }) => {
    if (!tool || !context || !context.state) {
      return undefined;
    }

    const toolName = tool.name;
    const agentName = context.agentName;
    const state = context.state;

    if (!response || typeof response !== 'object' || !('status' in response)) {
      return undefined;
    }

    // [1] Dynamic state management
    const attempts = (state.get<number>(VALIDATION_ATTEMPTS_KEY) || 0) + 1;
    state.set(VALIDATION_ATTEMPTS_KEY, attempts);

    // [2] Response modification
    const status = response.status || 'ERROR';
    if (status === 'ERROR' && attempts >= maxAttempts) {
      context.actions.escalate = true;

      return {
        status: 'FATAL_ERROR',
        message: fatalErrorMessage,
      };
    }
  };
}

In both project and decision subagents, I invoke createAfterToolCallback to create their own afterToolCallback to increment the validation attempts.

const projectAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately. Max validation attempts reached. Return the most accurate data found so far or empty strings if none.`,
);

const decisionAfterToolCallback = createAfterToolCallback(
  `STOP processing immediately and output the final JSON schema with verdict: "None".`,
);

Next, I call the projectAfterToolCallback in the afterToolCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      ... LLM instruction ....
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    ... 
  });

  return projectAgent;
}

Next, I call the decisionAfterToolCallback in the afterToolCallback stage of the decision subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    ...
  });

  return decisionTreeAgent;
}

Request/Response Modification

Prerequisite: The subagent uses tool calling to perform action
Callback used: AfterToolCallback

The same AfterToolCallback also modifies the response when the number of validation attempts exceeds the maximum iterations.

When the status is ERROR and the value of validation_attempts is at least maximum_iterations, the context.actions.escalate flag is set to true to break out of the loop. Moreover, the status is changed to FATAL_ERROR and the custom fatal message is returned.

Conditional Skipping of Steps

This is an important design pattern for avoiding unnecessary LLM executions.

Callback used: beforeModelCallback.

In my subagents, I run this callback to validate the session data. When a specific condition is met, the callback returns content immediately to skip the subsequent LLM flow.

Example 1

If the project agent is able to break down a project description into task, problem, constraint, and goal, the agent will return the breakdown immediately. Otherwise, the agent prompts Gemini to use reasoning to perform the breakdown.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted) {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(project),
          },
        ],
      },
    };
  }

  return undefined;
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createProjectAgent(model: string) {
  const projectAgent = new LlmAgent({
    name: 'ProjectAgent',
    model,
    description:
      'Analyzes the user-provided project description to extract and structure its core components, including the primary task, underlying problem, ultimate goal, and architectural constraints.',
    beforeAgentCallback: agentStartCallback,
    beforeModelCallback,
    instruction: (context) => {
      const { projectDescription } = getEvaluationContext(context);
      if (!projectDescription) {
        return '';
      }

      return generateProjectBreakdownPrompt(projectDescription);
    },
    afterToolCallback: projectAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateProjectTool],
    outputSchema: projectSchema,
    outputKey: PROJECT_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return projectAgent;
}

Example 2

The decision agent verifies the verdict property in the beforeModelCallback. If the verdict is not None, the callback returns the valid decision immediately. The verdict is None and the callback examines the project breakdown and the anti-patterns. When project breakdown and anti-patterns are provided, the callback returns undefined to trigger the LLM flow. Otherwise, the decision agent does not have valid inputs to derive the verdict. The callback returns None in this edge case.

import { SingleBeforeModelCallback } from '@google/adk';

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { decision } = getEvaluationContext(context);

  if (decision && decision.verdict !== 'None') {
    return {
      content: {
        role: 'model',
        parts: [
          {
            text: JSON.stringify(decision),
          },
        ],
      },
    };
  }

  const { project, antiPatterns } = getEvaluationContext(context);
  const { isCompleted } = isProjectDetailsFilled(project);

  if (isCompleted && antiPatterns) {
    return undefined;
  }

  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            verdict: 'None',
            nodes: [],
          }),
        },
      ],
    },
  };
};

Then, beforeModelCallback is hooked to the beforeModelCallback stage of the project subagent.

export function createDecisionTreeAgent(model: string) {
  const decisionTreeAgent = new LlmAgent({
    name: 'DecisionTreeAgent',
    model,
    beforeAgentCallback: [resetAttemptsCallback, agentStartCallback],
    beforeModelCallback,
    instruction: (context) => {
      ... instruction  of the LLM flow ...
    },
    afterToolCallback: decisionAfterToolCallback,
    afterAgentCallback: agentEndCallback,
    tools: [validateDecisionTool],
    outputSchema: decisionSchema,
    outputKey: DECISION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return decisionTreeAgent;
}

Example 3

The recommendation agent uses the beforeModelCallback to examine the project breakdown, anti-patterns and verdict. There are two scenarios that need LLM to generate the recommendation. The first scenario is valid project breakdown, anti-patterns, and non-None verdict. The second scenario is incomplete project breakdown and None verdict. The LLM is instructed to describe the missing field in the project breakdown and how important the missing field is to obtain a valid decision. For other cases, the callback returns a static recommendation immediately and skip the subsequent LLM flow.

function constructRecommendation(recommendation: string) {
  return {
    content: {
      role: 'model',
      parts: [
        {
          text: JSON.stringify({
            text: recommendation,
          }),
        },
      ],
    },
  };
}

const beforeModelCallback: SingleBeforeModelCallback = ({ context }) => {
  const { project, antiPatterns, decision } = getEvaluationContext(context);

  const { isCompleted } = isProjectDetailsFilled(project);
  const isDecisionNone = decision && decision.verdict === 'None';

  if ((isCompleted && antiPatterns && decision && decision.verdict !== 'None') || (!isCompleted && isDecisionNone)) {
    return undefined;
  } else if (isCompleted && isDecisionNone) {
    return constructRecommendation(
      '## Recommendation: Manual Review Required\n\n**Status:** Abnormal Case Detected\n\nThe provided project is complete and valid, but the decision tree could not reach a conclusive verdict (Result: `None`).\n\n**Possible Reasons:**\n- The requirements fall outside of known architectural patterns.\n- There are conflicting constraints and goals that cannot be resolved automatically.\n\n**Next Steps:**\n- Review and refine the constraints or goals.\n- Escalate for manual architectural review.',
    );
  }

  return constructRecommendation(
    '## Recommendation: Data Required\n\n**Status:** Abnormal Case Detected\n\nNo decision is reached.',
  );
};

Similar to the previous project and decision agents, the beforeModelCallback function is hooked to the beforeModelCallback stage of the recommendation agent.

export function createRecommendationAgent(model: string) {
  const recommendationAgent = new LlmAgent({
    name: 'RecommendationAgent',
    model,
    beforeModelCallback,
    beforeAgentCallback: agentStartCallback,
    instruction: (context) => {
      const { project, antiPatterns, decision } = getEvaluationContext(context);
      const { isCompleted, missingFields } = isProjectDetailsFilled(project);

      if (project) {
        if (!isCompleted && decision && decision.verdict === 'None') {
          console.log('RecommendationAgent -> generateFailedDecisionPrompt');
          return generateFailedDecisionPrompt(project, missingFields);
        } else if (isCompleted && antiPatterns && decision && decision.verdict !== 'None') {
          console.log('RecommendationAgent -> generateRecommendationPrompt');
          return generateRecommendationPrompt(project, antiPatterns, decision);
        }
      }
      return 'Skipping LLM due to missing data.';
    },
    afterAgentCallback: agentEndCallback,
    outputSchema: recommendationSchema,
    outputKey: RECOMMENDATION_KEY,
    disallowTransferToParent: true,
    disallowTransferToPeers: true,
  });

  return recommendationAgent;
}

These are the callback design patterns that the agent adopts. In the next section, I describe how to launch the agent to observe the logging messages in the terminal.

Environment Setup

I pulled the latest version of the MailHog Docker image from Docker Hub and started it locally to receive test emails and display them in the Web UI. The docker-compose.yml file contains the setup configuration.

services:
  mailhog:
    image: mailhog/mailhog
    container_name: mailhog
    ports:
      - '1025:1025' # SMTP port
      - '8025:8025' # HTTP (Web UI) port
    restart: always
    networks:
      - decision-tree-agent-network

networks:
  decision-tree-agent-network:

The SMTP server listens on port 1025, and the Web UI is accessible at http://localhost:8025.

docker compose up -d

Start MailHog in Docker.

Testing

Add scripts to package.json to build and start the ADK web interface.

  "scripts": {
    "prebuild": "rimraf dist",
    "build": "npx tsc --project tsconfig.json",
    "web": "npm run build && npx @google/adk-devtools web --host 127.0.0.1 dist/agent.js"
  },

Open a terminal and type npm run web to start the API server.
Open a new browser tab and type http://localhost:8000.
Paste the following text into the message box:

One of my favorite tech influencers just tweeted about a 'breakthrough in solid-state batteries.' Find which public company they might be referring to, check that company’s recent patent filings to see if it’s true, and then check their stock price to see if the market has already 'priced it in'.

Ensure the root agent executes and halts when the email agent terminates. The orchestrator and the project agent trigger the callback hooks to reset the session state, log performance metrics, validate session data, increment the validation attempts, and modify the response.

Similarly, both anti-patterns and decisions agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. They also used the beforeModelCallback to validate session data before calling the LLM to generate a response. Moreover, the decision agent incremented the validation attempts and modify the status to FATAL_ERROR when the validation attempts exceeded or equal to maximum iterations in the afterToolCallback.

Similarly, both recommendation and merger agents used the beforeAgentCallback and afterAgentCallback to log performance metrics. The recommendation agent also used the beforeModelCallback to validate session data before calling the LLM to generate a recommendation.

Conclusion

ADK supports lifecycle callback hooks that are applicable to different use cases. In this post, I describe logging performance metrics in the beforeAgentCallback and afterAgentCallback. I refactored the reset session state logic in the orchestrator's beforeAgentCallback to make the tool lean and cost efficient. The afterToolCallback escalated to a higher agent and modified the response status to FATAL_ERROR when the loop count in the validation-retry loop exceeded the maximum iterations. When skipping LLM calls conditionally, the beforeModelCallback returned custom content immediately. Then, the agent did not add unnecessary time to the agent flow and did not consume tokens.

The takeaway: follow the design patterns and best practices of callback at various stages of an agent to log performance metrics, reduce cost and latency, and modify response.

Resources

Deploying ADK Agents on Azure Fabric

xbill — Mon, 06 Apr 2026 13:07:27 +0000

Leveraging the Google Agent Development Kit (ADK) and the underlying Gemini LLM to build Agents in the Python programming language deployed on the Azure Cloud platform.

What is Gemini CLI?

The Gemini CLI is an open-source, terminal-based AI agent from Google that allows developers to interact directly with Gemini models, such as Gemini 2.5 Pro, for coding, content creation, and workflow automation. It supports file operations, shell commands, and connects to external tools via the Model Context Protocol (MCP).

The full details on Gemini CLI are available here:

Build, debug & deploy with AI

Azure Fabric

Microsoft Fabric is a comprehensive, AI-powered cloud platform that unifies data engineering, data warehousing, data science, and business intelligence (using Power BI) into a single SaaS solution, built on the OneLake storage system. It simplifies data management by reducing reliance on disparate, siloed tools.

More details are here:

What is Microsoft Fabric - Microsoft Fabric

Why would I want Gemini CLI with Azure? Isn’t that a Google Thing?

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

python --version
Python 3.13.12

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

azureuser@azure-new:~/gemini-cli-azure$ gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

To verify the Node version:

node --version
v25.9.0
nvm --version
0.40.3
npm --version
11.12.1

Agent Development Kit

The ADK can be installed from here:

Agent Development Kit (ADK)

Where do I start?

The strategy for starting Agent development is a incremental step by step approach.

The agents in the demo are based on the original code lab:

Create and deploy low code ADK (Agent Deployment Kit) agents using ADK Visual Builder | Google Codelabs

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal ADK Agent is built with the visual builder. Next — the entire solution is deployed to Google Cloud Run.

Setup the Basic Environment

At this point you should have a working Python environment and a working Gemini CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemini-cli-azure

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Azure Login

All of the deployment is done with the az CLI tool- but it also needs an active login on the Azure environment. To check the version:

xbill@penguin:~/gemini-cli-azure/adkui-aci$ az --version
azure-cli 2.84.0
core 2.84.0
telemetry 1.1.0

Dependencies:
msal 1.35.0b1
azure-mgmt-resource 24.0.0
Python location '/opt/az/bin/python3'
Config directory '/home/xbill/.azure'
Extensions directory '/home/xbill/.azure/cliextensions'
Python (Linux) 3.13.11 (main, Feb 25 2026, 02:29:30) [GCC 12.2.0]
Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

To login to Azure:

az login
You have logged into Microsoft Azure!
You can close this window, or we will redirect you to the Azure CLI documentation in 1 minute.

Verify The ADK Installation

To verify the setup, run the ADK CLI with Agent3:

xbill@penguin:~/gemini-cli-azure/adkui-fabric$ adk run Agent3
Log setup complete: /tmp/agents_log/agent.20260402_182048.log
To access latest log: tail -F /tmp/agents_log/agent.latest.log
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/utils/agent_loader.py:277: UserWarning: [EXPERIMENTAL] _load_from_yaml_config: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  if root_agent := self._load_from_yaml_config(actual_agent_name, agents_dir):
/home/xbill/.local/lib/python3.13/site-packages/google/adk/features/_feature_decorator.py:81: UserWarning: [EXPERIMENTAL] feature FeatureName.AGENT_CONFIG is enabled.
  check_feature_enabled()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/cli.py:204: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super(). __init__ ()
Running agent studio_director, type exit to exit.



  
  
  Running the ADK Web Interface


First- start the ADK interface:



xbill@penguin:~/gemini-cli-azure/adkui-fabric$ adk web --host 0.0.0.0
2026-04-02 18:22:15,410 - INFO - service_factory.py:266 - Using in-memory memory service
2026-04-02 18:22:15,410 - INFO - local_storage.py:84 - Using per-agent session storage rooted at /home/xbill/gemini-cli-azure/adkui-fabric
2026-04-02 18:22:15,410 - INFO - local_storage.py:110 - Using file artifact service at /home/xbill/gemini-cli-azure/adkui-fabric/.adk/artifacts
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/fast_api.py:192: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super(). __init__ ()
INFO: Started server process [27984]
INFO: Waiting for application startup.

+-----------------------------------------------------------------------------+
| ADK Web Server started |
| |
| For local testing, access at [http://0.0.0.0:8000](http://0.0.0.0:8000). |
+-----------------------------------------------------------------------------+

INFO: Application startup complete.
INFO: Uvicorn running on [http://0.0.0.0:8000](http://0.0.0.0:8000) (Press CTRL+C to quit)







  
  
  Connecting to the ADK


The ADK web interface is started with the place holder 0.0.0.0 address — which tells the service to listen on all interfaces. Depending on your local environment setup- you may need to use the actual local interface. There are 2 potential options:




http://0.0.0.0:8000

http://127.0.0.1:8000






This will bring up the ADK UI:




  
  
  Run a Sub Agent


Connect to the ADK and select “Agent3”:



The first interaction just verifies the connection to the agent:




what do you do?

I manage a linear pipeline of four ADK Sequential Agents to transform a user's seed idea into a fully rendered, responsive HTML5 comic book. If your request starts with "Create me a comic of...", I'll delegate the task to my sub-agent to begin the comic creation pipeline. Otherwise, I'll answer your question directly.






Give the Agent this prompt:




Create a comic of Bob and Clippy in a Battle Royal with Tux.






The sub agent will then generate the comic book images:




  
  
  Visual Build an Agent


To use the ADK visual builder- select the pencil Icon next to Agent 2. You can drill down into the Agent design:




  
  
  What did I just do?


The initial steps validate that the Python code and the ADK environment is functional and that the Agent code will run correctly. The next step is to deploy this environment to the Azure Fabric.


  
  
  Deploying to Azure Fabric


To start the deployment:




xbill@penguin:~/gemini-cli-azure/adkui-fabric$ make deploy
Ensuring Resource Group adk-rg-fabric exists...






You can validate the final result by checking the messages:




✦ The deployment to Azure Container Apps (Fabric integration) has completed successfully.

  Deployment Details:
   * Resource Group: adk-rg-fabric
   * ACR: adkacrpenguinv2
   * Container Apps Environment: adk-env-fabric
   * Container App: adk-comic-fabric
   * Endpoint: https://adk-comic-fabric.nicebush-151ee659.westus2.azurecontainerapps.io
   * Status: Running (Provisioning State: Succeeded)

  All environment variables (including API keys and project IDs) have been correctly applied to the Container App.
  You can now use this endpoint in your Microsoft Fabric Workload Manifest.






You can then get the endpoint:




xbill@penguin:~/gemini-cli-azure/adkui-fabric$ make endpoint
adk-comic-fabric.nicebush-151ee659.westus2.azurecontainerapps.io
xbill@penguin:~/gemini-cli-azure/adkui-fabric$ 







  
  
  Running the ADK Web Interface


Start a connection to the Azure Deployed ADK:



This will bring up the ADK UI.

Select the sub-agent “Agent3” and ask the Agent to create a comic about Bit:




Bit the Raccoon is a popular, unofficial community mascot for Microsoft Azure,
specifically representing the Cloud Advocate team and open-source initiatives. 
Created by Ashley McNamara, Bit is often depicted in tech-related, helpful 
scenarios to make complex cloud concepts more accessible and approachable 
to developers. X X +2 Role: Known as the "Developer Advocate Mascot," 
Bit is often used in presentations and content to represent the "open" 
aspect of Azure. Context: While Azure has an official anime-style mascot 
in Japan (Claudia Madobe), Bit is beloved within the broader developer 
community as a fun, tech-savvy character.  

Appearance: Bit is usually styled as an adorable, helpful raccoon, 
sometimes accompanied by other community mascots like "Brady the Ninja Sloth."
GitHub GitHub Bit is a favorite for stickers and frequently appears 
on social media to highlight new Azure features.






The Agent will get to work:




  
  
  Run the Viewer Agent


Connect to the ADK and select “Agent4”:




I've exported the comic as ADK artifacts for you!

You can now view the comic directly within the ADK environment. Please check the 'Artifacts' pane or tab in your ADK Visual Builder / Studio UI.

You'll find the following:

view_full_comic.html: This is a complete, self-contained HTML file with embedded images for the full comic experience.
comic_preview.md: A markdown summary of the comic.
Individual panel images (e.g., panel_panel_1.png).
If you are working in a local terminal, you can still view the original local files by opening output/comic.html in your browser or by running ./comic.sh. However, the ADK artifacts are the best way to view it while hosted!






The Agent will generate the artifacts:



The Comic can then be viewed:




  
  
  Summary


The Agent Development Kit was used to visually define a basic agent and added a tool that calls the ImageGen Gemini Model. This Agent was tested locally with the CLI and then with the ADK web tool. Then, the Agent was packaged and deployed to Azure Fabric. Several sample ADK agents were run directly from the deployment in Azure. This approach validates that cross cloud tools can be used — even with more complex agents.

Zero to V1

Jessica Rudd — Mon, 06 Apr 2026 12:03:17 +0000

We’ve all been there: a great app idea strikes while you’re mid-commute, but the gap between “cool idea” and “working prototype” often feels like an insurmountable wall of boilerplate and UI design.

In this “Data Byte,” we’re breaking down a high-speed workflow that turns scattered notes into a functional V1 in minutes. We’re building Folio —a privacy-first social app for digital postcards—using the new agentic trifecta: AI Studio , Stitch , and Antigravity.

Step 1: The Blueprint with Gemini 3 Flash

Every project starts with a spark. I use Apple Notes and Siri to capture my scattered thoughts, but the real magic begins in Google AI Studio.

I fed my raw notes into Gemini 3 Flash. Because Flash is optimized for speed and high-volume reasoning, it’s the perfect engine to distill messy ideas into structured prompts. I asked it for two specific outputs:

A Stitch design prompt to define the visual “vibe.”
An Antigravity implementation prompt to handle the technical build.

Leveraging Gemini 3 Flash in AI Studio to generate development prompts.

Step 2: “Vibe Designing” in Stitch (Gemini 3.1 Pro)

Next, I headed to Stitch to bring Folio’s “Premium Editorial” aesthetic to life.

Using Gemini 3.1 Pro within Stitch, I pasted the first prompt. Within seconds, it generated a gorgeous creator dashboard and a mobile-optimized gallery. The “vibe design” approach is a game-changer—you can iterate in plain English. I simply asked for a messaging feature, and 3.1 Pro updated the entire design system to include a refined chat interface.

Paste your prompt, make changes, add designs

Adding comprehensive messaging and chat interfaces in Stitch..

Step 3: Zero-Boilerplate Build in Antigravity (Gemini 3.1 Pro)

With the design locked, it was time to move to Antigravity , Google’s agentic IDE.

I opened the Agent Manager and pasted the second prompt.

Create a new workspace in the Antigravity agent manager and paste prompt 2 from AI Studio.

Using Gemini 3.1 Pro , the agent analyzed the design from Stitch and drafted a full implementation plan. It automatically handled:

Scaffolding a Next.js project.
Integrating Firebase Auth for secure logins.
Setting up Firestore for postcard storage.

Antigravity’s Agent Manager mapping the design to a Next.js architecture.

Within minutes, the agent finished the heavy lifting, and the app was running on localhost.

Success! The functional V1 of Folio is live on localhost.

The Takeaway

By offloading prompt engineering to Gemini 3 Flash and the complex execution to Gemini 3.1 Pro , the path from idea to execution is now a straight line. By the time I finished my morning coffee, Folio was a working application ready for its first users.

What’s next? I’ll be using the Firebase Console and Google Cloud to move this from localhost to the world.

Stay tuned for the next byte!

Building a Production-Ready Serverless App on Google Cloud (Part 2: The Data Contract)

Patricio Navarro — Sun, 05 Apr 2026 16:46:41 +0000

This is Part 2 of a 3-part series on building production-ready, data-intensive applications on Google Cloud. If you haven't read it yet, check out Part 1: Architecture to understand the foundational serverless components we are connecting today.

The Danger of Decoupling

In Part 1 of this series, we praised the decoupled architecture. By splitting our compute (Cloud Run) from our analytics (BigQuery) using a buffer (Pub/Sub), we created a system that scales infinitely and costs nothing when idle.

But decoupling introduces a massive architectural danger: The Data Swamp.

If your web application can throw any random JSON payload into a Pub/Sub topic, and that topic blindly dumps it into a data warehouse, your analytics team will spend 80% of their time cleaning malformed strings and fixing broken dashboards.

To prevent this, we must establish a strict Data Contract at the very edge of our ingestion layer.

The Bouncer: Enforcing the Pub/Sub Schema

A professional data pipeline does not rely on the application code to "hopefully" send the right data types. It enforces rules at the infrastructure level.

For the Dog Finder app, we attached a strict Apache Avro schema to our Pub/Sub topic. This acts as the "bouncer" for our data warehouse. If Cloud Run attempts to publish a sighting with a missing field or the wrong data type, Pub/Sub rejects it immediately.

By inspecting pubsub_schema.json, you can see standard Data Engineering practices enforced natively:

Precision Typing: We explicitly defined latitude and longitude as double precision. This prevents the backend from accidentally sending coordinates as strings, which would break spatial queries later.
Consistent Naming: We enforced snake_case for all fields, such as sighting_date and image_url.

The Vault: Designing the BigQuery Schema

BigQuery is where our data lives permanently. The schema here needs to mirror our Pub/Sub contract, but also provide the metadata necessary for reliable analytics.

If you look at bigquery_schema.json, we didn't just copy the business fields. We intentionally included metadata fields like message_id and publish_time. Because Pub/Sub guarantees "at-least-once" delivery, duplicate messages can occasionally occur. Capturing the message_id is essential for the analytics team to efficiently deduplicate records.

More importantly, we didn't just create a basic table. In our setup_resources.sh script, we enforced a partitioning strategy directly at creation:

bq mk --table \
    --time_partitioning_field sighting_date \
    --time_partitioning_type DAY \
    "${GOOGLE_CLOUD_PROJECT}:${BIGQUERY_DATASET}.${BIGQUERY_TABLE}" \
    "$PROJECT_ROOT/schemas/bigquery_schema.json"

By partitioning the table by sighting_date, we ensure that when a Looker Studio dashboard queries for "lost dogs this week" or an analyst performs research, BigQuery scans only the relevant daily partitions. This single command is the difference between a query that costs $1 and a query that costs $1,000 as your dataset grows.

The Serverless Bridge: Zero-Code Ingestion

Now for the architectural magic trick. We have a secure Pub/Sub topic and a partitioned BigQuery table. How do we move data between them?

Traditionally, developers write a Cloud Function or spin up a Dataflow job to consume from Pub/Sub, transform the payload, and insert it into BigQuery. That means writing code, managing deployments, and paying for intermediate compute.

Instead, we used a native BigQuery Subscription. This is a powerful serverless pattern that requires zero code. Here is the exact command from our setup script:

gcloud pubsub subscriptions create "$SUBSCRIPTION_ID" \
    --topic="$TOPIC_ID" \
    --bigquery-table="${GOOGLE_CLOUD_PROJECT}:${BIGQUERY_DATASET}.${BIGQUERY_TABLE}" \
    --use-topic-schema \
    --write-metadata \
    --project="$GOOGLE_CLOUD_PROJECT"

Notice the two critical flags:

--use-topic-schema: This tells the subscription to natively map the fields from our Avro schema directly to the BigQuery columns.
--write-metadata: This automatically populates those message_id and publish_time fields we added to our BigQuery schema for auditing.

Designing for Failure: The Dead Letter Topic (DLT)

But an architect must always design for failure. What happens if a schema evolution causes a mismatch, or BigQuery temporarily rejects an insert? By default, Pub/Sub will continually retry the delivery, but once the retention period or retry limit is exhausted, that message is dropped forever. Data loss in a production pipeline is unacceptable.

To prevent this, we must configure a Dead Letter Topic (DLT) alongside our subscription. This is a core defensive engineering practice.

By adding the --dead-letter-topic and --max-delivery-attempts flags to your subscription configuration, you create a safety net. If a message fails to write to BigQuery after, say, 5 attempts (perhaps due to an unforeseen schema mismatch), Pub/Sub automatically routes that specific message to the DLT and continues processing the rest of the queue.

Instead of losing the sighting, the malformed data is safely quarantined. You can set up an alert on the DLT, inspect the failing payload, patch your schema or application code, and then easily replay the dead-lettered message back into the main pipeline. Zero dropped records, zero panic.

With this configuration, GCP handles all the plumbing. As soon as the Cloud Run backend publishes a validated event to Pub/Sub, the infrastructure automatically streams it into BigQuery - securely and resiliently - with absolutely zero intermediate compute costs.

Conclusion

By enforcing a Data Contract via an Avro schema and utilizing native BigQuery subscriptions, we eliminated the "glue code" that normally plagues data pipelines. Our analytics team gets perfectly structured, partitioned data, and our application developers don't have to manage a single ingestion worker.

Deploying ADK Agents on Azure Kubernates Service (AKS)

xbill — Sat, 04 Apr 2026 22:47:18 +0000

Leveraging the Google Agent Development Kit (ADK) and the underlying Gemini LLM to build Agents in the Python programming language deployed on the Azure Cloud platform.

What is Gemini CLI?

The full details on Gemini CLI are available here:

Build, debug & deploy with AI

Azure Kubernates Service

More details are available here:

https://azure.microsoft.com/en-us/products/kubernetes-service

Isn’t that Overkill? A whole Cluster Just for some Agents?!

Why would I want Gemini CLI with Azure? Isn’t that a Google Thing?

Python Version Management

One of the downsides of the wide deployment of Python has been managing the language versions across platforms and maintaining a supported version.

The pyenv tool enables deploying consistent versions of Python:

GitHub - pyenv/pyenv: Simple Python version management

As of writing — the mainstream python version is 3.13. To validate your current Python:

xbill@penguin:~/gemini-cli-azure/adkui-aks$ python --version
Python 3.13.12

Gemini CLI

If not pre-installed you can download the Gemini CLI to interact with the source files and provide real-time assistance:

npm install -g @google/gemini-cli

Testing the Gemini CLI Environment

Once you have all the tools and the correct Node.js version in place- you can test the startup of Gemini CLI. You will need to authenticate with a Key or your Google Account:

gemini

▝▜▄ Gemini CLI v0.33.1
    ▝▜▄
   ▗▟▀ Logged in with Google /auth
  ▝▀ Gemini Code Assist Standard /upgrade

Node Version Management

Gemini CLI needs a consistent, up to date version of Node. The nvm command can be used to get a standard Node environment:

GitHub - nvm-sh/nvm: Node Version Manager - POSIX-compliant bash script to manage multiple active node.js versions

To verify the Node version:

xbill@penguin:~/gemini-cli-azure$ nvm --version
0.40.3
xbill@penguin:~/gemini-cli-azure$ node --version
v25.9.0
xbill@penguin:~/gemini-cli-azure$ npm --version
11.12.1

Agent Development Kit

The ADK can be installed from here:

Agent Development Kit (ADK)

Where do I start?

The strategy for starting Agent development is a incremental step by step approach.

The agents in the demo are based on the original code lab:

Create and deploy low code ADK (Agent Deployment Kit) agents using ADK Visual Builder | Google Codelabs

First, the basic development environment is setup with the required system variables, and a working Gemini CLI configuration.

Then, a minimal ADK Agent is built with the visual builder. Next — the entire solution is deployed to Google Cloud Run.

Setup the Basic Environment

At this point you should have a working Python environment and a working Gemini CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemini-cli-azure

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Azure Login

All of the deployment is done with the az CLI tool- but it also needs an active login on the Azure environment. To check the version:

xbill@penguin:~/gemini-cli-azure$ az --version
azure-cli 2.84.0

core 2.84.0
telemetry 1.1.0

Dependencies:
msal 1.35.0b1
azure-mgmt-resource 24.0.0

Python location '/opt/az/bin/python3'
Config directory '/home/xbill/.azure'
Extensions directory '/home/xbill/.azure/cliextensions'

Python (Linux) 3.13.11 (main, Feb 25 2026, 02:29:30) [GCC 12.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Unable to check if your CLI is up-to-date. Check your internet connection.

To login to Azure:

az login

Then you will see a message after successful authentication:

You have logged into Microsoft Azure!
You can close this window, or we will redirect you to the Azure CLI documentation in 1 minute.

Verify The ADK Installation

To verify the setup, run the ADK CLI with Agent3:

xbill@penguin:~/gemini-cli-azure/adkui-aks$ adk run Agent3
Log setup complete: /tmp/agents_log/agent.20260402_173311.log
To access latest log: tail -F /tmp/agents_log/agent.latest.log
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/utils/agent_loader.py:277: UserWarning: [EXPERIMENTAL] _load_from_yaml_config: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  if root_agent := self._load_from_yaml_config(actual_agent_name, agents_dir):
/home/xbill/.local/lib/python3.13/site-packages/google/adk/features/_feature_decorator.py:81: UserWarning: [EXPERIMENTAL] feature FeatureName.AGENT_CONFIG is enabled.
  check_feature_enabled()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/cli.py:204: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super(). __init__ ()
Running agent studio_director, type exit to exit.



  
  
  Running the ADK Web Interface


First- start the ADK interface:



xbill@penguin:~/gemini-cli-azure/adkui-aks$ adk web --host 0.0.0.0
2026-04-02 17:33:54,016 - INFO - service_factory.py:266 - Using in-memory memory service
2026-04-02 17:33:54,016 - INFO - local_storage.py:84 - Using per-agent session storage rooted at /home/xbill/gemini-cli-azure/adkui-aks
2026-04-02 17:33:54,016 - INFO - local_storage.py:110 - Using file artifact service at /home/xbill/gemini-cli-azure/adkui-aks/.adk/artifacts
/home/xbill/.local/lib/python3.13/site-packages/google/adk/cli/fast_api.py:192: UserWarning: [EXPERIMENTAL] InMemoryCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  credential_service = InMemoryCredentialService()
/home/xbill/.local/lib/python3.13/site-packages/google/adk/auth/credential_service/in_memory_credential_service.py:33: UserWarning: [EXPERIMENTAL] BaseCredentialService: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
  super(). __init__ ()
INFO: Started server process [23049]
INFO: Waiting for application startup.

+-----------------------------------------------------------------------------+
| ADK Web Server started |
| |
| For local testing, access at http://0.0.0.0:8000. |
+-----------------------------------------------------------------------------+

INFO: Application startup complete.







  
  
  Connecting to the ADK


The ADK web interface is started with the place holder 0.0.0.0 address — which tells the service to listen on all interfaces. Depending on your local environment setup- you may need to use the actual local interface. There are 2 potential options:




http://0.0.0.0:8000

http://127.0.0.1:8000






This will bring up the ADK UI:




  
  
  Run a Sub Agent


Connect to the ADK and select “Agent3”:



The first interaction just verifies the connection to the agent:




what do you do?

I manage a linear pipeline of four ADK Sequential Agents to transform a user's seed idea into a fully rendered, responsive HTML5 comic book. If your request starts with "Create me a comic of...", I'll delegate the task to my sub-agent to begin the comic creation pipeline. Otherwise, I'll answer your question directly.






Give the Agent this prompt:




Create a comic of Bob and Clippy in a Battle Royal with Tux.






The sub agent will then generate the comic book images:




  
  
  Visual Build an Agent


To use the ADK visual builder- select the pencil Icon next to Agent 2. You can drill down into the Agent design:




  
  
  What did I just do?


The initial steps validate that the Python code and the ADK environment is functional and that the Agent code will run correctly. The next step is to deploy this environment to the Azure Kubernates Service.


  
  
  Deploying to Azure Kubernates Service


To start the deployment:




✦ I will now run the deploy-aks.sh script 






You can validate the final result by checking the messages:




✦ The ADK application has been successfully deployed to AKS.

  Deployment Summary:
   - Deployment Status: adk-app is Ready (1/1).
   - Pod Status: adk-app-dcf865487-nbfrj is Running.
   - Service Endpoint: http://4.155.123.101 (http://4.155.123.101)

  The application is now accessible at the public IP listed above. Let me know if you need anything else!






You can then get the endpoint:




✦ The public endpoint for the ADK application is:
  4.155.123.101






The service will be visible in the Azure console:




  
  
  Running the ADK Web Interface


Start a connection to the Azure Deployed ADK.

This will bring up the ADK UI:



Select the sub-agent “Agent3” and ask the Agent to create a comic about Bit:




Bit the Raccoon is a popular, unofficial community mascot for Microsoft Azure,
specifically representing the Cloud Advocate team and open-source initiatives. 
Created by Ashley McNamara, Bit is often depicted in tech-related, helpful 
scenarios to make complex cloud concepts more accessible and approachable 
to developers. X X +2 Role: Known as the "Developer Advocate Mascot," 
Bit is often used in presentations and content to represent the "open" 
aspect of Azure. Context: While Azure has an official anime-style mascot 
in Japan (Claudia Madobe), Bit is beloved within the broader developer 
community as a fun, tech-savvy character.

Appearance: Bit is usually styled as an adorable, helpful raccoon, 
sometimes accompanied by other community mascots like "Brady the Ninja Sloth."
GitHub GitHub Bit is a favorite for stickers and frequently appears 
on social media to highlight new Azure features.

Create a story about bit






The Agent will get to work:




  
  
  Run the Viewer Agent


Connect to the ADK and select “Agent4”:




I've exported the comic as ADK artifacts for you!

You can now view the comic directly within the ADK environment. Please check the 'Artifacts' pane or tab in your ADK Visual Builder / Studio UI.

You'll find the following:

view_full_comic.html: This is a complete, self-contained HTML file with embedded images for the full comic experience.
comic_preview.md: A markdown summary of the comic.
Individual panel images (e.g., panel_panel_1.png).
If you are working in a local terminal, you can still view the original local files by opening output/comic.html in your browser or by running ./comic.sh. However, the ADK artifacts are the best way to view it while hosted!






The Agent will generate the artifacts:



The Comic can then be viewed:




  
  
  Summary


The Agent Development Kit was used to visually define a basic agent and added a tool that calls the Image Gemini Model. This Agent was tested locally with the CLI and then with the ADK web tool. Then, the Agent was packaged and deployed to the Azure Kubernates Service. Several sample ADK agents were run directly from the deployment in Azure. This approach validates that cross cloud tools can be used — even with more complex agents.