Forem: Yuya Mukai

Building ExpertLens: Real-time AI Coaching for Software You Control Directly

Yuya Mukai — Mon, 16 Mar 2026 07:32:06 +0000

Disclaimer: This post was created for the purposes of entering the Gemini Live Agent Challenge.

ExpertLens is a real-time voice and vision coaching agent for any software where the human must be the operator. Share your screen or point your camera, speak naturally, and get expert guidance for Blender, Affinity Photo, Unreal Engine, a mobile game, or any app an AI cannot run on your behalf.

This post covers the core insight behind the project, how it's built on Gemini Live API, and four specific technical challenges that required non-obvious solutions.

1. The Human-Control Gap

When deciding what to build for this hackathon, I mapped the landscape of AI assistance by tool type:

Browser-based apps (Figma, Canva, Google Docs): Playwright and Selenium can automate these. LLM agents can literally control them by reading the DOM and clicking elements. AI coaching adds limited value when AI can just do the task.

CLI and API-friendly tools (git, ffmpeg, AWS CLI): LLMs can call these directly via tool use. An agent with shell access can run git rebase -i for you. No coaching needed.

Everything else: desktop GUI apps (Blender, Affinity Photo, Unreal Engine, DaVinci Resolve), native mobile apps, professional hardware interfaces, games. There is no programmatic interface accessible to an AI. The application is a closed binary. The keyboard and touchscreen are the only way in, and only the human can use them. AI cannot automate anything. Coaching is the only viable form of AI assistance.

ExpertLens exists precisely in this gap. It watches your screen, listens to your voice, and advises you so you can command the application better. You stay in control. The AI makes you faster.

Mobile and cross-platform support

ExpertLens works on every device:

Desktop (Chrome): Full screen sharing via getDisplayMedia(). The coach sees your entire workspace.
Android (Chrome): Full screen sharing via getDisplayMedia(), same as desktop, works natively. Users can share any app window.
iOS (Safari): getDisplayMedia is not available on iOS. ExpertLens detects this at runtime and falls back to getUserMedia() with facingMode: "environment". The rear camera captures your physical screen. Point your phone at your monitor, and the coach sees exactly what you're looking at.

All three paths feed the same agent pipeline: same knowledge, same voice interaction, same Gemini Live session. No code changes on the agent side.

2. Why Not Just Use Gemini Live in AI Studio?

Gemini AI Studio has a built-in Live mode. You can share your screen, enable your microphone, and talk to Gemini about anything on screen. It works well. So why build ExpertLens?

Because general-purpose is not the same as expert. ExpertLens adds four concrete things:

Curated knowledge that is current. The seed sources include Blender 4.x-specific breaking changes: Auto Smooth was removed in 4.1 (replaced by the Smooth by Angle modifier), Bloom moved to the Compositor, keyframe shortcuts changed. General model training may not reflect these accurately. ExpertLens stuffs this knowledge directly into the system instruction at session start, adding zero additional latency.

User preferences. Every user is different. A shortcut-native power user needs different coaching than someone learning their first 3D software. ExpertLens supports interaction style (shortcuts-first vs. mouse-guided), tone (concise expert vs. calm mentor), response depth (short/medium/detailed), and proactivity (reactive/balanced/proactive). These are injected into every session's system instruction.

Cross-session memory. After each session, ExpertLens summarizes the coach's transcript using Gemini and stores it in Firestore. The next session loads the last three summaries and injects them as "## Previous Session Notes" into the system instruction. The coach picks up where you left off.

Software-specific coaching personas. The Blender coach is configured to lead every response with keyboard shortcuts, knows that Ctrl+2 applies Subdivision Surface Level 2, and treats the four Blender 4.x breaking changes as CRITICAL rules. A Blender session and an Affinity Photo session feel genuinely different.

3. Architecture

ExpertLens uses a two-layer grounding strategy designed for minimal latency:

Primary: Context stuffing. Curated knowledge is loaded into the system instruction at session start. Every coach has a dedicated prompt template (agent/prompts/coaches/<software>.py) with up to 50–70 pages of shortcuts, workflows, and common errors. Zero latency. No tool call needed for the most common questions.

Fallback: Firestore tool. The get_coach_knowledge(topic) ADK tool hits Firestore for deeper queries not covered by context. Latency: ~100–200ms.

The backend is FastAPI + ADK running on Cloud Run. The browser streams JPEG frames (~1fps, resized to 768×768) and PCM 16kHz audio over WebSocket. The Gemini Live session relays responses as PCM 24kHz audio back to the browser. The knowledge builder uses gemini-3-flash-preview with Google Search grounding to generate and keep coach knowledge current.

Deployment is fully automated: every push to main triggers a Cloud Build pipeline that builds both Docker images, pushes to Artifact Registry, deploys to Cloud Run, and persists CORS configuration. No manual steps.

4. Four Technical Challenges

Challenge 1: The 2-Minute Image Session Limit

Problem: Any image frame sent to Gemini Live API triggers "audio+video" mode, which has a 2-minute hard limit. A screen-sharing coaching session obviously needs to run longer than 2 minutes.

Solution: contextWindowCompression with SlidingWindow. Setting this in the Live session config tells Gemini to automatically compress older context when approaching the window limit. The session continues indefinitely. Without this, every screen-sharing session terminates after 2 minutes. It's a silent failure with no obvious error message.

config = LiveConnectConfig(
    context_window_compression=ContextWindowCompressionConfig(
        sliding_window=SlidingWindow(),
        trigger_tokens=25600,
    ),
    ...
)

Challenge 2: The ~10-Minute WebSocket Timeout

Problem: Gemini Live API WebSocket connections terminate after approximately 10 minutes. Cloud Run also has connection timeouts. A coaching session for learning a complex tool easily exceeds 10 minutes.

Solution: sessionResumption. When the server receives a GoAway signal, it stores the session handle and reconnects within 2 minutes (the handle's validity window). The client sees a brief "Reconnecting..." status and the session picks up with full context.

config = LiveConnectConfig(
    session_resumption=SessionResumptionConfig(handle=saved_handle),
    ...
)

The handle must be stored server-side and refreshed on every new_handle event. This needs to be designed in from the start, not added later.

Challenge 3: Zero-Latency Grounding

Problem: Grounding via RAG (vector search) adds 200–500ms per query in a live voice conversation. That latency is perceptible and breaks the conversational feel.

Solution: Context stuffing. All curated knowledge for a coach is pre-loaded into the system instruction at session start. The build_system_instruction_from_firestore function in agent/prompts/base.py assembles the full instruction including coach knowledge, user preferences, and session history before the WebSocket connection opens. The Firestore get_coach_knowledge tool exists as a fallback for deep queries, but most questions are answered from context.

Trade-off: system instruction size is bounded by the 128k context window. In practice, 50–70 pages of curated content fits well within this limit.

Challenge 4: Non-Blocking Tool Calls

Problem: ADK tool calls by default pause the audio stream while executing. If get_coach_knowledge takes 150ms, the coach goes silent mid-sentence. This is jarring in a live voice session.

Solution: NON_BLOCKING tool call mode with scheduling='WHEN_IDLE'. Tools execute between agent turns rather than interrupting them. The agent continues speaking; the tool result is incorporated into the next turn.

@FunctionTool
def get_coach_knowledge(topic: str) -> dict:
    ...

# In agent config:
tool_config = ToolConfig(
    function_calling_config=FunctionCallingConfig(
        mode=FunctionCallingConfigMode.NON_BLOCKING,
        scheduling=SchedulingMode.WHEN_IDLE,
    )
)

5. Cross-Session Memory: How It Works

The memory pipeline runs entirely in the background and adds less than 1 second to session start latency.

Accumulation: During a session, handler.py appends every coach turn to _coach_transcript, a list capped at 30 entries x 500 characters each (~15KB max). Only coach turns are kept; user audio is not transcribed.

Summarization: On session cleanup, summarize.py calls gemini-3.0-flash-preview with response_mime_type="application/json" and a typed schema. It returns a structured (summary, topics) object. The call has a 5-second timeout. If it fails, the session still ends cleanly.

Storage: The summary is written to Firestore under the session document, keyed by user_id (an anonymous UUID per browser connection) and coach_id.

Injection: At the start of the next session, base.py's build_system_instruction_from_firestore queries the last 3 session summaries with a strict 1-second timeout. If Firestore is slow, the session starts without history. This is acceptable graceful degradation. The summaries are injected as "## Previous Session Notes" in the system instruction, before the knowledge reference section.

The result: the coach opens each session with context about what the user was working on, without any user effort.

6. Authentication and Privacy

ExpertLens uses JWT-based authentication. Users log in with credentials, receive a signed token, and all subsequent API requests are authenticated via Bearer token. Each browser connection is assigned an anonymous UUID (user_id), which scopes session history and preferences to that user.

What's stored: Coach profiles, user preferences, and session summaries (generated text) are persisted in Firestore. Raw audio and video frames are never stored. They stream through the WebSocket to Gemini Live API and are discarded after processing. Session summaries contain only the coach's responses, not user audio transcriptions.

Coach ownership: Coaches created by a user are private to that user's account. The API enforces ownership checks on all coach CRUD operations.

7. What's Next

Two features are on the roadmap:

On-screen annotation. The coach could highlight UI elements in the user's screen share ("click this button" with a visual overlay). This requires a second WebSocket channel for screen coordinates and a browser-side overlay component.

Coach sharing. Users could publish custom coaches to a directory, so a DaVinci Resolve expert coach built by one user benefits everyone.

ExpertLens is live at: https://expertlens-frontend-pk4kcjevqa-uc.a.run.app

Source: https://github.com/TheIllusionOfLife/ExpertLens

#GeminiLiveAgentChallenge

Progressive Adoption of Kiro for 90s Website Generator

Yuya Mukai — Fri, 05 Dec 2025 14:15:56 +0000

Remember GeoCities? Angelfire? Tripod? The 1990s web was a magical chaos of Comic Sans, animated GIFs, MIDI music, visitor counters, and guestbooks. For the Kiroween hackathon, I decided to resurrect that era—and learned a lot about progressively adopting AI development tools along the way.

Live Demo: kiroween-mu.vercel.app
GitHub: github.com/TheIllusionOfLife/kiroween

The Project: 90s Website Generator

The app lets anyone create authentic 1990s-style personal homepages:

6 themes (Neon, Space, Rainbow, Matrix, GeoCities, Angelfire)
6 template presets ("90s Gamer Kid", "Elite Hacker", etc.)
Real-time live preview
Background music and sound effects
Working guestbook with real-time updates
Visitor tracking
Download as standalone HTML
Guest mode (no sign-in required)

Tech Stack & Architecture

Frontend: Next.js 16, React 19, TypeScript, Tailwind CSS, shadcn/ui, Zustand
Backend: Convex (real-time database), Clerk (authentication)
Testing: Vitest, fast-check (property-based testing)
Deployment: Vercel, Convex Cloud

The architecture follows a clean separation:

┌─────────────────────────────────────┐
│     Pages (Next.js App Router)      │
└─────────────────┬───────────────────┘
                  │
┌─────────────────▼───────────────────┐
│     Components (React + shadcn)     │
└─────────────────┬───────────────────┘
                  │
┌─────────────────▼───────────────────┐
│     State (Zustand) + Generator     │
└─────────────────┬───────────────────┘
                  │
┌─────────────────▼───────────────────┐
│     Backend (Convex real-time DB)   │
└─────────────────────────────────────┘

Progressive Adoption: My Journey with Kiro

Here's the key insight: Adopt Kiro's featuress progressively as you encounter specific problems.

Phase 1: Vibe Coding (Exploration)

I started with pure vibe coding—just chatting with Kiro to explore the problem space. This produced two experimental versions in vibe_coding/version1 (vanilla JS) and vibe_coding/version2 (Next.js).

These versions were messy but valuable. They helped me understand:

What features a 90s site generator actually needs
How to generate inline HTML with embedded CSS/JS
The tricky parts (iframe popups, audio autoplay policies)

Lesson: Vibe coding is great for exploration. Don't skip it.

Phase 2: Spec-Driven Development (Structure)

Once I understood what I was building, I formalized it with Kiro's spec-driven workflow:

Requirements (requirements.md): 24 user stories with EARS-formatted acceptance criteria
Design (design.md): Architecture, data models, 16 correctness properties
Tasks (tasks.md): 22 implementation tasks with requirement references

Example EARS requirement:

WHEN a user provides a name, hobby, and optional email 
THEN the System SHALL generate a complete HTML website incorporating these details

Example correctness property:

Property 1: Site generation incorporates all configuration
*For any* valid site configuration, the generated HTML should contain 
all specified values and include/exclude features according to the toggles.

Lesson: Specs eliminate ambiguity. You know exactly what "done" means.

Phase 3: Steering Docs (Consistency)

After a few commits, I noticed Kiro's suggestions weren't always consistent with my coding style. I added steering docs:

tech.md - Tech stack, commands, code conventions
structure.md - Directory layout, naming conventions
coding-standards.md - TypeScript patterns, React conventions
testing-guide.md - Property-based testing patterns with fast-check

The testing-guide.md was especially valuable. It taught Kiro my property testing patterns:

// **Feature: 90s-website-generator, Property 1: Site generation**
it('Property 1: Site generation incorporates all configuration', () => {
  fc.assert(
    fc.property(
      fc.record({
        name: fc.string({ minLength: 1, maxLength: 50 }),
        hobby: fc.string({ minLength: 1, maxLength: 100 }),
        theme: fc.constantFrom('neon', 'space', 'rainbow'),
      }),
      (config) => {
        const html = generateSiteHTML(config);
        expect(html).toContain(config.name);
      }
    ),
    { numRuns: 100 }
  );
});

Lesson: Steering docs compound over time. Document patterns as you discover them.

Phase 4: Agent Hooks (Automation)

After forgetting to run tests a few times (and pushing broken code), I added 7 agent hooks:

Run Tests on Save - Auto-runs tests when any .ts/.tsx file is saved
Run Property Tests on Test Change - Ensures property tests pass
Check Convex Schema on Change - Reminds to update types when schema changes
Security Check on User Input - Prompts security review when touching input handling
Task Completion Reminder - Prompts to update task status
Update Spec on Requirements Change - Keeps spec documents in sync
Validate Before Commit - Manual hook for pre-commit validation

The security hook was a lifesaver—it reminded me to add HTML escaping to the site generator when I might have shipped an XSS vulnerability.

Lesson: Hooks are "set and forget" safety nets. Add them when you keep forgetting something.

Phase 5: MCP (Extended Capabilities)

Finally, I added the Playwright MCP for end-to-end testing. This let Kiro:

Navigate to the deployed site
Test complete user flows
Verify the guestbook actually works
Test authentication flows

Lesson: Add MCP when you hit a wall that unit tests can't solve.

Implementing 90s Web Features

Now let's talk about the fun part—recreating authentic 90s web features with modern tech.

The HTML Generator

The core of the app is lib/site-generator.ts—a function that takes a config object and returns a complete HTML string with inline CSS and JavaScript.

export function generateSiteHTML(config: SiteConfig): string {
  const theme = themes[config.theme];

  return `<!DOCTYPE html>
<html>
<head>
  <title>${escapeHtml(config.name)}'s Homepage</title>
  <style>
    body {
      background: ${theme.background};
      color: ${theme.textColor};
      font-family: 'Comic Sans MS', cursive;
    }
    /* ... 200+ lines of inline CSS ... */
  </style>
</head>
<body>
  ${generateHeader(config)}
  ${generateAboutSection(config)}
  ${generateLinksSection(config)}
  ${generateGuestbookSection(config)}
  ${generateFooter(config)}

  <script>
    ${generateJavaScript(config)}
  </script>
</body>
</html>`;
}

Key challenges:

Everything inline: No external files, so the downloaded HTML works standalone
XSS prevention: User inputs must be escaped with escapeHtml()
Iframe detection: Popups must be suppressed in preview mode

Authentic 90s Visual Effects

Rainbow text with CSS animations:

@keyframes rainbow {
  0% { color: red; }
  17% { color: orange; }
  33% { color: yellow; }
  50% { color: green; }
  67% { color: blue; }
  83% { color: indigo; }
  100% { color: violet; }
}
.rainbow-text {
  animation: rainbow 3s infinite;
}

Marquee scrolling (yes, it still works):

<marquee behavior="scroll" direction="left">
  Welcome to my homepage! You are visitor #${visitorCount}!
</marquee>
</html>

Blinking text:

@keyframes blink {
  0%, 50% { opacity: 1; }
  51%, 100% { opacity: 0; }
}
.blink { animation: blink 1s infinite; }

The Guestbook System

The guestbook was surprisingly complex. It needed to:

Store entries in a real database (Convex)
Update in real-time when new entries are added
Validate input lengths (name 1-50 chars, message 1-500 chars)
Display entries chronologically

Convex schema:

guestbookEntries: defineTable({
  siteId: v.id("sites"),
  name: v.string(),
  message: v.string(),
  email: v.optional(v.string()),
  website: v.optional(v.string()),
  createdAt: v.number(),
}).index("by_site", ["siteId"]),

The real-time updates come for free with Convex's reactive queries:

const entries = useQuery(api.guestbook.getEntries, { siteId });
// Automatically updates when new entries are added!

Iframe Popup Suppression

Generated sites can have alert() and confirm() dialogs—authentic 90s behavior! But these break the live preview iframe.

Solution: Detect iframe context and suppress popups:

const isInIframe = window.self !== window.top;

if (!isInIframe && config.addPopups) {
  alert('Welcome to ' + config.name + '\\'s homepage!');

  window.onbeforeunload = function() {
    return 'Are you sure you want to leave this awesome page?';
  };
}

Audio Support

90s sites had MIDI music! We support background music and sound effects:

if (config.bgmTrack) {
  html += `
    <audio id="bgm" autoplay loop>
      <source src="${config.bgmTrack}" type="audio/mpeg">
    </audio>
    <div class="audio-controls">
      <button onclick="document.getElementById('bgm').paused ? 
        document.getElementById('bgm').play() : 
        document.getElementById('bgm').pause()">
        🎵 Toggle Music
      </button>
    </div>
  `;
}

Results

The progressive adoption approach worked:

35 tests passing (13 property-based, 100+ iterations each)
24 requirements formally specified
16 correctness properties validated
7 agent hooks automating the workflow
Production deployment at kiroween-mu.vercel.app

Key Takeaways

Start with vibe coding to explore the problem space
Add specs when you know what you're building
Add steering docs when you want consistency
Add hooks when you keep forgetting things
Add MCP when you hit capability walls
Don't adopt everything at once—each layer solves a specific problem

The 90s web is back. And it's tested.

Built for Kiroween hackathon. Try it at kiroween-mu.vercel.app

Building a Multi-Agent Marketing System with Google ADK and Cloud Run

Yuya Mukai — Mon, 10 Nov 2025 20:03:08 +0000

How I built an AI-powered marketing automation platform with human oversight using Google's Agent Development Kit

The Challenge: AI Needs Human Oversight

We've all seen the promise of AI-generated content. Marketing teams everywhere are experimenting with tools that can write social media captions, generate images, and even create videos. But there's a critical problem: AI makes mistakes.

A fully automated marketing system might:

Generate off-brand content
Create inappropriate imagery
Misunderstand campaign goals
Waste budget on poor-quality assets

The solution? Human-in-the-Loop (HITL) — AI generates the content, but humans approve before execution.

For the Cloud Run Hackathon, I built Promote Autonomy: a multi-agent marketing automation system that combines Google's Agent Development Kit (ADK) with a mandatory human approval workflow, all deployed on Cloud Run.

Try the live demo: https://frontend-909635873035.asia-northeast1.run.app

Architecture Overview: Three Services, One Workflow

The system is built on three independent Cloud Run services that communicate asynchronously via Pub/Sub:

1. Frontend (Next.js)

User-facing web interface for goal submission
Real-time job status updates via Firestore listeners
Approval/rejection workflow for generated plans
Display of final assets (captions, images, videos)

2. Strategy Agent (FastAPI + Gemini)

Receives marketing goals from users
Uses Gemini 2.5 Flash to generate comprehensive marketing plans
Saves plans to Firestore with status = pending_approval
Critical HITL step: Only publishes to Pub/Sub after human approval
Implements atomic Firestore transactions to prevent race conditions

3. Creative Agent (FastAPI + ADK)

Receives approved tasks via Pub/Sub push subscription
ADK Multi-Agent Coordinator orchestrates three specialized sub-agents:
- Copy Writer Agent: Generates social media captions using Gemini
- Image Creator Agent: Generates promotional images using Imagen 4.0
- Video Producer Agent: Generates promotional videos using Veo 3.0
All three agents execute in parallel for efficiency
Results uploaded to Cloud Storage with public URLs

The Data Flow

User → Frontend → Strategy Agent → Gemini → Firestore (pending_approval)
                                                ↓
User reviews plan → Approves → Firestore transaction + Pub/Sub publish
                                                ↓
Creative Agent ← Pub/Sub ← ADK Coordinator delegates to 3 sub-agents
                                                ↓
Gemini (captions) + Imagen (images) + Veo (videos) → Cloud Storage
                                                ↓
Firestore (completed) → Frontend displays assets

Key Innovation: The atomic Firestore transaction ensures that approval state and Pub/Sub message publishing happen together — preventing duplicate executions or lost messages.

Why Google ADK? The Multi-Agent Orchestration Problem

Initially, I orchestrated the three asset generation tasks using basic asyncio.gather():

# Original approach: Manual orchestration
captions_url, image_url, video_url = await asyncio.gather(
    generate_captions_task(),
    generate_image_task(),
    generate_video_task(),
)

This worked, but had limitations:

No built-in retry logic
Manual error handling for each task
Hard to add new agents or modify workflow
Difficult to test individual agent behavior

Enter Google ADK

Google's Agent Development Kit (ADK) provides a framework for building multi-agent systems with automatic orchestration. Here's how I restructured the Creative Agent:

1. Define Tools as Regular Functions

ADK automatically wraps Python functions as tools for agents:

# creative-agent/app/agents/tools.py

async def generate_captions_tool(
    config: dict[str, Any],
    goal: str,
    event_id: str
) -> dict[str, str]:
    """Generate social media captions using Gemini and upload to Cloud Storage.

    Args:
        config: Caption configuration (n, style)
        goal: Marketing goal for context
        event_id: Job ID for storage path

    Returns:
        Dict with captions_url key
    """
    caption_config = CaptionTaskConfig(**config)
    copy_service = get_copy_service()
    storage_service = get_storage_service()

    # Generate captions
    captions = await copy_service.generate_captions(caption_config, goal)

    # Upload to storage
    captions_json = json.dumps(captions, indent=2).encode("utf-8")
    url = await storage_service.upload_file(
        event_id=event_id,
        filename="captions.json",
        content=captions_json,
        content_type="application/json",
    )

    return {"captions_url": url}

No decorators needed — ADK inspects function signatures, docstrings, and type hints to automatically generate tool schemas for the LLM.

2. Create Specialized Sub-Agents

Each sub-agent is an LlmAgent with a specific role and tools:

# creative-agent/app/agents/coordinator.py

def create_copy_agent() -> LlmAgent:
    """Create ADK agent for caption generation."""
    return LlmAgent(
        name="copy_writer",
        model="gemini-2.5-flash",
        instruction="""You are a creative copywriter specialized in social media captions.
        Generate engaging, on-brand captions that match the requested style.
        Always call the generate_captions_tool to create captions.""",
        description="Generates social media captions using Gemini",
        tools=[generate_captions_tool],
    )

def create_image_agent() -> LlmAgent:
    """Create ADK agent for image generation."""
    return LlmAgent(
        name="image_creator",
        model="gemini-2.5-flash",
        instruction="""You are a visual designer specialized in promotional graphics.
        Generate images that match the requested prompt, size, and aspect ratio.
        Always call the generate_image_tool to create images.""",
        description="Generates promotional images using Imagen",
        tools=[generate_image_tool],
    )

def create_video_agent() -> LlmAgent:
    """Create ADK agent for video generation."""
    return LlmAgent(
        name="video_producer",
        model="gemini-2.5-flash",
        instruction="""You are a video producer specialized in short-form promotional videos.
        Generate videos that match the requested prompt, duration, and aspect ratio.
        Always call the generate_video_tool to create videos.""",
        description="Generates promotional videos using Veo",
        tools=[generate_video_tool],
    )

3. Create the Coordinator Agent

The coordinator delegates tasks to the specialized sub-agents:

def create_creative_coordinator() -> LlmAgent:
    """Create ADK coordinator agent that orchestrates all creative agents."""
    copy_agent = create_copy_agent()
    image_agent = create_image_agent()
    video_agent = create_video_agent()

    coordinator = LlmAgent(
        name="creative_director",
        model="gemini-2.5-flash",
        instruction="""You are a creative director coordinating asset generation.

        Your job is to delegate tasks to specialized agents:
        - copy_writer: For captions
        - image_creator: For images
        - video_producer: For videos

        Delegate ALL tasks in parallel for efficiency.
        Collect all results and return them in a structured format.

        If any agent fails, continue with others and report the error.""",
        description="Coordinates creative asset generation across multiple specialized agents",
        sub_agents=[copy_agent, image_agent, video_agent],
    )

    return coordinator

4. Run the Coordinator

Using ADK's Runner API to execute the multi-agent workflow:

from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

async def generate_assets_with_adk(event_id: str, task_list: TaskList) -> dict:
    """Generate assets using ADK multi-agent orchestration."""
    coordinator = get_creative_coordinator()

    # Prepare prompt for coordinator
    prompt = f"""Generate creative assets for this marketing campaign:

Goal: {task_list.goal}
Target Platforms: {', '.join(p.value for p in task_list.target_platforms)}
Event ID: {event_id}

Tasks to complete:
{json.dumps(task_list.tasks, indent=2)}

Delegate each task to the appropriate specialized agent.
Run all tasks in parallel for efficiency.
Return URLs for all generated assets."""

    # Run ADK coordinator
    session_service = InMemorySessionService()
    runner = Runner(
        agent=coordinator,
        app_name="creative-agent",
        session_service=session_service
    )

    user_message = types.Content(
        role='user',
        parts=[types.Part(text=prompt)]
    )

    # Execute and collect results
    def run_agent():
        events = runner.run(
            user_id="creative-agent",
            session_id=event_id,
            new_message=user_message
        )
        for event in events:
            if event.is_final_response() and event.content:
                return event.content.parts[0].text
        return ""

    result = await asyncio.to_thread(run_agent)

    # Parse URLs from response
    return {
        "captions_url": extract_url(result, "captions"),
        "image_url": extract_url(result, "image"),
        "video_url": extract_url(result, "video"),
    }

Benefits of ADK Orchestration

Automatic Parallel Execution: ADK handles running sub-agents concurrently
Built-in Error Handling: Coordinator can decide how to handle failures
Extensible: Easy to add new agents (e.g., brand style guide checker)
Testable: Can test each agent independently
Observability: ADK provides logging and tracing out of the box

Technical Challenges & Solutions

Challenge 1: Atomic HITL Approval

Problem: The approval workflow required updating Firestore state AND publishing to Pub/Sub. If either operation failed, the system would be in an inconsistent state.

Solution: Firestore transactions with conditional updates:

# strategy-agent/app/routers/approve.py

@router.post("/approve")
async def approve_strategy(request: ApproveRequest):
    """Approve pending strategy with atomic state transition."""
    firestore_service = get_firestore_service()
    pubsub_service = get_pubsub_service()

    # Atomic transaction: only update if status is pending_approval
    @firestore.transactional
    def update_in_transaction(transaction, job_ref):
        snapshot = job_ref.get(transaction=transaction)

        if not snapshot.exists:
            raise HTTPException(status_code=404, detail="Job not found")

        job = snapshot.to_dict()

        # Conditional update: only proceed if pending approval
        if job["status"] != "pending_approval":
            raise HTTPException(
                status_code=409,
                detail=f"Job already {job['status']}"
            )

        # Update status to processing
        transaction.update(job_ref, {
            "status": "processing",
            "approved_at": firestore.SERVER_TIMESTAMP,
        })

    # Execute transaction
    job_ref = firestore_service.db.collection("jobs").document(event_id)
    transaction = firestore_service.db.transaction()
    update_in_transaction(transaction, job_ref)

    # Only publish to Pub/Sub after successful transaction
    await pubsub_service.publish_task(event_id, task_list)

    return {"status": "approved", "event_id": event_id}

Key Insight: The transaction prevents duplicate approvals, and we only publish to Pub/Sub after the transaction succeeds. This ensures exactly-once approval semantics.

Challenge 2: Long-Running AI Models

Problem: Veo video generation can take 2-5 minutes. Without proper timeout handling, requests would hang indefinitely.

Solution: Layered timeout protection:

# creative-agent/app/services/video.py

async def generate_video(self, task: VideoTaskConfig) -> bytes:
    """Generate video with timeout protection."""
    try:
        # Timeout at service level (120 seconds)
        video_bytes = await asyncio.wait_for(
            asyncio.to_thread(self._generate_video_sync, task),
            timeout=self.settings.VEO_TIMEOUT_SEC
        )
        return video_bytes
    except asyncio.TimeoutError:
        logger.error(f"Veo generation timed out after {self.settings.VEO_TIMEOUT_SEC}s")
        raise HTTPException(status_code=504, detail="Video generation timed out")

Additionally, Cloud Run services have request timeouts (default 5 minutes), providing another layer of protection.

Challenge 3: ADK Runner is Synchronous

Problem: ADK's Runner.run() is a synchronous generator, but our FastAPI app is async.

Solution: Use asyncio.to_thread() to run the synchronous ADK code in a thread pool:

def run_agent():
    events = runner.run(
        user_id="creative-agent",
        session_id=event_id,
        new_message=user_message
    )
    for event in events:
        if event.is_final_response() and event.content:
            return event.content.parts[0].text
    return ""

result = await asyncio.to_thread(run_agent)

This prevents blocking the async event loop while maintaining compatibility with FastAPI.

Challenge 4: Pub/Sub Authentication

Problem: The Creative Agent's /consume endpoint needs to verify that incoming requests actually come from Pub/Sub (not malicious actors).

Solution: OIDC token verification:

# creative-agent/app/routers/consume.py

@router.post("/consume")
async def consume_task(
    pubsub_message: PubSubMessage,
    request: Request,
    authorization: str | None = Header(None),
):
    """Consume task from Pub/Sub with OIDC authentication."""
    if not authorization:
        raise HTTPException(status_code=401, detail="No authorization header")

    # Verify OIDC token from Pub/Sub
    try:
        token = authorization.split("Bearer ")[1]
        audience = str(request.url)

        # Verify token with Google's token verification
        id_token.verify_oauth2_token(
            token,
            google_requests.Request(),
            audience=audience
        )
    except Exception as e:
        logger.error(f"OIDC verification failed: {e}")
        raise HTTPException(status_code=403, detail="Invalid token")

    # Token verified - process message
    # ...

The Pub/Sub subscription is configured with a service account that has permission to invoke the Cloud Run service, and Pub/Sub automatically includes an OIDC token in the Authorization header.

Results & Learnings

What Works Well

End-to-End Workflow: The full pipeline (goal submission → plan generation → human approval → asset creation → display) completes in 60-120 seconds for a typical job.

ADK Integration: The multi-agent coordinator makes the Creative Agent significantly more maintainable. Adding a new agent type (e.g., brand compliance checker) would take ~30 minutes.

Real-Time Updates: Firestore listeners provide instant feedback to users as job status changes, creating a responsive UX without polling.

Production Ready: 83 passing tests, full CI/CD pipeline, comprehensive error handling, and security hardening (Firebase Auth, OIDC, Firestore security rules).

Performance Metrics

Strategy Generation: 3-10 seconds (Gemini API call)
Caption Generation: 5-15 seconds (Gemini API call)
Image Generation: 20-40 seconds (Imagen 4.0)
Video Generation: 2-5 minutes (Veo 3.0 is slow but produces high-quality results)
Total Workflow: 60-120 seconds for captions + image, 3-7 minutes with video

Key Learnings

ADK Coordinator Pattern is Powerful: Separating orchestration logic (coordinator) from execution logic (tools) makes the system extensible and testable.
Atomic Transactions are Critical: In distributed systems, ensuring state consistency requires careful use of transactions. Firestore transactions prevented race conditions in the approval workflow.
Async/Sync Bridging: Integrating synchronous libraries (ADK) with async frameworks (FastAPI) requires asyncio.to_thread() or similar patterns.
HITL is Non-Negotiable: Every test run where I skipped the approval step generated at least one piece of content I wouldn't want published. Human oversight is essential for AI-generated marketing.
Cloud Run is Perfect for AI Workloads: Auto-scaling handles variable AI model latency (Imagen takes 30s, Veo takes 3 minutes). Pay-per-use pricing means we only pay when jobs are running.

Future Enhancements

The current system is an MVP, but there are several natural next steps:

1. Multi-Modal Input (Product Photos)

Currently, users describe products in text. Next version will support uploading product photos that Gemini analyzes for visual context, ensuring generated images match the actual product.

2. Platform-Specific Configuration

Generate assets tailored to each platform:

Instagram: 1:1 square images, 15-second videos
X (Twitter): 16:9 images, 2:20 max videos
LinkedIn: 1.91:1 images, professional tone

3. Brand Style Guide Integration

Add a fourth agent (brand guardian) that validates all generated content against uploaded brand guidelines (colors, fonts, tone, taglines).

4. Performance Feedback Loop

Track which generated assets perform well (likes, shares, conversions) and feed this data back to improve future generations.

Try It Yourself

Live Demo: https://frontend-909635873035.asia-northeast1.run.app
Source Code: https://github.com/TheIllusionOfLife/promote_autonomy
Architecture Diagram: View on GitHub

Local Development Setup

# Clone the repository
git clone https://github.com/TheIllusionOfLife/promote_autonomy.git
cd promote_autonomy

# Install shared schemas
cd shared && uv sync

# Set up Strategy Agent
cd ../strategy-agent
cp .env.example .env
# Edit .env with your Google Cloud credentials
uv sync
uv run pytest  # Run tests

# Set up Creative Agent
cd ../creative-agent
cp .env.example .env
# Edit .env with your credentials
uv sync
uv run pytest  # Run tests

# Set up Frontend
cd ../frontend
npm install
cp .env.local.example .env.local
# Edit .env.local with Firebase config
npm run dev

The system supports mock mode for rapid development without API costs:

USE_MOCK_GEMINI=true (no Gemini API calls)
USE_MOCK_IMAGEN=true (placeholder images)
USE_MOCK_VEO=true (text briefs only)

Conclusion

Building Promote Autonomy taught me that multi-agent systems are the future of AI applications. Rather than building monolithic AI systems, we can compose specialized agents that work together, each focused on one task.

Google's ADK makes this pattern accessible. The coordinator pattern, automatic tool wrapping, and built-in orchestration eliminate much of the boilerplate code required for multi-agent systems.

Cloud Run provides the perfect deployment platform: auto-scaling handles variable AI workload patterns, independent service deployment enables microservices architecture, and pay-per-use pricing keeps costs low during development.

Most importantly, the Human-in-the-Loop approval workflow demonstrates that AI augmentation beats AI automation. Rather than replacing human marketers, this system makes them more productive by handling the repetitive work while keeping humans in control of strategy and quality.

If you're building AI applications, consider:

Break complex tasks into specialized agents (coordinator pattern)
Use ADK for orchestration (handles parallel execution, error handling, observability)
Deploy on Cloud Run (serverless, auto-scaling, cost-effective)
Keep humans in the loop (AI generates, humans approve)

About the Hackathon

This content was created for the purposes of entering the Cloud Run Hackathon in the AI Agents category. The hackathon challenges developers to build innovative applications using Cloud Run and Google's Agent Development Kit (ADK). Promote Autonomy demonstrates how multi-agent systems can solve real-world problems while maintaining human oversight.

Technology Stack: Google ADK, Cloud Run, Gemini 2.5 Flash, Imagen 4.0, Veo 3.0, Firestore, Pub/Sub, Cloud Storage, Next.js, FastAPI, Python 3.11+

Have questions about the implementation? Want to contribute? Find me on GitHub or check out the live demo!