Forem: omkar

Building NagarDrishti: A Live AI Agent for Road Safety in India

omkar — Mon, 16 Mar 2026 23:44:21 +0000

Disclaimer: This blog post was created for the purposes of entering the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

The Problem: Monsoon Potholes Kill

Pune, India — 7 million people, 2,273 kilometers of roads. During monsoon season, potholes and waterlogging cause hundreds of accidents every year. In October 2025, the Bombay High Court mandated that every pothole must be repaired within 48 hours after multiple fatalities, with personal liability on senior officials for delays.

Yet citizens have no efficient way to report hazards, and municipal officers lack real-time situational awareness.

The Solution: NagarDrishti (City Vision)

We built NagarDrishti — an AI-powered platform where citizens report road hazards through video or natural voice conversation, and municipal officers manage them through an intelligent dashboard with AI chat capabilities.

The Live Agent: Commute Mode

The heart of NagarDrishti is Commute Mode — a hands-free, always-listening voice agent powered by Gemini Live API.

How It Works

Tap once to activate Commute Mode while driving
Speak naturally — "There's a big pothole ahead"
AI responds instantly via audio — "Got it. Pothole logged. Stay safe."
Report created with GPS, severity, and jurisdiction — all without touching your phone

What Makes It "Live"

This isn't a chatbot. It's a real-time AI agent with:

Bidirectional audio streaming via WebSocket (not turn-based text)
Voice Activity Detection tuned for noisy traffic (800ms silence threshold)
Function calling during live audio — creates database records while you speak
Interruptible — you can speak mid-response, agent handles gracefully
Context-aware — knows your GPS, nearby hazards, previous reports this session
Multilingual — English, Hindi, Marathi system prompts

Response latency: 2–4 seconds from speech to AI audio reply.

The AI Pipeline: 5 Steps in <15 Seconds

When a citizen submits a video report, the AI pipeline processes it automatically:

Classification — Gemini Vision analyzes video frames → hazard type + severity (1–5)
Jurisdiction Routing — GPS-based routing to PMC/PCMC/PMRDA/Cantonment
Deduplication — BigQuery spatial query (20m radius) + Gemini LLM similarity
Weather Intelligence — Open-Meteo 48h rain forecast → severity multiplier (1.0×–2.0×)
Dimension Estimation — Gemini Vision estimates length/width/depth + repair cost

Result: Verified report in BigQuery, ready for officer action.

The Dashboard: AI Chat for Officers

Municipal officers use the PMC Dashboard — a Next.js web app with:

Live heatmap with severity-colored dots (Mapbox)
AI Chat powered by Gemini with 11 BigQuery function-calling tools
Agent-as-tool pattern — generate_ward_report is a sub-agent that queries BigQuery + Gemini to produce structured ward reports on demand

Example Queries

Officer: "Which areas have the most critical hazards?"

AI: (queries BigQuery) "Dapodi ward has 12 severity-4 potholes, Baner has 8..."

Officer: "Generate report for ward 15"

AI: (invokes sub-agent) "Ward 15 Report: 23 active hazards, 18 overdue, top type: waterlogging..."

Real Data, Real Roads

Every report in the system is from actual road hazards in Pune — no synthetic or mock data.

Our team physically traveled across Pune on bikes to collect 24+ real field reports from:

PCMC: Dapodi, Sangvi, Pimple Saudagar, Chinchwad
PMC: Aundh, Baner, Kothrud, Pashan

We drove through these areas and recorded hazards using both video capture and Commute Mode voice reporting.

Technical Highlights

Google Cloud Stack

Cloud Run — FastAPI backend + Next.js frontend (asia-south1)
BigQuery — Hazard reports, users, verifications (45 reports currently)
Cloud Storage — Video evidence, thumbnails (nagardrishti-media bucket)
Terraform — Infrastructure-as-Code for all GCP resources

Gemini Models

Gemini 3 Flash (gemini-3-flash-preview) — Vision + Text classification
Gemini 2.5 Flash Native Audio (gemini-2.5-flash-native-audio-preview-12-2025) — Live API audio streaming

Mobile App

Flutter (Android) with Riverpod state management
Features: Video capture (10s auto-stop), Commute Mode, heatmap, leaderboard, proximity alerts
Gamification: Points system (+10 report, +5 verify, +2 endorse) with tier progression (Bronze → Diamond)

Challenges We Solved

1. Gemini Live API Latency

Problem: Initial voice response took 10–15 seconds.

Solution: Tuned VAD parameters:

silence_duration: 1800ms → 800ms
prefix_padding: 300ms
Disabled dynamic thinking in Live API config

Result: 2–4 second response latency.

2. Duplicate Detection at Scale

Problem: Simple distance-based matching caused false positives.

Solution: Combined BigQuery spatial query (20m radius) + Gemini LLM comparison of hazard descriptions + hazard type matching.

Result: Precision deduplication with community confirmation boost.

3. Jurisdiction Routing Accuracy

Problem: GPS coordinates near municipal boundaries gave incorrect assignments.

Solution: Deterministic polygon-based routing instead of AI classification.

Result: 100% accurate jurisdiction assignment.

What We Learned

Gemini Live API's native audio capabilities are remarkably good at understanding Indian English accents and Hindi/Marathi mixed speech
Voice Activity Detection tuning is critical for real-world driving conditions — default settings are designed for quiet environments
BigQuery's serverless architecture handles the read-heavy pattern of a dashboard perfectly — no infrastructure management needed
Real field data collection is irreplaceable — AI models behave very differently with actual monsoon road conditions vs. clean test videos

Try It Yourself

Live Dashboard: https://nagardrishti-frontend-692377565465.asia-south1.run.app
Demo Video: https://youtu.be/Rn7eJxBdWe4
Mobile APK: https://drive.google.com/file/d/1xkoClmcbUo0uvFUaYITg4GTM5RkXIZMj/view?usp=sharing

Demo Credentials

PMC Dashboard (Web):

Email: admin@pmc.gov.in
Password: pmc2026

Mobile App:

Email: test@email.com
Password: test123

What's Next

FCM push notifications when report status changes
Multi-city expansion beyond Pune (Mumbai, Nagpur)
Integration with PMC's existing grievance portal (SARATHI)
Community verification rewards with blockchain-anchored proof

Built With

Python • FastAPI • Flutter • Next.js • TypeScript • TailwindCSS • Google Gemini API • Gemini Live API • Google GenAI SDK • Google Cloud Run • Google BigQuery • Google Cloud Storage • Terraform • Docker • Mapbox

This project was built for the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge

Project Category: Live Agents

Why Elasticsearch Is the Best Memory for AI Agents: A Deep Dive into Agentic Architecture

omkar — Fri, 27 Feb 2026 15:59:13 +0000

This blog post was submitted to the Elastic Blogathon Contest and is eligible to win a prize.

I've been researching how developers are building AI agents in 2026 — not chatbots, not search bars, but autonomous systems that investigate, reason, and act. In the emerging discipline of **context engineering* — the defining paradigm of AI development in 2026, focused on giving agents the right information at the right time — one pattern keeps emerging: the best agents don't just search. They remember.*

And the memory layer they're choosing? Elasticsearch.

The Elastic Stack for Agentic AI

After months of hands-on research and building agentic architectures, I've identified the Elastic features that matter most for production AI agents in 2026:

Elastic Feature	Role in Agent Architecture
Agent Builder	Core reasoning orchestration — the brain
Temporal analytics — episodic memory
Semantic/Vector Search	Meaning retrieval — semantic memory
Elastic Workflows	Automated actions — procedural memory
ELSER/ELSER-2	Zero-config embeddings via Inference Service
MCP Server	Tool integration for external agent frameworks
Hybrid Search (BM25 + kNN + RRF)	Best-of-both-worlds retrieval

The combination that matters most? Agent Builder + ES|QL + vector search — together they transform Elasticsearch from a database into an agent brain. Not a tool that finds documents, but a system that gives AI agents institutional knowledge, contextual reasoning, and the ability to act.

This blog explores three memory layers that define production-ready agentic architectures, the trust patterns that separate toy demos from deployable systems, and why Elasticsearch's convergent platform gives it an edge that no standalone vector database can match

The Three Memory Layers of an Elasticsearch Agent Brain

The most sophisticated agentic systems aren't just storing vectors. They're building layered memory systems — eerily similar to how the human brain organizes knowledge. Here's the pattern:

Layer 1: Episodic Memory — "What Happened"

Elastic Feature: ES|QL + Time-Series Indices

Episodic memory stores specific events with temporal context — what happened, when, and in what sequence. In Elasticsearch, this maps perfectly to time-series indices queried via ES|QL.

Consider a self-healing infrastructure agent. Its anomaly detection component uses parameterized ES|QL to detect anomalies against rolling baselines:

FROM metrics-*
| WHERE @timestamp > NOW() - 15 minutes
| STATS avg_cpu = AVG(system.cpu.percent),
        avg_memory = AVG(system.memory.used.pct)
  BY host.name
| WHERE avg_cpu > 85.0 OR avg_memory > 90.0

This gives the agent temporal awareness — it doesn't just know something is wrong, it knows how the system's behavior has changed over time. It can detect that CPU usage has been climbing steadily for the past hour, not just that it's currently high.

Why it matters: ES|QL transforms agents from stateless chatbots into operationally aware tools that reason over real data. The consistent insight from my research? ES|QL + LLM reasoning = grounded intelligence.

🔬 Working Demo: ES|QL Aggregation on Live Elastic Cloud

I built a working demo on Elastic Cloud Serverless to demonstrate this. Here's a real ES|QL query aggregating incident counts and resolution times by service:

Layer 2: Semantic Memory — "What Things Mean"

Elastic Feature: ELSER / Vector Search / semantic_text

Semantic memory stores meaning — not exact words, but concepts and relationships. This is where Elasticsearch's vector capabilities shine, and where the most powerful agent capabilities emerge.

The "Zero Keywords" Moment: Imagine an SRE agent that needs to match the production error "NullPointerException in PaymentProcessor" to a Git commit message "Removed null safety checks". There is zero keyword overlap between these two strings. Yet ELSER finds the connection because it understands meaning, not words.

Setting this up requires just a few lines in your index mapping:

PUT /incidents
{
  "mappings": {
    "properties": {
      "description": { "type": "semantic_text" },
      "root_cause": { "type": "semantic_text" },
      "severity": { "type": "keyword" },
      "timestamp": { "type": "date" }
    }
  }
}

That's it. On Elastic Cloud Serverless, semantic_text defaults to the .elser-2-elastic inference endpoint — Elastic's hosted ELSER model via the Elastic Inference Service (EIS). Embedding generation, text chunking, and storage are handled automatically at ingest time. No separate pipeline, no external embedding service, no vector dimension configuration.

Want to use a different model? Elastic's Inference Endpoints API lets you swap in Cohere, OpenAI, or the multilingual E5 model with a single inference_id parameter:

// First, create a custom inference endpoint
PUT _inference/text_embedding/my-cohere-endpoint
{
  "service": "cohere",
  "service_settings": {
    "api_key": "{{COHERE_API_KEY}}",
    "model_id": "embed-english-v3.0"
  }
}

// Then reference it in your mapping
PUT /incidents-multilingual
{
  "mappings": {
    "properties": {
      "description": {
        "type": "semantic_text",
        "inference_id": "my-cohere-endpoint"
      }
    }
  }
}

This pluggable inference architecture is a massive advantage — you can switch embedding models without rewriting application code, test ELSER vs. Cohere vs. E5 on the same data, or use multilingual models for international deployments.

Ingestion at Scale: Elasticsearch's Open Web Crawler can feed directly into semantic_text indices, automatically generating embeddings at ingest. This means an agent can continuously crawl and semantically index an entire corporate knowledge base — docs, wikis, runbooks — without any custom ETL pipeline.

Scaling the Memory Layer: At enterprise scale, agent memory can grow to billions of vectors. Elasticsearch addresses this with Better Binary Quantization (BBQ) — reducing float32 vectors to bits with a 95% memory reduction while preserving ranking quality (GA in ES 9.1). The newer DiskBBQ algorithm (ES 9.2) eliminates the need to keep entire vector indexes in RAM, sustaining ~15ms query latencies at just 100MB of memory. This means your agent's semantic memory can scale to petabytes without proportionally scaling infrastructure costs.

🔬 Working Demo: Semantic Search with Zero Keyword Overlap

I tested this on my Elastic Cloud project. I queried "null pointer exception in payment processing" against the root_cause field. The top result? "Removed null safety checks in payment module during refactoring" with a score of 13.44 — matched purely by meaning, not keywords:

What makes this even more powerful is the dual memory architecture. The best agentic systems I've studied implement a pattern inspired by the actual hippocampus:

Episodic layer: Recent operational events with a 90-day ILM expiry policy
Semantic layer: Persistent consolidated knowledge, distilled from episodes

A background consolidation loop periodically converts episodic memories into semantic ones. The agent uses ES|QL joins for "domain density scoring" — measuring how much verified experience exists per topic. When density is too low, the agent genuinely refuses to answer: "I don't have enough evidence on this topic."

Layer 3: Procedural Memory — "How to Act"

Elastic Feature: Elastic Workflows + Agent Builder

Procedural memory stores how to do things — learned routines, proven playbooks, tested remediation steps. In Elasticsearch, this maps to Elastic Workflows (YAML-defined automation, currently in technical preview) triggered by Agent Builder reasoning.

Here's the architecture pattern I've seen work best for incident response agents — chaining everything together with a confidence gate (simplified for illustration):

Workflow receives webhook (alert fires)
Agent runs 3 ES|QL queries (regional errors, latency anomalies, error fingerprinting)
Agent retrieves SOP via hybrid search (BM25 + ELSER)
Agent produces RemediationPlan with deterministic confidence score
Safety gate evaluates confidence:

steps:
  - id: evaluate_confidence
    action: condition
    if:
      all:
        - field: agent.confidence
          gte: 0.90
        - field: agent.thresholds_met
          eq: true
    then:
      - action: http_request
        url: "{{remediation_endpoint}}"
        method: POST
        body: "{{agent.remediation_plan}}"
    else:
      - action: slack_message
        channel: "#incidents"
        text: "⚠️ Confidence {{agent.confidence}} < 0.90. Human review required."

The result: Alert to fix in under 90 seconds. 100% Elastic-native — no LangChain, no external orchestration, no data leaving the cluster.

Agent Builder also supports the Agent-to-Agent (A2A) protocol — allowing your incident response agent to delegate tasks to specialized sub-agents (e.g., a "forensics agent" and a "remediation agent") that coordinate autonomously while sharing a unified Elasticsearch context. Combined with MCP for external tool integration, this creates a fully interoperable multi-agent ecosystem.

The Trust Problem — Why the Best Agents Refuse to Act

The #1 lesson from studying production agentic systems isn't about speed or accuracy. It's about trust.

The most production-ready systems all share one characteristic: they build in mechanisms for the agent to refuse when it isn't confident enough. Here are the five trust patterns I've identified:

Pattern 1: Evidence Gates

Require ≥2 independent citations from different indices before taking any action. Use Reciprocal Rank Fusion (RRF) to combine BM25 lexical scores with kNN vector similarity scores into a single ranking. If you can't find two independent sources that corroborate an answer, stop.

A more sophisticated approach uses intent-based query routing: the agent classifies each query's intent before searching. Factual queries (e.g., "What is the SLA for service X?") are routed to BM25 for exact matches, while reasoning queries (e.g., "Why did latency spike?") prioritize vector similarity via the Linear Retriever (GA in ES 8.18), which allows weighted score normalization between lexical and semantic signals. This adaptive fusion maximizes relevance for each question type — something no standalone vector DB offers.

"What if an AI system refused to act unless it had independent evidence?"

Pattern 2: Adversarial Self-Review

Run a multi-agent swarm where one agent's entire job is to disprove the findings of the others. It searches for exception patterns, checks IOC databases, cross-references asset inventory. Only when it cannot explain a finding does the conclusion stand.

In cybersecurity threat hunting, this means having a "devil's advocate" agent that adversarially challenges every finding. In healthcare applications, it means a verification agent that challenges every clinical recommendation. As one architect noted: "A review where nothing is questioned is a rubber stamp, not a safeguard."

Pattern 3: The "No Query, No Number" Policy

Enforce a radical rule: every number presented to the user must come from an ES|QL aggregation (SUM, COUNT_DISTINCT, AVG). If there are zero records, the agent returns no answer — never a fabricated one. I've seen this validated across three different LLMs with zero hallucinations in testing.

"Confidence without data is a well-dressed lie."

Pattern 4: CONFIRM Gates

Implement explicit human confirmation for privileged actions (closing incidents, reopening tickets) in compliance-sensitive environments. The MCP server acts as the single source of truth for incident lifecycle state.

Pattern 5: Investigation ≠ Decision-Making

The best architectural principle I've found: "Investigation (AI) ≠ Decision-making (workflow)."

The agent investigates — searching logs, correlating signals, matching patterns. But priority classification, severity assignment, and team routing are handled by deterministic Elastic Workflows with YAML if steps and Liquid templates — never by the LLM.

"LLMs should never decide severity."

The Elasticsearch Advantage — Why Not a Standalone Vector DB?

If you only need vector search, any vector database will do. But real-world agentic systems never only need vector search. They need:

What Agents Need	Elasticsearch	Standalone Vector DB
Vector + keyword + structured in one query	✅ Hybrid Search (BM25 + kNN + RRF + Linear Retriever)	❌ Separate systems
Time-series analytics over operational data	✅ ES\	QL: `BUCKET`, `STATS`, date math
Automated actions triggered by agent reasoning	✅ Elastic Workflows (YAML)	❌ Custom orchestration code
Built-in agent framework	✅ Agent Builder (GA Jan 2026)	❌ Requires LangChain/LlamaIndex wrapper
Semantic search with zero config	✅ ELSER `semantic_text` + Inference Endpoints	❌ Bring your own embedding pipeline
Geospatial + vector in one index	✅ `geo_point` + `dense_vector`	❌ Separate stores
IDE integration & multi-agent	✅ MCP + Agent-to-Agent (A2A) protocol	❌ API-only
Lifecycle management (ILM/retention)	✅ Native ILM policies	❌ Manual TTL management

The key insight: ES|QL is the #1 differentiator. It lets agents investigate data with time-window correlations, aggregation pipelines, and structured analytics — capabilities that transform a chatbot into an operational tool. No standalone vector DB offers this.

What Practitioners Are Saying

"We kept overcomplicating the architecture — a single well-crafted Elasticsearch Tool definition could do it better, faster, and with zero hallucinations." — a common realization among practitioners

"Simplicity wins — we removed an entire tool from our agent, and it became simpler and just as functional."

"The power of an AI agent lies not in prompt complexity, but in the quality of retrieved context."

"Elasticsearch is far more than a search engine — it is a high-performance Analytical Memory Store." — a sentiment echoed across the developer community

Five Patterns You Should Steal

Let me close with five creative patterns that go beyond standard RAG tutorials and show what's possible when you treat Elasticsearch as an agent brain.

1. 🧠 Neuroscience-Inspired Cognitive Filtering

Result: 91% noise reduction

Most agents index everything. A smarter approach applies neuroscience research to filter signals before they reach Elasticsearch:

Habituation Filter (Thompson & Spencer, 1966): Repeated similar events raise the threshold for alerting
Circadian Rhythm (Borbely, 1982): Time-of-day vigilance adjustment
Salience Network (Corbetta & Shulman, 2002): Only novel, significant events pass through

In a 48-hour live test, 2,200 sensor events were reduced to 173 meaningful alerts. Each indexed document carries cognitive metadata (habituation state, circadian phase, priority score) that improves downstream search relevance.

2. 🔍 Semantic Commit Matching

Result: 3 hours → 3 minutes

When a production error like "NullPointerException in PaymentProcessor" occurs, use ELSER semantic_text fields to search recent Git commits. ELSER matched the error to the commit "Removed null safety checks" with zero shared keywords — as we demonstrated in our live demo above. Traditional keyword search would have found nothing. The agent can then automatically create a revert PR via GitHub Actions and notify the team on Slack.

3. 🏥 Geographic Impossibility Detection

Result: $195K/year savings, lives saved

In healthcare systems with distributed facilities, duplicate patient records waste resources and compromise care. An agent using ES|QL can detect "geographic impossibility" — a patient can't realistically be tested at two distant facilities on the same day. From 1,010 records, it found 131 duplicates including 5 same-day multi-facility cases, all in under 10 seconds.

4. 🛡️ Active Cyber Deception

Result: 258 days → 15 seconds

Instead of passively alerting on threats, build an active defense system:

Elastic Watcher monitors live logs for intrusion patterns
ES|QL runs forensics to isolate attacker IP and payload
ELSER searches source code semantically to find the exact vulnerable file
Nginx reroutes the attacker into a Docker honeypot (fake environment)
Cryptographic patch is applied with human approval

The average breach lifecycle drops from 258 days to 15 seconds.

5. ⚖️ Adversarial Agent Debate

Innovation: Agents argue with each other

Build four specialized agents that work in a structured debate:

SCANNER: Finds initial compromise signals
TRACER: Traces lateral movement and privilege escalation
ADVOCATE (devil's advocate): Actively tries to disprove every finding using exception patterns and asset inventory
COMMANDER: Resolves disagreements, assigns final confidence scores

When the ADVOCATE cannot explain a finding, the threat stands. When it can explain it (e.g., "that was a scheduled admin action"), a false positive is prevented. This mirrors how real investigative teams work — and it runs entirely on Elasticsearch Agent Builder.

The Vector Is Just the Beginning

Here's the mental model shift:

Vectors are how you represent knowledge. Memory is how you use it.

A vector tells you that two documents are semantically similar. But memory tells you:

When something last happened (episodic, via ES|QL time-series)
What it means in context (semantic, via ELSER)
How to respond (procedural, via Elastic Workflows)
Whether to trust the response (evidence gates, adversarial review, CONFIRM gates)

Elasticsearch in 2026 isn't just a vector database. It's the memory layer — the system that gives AI agents the ability to investigate, reason, question their own conclusions, refuse to act without evidence, and learn from every interaction.

The best agents don't just search. They remember.

🎥 Watch the Demo

See the complete Elasticsearch memory layer in action — live ES|QL queries, ELSER semantic search matching "null pointer exception" to "removed null safety checks" with zero keyword overlap, and the full agentic architecture walkthrough:

Ready to build? Start with Elastic Agent Builder and explore Elastic's vector search capabilities to create your own agentic memory layer.

Disclaimer: This Blog was submitted as part of the Elastic Blogathon.

Tags: #ElasticBlogathon #VectorizedThinking #AgentBuilder #ELSER #ESQL #RAG #AgenticAI #VectorSearch #SemanticSearch #VectorDB #VectorSearchwithElastic #ElasticWorkflows

ClimateIQ AI – 11 Mux Features Power Climate Action for 2B+ Users

omkar — Mon, 05 Jan 2026 07:35:59 +0000

This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux

What I Built

ClimateIQ AI is a production-grade climate intelligence platform transforming satellite data into actionable insights for farmers, cities, and communities worldwide.

🌍 The Full Platform (Beyond Video)

ClimateIQ already has deep climate features:

Module	Features
🌾 Smart Agriculture	Multi-agent AI Crop Advisor (SoilAgent + WeatherAgent + MarketAgent), Carbon Footprint Calculator (IPCC emission factors), Solar Irrigation Planner (NREL API)
🗺️ Climate Map	6 satellite layers (NASA FIRMS fires, AQI, temperature, precipitation, NREL solar, NDVI vegetation), AI Climate Alert System (heatwave/flood/AQI warnings)
🏙️ Urban Sustainability	Gemini Vision Waste Scanner, Smart Home Tracker (water/electricity), Solar ROI Calculator, Eco-Advisor with weather-based tips
📚 Climate Education	Storyblok CMS learning modules, AI-generated quizzes, climate news feed
♿ Accessibility	Keyboard shortcuts, voice navigation, high contrast themes

Tech Stack: Next.js 14, TypeScript, Supabase, Google Gemini AI, Mapbox, NASA FIRMS, NREL PVWatts, Planet API, SoilGrids

📹 NEW: Community Hub (Powered by Mux)

The missing piece was community-generated content. Rural communities couldn't share drought videos. Cities couldn't broadcast awareness campaigns. Climate educators needed multilingual reach.

Mux solved this with 11 integrated features:

📹 Climate Stories – Upload video testimonials with auto-generated captions
🌍 Climate Live – Low-latency streaming for real-time expert consultations
🌐 AI Translation – Captions in Hindi, Spanish, French for global reach
🎙️ Audio Dubbing – ElevenLabs voice cloning + Mux multi-track audio

My Pitch Video

Demo

🔗 Live Demo: https://climate-iq-git-feature-omkardongres-projects.vercel.app/

📦 GitHub: https://github.com/omkardongre/climate-iq/tree/feature

Test Credentials:

Email: test@gmail.com
Password: 12345678

🔓 Without Login: View videos, watch with captions, switch dubbed audio tracks, watch live streams

🔐 With Login: Upload videos, generate chapters, translate captions, dub audio, go live

How to Test Mux Features (All from the UI)

Upload Video: Go to Community Hub → Click "Share Your Story" button → Select a video → Watch it upload with progress bar
AI Chapters: After upload completes, click on your video → Click "Generate AI Chapters" button → See chapters appear below the player
Translate Captions: Click "Translate Captions" button → Select Hindi/Spanish from dropdown → After ~40 seconds, check the CC menu in the video player
Dub Audio: Click "Dub Audio" button → Select language → Wait ~2 minutes → Check the Audio/Language menu in the player settings (gear icon)
Live Streaming: Click "Go Live" button → Copy the RTMP URL and Stream Key → Use OBS or any streaming software to broadcast

The Story Behind It

The Climate Data Gap: Satellites collect terabytes of climate data daily. But farmers facing crop failures, cities battling pollution, and communities seeking education can't access it. ClimateIQ bridges this gap.

Why Video Was Missing:

A community in India showing drought damage creates more impact than statistics
Live broadcasts enable real-time expert consultations during crises
2B+ potential users need content in their language

Why Mux:

We needed auto-captions for accessibility
Low-latency for real-time consultations
Multi-language support for global reach
Production-ready infrastructure in days, not months

Technical Highlights

11 Mux Features Integrated:

#	Mux Feature	Implementation
1	`@mux/mux-player-react`	All video playback with ClimateIQ theming
2	`@mux/mux-uploader-react`	Direct uploads (no backend file handling)
3	Auto-Generated Captions	`generated_subtitles` in upload config
4	Live Streaming API	`muxVideo.liveStreams.create()`
5	Low Latency	`reduced_latency: true` for live broadcasts
6	Image API	Thumbnails + animated GIFs on hover
7	`@mux/ai` Chapters	Official SDK for chapter generation via Gemini
8	Text Tracks API	`assets.createTrack()` for translated captions
9	Audio Tracks API	`assets.createTrack({ type: 'audio' })` for dubs
10	Master Access	Temporary signed URLs for AI processing
11	Mux Data Analytics	Custom dimensions: `climate_topic`, `user_region`

Use of Mux (Additional Prize Category)

Mux isn't just hosting videos – it's the backbone of our entire community feature.

🎬 Beyond Basic Hosting

1. AI-Powered Chapters (@mux/ai SDK)

import { generateChapters } from '@mux/ai/workflows';
const chapters = await generateChapters(assetId, 'en', { provider: 'google' });

Climate videos get auto-chapters like "Impact on Agriculture" or "Mitigation Strategies" – helping users navigate long content.

2. Multi-Track Audio Dubbing

await muxVideo.assets.createTrack(assetId, {
  type: 'audio',
  url: supabaseAudioUrl,
  language_code: 'hi',
  name: 'AI Dubbed (hi)'
});

ElevenLabs generates dubbed audio → we add it to the Mux asset → users switch languages natively in the player.

3. Low-Latency Live Streaming

await muxVideo.liveStreams.create({
  reduced_latency: true,
  reconnect_window: 60,
  generated_subtitles: [{ language_code: 'en', name: 'English (auto)' }]
});

Users broadcast with low-latency. Experts provide real-time advice during live sessions.

4. Auto-Captions + AI Translation Pipeline

Upload with generated_subtitles → Mux generates English captions
Fetch VTT from https://stream.mux.com/{playbackId}/text/{trackId}.vtt
Gemini translates to Hindi/Spanish
Upload to Supabase + add via assets.createTrack()

📊 What Makes This Deep Integration

We're not just hosting videos. We're using:

Official @mux/ai SDK for chapter generation
Multi-Track Audio for dubbed languages
Text Track API for translated captions
Master Access for AI processing pipelines
Low-Latency Live Streaming for real-time advice
Custom Analytics Dimensions for climate insights

🌍 Real Impact

User	How Mux Helps
Rural Communities	Upload climate impact videos with auto-captions
Experts	Watch live broadcasts, give real-time advice
Global Audience	Content in Hindi, Spanish, French via AI translation
Non-Readers	Audio dubbing for accessibility

Mux handles encoding, CDN, multi-track, and AI – so we focus on climate impact.

❗ By submitting this project, I confirm that my video adheres to Mux's terms of service: https://www.mux.com/terms

AdForge AI - Enterprise Visual Production Platform

omkar — Fri, 02 Jan 2026 11:34:53 +0000

This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux

What I Built

AdForge AI is a complete enterprise visual production platform that transforms brand guidelines and campaign briefs into production-ready marketing assets at scale using an automated multi-agent AI pipeline.

Marketing teams spend weeks creating campaign visuals—coordinating designers, waiting for revisions, losing brand consistency across platforms. AdForge AI solves this with deterministic, controllable, and reproducible visual generation.

Key Capabilities:

🎨 AI-powered brand DNA extraction from minimal input
🚀 Automated multi-agent pipeline for asset generation
🎛️ JSON-native visual controls (camera, lighting, composition)
🔄 Natural language refinement without full regeneration
🛡️ Deterministic reproducibility with seeds
📤 Multi-destination export (PDF, Slack, HDR)

My Pitch Video

Demo

🔗 Live Demo: https://ad-forge-ai.vercel.app
🔗 GitHub: https://github.com/omkardongre/AdForge-AI

No login required - fully accessible for testing.

The Story Behind It

As a developer working with marketing teams, I watched them spend weeks creating campaign visuals - coordinating between designers, waiting for revisions, and losing brand consistency across platforms. It's slow, expensive, and broken.

Most AI tools just generate random outputs. Marketing teams need deterministic, controllable, and reproducible visual generation. That's why I built AdForge AI.

What Makes It Special

1. Multi-Agent Pipeline

Instead of a single AI call, AdForge uses specialized agents working together:

Brand DNA Extractor: Analyzes minimal input to create complete brand guidelines
Scene Composer: Creates visual concepts from campaign briefs
JSON Generator: Builds structured prompts with camera/lighting/composition parameters
Variation Generator: Creates multiple versions with parameter variations
Quality Assurance: Validates brand compliance

2. JSON-Native Visual Controls

Unlike traditional AI tools that regenerate everything from scratch, AdForge gives you direct control over generation parameters:

Camera Settings: Angle (eye-level, low, high, bird's eye), FOV (10-180°), focal length (24mm-85mm), distance
Lighting Settings: Setup (studio soft, dramatic, golden hour, natural), direction (front, side, back), intensity (0-2), color temperature
Color Settings: Brand color palette integration, temperature, saturation (0-2), contrast (0-2)
Composition Settings: Rule of thirds, depth of field, framing, negative space

Camera angle, lighting setup, and light direction use dropdown selectors. Saturation and contrast use visual sliders. All parameters map to structured JSON sent to the generation API.

3. Deterministic Reproducibility

Every generated image has a reproducibility seed. Same seed + same JSON = identical output. Enterprises can recreate approved assets months later with pixel-perfect accuracy.

Reproducibility Proof Feature:

Every asset displays a Shield icon showing its seed value
Copy seed → paste in Visual Controls → Regenerate = identical image
Proves deterministic output capability

4. Three Generation Modes

Generate Mode:

Text-to-image with structured JSON prompts
5 camera presets (product, lifestyle, portrait, wide, dramatic)
5 lighting presets automatically applied based on scene analysis

Refine Mode:

Iterative modifications via natural language
Example: "make the lighting warmer" or "change camera angle to low"
Updates only specified parameters while preserving everything else
Pre-built instruction examples in UI

Inspire Mode (Vision AI-Powered):

Upload reference image OR enter creative direction OR both
Gemini Vision AI analyzes reference images to extract:
- Subject attributes, lighting conditions, shadows
- Composition, color scheme, mood/atmosphere
- Camera angle, depth of field, focal length
Generates new images inspired by reference style

5. AI-Powered Campaign Analysis

Select any two campaign assets for side-by-side comparison. Gemini AI analyzes both with full campaign context:

Brand alignment assessment
Campaign objective fit
Technical quality scores (1-10)
Lighting, composition, color/mood comparison
Winner recommendation with reasoning

Core Features

Dashboard

Real-time statistics showing total brands, active campaigns, and generated assets. Quick Actions for creating brands, campaigns, and viewing assets.

4-Step Brand Wizard

Guided workflow (Brand Info → Colors → Visual Style → Review):

Enter minimal brand information
Pick 3 brand colors - AI expands into complete palette (primary, secondary, background, text)
Define visual style preferences
Review and confirm brand DNA
LLM expands minimal input into complete brand guidelines

Campaign Creation

Select existing brand
Set campaign name, objective (awareness/conversion), key message, call-to-action
Choose target platforms: Instagram Feed/Story, Facebook Feed, LinkedIn Post, Google Display
Configure generation settings (scenes count, variations per scene)

Multi-Platform Generation

Automatically generates assets at correct dimensions for each selected platform:

Instagram: 1080×1080 (Feed), 1080×1920 (Story)
LinkedIn: 1200×627
Facebook: 1200×630
Google Display: Various standard sizes

Asset Gallery

Grid view of all generated assets
Quality scores for brand compliance
Platform dimension badges
Action buttons for each asset (Refine, Compare, Download, Export)

Export Panel

Multi-Destination Export:

PDF Storyboard: Professional PDF document with all campaign assets
Slack Notifications: Team alerts when exporting assets

Print-Ready Format Export:

TIFF: 300 DPI with enhanced sharpness and contrast for professional print
PNG: High-quality with DPI metadata
JPEG: 300 DPI high-quality compression

Technical Highlights

User Journey

AdForge AI follows a structured 5-stage workflow: **Onboarding* → Campaign → Generate → Refine → Export*

Backend (Python/FastAPI)

Multi-Agent Architecture:
5 specialized AI agents working in pipeline:

Brand DNA Extractor: Uses LLM (Gemini/OpenAI/Anthropic) to extract complete brand identity
Scene Composer: Creates visual scene descriptions from campaign briefs
JSON Generator: Builds structured prompts with camera/lighting/color/composition parameters
Variation Generator: Creates parameter-sweep variations
Quality Assurance: Validates brand compliance, calculates quality scores

Technology Stack:

FastAPI for async/await API endpoints
Bria API for structured JSON-based image generation
Google Gemini for Vision AI (Inspire mode, AI Compare) and LLM orchestration
SQLite with async support for database
ReportLab for PDF generation
slack_sdk for team notifications

Frontend (React/Vite/TypeScript)

UI Pages and Modals:

Pages: Dashboard, Brands List, Brand Detail, Brand Wizard, Campaigns List, Campaign Detail, Campaign Create, Gallery, Export Panel, Settings, AI Compare, Semantic Search
Modals: Refine Modal, Inspire Modal, Visual Controls (JSON Editor)

Key UI Components:

4-step Brand Wizard with color pickers and visual style selectors
Campaign creation with brand selector and platform checkboxes
Asset gallery with quality score badges
Visual Controls modal with parameter sliders
AI Compare page with side-by-side analysis
ConsistencyBadge for reproducibility proof
Export panel with multi-destination options

Technology Stack:

React 18 with TypeScript
Vite for fast development and builds
TailwindCSS for styling
React Router for navigation
Axios for API communication

Challenges I Ran Into

JSON Schema Design: Designing the precise JSON structure for structured image generation required extensive experimentation with the Bria API's capabilities and constraints.
Multi-Platform Aspect Ratios: Generating consistent brand visuals across multiple platform dimensions while maintaining composition quality was challenging. Solved with platform-specific camera presets that adjust framing based on aspect ratio.
Vision-Based Inspire Mode: The Bria API doesn't directly accept image URLs. Integrated Gemini Vision AI to analyze reference images and build structured prompts from the visual analysis.
Reproducibility Implementation: Implementing seed-based deterministic generation with a user-friendly UI for copying and reusing seeds across different generation sessions.
Brand Consistency: Ensuring generated assets maintained brand compliance across different scenes and variations. Built the Quality Assurance agent to validate brand DNA adherence.

Accomplishments I'm Proud Of

All 3 Generation Modes Fully Integrated: Generate, Refine, and Inspire working end-to-end with production-quality results
JSON-Native Control System: Visual sliders for every generation parameter (camera angle, lighting, saturation, contrast) directly mapped to API parameters
Vision AI Integration: Gemini analyzes reference images to extract visual characteristics and build generation prompts
Campaign-Aware AI Compare: Context-aware asset comparison that considers brand guidelines and campaign objectives
Reproducibility Proof: Every asset has a verifiable seed for exact recreation, with UI indicators
Complete Multi-Agent Pipeline: 5 specialized agents working together seamlessly from brief to production assets
Production Deployment: Fully deployed and working live on Vercel (frontend) and Render (backend)
Multi-Destination Export: PDF storyboards and Slack notifications working in production

What I Learned

Determinism is Critical for Enterprise AI: Random outputs don't work in professional workflows. Reproducible generation with seeds is essential for brand consistency and client approvals.
Multi-Agent Systems Are Powerful: Breaking complex creative workflows into specialized agents produces better results than monolithic AI calls. Each agent can focus on its specific task with clear inputs/outputs.
Vision AI Enables New Workflows: Gemini's image analysis unlocks reference-based generation and intelligent asset comparison, bridging the gap between traditional design workflows and AI generation.
JSON-Native Control Changes Everything: Direct parameter control (vs. prompt engineering) is a game-changer for enterprise workflows. Designers need precise, predictable control.
Real-World Tools Need Export: Production tools must integrate with existing workflows (PDF, Slack, print formats) to be truly useful. Generation is only half the story.

What's Next

Real-time Collaboration: Multi-user editing with live preview and commenting
A/B Test Analytics: Performance analytics on generated variations with engagement metrics
DAM Integration: Connect with existing Digital Asset Management systems (e.g., Bynder, Widen)
Batch Processing: Generate hundreds of variations in parallel for large-scale campaigns
Enhanced Semantic Search: Natural language asset search using vector embeddings (ChromaDB)
Version Control: Git-like versioning for brand guidelines and campaign changes
API Access: RESTful API for programmatic asset generation

ClimateIQ - AI Acceleration

omkar — Mon, 08 Dec 2025 07:34:21 +0000

This is a submission for the AI Challenge for Cross-Platform Apps - AI Acceleration

What I Built

I built ClimateIQ — a comprehensive climate intelligence platform that demonstrates how AI coding assistants can accelerate the development of complex, production-quality cross-platform applications.

APIs Used:

Google Gemini AI — Powers climate alerts, crop recommendations, eco-tips, waste scanning
NASA FIRMS — Real-time fire/thermal anomaly data
OpenWeather API — Temperature, air quality, precipitation data
NREL PVWatts — Solar potential calculations
Planet Labs — Vegetation health (NDVI) data
Storyblok CMS — Community content, events, learning modules

The Challenge: Build a feature-rich climate app with 15+ tools, 6 real-time data layers, AI integrations, and CMS-powered content — all running on multiple platforms from a single codebase.

The Solution: Leverage AI coding assistants with Uno Platform MCP for contextual, grounded guidance throughout development.

Demo

🔗 GitHub Repository: https://github.com/omkardongre/ClimateIQ-App

🌐 Live WebAssembly Demo: roaring-gumption-b8bf96.netlify.app

AI Development Screenshots

IDE with AI Assistant

Google Antigravity IDE with Gemini AI researching Mapbox integration and recommending Mapsui as the best map solution for Uno Platform

Windsurf IDE with Cascade AI fixing XAML build errors and adding Uno Toolkit to multiple pages simultaneously

Uno Platform MCP in Action

Uno Platform MCP searching documentation for Material Controls Styles and Toolkit UI components

Uno Platform MCP fetching Material Toolkit theme setup guidance while AI updates App.xaml resources

App Screenshots

Home Page - Feature Discovery

Interactive Climate Map - 6 Data Layers

Real-time visualization with NASA FIRMS fire data, air quality, flood risk

AI Climate Alerts - Gemini Powered

Personalized alerts with severity indicators and actionable recommendations

Smart Agriculture Hub - Multi-Agent AI

AI Crop Advisor

Carbon Calculator

Solar Irrigation Calculator

Urban Sustainability Hub

Waste Scanner, Solar Savings Calculator, AI Eco-Advisor, Smart Home Tracker

Waste Scanner

Solar Savings Calculator

AI Eco-Advisor

Community Hub - Storyblok CMS

Environmental events, news, learning modules

Cross-Platform Testing

Linux Desktop (Ubuntu)

ClimateIQ running as a native desktop app on Ubuntu Linux with Skia renderer

WebAssembly (Browser)

Same app running in the browser via WebAssembly, deployed to Netlify

AI Tooling in Action

AI Agents Used

Windsurf IDE with Claude — Primary AI coding assistant for code generation, debugging, and architecture
Google Antigravity IDE with Gemini 3.0 Pro — Additional AI assistance for rapid prototyping and API research

I used AI coding assistants throughout the entire development process. Here's how AI accelerated my workflow:

1. Uno Platform MCP Integration

The Uno Platform MCP Server provided contextual, grounded guidance for:

XAML layout patterns and best practices
Cross-platform compatibility considerations
Material Design integration with Uno Toolkit
Navigation patterns and state management
Platform-specific adaptations

Example Interaction:

Me: "How do I create a responsive card layout with shadows?"
MCP: [Provided specific Uno Platform guidance on ThemeShadow,
      Border styling, and responsive Grid layouts]

2. Code Generation Acceleration

Before AI: Manually writing 100+ XAML files, ViewModels, Services, and Models would take weeks.

With AI: The AI assistant generated production-quality code efficiently:

Component	Approximate Lines	AI Contribution
XAML Pages	~3,000 lines	~90% AI-generated, human-refined
ViewModels	~2,500 lines	~85% AI-generated
Services	~2,000 lines	~80% AI-generated
Models	~500 lines	~95% AI-generated

3. Real-Time Problem Solving

Challenge: Emojis not rendering on Linux Skia renderer.

AI Solution: The AI researched the issue, identified the root cause (font fallback limitations), and implemented a comprehensive fix — replacing all emojis with styled text badges across 10+ pages in a single session.

Challenge: Complex multi-step wizard for Solar Savings Calculator.

AI Solution: The AI designed the 4-step wizard architecture, implemented progress tracking, and integrated NREL API calls — all following Uno Platform best practices from MCP guidance.

Challenge: JSON deserialization failing in WebAssembly Release builds.

AI Solution: The AI identified the JsonSerializerIsReflectionDisabled error caused by .NET trimming, and replaced all reflection-based JSON operations with manual JsonDocument parsing across 7 service files.

4. API Integration Patterns

The AI helped integrate 6 different APIs with proper:

Error handling and fallbacks
Rate limiting considerations
Response parsing and mapping
Caching strategies

// Example: AI-generated NASA FIRMS integration
public async Task<IEnumerable<ClimateDataPoint>> GetFireDataAsync(double lat, double lon)
{
    // AI generated this with proper error handling,
    // CSV parsing, and data point mapping
}

5. UI/UX Refinement

The AI helped maintain consistent design patterns:

Gradient headers on every page
Card-based layouts with proper spacing
Accessible color contrasts
Responsive breakpoints

Key AI Acceleration Metrics

Metric	Traditional Estimate	With AI
Initial prototype	~2 weeks	~2 days
Full feature set	~2 months	~2 weeks
Bug fixes	Hours each	Minutes each
Cross-platform testing	Days	Hours

MCP-Grounded Development

The Uno Platform MCP ensured that AI suggestions were:

✅ Compatible with Uno Platform's Skia renderer
✅ Following MVVM patterns correctly
✅ Using proper XAML syntax for cross-platform
✅ Leveraging Uno Toolkit components appropriately

Targets

ClimateIQ runs on multiple platforms from a single codebase:

Platform	Framework	Status
🪟 Windows	net9.0-desktop	✅ Working
🐧 Linux	net9.0-desktop (Skia)	✅ Working
🍎 macOS	net9.0-desktop	✅ Builds
🌐 WebAssembly	net9.0-browserwasm	✅ Working

Build Commands

# Desktop (Linux/Windows/macOS)
dotnet run -f net9.0-desktop

# WebAssembly
dotnet run -f net9.0-browserwasm

Development Experience

The AI development experience was transformative:

AI + MCP = Grounded Intelligence — The Uno Platform MCP kept AI suggestions relevant and accurate
Iterative Refinement Works — Quick feedback loops with AI accelerated learning
Complex Apps Are Achievable — What seemed like months of work became weeks
Cross-Platform Is Real — Write once, run everywhere actually works with Uno Platform

Built with Uno Platform, .NET 9, Google Gemini AI, Windsurf AI Assistant, and a passion for climate action. 🌍

ClimateIQ - WOW Factor

omkar — Mon, 08 Dec 2025 07:02:30 +0000

This is a submission for the AI Challenge for Cross-Platform Apps - WOW Factor

What I Built

I built ClimateIQ — an AI-powered climate intelligence platform that combines real-time satellite data, smart agriculture tools, and urban sustainability features into a stunning, cohesive experience. Think of it as a "climate command center" that empowers communities to understand, adapt to, and combat climate change.

The theme is environmental sustainability meets modern design — using a calming green gradient palette with clean card-based interfaces that make complex climate data accessible and beautiful. Every screen is designed to make users feel empowered rather than overwhelmed by climate information.

What makes it special:

6 Real-Time Data Layers on an interactive climate map (Temperature, Air Quality, Flood Risk, Solar Potential, Vegetation Health, Thermal Anomalies)
AI-Powered Features using Google Gemini for personalized climate alerts, crop recommendations, and eco-tips
Production-Quality UI with smooth animations, gradient headers, card shadows, and responsive layouts
Real API Integrations — NASA FIRMS, OpenWeather, NREL Solar, Planet Labs, Storyblok CMS

Demo

🔗 GitHub Repository: https://github.com/omkardongre/ClimateIQ-App

🌐 Live WebAssembly Demo: roaring-gumption-b8bf96.netlify.app

Screenshots

Home Page - Feature Discovery

Interactive Climate Map - 6 Data Layers

Real-time visualization with NASA FIRMS fire data, air quality, flood risk

AI Climate Alerts - Gemini Powered

Personalized alerts with severity indicators and actionable recommendations

Smart Agriculture Hub - Multi-Agent AI

AI Crop Advisor

Carbon Calculator

Solar Irrigation Calculator

Urban Sustainability Hub

Waste Scanner, Solar Savings Calculator, AI Eco-Advisor, Smart Home Tracker

Waste Scanner

Solar Savings Calculator

AI Eco-Advisor

Community Hub - Storyblok CMS

Environmental events, news, learning modules

Cross-Platform Testing

ClimateIQ runs natively on multiple platforms from a single C#/XAML codebase. Here's proof of cross-platform functionality:

Linux Desktop (Ubuntu)

ClimateIQ running as a native desktop app on Ubuntu Linux with Skia renderer

WebAssembly (Browser)

Same app running in the browser via WebAssembly, deployed to Netlify

Platform Support

Platform	Status	Notes
✅ Windows	Working	Full desktop experience
✅ Linux	Working	Skia renderer, tested on Ubuntu
✅ macOS	Builds	Native macOS app (same codebase)
✅ WebAssembly	Working	Browser-based, deployed to Netlify

The Magic Moment: I wrote the interactive climate map visualization once — with custom data point rendering, layer toggles, and real-time API calls — and it worked identically across desktop and web. The Uno Platform's Skia renderer ensured pixel-perfect consistency.

# Build for any platform from the same codebase
dotnet run -f net9.0-desktop      # Linux/Windows/macOS
dotnet run -f net9.0-browserwasm  # WebAssembly

Interactive Features

🗺️ Interactive Climate Map

6 toggleable data layers with real-time API data
Click-to-inspect data points with detailed info cards
AI Analysis button generates Gemini-powered climate risk assessment
Search area to fetch data for any location

🤖 AI-Powered Tools

Multi-Agent Crop Advisor — 3 AI specialists (Soil, Weather, Market) analyze your farm
AI Waste Scanner — Upload photos to identify waste type and get disposal instructions
AI Eco-Advisor — Personalized eco-tips based on your location and weather

📊 Calculators & Trackers

Solar Savings Calculator — 4-step wizard with NREL data integration
Carbon Footprint Calculator — Farm emissions tracking with AI recommendations
Smart Home Tracker — Electricity and water usage monitoring with appliance breakdown
Solar Irrigation Calculator — Pump sizing and cost analysis for farmers

📚 Learning & Community

Interactive Learning Modules — Climate basics and sustainability practices with lesson progression
Community Events — Storyblok CMS integration with event images and registration

🎨 Visual Polish

Gradient headers on every page
Card shadows using ThemeShadow
Smooth navigation with back buttons
Responsive layouts that adapt to screen size
Styled icon badges for cross-platform consistency

The Wow Factor

The moment that makes people say "Wow": Opening the Interactive Climate Map and seeing 6 different real-time data layers — fire hotspots from NASA satellites, air quality readings, flood risk zones, solar potential, vegetation health — all visualized on a single map, fetched from real APIs, running in a browser or on your desktop from the same codebase.

What I'm most proud of:

Real Data, Real Impact — This isn't mock data. Every data point comes from production APIs (NASA FIRMS, OpenWeather, NREL, Planet Labs). Users can actually use this to understand their local climate conditions.
AI That Helps — The Gemini integration provides genuinely useful recommendations. The Multi-Agent Crop Advisor doesn't just give generic advice — it considers your specific location, soil type, and market conditions.
Production-Quality UI — Every screen looks polished. The gradient headers, card shadows, consistent spacing, and thoughtful color palette create an experience that feels like a commercial product, not a hackathon project.
Comprehensive Feature Set — 5 major hubs, 10+ individual tools, real API integrations, CMS-powered content — all from a single Uno Platform codebase running on multiple platforms.

The "Wow" is the combination: Beautiful design + Real data + AI intelligence + Cross-platform magic = A climate app that actually makes you want to take action.

Built with Uno Platform, .NET 9, Google Gemini AI, and a passion for climate action. 🌍

From $50K Reports to AI Automation: Securing Multi-Agent ESG Compliance with Auth0

omkar — Mon, 27 Oct 2025 06:02:08 +0000

This is a submission for the Auth0 for AI Agents Challenge

What I Built

ESG Copilot is an autonomous multi-agent AI system that automates ESG (Environmental, Social, Governance) compliance for small and medium businesses, secured end-to-end with Auth0 for AI Agents.

The Problem

SMBs face a $10K-$50K cost for ESG reports, struggle with complex regulations across jurisdictions, and lack in-house expertise. Manual compliance is time-consuming and error-prone.

The Solution

An autonomous AI agent system that:

🔍 Researches regulations via real-time web search (Google Search grounding)
📊 Collects ESG data from EPA, web scraping, and AI estimation
🌍 Calculates emissions using Climatiq API (Scope 1, 2, 3)
📄 Generates reports in GRI, SASB, TCFD formats with iterative refinement
💬 Answers questions via permission-aware RAG (Retrieval-Augmented Generation)

All secured by Auth0 - every agent action is authenticated, authorized, and audited.

Demo

🔗 GitHub Repository

github.com/omkardongre/ESG-Copilot

🔗 Live Demo

https://esg-copilot.vercel.app/

📸 Screenshots

Homepage

Company Dashboard

Regulation Research Agent

ESG Data Collection Agent

Emissions Calculator Agent

Report Generator Agent

Chat Agent (RAG)

Generated Report

Auth0 Users

Token Vault in Action

Auth0 FGA Store

▶️ Try It Yourself

Test User Accounts (Password: Test@1234):

Email	Role	Company	Agent Access	Document Access	Chat Context
`patagonia@company.com`	Company Admin	Patagonia Inc	All 5 agents	Own company only	Patagonia docs
`tesla@company.com`	Company Admin	Tesla Inc	All 5 agents	Own company only	Tesla docs
`consultant@esgfirm.com`	ESG Consultant	N/A	All 5 agents	All companies	All docs
`auditor@sustainapilot.com`	Auditor	N/A	Read-only	All companies	All docs (read-only)
`regulator@epa.gov`	Regulator	N/A	Read-only	All companies	All docs (read-only)

Try Different Scenarios:

Login as Patagonia Admin → Execute agents, see only Patagonia data
Login as Tesla Admin → Execute agents, see only Tesla data (zero data leakage)
Login as ESG Consultant → Execute agents for any company, see all data
Login as Auditor → Read-only access, cannot execute agents
Login as Regulator → Read-only access, view all companies for compliance oversight

📋 Complete Access Rules Matrix

Agent Execution Permissions:

Agent	Company Admin	ESG Consultant	Auditor	Regulator
Regulation Research	✅ Own company	✅ Any company	❌	❌
ESG Data Collection	✅ Own company	✅ Any company	❌	❌
Emissions Calculator	✅ Own company	✅ Any company	❌	❌
Report Generator	✅ Own company	✅ Any company	❌	❌
Chat Agent (RAG)	✅ Own docs	✅ All docs	✅ All docs	✅ All docs

Document Access Control:

Document Type	Company Admin	ESG Consultant	Auditor	Regulator
Own Company ESG Data	✅ Read/Write	✅ Read/Write	✅ Read-only	✅ Read-only
Other Company ESG Data	❌ No access	✅ Read/Write	✅ Read-only	✅ Read-only
Compliance Reports	✅ Own company	✅ All companies	✅ All companies	✅ All companies
Audit Logs	✅ Own company	✅ All companies	✅ All companies	✅ All companies

Token Vault API Key Distribution:

API Key	Company Admin	ESG Consultant	Auditor	Regulator
Climatiq (Emissions)	✅	✅	❌	❌
Pinecone (RAG)	✅	✅	✅	✅
SendGrid (Email)	✅	✅	❌	❌

Example Scenarios:

Cross-Company Privacy:

Patagonia Admin asks chat: "Show me Nike's emissions data"
AI Response: "No relevant documents found" (FGA blocks access)
Audit: Query logged but no Nike documents accessed

Role-Based Agent Restrictions:

Auditor tries to execute "Calculate Emissions" agent
Result: 403 Forbidden (no execute:agents permission)
Audit: Attempt logged with failure reason

Token Vault Security:
- Auditor tries to access Climatiq API
- Result: API key not in JWT (Token Vault blocks)
- Audit: No API key injection for read-only roles

How I Used Auth0 for AI Agents

1. 🔐 Authenticate the User

Challenge Requirement: Secure the human prompting the agents

Implementation:

// frontend/src/app/api/auth/[auth0]/route.ts
import { handleAuth } from "@auth0/nextjs-auth0";

export const GET = handleAuth();

Features:

✅ Auth0 Universal Login (no custom auth code)
✅ JWT tokens with user identity, roles, and permissions
✅ 4 roles: Company Admin, ESG Consultant, Auditor, Regulator
✅ 9 permissions: execute:agents, read:companies, write:reports, etc.
✅ Every agent action tied to authenticated user

Why This Matters: Each AI agent execution is traceable to a specific user with specific permissions. No anonymous agent actions.

2. 🔑 Control the Tools

Challenge Requirement: Manage which APIs your agents can call

Implementation:

// Auth0 Action: Token Vault (Post-Login)
exports.onExecutePostLogin = async (event, api) => {
  const namespace = "https://esg-copilot.com";

  // Add email and roles to custom claims
  api.accessToken.setCustomClaim(`${namespace}/email`, event.user.email);
  api.accessToken.setCustomClaim(`${namespace}/roles`, event.authorization?.roles || []);

  // Check if user has agent execution permission
  const permissions = event.authorization?.permissions || [];
  const roles = event.authorization?.roles || [];

  const hasAccess =
    permissions.includes("execute:agents") ||
    roles.includes("ESG Consultant") ||
    roles.includes("Company Admin");

  if (hasAccess) {
    // Inject API keys into JWT from Auth0 Secrets
    const apiKeys = {
      climatiq_api_key: event.secrets.CLIMATIQ_API_KEY,
      sendgrid_api_key: event.secrets.SENDGRID_API_KEY,
      pinecone_api_key: event.secrets.PINECONE_API_KEY,
    };

    api.accessToken.setCustomClaim(`${namespace}/api_keys`, apiKeys);
  }

  // Add company_id for Company Admins
  api.accessToken.setCustomClaim(`${namespace}/company_id`, 
    event.user.user_metadata?.company_id || null);
};

Backend Extraction & Usage:

// src/middleware/auth0.js - Extract from JWT
const extractUserInfo = (req, res, next) => {
  if (req.auth?.payload) {
    const apiKeys = req.auth.payload['https://esg-copilot.com/api_keys'] || {};

    req.user = {
      id: req.auth.payload.sub,
      email: req.auth.payload['https://esg-copilot.com/email'],
      roles: req.auth.payload['https://esg-copilot.com/roles'] || [],
      companyId: req.auth.payload['https://esg-copilot.com/company_id'],
      permissions: req.auth.payload.permissions || [],
      api_keys: apiKeys, // ✅ Available to all agents
    };
  }
  next();
};

// src/routes/chat.js - Agent retrieves keys from Token Vault
const apiKeys = req.user.api_keys || {};
const pineconeApiKey = apiKeys.pinecone_api_key;

if (!pineconeApiKey) {
  return res.status(400).json({
    error: 'Pinecone API key not found in Token Vault'
  });
}

Features:

✅ 3 API keys secured in Token Vault (Climatiq, SendGrid, Pinecone)
✅ Keys only injected for authorized users
✅ No hardcoded credentials anywhere
✅ Agents retrieve keys from JWT at runtime

Why This Matters: Agents can only call APIs if the user has permission. A Company Admin can calculate emissions, but an Auditor cannot.

3. 🎯 Limit Knowledge

Challenge Requirement: Ensure agents only access authorized data

Implementation:

// src/services/fga-store-service.js
const { OpenFgaClient } = require("@openfga/sdk");

class FGAStoreService {
  constructor() {
    this.client = new OpenFgaClient({
      apiUrl: process.env.FGA_API_URL,
      storeId: process.env.FGA_STORE_ID,
      credentials: {
        method: "client_credentials",
        config: {
          apiTokenIssuer: "auth.fga.dev",
          apiAudience: process.env.FGA_API_AUDIENCE,
          clientId: process.env.FGA_CLIENT_ID,
          clientSecret: process.env.FGA_CLIENT_SECRET,
        },
      },
    });
  }

  // Check if user can view a document
  async canViewDocument(userId, documentId) {
    const { allowed } = await this.client.check({
      user: `user:${userId}`,
      relation: "can_view",
      object: `document:${documentId}`,
    });

    return allowed;
  }
}

Authorization Model (FGA Store):

type user

type company
  relations
    define admin: [user]
    define member: [user]
    define viewer: [user] or member or admin

type document
  relations
    define company: [company]
    define owner: [user]
    define viewer: [user] or owner or viewer from company
    define can_view: viewer

RAG Pipeline with FGA:

// src/services/rag-service.js - Two-layer authorization
async queryWithFGAStore({ query, userId, userEmail, companyId, topK = 5 }) {
  // Step 1: Generate embedding
  const queryEmbedding = await this.embeddings.embedQuery(query);

  // Step 2: Query Pinecone with company filter (first layer)
  const results = await this.index.query({
    vector: queryEmbedding,
    topK: topK * 3, // Fetch more, then filter by FGA
    filter: { company_id: { $eq: companyId } },
    includeMetadata: true,
  });

  // Step 3: Filter by FGA Store authorization (second layer)
  const authorizedDocuments = [];

  for (const match of results.matches) {
    const documentId = match.metadata.document_id;

    // Real-time FGA check via Auth0 FGA Store API
    const canView = await fgaStoreService.canViewDocument(userEmail, documentId);

    if (canView) {
      authorizedDocuments.push(match);
    }
  }

  return authorizedDocuments.slice(0, topK);
}

Features:

✅ Auth0 FGA Store for fine-grained authorization
✅ Company-level data isolation (Pinecone filters)
✅ Document-level access control (FGA checks)
✅ Real-time authorization on every query
✅ Zero knowledge leakage between companies

6 Authorization Rules:

Company Admin: See only own company documents
ESG Consultant: See all companies (global viewer)
Auditor: Read-only access to all documents
Regulator: Read-only access to all documents
Document ownership: Creator is owner
Company-document relationship: Documents belong to companies

Why This Matters: A Company Admin for Patagonia cannot see Nike's ESG data. The AI agent respects these boundaries automatically.

Technical Architecture

Tech Stack

Frontend:

Next.js 14 (App Router)
Auth0 Next.js SDK (@auth0/nextjs-auth0)
TailwindCSS

Backend:

Node.js + Express
Google Gemini 2.5 Flash (with Search grounding)
LangChain.js (RAG pipeline)

Data & Storage:

BigQuery (agent logs, ESG data, compliance requirements)
Pinecone (vector database for RAG)
Auth0 FGA Store (external authorization service)

External APIs (via Token Vault):

Climatiq API (carbon footprint calculations)
SendGrid API (email delivery)
Pinecone API (vector search)

Multi-Agent Architecture

5 Core Agents:

Regulation Research Agent (3 sub-agents in parallel)

Jurisdiction Analyzer (web search)
Framework Mapper (GRI/SASB/TCFD)
Deadline Calculator (compliance dates)

ESG Data Collection Agent (3 sub-agents in parallel)

EPA Data Collector (government data)
Web Scraper (company sustainability pages)
AI Estimator (industry benchmarks)

Emissions Calculator Agent

Scope 1, 2, 3 emissions via Climatiq API
Activity data extraction from ESG metrics

Report Generator Agent (4 sub-agents + iterative refinement)

GRI Writer, SASB Writer, TCFD Writer
Review & Critique Agent (up to 3 iterations)

Chat Agent (RAG-based)
- Permission-aware document retrieval
- Company-scoped knowledge base
- Real-time FGA authorization

Agentic Patterns:

✅ Parallel fan-out/gather (regulation research)
✅ Iterative refinement (report quality loop)
✅ Multi-agent orchestration (service-level coordination)

Lessons Learned and Takeaways

🎓 Key Insights

1. Auth0 FGA Store is Production-Ready

External authorization service (not just role logic)
Real-time checks with low latency
Scales to millions of tuples
No need to build custom RBAC

2. Token Vault Simplifies Agent Security

API keys in JWT = no backend secret management
Permission-based key injection
Agents are stateless and secure

3. Multi-Agent Systems Need Orchestration

Service-level coordination > agent-to-agent messaging
Parallel execution for independent tasks
Sequential execution for dependent tasks

🚧 Challenges Faced

1. FGA Store Learning Curve

Understanding tuple relationships took time
Authorization model design is critical
Testing with FGA Query Tool was essential

2. RAG Authorization Performance

Initial approach: Check every document (slow)
Solution: Pinecone filters + FGA checks (fast)
Result: Smaller query time with 10+ documents

3. Agent Debugging

BigQuery audit trail provides complete action history
Structured logging helps identify agent execution flow

💡 Key Takeaways

For Developers:

Start with Auth0 Universal Login (don't build custom auth)
Use FGA Store for document-level permissions
Token Vault is perfect for AI agent API keys
Audit logging is non-negotiable for production

For AI Agent Builders:

Permission-aware RAG prevents data leakage
Multi-agent orchestration needs clear patterns
Real-time web search > static knowledge bases
Iterative refinement improves output quality

🎯 What I'm Proud Of

✅ Agent API keys via Token Vault - Climatiq, SendGrid, Pinecone secured in JWT
✅ Production-ready security - FGA Store, audit logs, JWT, environment variables
✅ Real-world use case - solves actual SMB pain point
✅ No dummy data - live web search, real APIs
✅ 5 agents + 9 sub-agents - complex multi-agent system

Conclusion

ESG Copilot demonstrates how Auth0 for AI Agents enables production-ready AI systems with:

🔐 User authentication (Universal Login + JWT with roles/permissions)
🔑 Tool control (Token Vault for secure API key management)
🎯 Knowledge limits (FGA Store + two-layer RAG authorization)

The result: An autonomous multi-agent AI system that SMBs can trust with sensitive ESG data, knowing every action is authenticated, authorized, and audited.

Key Numbers:

5 core agents + 9 sub-agents
4 roles with 9 permissions
3 API keys secured via Token Vault
2-layer authorization (Pinecone filters + FGA checks)

Try it yourself: GitHub Repository

🙏 Acknowledgments

Thanks to Auth0 and DEV Community for this challenge! Building with Auth0 for AI Agents transformed how I think about securing multi-agent systems. The combination of Token Vault and FGA Store makes production-ready AI security achievable without building custom infrastructure.

MediVision Assistant

omkar — Sun, 14 Sep 2025 18:28:01 +0000

This is a submission for the Google AI Studio Multimodal Challenge

What I Built

MediVision Assistant - An AI-powered healthcare companion that makes medical assistance accessible to everyone, especially those with visual impairments or accessibility needs. The app combines computer vision, voice recognition, and AI chat to provide comprehensive health monitoring and assistance.

Key Features:

🖼️ AI Skin Analysis - Upload photos and videos for instant skin condition assessment
🎨 AI Health Infographics - Generate professional medical infographics using Imagen 4.0
🎤 Voice Symptom Logger - Record and transcribe health symptoms using speech-to-text
💊 Medication Scanner - OCR-powered medication identification and management
💬 AI Health Chat - Conversational AI for health questions and guidance
🔗 Seamless Analysis-to-Chat Integration - Continue conversations with AI based on analysis results
♿ Full Accessibility Support - Voice navigation, screen reader compatibility, high contrast mode
📱 Progressive Web App - Works offline, installable on any device

Demo

Live Application: https://medivision.omkard.site (Custom domain mapped via Google Cloud Run)
Backup Link: https://medivision-assistant-968390101733.us-central1.run.app (Direct Cloud Run URL)

GitHub Repository: https://github.com/omkardongre/medi-vision-assistant-ai

Demo : https://youtu.be/kxGtnp9X_48?si=rvrUcb-HwdogB7pS

Screenshots

Homepage Dashboard: Clean, accessible dashboard with health summary and quick actions

Skin Analysis: AI-powered skin condition analysis with detailed insights

Voice Logger: Voice-to-text symptom recording with transcription

Health Chat: Conversational AI for health questions

AI Health Infographics: Professional medical infographics

Health Records

Accessibility Features: Comprehensive accessibility toolbar with voice navigation

How I Used Google AI Studio

I leveraged Google AI Studio extensively to power the multimodal capabilities:

1. Gemini 2.5 Flash for Skin Analysis (Image + Video)

Integrated Gemini's vision capabilities to analyze uploaded skin photos and videos
Provides detailed assessments of skin conditions, moles, rashes, and other dermatological concerns
Supports video analysis for dynamic skin condition monitoring and movement patterns
Returns structured health insights with confidence scores and recommendations
Supports multiple video formats (MP4, MOV, AVI, WebM) up to 25MB

2. Gemini 2.5 Flash for Health Chat

Powers the conversational AI health assistant
Processes natural language health questions and provides evidence-based responses
Maintains conversation context for follow-up questions

3. Imagen 4.0 for Health Infographics

Integrated Google Imagen 4.0 for professional medical infographic generation
Creates medication schedules, health progress charts, and symptom tracking visuals
Generates accessible, high-contrast infographics with professional medical styling
Supports download and sharing of AI-generated health content
Uses latest Imagen for cutting-edge image generation

4. Multimodal Integration

Combined text, image, video, voice, and AI-generated visual content for comprehensive health monitoring

Multimodal Features

🎥 Video + Text Analysis (Skin Analysis Page)

Video Skin Monitoring: Users upload videos for dynamic skin condition analysis and movement patterns
Symptom Documentation: Video recordings of skin symptoms for detailed medical assessment

🖼️ Image + Text Analysis

Skin Photo Analysis: Users upload photos of skin conditions, and Gemini analyzes them for potential health concerns
Medication OCR: Scans medication labels and bottles to extract drug information, dosages, and instructions

🎤 Voice + Text Processing

Voice Symptom Logger: Records audio descriptions of symptoms and converts them to structured text
Voice Navigation: Complete app navigation using voice commands ("go home", "skin analysis", "emergency")
Audio Feedback: Text-to-speech responses for accessibility

💬 Conversational AI

Contextual Health Chat: AI remembers previous conversations and provides personalized health guidance
Seamless Analysis Integration: After any analysis (skin, medication, voice logger), users can click "Discuss with AI Assistant" to continue the conversation with full context of their analysis results

♿ Accessibility-First Design

Screen Reader Compatible: Full ARIA labels and semantic HTML
Voice Commands: Navigate the entire app using voice ("skin analysis", "medication scanner", "help")
High Contrast Mode: Enhanced visibility for users with visual impairments
Font Scaling: Adjustable text size up to 300%
Keyboard Navigation: Complete app functionality without mouse

🎨 AI-Generated Visual Content

Health Infographics: Professional medical charts and schedules generated by Imagen 4.0
Medication Schedules: Visual medication timing and dosage charts
Progress Tracking: Health milestone and achievement visualizations
Symptom Charts: Color-coded symptom monitoring and tracking graphics
Download & Share: Export AI-generated infographics for medical consultations

🔄 Data Integration

Health Records: All multimodal inputs (videos, images, voice, chat, infographics) are stored and organized
Export Capabilities: Users can export their health data and AI-generated infographics for medical consultations
Video Storage: Secure video analysis results

Technical Implementation

Frontend: Next.js 15 with TypeScript and Tailwind CSS
AI Integration: Google AI Studio with Gemini 2.5 Flash (video, image, text, audio) and Imagen 4.0 (infographics)
Voice Processing: Web Speech API for speech-to-text and text-to-speech
Image Processing: Canvas API for image optimization and preprocessing
Deployment: Google Cloud Run with automatic scaling
Database: Supabase for health records and user data
Accessibility: WCAG 2.1 AA compliant with comprehensive testing

Impact & Accessibility

This project demonstrates how AI can make healthcare more accessible to everyone, particularly:

Visually impaired users who can navigate entirely by voice
Elderly users who may have difficulty with complex interfaces
Users with motor disabilities who rely on voice commands
Non-native speakers who can describe symptoms in their own words

The multimodal approach ensures that health monitoring is not limited by traditional input methods, making medical assistance truly inclusive.

Built with ❤️ for the Google AI Studio Multimodal Challenge