🚀Mission Accomplished: Building a Real-Time AI Spatial Agent | Vision Possible Hackathon & The Power of Learning by Building

Swagatika Beura — Sun, 01 Mar 2026 04:09:34 +0000

When curiosity meets constraints, innovation happens.

And when you choose to just learn by building and enjoy the journey — growth becomes unstoppable.

Recently, I built VisionMate AI as my submission for the Vision Possible: Agent Protocol Hackathon — an event that challenged developers worldwide to build cutting-edge generative AI solutions using multimodal agents.

Hosted by WeMakeDevs , the hackathon challenged participants to push the limits of what’s possible with real-time AI — from blazing-fast inference to true multimodal interactions.

And that’s exactly where VisionMateAI was born.

🚀 What is VisionMate AI?

Standard computer vision is entirely passive. If you are a cyclist, an e-scooter rider, or someone using a smart dashcam, knowing **"there is a car" **in a video feed is useless. You already know there are cars on the road!

The real problem is knowing if a specific car or bus is approaching too fast or getting dangerously close to you.

VisionMate AI is a comprehensive mobility platform that allows users to turn any standard camera into an active spatial hazard agent — transforming the way we handle personal safety on the road.

It’s built with Streamlit, OpenCV, and Python, powered by Stream's Vision Agents SDK and Ultralytics YOLOv8, running on the Gemini Realtime LLM for unparalleled inference speed and voice interaction.

🧠 How I Used Stream Vision Agents & Gemini Together :

At the core of ** VisionMate AI **lies a unique combination:

🧩 Ultralytics YOLOv8 for highly accurate entity detection and bounding box generation.
⚡ Stream Vision Agents SDK for lightning-fast edge network routing — ensuring real-time, snappy voice-to-vision pipelines.
🗣️ Gemini Realtime LLM for instant, conversational audio warnings.

⚙️ Architecture Highlights:

Instead of just identifying objects passively, VisionMate AI actively measures spatial threats across three core modes:

📷 1. Image Processing (High-Fidelity Frame Inspection)

Contextual safety thrives on precision. You simply upload an image, and the model instantly processes it to accurately detect, label, and draw bounding boxes around the objects present in the frame. It maps out the immediate environment, providing structured, high-quality safety responses in milliseconds.

🎬 2. Video Pipeline (Time-Series Hazard Tracking)

This feature is where the custom spatial math truly shines. You upload a pre-recorded dashcam video, and the system processes it frame-by-frame. It features a synchronous dashboard with a live telemetry HUD that successfully detects and tracks objects throughout the footage, exporting a final security audit in JSON format.

🎥 3. Live Protocol (Real-Time Agent Interface)

This is where the custom hackathon logic comes in. It features a Native Hardware Test for local proximity calculations and a WebRTC Edge Network. While the AI seamlessly detects everything in your live camera feed, it isolates true spatial threats in real-time.

I wrote the "40% Rule"—a custom proximity math algorithm inside the bounding box processor:


# The Core Spatial Hazard Math

def process_bounding_boxes(image, results):

    # ... setup code ...

    for box in results.boxes:

        x1, y1, x2, y2 = map(int, box.xyxy[0])

        vertical_span = (y2 - y1) / img_h # Object height relative to frame



        if vertical_span > 0.4:

            status, color, thickness = "HAZARD", (0, 0, 255), 4

        elif vertical_span > 0.2:

            status, color, thickness = "MID", (0, 255, 255), 2

        else:

            status, color, thickness = "FAR", (0, 255, 0), 2

If an object (like a car or bus) suddenly takes up more than 40% of the camera frame, my system mathematically determines it is an imminent collision risk and triggers the Gemini agent to issue an instant gTTS voice alert!

🧩 Tech Stack Overview :

Frontend: Streamlit, Custom CSS UI Components
Core Logic: Python, OpenCV, NumPy
Authentication: Bcrypt, JSON-based Local DB
AI & Vision: Stream Vision Agents SDK, Ultralytics YOLOv8, Gemini Realtime LLM
Audio: Google Text-to-Speech (gTTS)
Deployment/Environment: Dotenv, Nest_Asyncio

⚡ The Experience — When Curiosity Met Constraints :

The Vision Possible: Agent Protocol Hackathon was centered around one powerful idea: building real-time multimodal AI agents that go beyond traditional cloud pipelines.

For me, this was an opportunity to test a belief I’ve always lived by:

"You don’t need to be an expert to start. You just need to start doing."

I joined the hackathon organized by WeMakeDevs almost spontaneously. There was no massive roadmap—just curiosity and a desire to build something meaningful.

🛠️ Engineering Around the "OOM" Wall

To be honest, my development machine is not a high-end workstation. Running Ultralytics YOLOv8 locally was a struggle, and under strict 1GB cloud RAM limits, I repeatedly faced Out of Memory (OOM) crashes.

Instead of scaling down, I treated these constraints as engineering challenges:

Custom OpenCV Scaler: I engineered an aspect-ratio scaler to intelligently reduce the memory footprint without compromising spatial accuracy.
Memory Management: I minimized redundant tensor allocations and tuned inference batching to keep the system stable on low-tier instances.
WebRTC Integration: The real breakthrough came with Stream’s Vision Agents SDK and Google’s Gemini Realtime LLM.

🚀 The Result: Sub-30ms Latency :

Rather than relying on a laggy HTTP pipeline, the Vision Agents SDK enabled a WebRTC-based streaming architecture over Stream’s Edge network. This allowed YOLO detections to bind directly with Gemini’s reasoning engine.

Stream handled efficient edge routing of heavy video data, while Gemini provided refined multimodal understanding—allowing the AI to “see” through the camera and “speak” contextual warnings in real-time.

🏁 Looking back, this was about proving something to myself.

✅ You don’t need a perfect plan.
✅ You don’t need elite hardware.
✅ You don’t need to feel fully ready.

You need curiosity. You need resilience. And you need to enjoy the journey while building.

Because when curiosity meets constraints — and you choose to build anyway — innovation becomes inevitable.

🌍 What’s Next for VisionMate AI :

This is only the beginning. I’m working on turning VisionMate AI into a production-grade safety ecosystem where:

Edge Deployment: Porting the weights to run natively on NPU-enabled hardware like Raspberry Pi for dedicated smart dashcams.
Predictive Analytics: Moving beyond the "40% Rule" to include velocity-based trajectory prediction for higher accuracy.
Unified Safety Dashboard: Allowing users to review past hazard logs, security audits, and spatial data in one centralized interface.

🙌 A Huge Thanks to WeMakeDevs :

A massive shoutout to WeMakeDevs and Kunal Kushwaha for organizing such a well-structured and inspiring hackathon experience.

From the very beginning, they ensured that participants had everything they needed—from technical resources and community support to sessions with the Stream team explaining their technology in depth.

They hosted interactive workshops on how to effectively use the Vision Agents SDK, which really helped me understand how to build a low-latency, impactful project. The WeMakeDevs community was always active and approachable, answering doubts instantly and fostering an environment where learning and innovation could thrive.

I’m truly grateful for the opportunity to be a part of such an event — it wasn’t just a competition, it was a collaborative learning experience that empowered me to build, explore, and create with real-time AI.

</> Github Repo

Github Repo
⚠️ Disclaimer: This project was developed during the Vision Possible Hackathon. The current architecture is optimized for low-memory environments (1GB RAM) to ensure accessibility for users without high-end hardware.

VisionMate AI is an active spatial hazard agent that empowers users to turn any camera into a real-time safety tool. With advanced CV capabilities, sub-30ms WebRTC latency, and conversational AI alerts, VisionMate transforms passive video feeds into life-saving interactions.

Demo Video
Live Application
Built with ❤️ by Swagatika Beura for the Vision Possible Hackathon.

wemakedevs

Swagatika Beura — Sun, 01 Mar 2026 01:11:30 +0000

2025 Wrap Up

Swagatika Beura — Wed, 31 Dec 2025 17:23:21 +0000

1.Explored many fields and worked on some amazing 2.stuff — tried, experimented, and learned by doing.
Learned a lot this year, not just technical skills but also how things work in the real world.
3.Became part of some incredible communities from around the world 🌍 — open source, tech, and beyond.
4.Midway through the year, I finally found clarity about my path and what I truly want to focus on.
5.Failed many times. Restarted again and again. Each failure taught me something new and helped me discover some pretty cool things.
6.This year taught me how to handle myself — how to manage pressure, stay disciplined, and keep going even in difficult situations.
7.Faced some really bad situations, but those moments showed me how growth actually happens. One lesson stayed with me: keep your head down, stay focused, and let your work speak.
Grateful for every experience — the good, the bad, and everything in between.
Still learning. Still building. 🚀

2025WrapUp #Growth #DevLife

✨ A few days ago, my first PR on Kestra got merged! ✨

Swagatika Beura — Mon, 29 Dec 2025 12:29:31 +0000

I was completely new to this project, and honestly, it wasn’t easy. But this journey taught me something important: you don’t need to be an expert to start—you just need to start.
I failed many times, tried repeatedly, and kept pushing with the hope that one day it would be solved. And finally, that day came—my PR got merged!
Huge thanks to the #Kestra team for their support.
Lesson learned: don’t give up, always try, and learn from failure. One day, your time will come—you don’t need to chase it; it will reach you.

Start First, Learn by Doing: My Thoughts on Growth 🚀

Swagatika Beura — Fri, 17 Oct 2025 14:01:28 +0000

Hello Dev Community! 👋

I want to share a simple but important thought I’ve realized on my journey:

You don’t need to be perfect to start.

I’m not perfect, and I don’t know everything yet — but I’ve decided to start doing things instead of overthinking. I’m exploring Python, building beginner-friendly projects, contributing to open source, participating in communities, and practicing DSA.

The key is learning by doing practical things. Every project, every contribution, and every small step teaches me more than endless planning ever could.

If you’re also learning something new or thinking about starting, just start. Don’t wait for perfection — growth comes from action and experience 🌱💻

Forem: Swagatika Beura