Forem: Quincy Oghenetejiri

Building a Real-Time Security Dashboard with Stream Vision Agents and YOLO11

Quincy Oghenetejiri — Tue, 17 Feb 2026 16:15:28 +0000

Traditional security camera stacks built with OpenCV and Flask often break down under real-world load. When video processing and streaming all run inside a blocking while True loop, a single slow operation can stall the entire pipeline—freezing the live feed for every connected viewer.

In this tutorial, we will build a responsive, scalable security dashboard that avoids these limitations by separating video delivery from detection logic. We will use:

Stream Vision Agents: For global, low-latency video infrastructure.
YOLO11: For human pose estimation and object detection.
React: For a real-time monitoring and alerting dashboard.

This architecture ensures that the video feed remains smooth for the end users.

What We're Building

By the end of this tutorial, you will have a security monitoring dashboard with the following capabilities:

Live Video Panel: A real-time WebRTC stream with pose-skeleton overlays generated by YOLO
Event Log Panel: Timestamped intrusion alerts that update instantly as events occur
Smart Detection: Automatic alerts triggered when a person enters the monitored area
Visual Feedback: Immediate UI cues, such as a red video border, when a threat is detecte

Technical Prerequisites

Stream API credentials
LLM API Key (Gemini, OpenAI or Anthropic)
Python 3.11+
Node.js 18+ (for React tooling)

Setup & Installation

Backend Setup (Python)

We use uv, a modern, high-speed Python package manager, to handle dependencies. Its significantly faster resolution times and built-in environment handling make it ideal for agent-based workflows.

# 1. Create the project folders
mkdir security-agent
cd security-agent

# 2. Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 3. Initialise the project and install dependencies
uv init
uv add "vision-agents[getstream, gemini]" vision-agents-plugins-ultralytics ultralytics

Add the necessary keys to an .env file

STREAM_API_KEY=your_stream_key_here
STREAM_API_SECRET=your_stream_secret_here
GEMINI_API_KEY=your_llm_key_here

Models auto-download on first run, so you can skip this step. However, running this now ensures your agent starts instantly without waiting for downloads:

uv run python -c "from ultralytics import YOLO; YOLO('yolo11n.pt'); YOLO('yolo11n-pose.pt')"

Frontend Setup (React)

Now, let's create our React application with Vite and install the needed dependencies:

npm create vite@latest security-frontend -- --template react-ts
cd security-frontend
npm install @stream-io/video-react-sdk

System Architecture

The system is split into three distinct layers:

Frontend (React): Handles the WebRTC connection and listens for WebSocket events (intruder_alert).
Edge (Stream): Ingests the camera feed and routes frames to the backend via a low-latency protocol.
Backend (Python): Runs the Vision Agent.

The Optimisation Strategy

We use two YOLO models working together to balance visual feedback with performance:

YOLO11 Pose (yolo11n-pose.pt): Responsible for rendering pose skeletons for visual confirmation. This runs on every frame and prioritises clarity over speed.
YOLO11 Detection (yolo11n.pt): Optimised for fast person detection, used only for logical decisions and alerting.

The Filtering Logic

When YOLO processes a frame, it detects everything—including chairs, cups, plants, and people. We filter out this noise by checking whether class_id == 0, since YOLO models are trained on the COCO (Common Objects in Context) dataset, which assigns a numeric class ID to each detectable object.

This approach transforms a generic object detector into a specialised security guard that ignores your cat running past but triggers instantly for humans

# YOLO returns ALL detected objects
results = model(frame)

person_count = 0
for box in results[0].boxes:
    # Check the class ID
    class_id = int(box.cls)

    if class_id == 0:  # 0 = Person
        person_count += 1
        # This is our target - count it, draw it, alert!
    else:
        # Ignore chairs, tables, pets, etc.
        pass

if person_count > 0:
    send_alert("Intruder detected!")

Building the Security Agent (Backend)

Project Structure

The backend is organised around a single Python agent responsible for video processing, detection logic, and event emission.

security-agent/
├── .venv/
├── .env                  # API keys and secrets
├── generate_token.py     # Development utility for creating access tokens
├── main.py               # Vision agent entry point
├── yolo11n.pose.pt       # YOLO pose model (excluded from version control)
└── yolo11n.pt            # YOLO detection model (excluded from version control)

Generating Tokens for Development

Since we don’t have a full authentication backend for this demo, we need to generate a Stream access token manually so the frontend can connect to the video stream.

Choosing a User ID

You need to decide on a user_id for your admin viewer.

Existing Users: If you already have a user in your Stream dashboard, reuse that ID.
New Users: If not, simply pick a name (e.g., admin_user). Stream will automatically create this user the first time they connect with a valid token.

Create a file named get_token.py in your backend directory. This script uses your server-side API secret to sign a token for the chosen user.

Security Note: Tokens must always be generated server-side using your Stream API secret. In this demo, we manually generate a temporary development token. In a production application, this token would be issued by your authentication or login endpoint.

import os
from getstream import Stream

# 1. Get keys from the environment (loaded by uv)
key = os.getenv("STREAM_API_KEY")
secret = os.getenv("STREAM_API_SECRET")

if not key or not secret:
    print("❌ Error: Could not find API keys. Make sure you run with --env-file .env")
    exit(1)

# 2. Initialize Stream
client = Stream(key, secret)

# 3. Generate Token for Shaq12
user_id = "Shaq12"
token = client.create_token(user_id)

print(f"\n✅ TOKEN FOR {user_id}:")
print(token)
print("\nCopy the string above and paste it into App.tsx")

Run the script using:

uv run --env-file .env generate_token.py

The generated token will be used later in the frontend configuration.

The Foundation (Imports & Config)

First, we need to import our tools and load the API keys.

import asyncio
import os
import time
import logging
import sys
from vision_agents.core import agents, User
import vision_agents.plugins.getstream as getstream_plugin
from getstream import AsyncStream
from vision_agents.plugins.ultralytics import YOLOPoseProcessor
from ultralytics import YOLO
import vision_agents.plugins.gemini as gemini

# Setup logging to see what's happening in the terminal
logging.basicConfig(level=logging.INFO, format='%(message)s', stream=sys.stdout)

# Global Config
ACTIVE_CALL_ID = "security-demo-working"
STREAM_CLIENT = None  # We will fill this later

The Custom Processor

To make the agent intelligent, we override the default video processor and introduce a custom SecurityYOLOProcessor. Running object detection on every frame is computationally expensive and can quickly introduce lag. Instead of processing all 30 frames per second, we split the workload into two distinct paths: visuals and logic.

For visuals, we call the parent YOLOPoseProcessor on every frame to render the skeleton overlay at 30 FPS, ensuring smooth and responsive video output.

For decision-making, we run the detection model only every fifth frame using a simple modulo check (frame_count % 5 == 0). This means the agent “thinks” about intruder detection just six times per second—dramatically reducing CPU usage while remaining fast enough to react to humans in the scene.

class SecurityYOLOProcessor(YOLOPoseProcessor):
    def __init__(self, model="yolo11n-pose.pt", confidence=0.5, cooldown=5.0):
        super().__init__(model=model, confidence=confidence)
        self.cooldown = cooldown
        self.last_alert_time = 0
        self.frame_count = 0

        print("📦 Loading detection model...", flush=True)
        # Load the secondary model for counting
        self.detect_model = YOLO("yolo11n.pt") 

    async def _process_pose_async(self, frame_data):
        # 1. Let the parent draw the skeleton (Visuals)
        annotated = await super()._process_pose_async(frame_data)

        # 2. Run our Counter Logic (Optimization)
        self.frame_count += 1
        if self.frame_count % 5 == 0:  # Check every 5th frame
            await self._run_detection(frame_data)

        return annotated

    async def _run_detection(self, frame_data):
        # ... (Detection logic to count people) ...
        # If person_count > 0, call self._send_alert()
        pass

The Agent Assembly

Next, we wire the video processing and reasoning layers together using the agents.Agent class. This wrapper acts as the bridge between two core components:

The edge layer, which manages the real-time video connection using your Stream credentials.
The LLM layer (e.g., Gemini) provides the core reasoning and runtime dependency for Vision Agents. Even if you don’t use language or reasoning capabilities in this dashboard, an LLM must still be configured so the agent runtime initialises properly and can orchestrate video processing and events.

We then inject our SecurityYOLOProcessor into the processors list. This tells the agent to apply our custom video-processing logic before publishing the video stream.

async def main():
    global STREAM_CLIENT

    # 1. Authenticate
    api_key = os.getenv("STREAM_API_KEY")
    api_secret = os.getenv("STREAM_API_SECRET")
    client = AsyncStream(api_key=api_key, api_secret=api_secret)
    STREAM_CLIENT = client

    # 2. Join the Call
    call = client.video.call("default", ACTIVE_CALL_ID)
    await call.get_or_create(data={"created_by": {"id": "security-bot"}})

    # 3. Assemble the Agent
    agent = agents.Agent(
        edge=getstream_plugin.Edge(api_key=api_key, api_secret=api_secret),
        llm=gemini.LLM(model="gemini-1.5-flash", api_key=os.getenv("GEMINI_API_KEY")),
        agent_user=User(id="security_agent", name="Security Guard"),
        processors=[SecurityYOLOProcessor()] # <--- Inject our custom processor
    )

    await agent.join(call)

    # Keep alive
    try:
        while True: await asyncio.sleep(1)
    except KeyboardInterrupt: pass

if __name__ == "__main__":
    asyncio.run(main())

Complete Backend Code (`main.py`)

Now that we’ve defined the custom processor and the agent configuration, we can combine them into the final executable script. Create a file named main.py and paste in the code below.

import asyncio
import os
import time
import logging
import sys
from vision_agents.core import agents, User
import vision_agents.plugins.getstream as getstream_plugin
from getstream import AsyncStream
from vision_agents.plugins.ultralytics import YOLOPoseProcessor
from ultralytics import YOLO
import vision_agents.plugins.gemini as gemini


logging.basicConfig(level=logging.INFO, format='%(message)s', stream=sys.stdout)
logging.getLogger("vision_agents").setLevel(logging.CRITICAL)



print("\n🚀 SECURITY DASHBOARD - ARGUMENT ORDER FIX\n", flush=True)


# --- GLOBAL VARIABLES ---
ACTIVE_CALL_ID = "security-demo-working"
STREAM_CLIENT = None


class SecurityYOLOProcessor(YOLOPoseProcessor):
    def __init__(self, model="yolo11n-pose.pt", confidence=0.5, cooldown=5.0):
        super().__init__(model=model, confidence=confidence)
        self.cooldown = cooldown
        self.last_alert_time = 0
        self.frame_count = 0

        print("📦 Loading detection model...", flush=True)
        self.detect_model = YOLO("yolo11n.pt")
        print("✅ Processor Ready", flush=True)

    async def _process_pose_async(self, frame_data):
        # 1. Parent Logic
        annotated = await super()._process_pose_async(frame_data)

        self.frame_count += 1
        if self.frame_count % 30 == 0:
            print(f"💓 Frame {self.frame_count}", flush=True)

        # 2. Detection Logic
        try:
            if self.frame_count % 5 == 0:
                results = self.detect_model(frame_data, verbose=False)
                person_count = 0
                for box in results[0].boxes:
                    if int(box.cls) == 0:
                        person_count += 1

                if person_count > 0:
                    await self._send_alert(person_count)
        except Exception as e:
            pass

        return annotated

    async def _send_alert(self, person_count):
        now = time.time()

        if (now - self.last_alert_time) > self.cooldown:
            print(f"\n🚨 {person_count} PERSON(S) DETECTED! Sending Alert...", flush=True)

            try:
                if STREAM_CLIENT:
                    # 🛠️ FIX: SIMPLE FLAT PAYLOAD
                    # We don't try to nest "custom" inside "custom".
                    # We just send the data directly.
                    payload = {
                        "alert_trigger": "intrusion", # <--- The Key we will look for
                        "message": f"{person_count} Intruder(s) Detected",
                        "person_count": person_count,
                        "timestamp": now
                    }


                    await STREAM_CLIENT.video.send_call_event(
                        "default",         # Call Type
                        ACTIVE_CALL_ID,    # Call ID
                        "security_agent",  # User ID
                        payload            # The Data
                    )

                    print(f"✅ ALERT SENT!", flush=True)
                else:
                    print(f"❌ CRITICAL: Global CLIENT variable is missing!", flush=True)

                self.last_alert_time = now

            except Exception as e:
                print(f"❌ Alert Failed: {e}", flush=True)


async def main():
    global STREAM_CLIENT

    api_key = os.getenv("STREAM_API_KEY")
    api_secret = os.getenv("STREAM_API_SECRET")
    gemini_key = os.getenv("GEMINI_API_KEY")


    if not api_key: raise ValueError("❌ Check .env")


    client = AsyncStream(api_key=api_key, api_secret=api_secret)
    STREAM_CLIENT = client

    call = client.video.call("default", ACTIVE_CALL_ID)
    await call.get_or_create(data={"created_by": {"id": "security-bot"}})

    print(f"📹 Call ID: {ACTIVE_CALL_ID}", flush=True)

    processor = SecurityYOLOProcessor(
        model="yolo11n-pose.pt",
        confidence=0.5,
        cooldown=5.0
    )

    edge = getstream_plugin.Edge(api_key=api_key, api_secret=api_secret)
    llm = gemini.LLM(model="gemini-1.5-flash", api_key=gemini_key)

    agent = agents.Agent(
        edge=edge,
        llm=llm,
        agent_user=User(id="security_agent", name="Security Guard"),
        processors=[processor]
    )

    print("🤖 Agent joining...", flush=True)
    await agent.join(call)
    print("✅ AGENT LIVE!", flush=True)

    try:
        while True: await asyncio.sleep(1)
    except KeyboardInterrupt: pass


if __name__ == "__main__":
    asyncio.run(main())

The Frontend: Building the Dashboard UI

With the backend successfully broadcasting events, we need a dashboard to visualise them. Our React application acts as the command centre, combining a live video feed with real-time threat logs.

Our dashboard is built on three pillars:

Video Panel: Renders the WebRTC stream, overlaying the YOLO detection boxes from the backend.
Event Listener: A background process that subscribes to the custom WebSocket events sent by our Python backend.
Event Log Panel: A chronological list of security alerts.

Environment Setup

Instead of hardcoding our secrets, we will store them safely. Create a file named .env in the root of your frontend folder:

# .env
VITE_STREAM_API_KEY=api_key
# Paste the token you generated in the get_token.py script
VITE_STREAM_TOKEN=token_generated

Project Structure

Here’s how the frontend project is organised. The dashboard logic lives in a single React entry point for simplicity, while sensitive credentials are stored securely in an environment file.

security-dashboard/
├── src/
│   ├── App.tsx      # Dashboard UI and logic
│   └── main.tsx
└── .env             # API keys and access token

Imports & Setup

We import the required Stream Video components along with the necessary React hooks. The API credentials—VITE_STREAM_API_KEY and VITE_STREAM_TOKEN—are loaded from environment variables, allowing the application to authenticate with Stream securely.

import { useEffect, useState } from 'react';
import {
  StreamVideo,
  StreamVideoClient,
  StreamCall,
  StreamTheme,
  useCallStateHooks,
  CallControls,
  SpeakerLayout,
  useCall,
} from '@stream-io/video-react-sdk';
import '@stream-io/video-react-sdk/dist/css/styles.css';
const streamKey = import.meta.env.VITE_STREAM_API_KEY;
const streamToken = import.meta.env.VITE_STREAM_TOKEN;

Configuration

We configure the video call using fixed identifiers for demonstration purposes:

callId: 'security-demo-working' — uniquely identifies the video call session.
userId: 'Shaq12' — represents the current user joining the call.

In a production environment, these values would typically be generated dynamically based on the authenticated user and session context.

const apiKey = streamKey; 
const callId = 'security-demo-working';
const userId = 'Shaq12';
const token = streamToken; 

const user = { id: userId, name: 'Admin Viewer' };

Client & Call Initialisation

Next, we initialise the Stream Video client using the API key and token generated earlier. This establishes a secure connection to the Stream servers and enables real-time video and event delivery.

const videoClient = new StreamVideoClient({ 
  apiKey, 
  user, 
  token,
  options: { logLevel: 'warn' } 
});

const call = videoClient.call('default', callId);
call.join({ create: true });

The Event Listener (Alert Detection)

This is the bridge between the Python backend and the React frontend. We use the useCall() hook to access the active call instance, which also serves as our WebSocket transport.

The backend sends structured JSON payloads as custom events, and the frontend subscribes specifically to this channel.

When an event arrives, we inspect the alert_trigger field to confirm whether an intrusion has been detected.

useEffect(() => {
  if (!currentCall) return;

  const handleEvent = (event: any) => {
    const data = event.custom || {};

    if (data.alert_trigger === 'intrusion') {
      console.log("🚨 INTRUDER CONFIRMED:", data);

      const timestamp = new Date().toLocaleTimeString();
      const message = data.message || "Intruder Detected";
      const count = data.person_count || 1;

      const logEntry = `⚠️ ${timestamp}: ${message}`;

      setLogs((prev) => [logEntry, ...prev.slice(0, 9)]);
    }
  };

  currentCall.on('custom', handleEvent);

  return () => {
    currentCall.off('custom', handleEvent);
  };
}, [currentCall]);

Data Flow

The data flow through the frontend follows a simple, predictable pipeline:

Header Section

The header provides immediate system visibility, displaying the dashboard title and the current agent status.

<div style={{ padding: '20px', borderBottom: '1px solid #333', display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
  <h2 style={{ margin: 0 }}>🛡️ Security Control Center</h2>
  <div style={{ fontSize: '14px', color: '#aaa' }}>
    System Status: <span style={{ color: agent ? '#0f0' : '#f00', fontWeight: 'bold' }}>
      {agent ? 'ONLINE' : 'OFFLINE'}
    </span>
  </div>
</div>

Video Feed Panel

The video panel renders the live WebRTC stream and provides immediate visual feedback when a security event occurs.

The container’s border colour is driven directly by application state. When the logs array contains one or more alerts, the border turns red, instantly signalling that attention is required. When no alerts are present, the border returns to its neutral state.

This approach ensures that visual feedback is both reactive and consistent, without requiring additional UI state or side effects.

<div style={{ flex: 3, padding: '20px', display: 'flex', flexDirection: 'column', gap: '10px' }}>
  <div style={{ 
    flex: 1, 
    borderRadius: '15px', 
    overflow: 'hidden', 
    border: logs.length > 0 ? '2px solid red' : '1px solid #333',
    transition: 'border 0.3s ease'
  }}>
    <SpeakerLayout participantsBarPosition="bottom" />
  </div>
  <div style={{ display: 'flex', justifyContent: 'center' }}>
    <CallControls />
  </div>
</div>

Event Log Panel

The Event Log serves as the system’s source of truth for all detected security events. It records each alert in a clear, chronological format, allowing operators to review activity over time and quickly understand what triggered a visual warning.

Each log entry includes:

A timestamp indicating when the event occurred
A short descriptive message provided by the backend
Visual emphasis to distinguish alerts from normal system activity

To keep the interface readable during high-activity periods, the log is managed as a First-In, First-Out (FIFO) queue. New alerts are added to the top of the list, while older entries are truncated once the maximum display limit is reached. This ensures that the most relevant information is always visible without overwhelming the user.

In addition to alerts, the panel also displays lightweight system metadata—such as the number of active viewers and the connected agent ID—providing useful operational context at a glance.

When no alerts are present, the UI enters a passive monitoring state, clearly indicating that the system is active and scanning for intrusions.

<div style={{ flex: 1, background: '#1a1a1a', padding: '20px', borderLeft: '1px solid #333', overflowY: 'auto' }}>
  <h3 style={{ marginTop: 0, borderBottom: '1px solid #444', paddingBottom: '10px' }}>🚨 Event Log</h3>

  <div style={{ fontSize: '12px', color: '#666', marginBottom: '15px', padding: '10px', background: '#111', borderRadius: '5px' }}>
    <div>Active Viewers: {participants.length}</div>
    <div>Bot ID: {agent ? agent.userId : 'Searching...'}</div>
  </div>

  {logs.length === 0 ? (
    <div style={{ color: '#666', fontStyle: 'italic', marginTop: 20 }}>
      Waiting for alerts...<br/>
      <small>System is scanning for intruders.</small>
    </div>
  ) : (
    <ul style={{ listStyle: 'none', padding: 0 }}>
      {logs.map((log, i) => (
        <li key={i} style={{ 
          padding: '10px', marginBottom: '8px', 
          background: 'rgba(255, 0, 0, 0.15)', 
          borderLeft: '3px solid #ff4444', 
          color: '#ffcccc',
          borderRadius: '0 4px 4px 0',
          animation: 'fadeIn 0.3s ease-in'
        }}>
          {log}
        </li>
      ))}
    </ul>
  )}
</div>

The Complete Source Code(`App.tsx`)

We have examined the core logic blocks individually. Now, let’s assemble them into the final implementation file.

Copy the code below into App.tsx.

import { useEffect, useState } from 'react';
import {
  StreamVideo,
  StreamVideoClient,
  StreamCall,
  StreamTheme,
  useCallStateHooks,
  CallControls,
  SpeakerLayout,
  useCall, // <--- IMPORTANT: We need this hook to listen to events
} from '@stream-io/video-react-sdk';
import '@stream-io/video-react-sdk/dist/css/styles.css';


const streamKey = import.meta.env.VITE_STREAM_API_KEY;
const streamToken = import.meta.env.VITE_STREAM_TOKEN;

// --- CONFIGURATION ---
const apiKey = streamKey; 
const callId = 'security-demo-working'; // MUST MATCH BACKEND
const userId = 'Shaq12';
const token = streamToken; 

const user = { id: userId, name: 'Admin Viewer' };

// --- CLIENT SETUP ---
const videoClient = new StreamVideoClient({ 
  apiKey, 
  user, 
  token,
  options: { logLevel: 'warn' } 
});

const call = videoClient.call('default', callId);
call.join({ create: true });

// --- MAIN LAYOUT COMPONENT ---
const SecurityLayout = () => {
  const { useParticipants } = useCallStateHooks();
  const participants = useParticipants();
  const [logs, setLogs] = useState<string[]>([]);

  // 1. GET THE ACTIVE CALL OBJECT
  const currentCall = useCall(); 

 // 2. LISTEN FOR ALERTS (Simplified)
  useEffect(() => {
    if (!currentCall) return;

    const handleEvent = (event: any) => {
      // DEBUG: Log everything so we can see what arrives
      // console.log("📨 Event Received:", event);

      // 1. Grab the custom data packet
      // Stream puts our payload inside 'custom'
      const data = event.custom || {};

      // 2. Check for our specific trigger key
      if (data.alert_trigger === 'intrusion') {
        console.log("🚨 INTRUDER CONFIRMED:", data);

        const timestamp = new Date().toLocaleTimeString();
        const message = data.message || "Intruder Detected";
        const count = data.person_count || 1;

        const logEntry = `⚠️ ${timestamp}: ${message}`;

        setLogs((prev) => [logEntry, ...prev.slice(0, 9)]);
      }
    };

    currentCall.on('custom', handleEvent);

    return () => {
      currentCall.off('custom', handleEvent);
    };
  }, [currentCall]);

  // Check if the Security Bot is in the room
  const agent = participants.find((p) => 
    p.userId.includes('security') || p.userId.includes('agent')
  );

  return (
    <div style={{ width: '100vw', height: '100vh', background: '#0e0e0e', color: 'white', fontFamily: 'Arial, sans-serif' }}>

      {/* HEADER */}
      <div style={{ padding: '20px', borderBottom: '1px solid #333', display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
        <h2 style={{ margin: 0 }}>🛡️ Security Control Center</h2>
        <div style={{ fontSize: '14px', color: '#aaa' }}>
          System Status: <span style={{ color: agent ? '#0f0' : '#f00', fontWeight: 'bold' }}>
            {agent ? 'ONLINE' : 'OFFLINE'}
          </span>
        </div>
      </div>

      <div style={{ display: 'flex', height: 'calc(100vh - 80px)' }}>

        {/* VIDEO FEED (Left Panel) */}
        <div style={{ flex: 3, padding: '20px', display: 'flex', flexDirection: 'column', gap: '10px' }}>
          <div style={{ 
            flex: 1, 
            borderRadius: '15px', 
            overflow: 'hidden', 
            border: logs.length > 0 ? '2px solid red' : '1px solid #333',
            transition: 'border 0.3s ease'
          }}>
             <SpeakerLayout participantsBarPosition="bottom" />
          </div>
          <div style={{ display: 'flex', justifyContent: 'center' }}>
            <CallControls />
          </div>
        </div>

        {/* LOG PANEL (Right Panel) */}
        <div style={{ flex: 1, background: '#1a1a1a', padding: '20px', borderLeft: '1px solid #333', overflowY: 'auto' }}>
          <h3 style={{ marginTop: 0, borderBottom: '1px solid #444', paddingBottom: '10px' }}>🚨 Event Log</h3>

          <div style={{ fontSize: '12px', color: '#666', marginBottom: '15px', padding: '10px', background: '#111', borderRadius: '5px' }}>
            <div>Active Viewers: {participants.length}</div>
            <div>Bot ID: {agent ? agent.userId : 'Searching...'}</div>
          </div>

          {logs.length === 0 ? (
            <div style={{ color: '#666', fontStyle: 'italic', marginTop: 20 }}>
              Waiting for alerts...<br/>
              <small>System is scanning for intruders.</small>
            </div>
          ) : (
            <ul style={{ listStyle: 'none', padding: 0 }}>
              {logs.map((log, i) => (
                <li key={i} style={{ 
                  padding: '10px', marginBottom: '8px', 
                  background: 'rgba(255, 0, 0, 0.15)', 
                  borderLeft: '3px solid #ff4444', 
                  color: '#ffcccc',
                  borderRadius: '0 4px 4px 0',
                  animation: 'fadeIn 0.3s ease-in'
                }}>
                  {log}
                </li>
              ))}
            </ul>
          )}
        </div>
      </div>
    </div>
  );
};

export default function App() {
  return (
    <StreamVideo client={videoClient}>
      <StreamTheme>
        <StreamCall call={call}>
          <SecurityLayout />
        </StreamCall>
      </StreamTheme>
    </StreamVideo>
  );
}

Running the Application

Now that the code is built, let's see it in action.

Start the Python Agent (Backend)

In your backend directory, run the command below, which ensures the env files are loaded:

uv run --env-file .env main.py

You should see:

Here is how the alert looks in the backend logs when an intruder is detected and when it is sent to the frontend:

Start the React App (Frontend)

npm run dev

You should see:

Click the http://localhost:5173 link to open your dashboard.

Testing the Security System

Clear the Camera: Step out of the frame so the camera sees no one.
Enter the Frame: Walk into view.
Watch the Logs: You should see the backend print 🚨 1 PERSON(S) DETECTED! and the frontend log panel update instantly with a red border flash.

Congratulations! You have just built a real-time AI security agent.

Complete Source Code

We have built the Backend Security Agent and the Frontend Dashboard as two distinct services. You can find the complete source code for both parts in the repositories below:

Backend Security Agent (Python)
- Contains: The custom YOLO processor, agent configuration, and token generation script.
Frontend Dashboard (React)
- Contains: The video player component, event listeners, and alert UI logic.

Conclusion

In this tutorial, we built a real-time security dashboard using Python and Vision Agents on the backend and React with Stream Video on the frontend. Along the way, we covered

Implementing human-focused detection logic
Setting up a real-time video interface
Broadcasting and displaying security alert events

Although this example uses standard YOLO models, Stream Vision Agents also support integrations with other vision and multimodal systems such as Roboflow, MoonDream, and Decart.

Want to keep building with Vision Agents? Explore these tutorials next:

Build an Accessible Chat and Video App in React

Quincy Oghenetejiri — Tue, 18 Nov 2025 17:55:06 +0000

Imagine joining a team meeting but having no idea who’s speaking. Or trying to follow a busy chat thread, only for your screen reader to miss half the messages.

For the 1.3 billion people living with disabilities, this is a daily experience with most communication tools. Real-time apps like chat and video platforms often overlook accessibility, leading to issues such as:

Dynamic updates that screen readers can’t keep up with
Visual-only indicators like reactions or hand raises
Audio-only conversations that exclude deaf or hard-of-hearing users

Building accessible real-time apps means designing for everyone, not just those who can see, hear, or interact in typical ways.

What We're Building

In this article, we'll build a production-ready chat and video application that's fully accessible:

Real-time video conferencing with accessible controls
Live captions powered by Stream Video's transcription API
Text chat with file sharing that works seamlessly with screen readers
Complete keyboard navigation - no mouse required
WCAG 2.1 AA compliant - meeting international accessibility standards

Here is a quick demonstration of the seamless keyboard-only flow we will achieve.

Technologies Used

React with TypeScript
Stream Video SDK for video conferencing
Stream Chat SDK for messaging
Custom accessibility hooks and utilities
Semantic HTML and ARIA labels

Technical Prerequisite

Before we begin, ensure you have the following:

Free Stream account
Node.js 14 or higher installed
Basic React and TypeScript knowledge
A general understanding of accessibility principles.

Project Architecture

Setup: Project Scaffolding and Backend Settings

Backend Setup (Token Generation)

While this project primarily involves frontend development, a backend setup is necessary for fundamental functionalities such as generating tokens for user authentication and retrieving user lists.

We’ll start by setting up the backend:

# backend
npm init
npm install express dotenv stream-chat nodemon cors

Next, create a .env file in your project root:

# .env
STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret

This is the auth route that involves token generation in the Express server.

// Authentication endpoint
app.post("/auth", async (req, res) => {
  const { id, name } = req.body;

  try {
    await chatServer.upsertUser({ id, name });
    const token = chatServer.createToken(id);

    res.json({
      apiKey: STREAM_API_KEY,
      user: { id, name },
      token,
    });
  } catch (err) {
    console.error("Auth error:", err);
    res.status(500).json({ error: "Authentication failed" });
  }
});

Once the code is run, the terminal will look like this:

Frontend Setup

Now, let's create our React application with Vite for fast development:

# Create new Vite + React + TypeScript project
npm create vite@latest accessible-video-chat -- --template react-ts

cd accessible-video-chat

# Install Stream SDKs
npm install stream-chat stream-chat-react @stream-io/video-react-sdk

Next, you will create a .env that will contain the backend URL

VITE_API_BASE_URL="http://localhost:4000"

You might be wondering why we don't include the Stream keys directly in the frontend. The reason is that these keys are securely obtained from the backend through a token. We will discuss the specific hook responsible for this process in the following section.

Accessible Core: Utility Hooks

Understanding Semantic HTML and ARIA

Before diving into the code, let's understand the foundation of accessible web applications.

Semantic HTML provides meaning to content:

<article> for self-contained content (messages)
<time> for timestamps with machine-readable dates
<button> for interactive elements
<form> for data submission

ARIA (Accessible Rich Internet Applications) enhances accessibility when semantic HTML isn't enough:

role defines what an element is (e.g., role="log" for chat history, role="toolbar" for video controls)
aria-label provides a text alternative for screen readers
aria-live announces dynamic content updates
aria-describedby associates descriptive text with elements

Screen Reader Announcement Hook

Screen readers are essential for blind and low-vision users, but they can only announce content that's properly exposed. In real-time apps, where messages, typing indicators, and status changes happen dynamically, proper announcements are the backbone of accessibility.

This pattern centres on ARIA Live Regions, which are hidden elements designed to announce content updates:

aria-live="polite": For non-urgent updates (typing indicators, character counts, new messages).
aria-live="assertive": For critical updates (errors, connection lost, important status changes).
role="alert": This is an assertive live region for immediate error announcement.
role="status": This is used for non-critical, continuous updates, such as "Loading users..."
role="list"/ role="listitem": These are used to give structure to a collection of clickable items, helping screen readers understand the total count and navigation context.

This ScreenReaderAnnouncer class creates a hidden DOM element with aria-live, which is completely invisible to sighted users and designed for screen readers only. It utilises the Singleton pattern, ensuring only one instance for the entire application, and directly announces information to assistive technology.

// utils/screenReader.ts
export class ScreenReaderAnnouncer {
  private static instance: ScreenReaderAnnouncer;
  private container: HTMLDivElement | null = null;

  private constructor() {
    this.createContainer();
  }

  static getInstance(): ScreenReaderAnnouncer {
    if (!ScreenReaderAnnouncer.instance) {
      ScreenReaderAnnouncer.instance = new ScreenReaderAnnouncer();
    }
    return ScreenReaderAnnouncer.instance;
  }

  private createContainer(): void {
    if (typeof window === 'undefined') return;

    this.container = document.createElement('div');
    this.container.setAttribute('aria-live', 'polite');
    this.container.setAttribute('aria-atomic', 'true');
    this.container.className = 'sr-only';
    this.container.style.cssText = `
      position: absolute;
      left: -10000px;
      width: 1px;
      height: 1px;
      overflow: hidden;
    `;

    document.body.appendChild(this.container);
  }

  announce(message: string, priority: 'polite' | 'assertive' = 'polite'): void {
    if (!this.container) return;

    // Update priority if needed
    if (this.container.getAttribute('aria-live') !== priority) {
      this.container.setAttribute('aria-live', priority);
    }

    // Clear previous message
    this.container.textContent = '';

    // Add new message after brief delay
    setTimeout(() => {
      if (this.container) {
        this.container.textContent = message;
      }
    }, 10);

    // Clear after announcement
    setTimeout(() => {
      if (this.container) {
        this.container.textContent = '';
      }
    }, 1000);
  }
}

// React hook wrapper
export const useScreenReader = () => {
  const announcer = ScreenReaderAnnouncer.getInstance();

  return {
    announce: (message: string, priority?: 'polite' | 'assertive') => 
      announcer.announce(message, priority)
  };
};

`useFocusManager.ts` Hook

This custom hook gives precise and programmatic control over keyboard focus in the application. Keyboard focus refers to the active element on the screen targeted to receive user input (e.g., keystrokes). For users relying on keyboards or screen readers, the path of this focus (indicated by the visible outline) must be predictable. When standard browser focus gets lost or jumps randomly, the application becomes unusable.

The hook’s primary purpose is to overcome the limitations of the standard browser tabbing, especially when dealing with elements like modals, sidebars, or when switching between major UI sections(like the chat and video views).

This is carried out through these four distinct utility functions:

saveFocus() remembers the focused element before opening a modal or dialogue.
restoreFocus() returns focus to that element after closing the modal.
manageFocus() programmatically sets focus on an element.
trapFocus() keeps focus within a modal, preventing tabbing outside of it.

// hooks/useFocusManager.ts
import { useRef, useCallback } from 'react';
import type { FocusManager } from '../types/accessibility';

export const useFocusManager = (): FocusManager => {
  const previousFocusRef = useRef<HTMLElement | null>(null);

  const saveFocus = useCallback((): void => {
    previousFocusRef.current = document.activeElement as HTMLElement;
  }, []);

  const restoreFocus = useCallback((): void => {
    if (previousFocusRef.current && typeof previousFocusRef.current.focus === 'function') {
      previousFocusRef.current.focus();
    }
  }, []);

  const manageFocus = useCallback((element: HTMLElement | null): void => {
    if (element && typeof element.focus === 'function') {
      element.focus();
    }
  }, []);

  const trapFocus = useCallback((containerElement: HTMLElement | null): (() => void) | void => {
    if (!containerElement) return () => {};

    const focusableElements = containerElement.querySelectorAll<HTMLElement>(
      'button, [href], input, select, textarea, [tabindex]:not([tabindex="-1"])'
    );

    const firstElement = focusableElements[0];
    const lastElement = focusableElements[focusableElements.length - 1];

    const handleKeyDown = (event: KeyboardEvent): void => {
      if (event.key !== 'Tab') return;

      if (event.shiftKey) {
        if (document.activeElement === firstElement) {
          event.preventDefault();
          lastElement.focus();
        }
      } else {
        if (document.activeElement === lastElement) {
          event.preventDefault();
          firstElement.focus();
        }
      }
    };

    containerElement.addEventListener('keydown', handleKeyDown);

    // Focus first element
    if (firstElement) {
      firstElement.focus();
    }

    return (): void => {
      containerElement.removeEventListener('keydown', handleKeyDown);
    };
  }, []);

  return {
    saveFocus,
    restoreFocus,
    manageFocus,
    trapFocus
  };
};

`useStreamConnection.ts` Hook

This custom hook manages the connection to Stream's Chat and Video services by handling user authentication with a backend server, making POST requests to the /auth endpoint, and managing connection states such as isConnecting and error.

It initialises both StreamChat and StreamVideoClient instances, connects authenticated users to both services, and returns these connected clients for use throughout the application. Additionally, the hook provides robust error handling, including error state management, throwing errors for failed connections, and offering type-safe error messages.

// hooks/useStreamConnection.ts
import { useState, useCallback } from 'react';
import { StreamChat } from 'stream-chat';
import { StreamVideoClient } from '@stream-io/video-react-sdk';
import type { StreamUser, AuthResponse } from '../types';

`useAccessibilitySettings.ts` Hook

This hook helps in the automatic detection of the user’s system accessibility preferences, including reduced motion, high contrast mode, and basic screen reader usage.

// hooks/useAccessibilitySettings.ts
import { useState, useEffect, useCallback } from 'react';
import type { AccessibilitySettings } from '../types/accessibility';

`useCallManagement.ts` Hook

This custom hook contains all meeting join/leave logic, handles transcription setup, and also ensures proper cleanup on meeting exit.

// hooks/useCallManagement.ts
import { useState, useCallback } from 'react';
import type { StreamVideoClient, Call } from '@stream-io/video-react-sdk';
import { cleanupMediaTracks, disableCallDevices, enableCallDevices } from '../utils';
import { DEVICE_ENABLE_DELAY } from '../utils/constants';

Building the Application Shell

Before users can chat or video call, they need to authenticate and navigate the application. Let's build the foundational components that tie everything together.

Authentication Flow

Users must sign up or sign in to access any features. The authentication system has three components:

1. AuthView - The Container

This component is the central component that manages switching between signUp and signIn modes.

Its key accessibility features include role="main" for primary page content, and role="region" with aria-labelledby to group related form content.

// components/Auth/AuthView.tsx
import React, { useState } from 'react';
import { LoginForm } from './LoginForm';
import { SignupForm } from './SignupForm';
import type { AuthMode, Credentials } from '../../types';

2. Signup and Login Form

Forms use strong accessibility patterns for clarity, validation, and immediate error feedback.

Input Accessibility Pattern: Fields are clearly linked to their labels. aria-required="true" indicates to screen readers that a field is mandatory, allowing users to understand validation requirements immediately.

Assertive Error Pattern: Errors are announced instantly after a failed submission. The error message is contained within a container with the role="alert" attribute. This is an ARIA Live Region with assertive priority that interrupts the user to ensure they hear and fix the error immediately.

<form onSubmit={onSubmit} className="auth-form" aria-label="Sign up form">
      <label className="field">
        <span>User ID</span>
        <input
          className="input"
          placeholder="Enter user id"
          value={credentials.id}
          onChange={e => onChange({ ...credentials, id: e.target.value })}
          required
          aria-required="true"
          disabled={isLoading}
        />
      </label>

      <label className="field">
        <span>Name</span>
        <input
          className="input"
          placeholder="Enter display name"
          value={credentials.name}
          onChange={e => onChange({ ...credentials, name: e.target.value })}
          required
          aria-required="true"
          disabled={isLoading}
        />
      </label>

      <button
        type="submit"
        className="button primary"
        disabled={isLoading || !credentials.id || !credentials.name}
      >
        {isLoading ? 'Connecting...' : 'Create account'}
      </button>

      {error && (
        <div role="alert" className="error-text">
          {error}
        </div>
      )}
    </form>

Navigation Hub

The HubView is responsible for communicating feature status accessibility. This means it enforces a clear hierarchy of information: it uses role="navigation" to clearly identify the main links, and it ensures secondary information (like the "Live Captions Available" indicator) is handled correctly.

This indicator is placed in an element with role="status", which is a polite ARIA Live Region that provides a subtle announcement to the screen reader, informing the user without interrupting their current task of reading or navigating.

// Hub/HubView.tsx - Status Announcement Snippet

// ... (inside the component's JSX)

<div className="user-welcome">
  <h2>Welcome, {userName}</h2>
  {transcriptionAvailable && (
    <span 
      className="feature-badge" 
      // ARIA: role="status" is a polite live region
      role="status"
    >
      Live Captions Available
    </span>
  )}
</div>
// ... (rest of the component)

Building Accessible Chat UI

With the application foundation in place, let's build the chat interface.

The chat experience has three stages:

Select a user to chat with (UserSelectView.tsx)
View message history (MessageList.tsx)
Send new messages (MessageInput.tsx)

We’ll build each stage with full accessibility support.

User Selection - Choosing Who to Chat With

When users click "Chat with user" from the hub, they must select the user they want to chat with.

The UserSelectView.tsx component handles this. The component ensures clear status feedback. When fetching users, we use a polite live region (role="status") to announce 'Loading users...'. If the fetch fails, the error message is placed inside a container with the role="alert" to trigger an assertive announcement.

// components/UserSelect/UserSelectView.tsx  
return (
    <section className="panel" aria-label="Select a user to chat">
      <h2 className="panel-title">Choose a user</h2>

      {error && (
        <div className="error-text" role="alert">
          {error}
        </div>
      )}

      <div className="user-list" role="list">
        {!users.length && !isLoading && (
          <div className="empty-state">
            <p>Load available users to start chatting</p>
            <button
              className="button primary"
              onClick={loadUsers}
              type="button"
            >
              Load users
            </button>
          </div>
        )}

        {isLoading && (
          <div className="hint" role="status" aria-live="polite">
            Loading users...
          </div>
        )}

        {filteredUsers.map(user => (
          <button
            key={user.id}
            className="user-item"
            role="listitem"
            onClick={() => handleUserSelect(user)}
            aria-label={`Chat with ${user.name || user.id}`}
            type="button"
          >
            {user.image ? (
              <img
                src={user.image}
                alt=""
                className="user-avatar"
                aria-hidden="true"
              />
            ) : (
              <span className="user-avatar" aria-hidden="true">
                {getUserInitial(user)}
              </span>
            )}
            <span className="user-name">{user.name || user.id}</span>
          </button>
        ))}
      </div>

      <div className="panel-actions">
        <button
          className="button"
          onClick={onBack}
          type="button"
        >
          Back
        </button>
      </div>
    </section>
  );
};

Message List: Semantic Structure with ARIA

The MessageList.tsx implements the crucial Live Log Accessibility Pattern and advanced keyboard review.

Live Log Pattern: The main message container is assigned role="log". This is essential for dynamic chat, as it tells screen readers that new, ordered content (messages) will be added frequently. This ensures new messages are announced politely without interrupting the user's focus on the input field.
Keyboard Navigation: The list container is made focusable (tabIndex={0}) and includes custom logic that enables users to navigate the entire message history using the vertical Arrow Keys (Up/Down), as well as the Home and End keys to jump to the beginning or end of the conversation.
Accessibility Labelling: The container uses aria-describedby to link to a hidden element containing instructions for keyboard navigation.

// components/AccessibleChat/MessageList.tsx
import React, { useRef, useCallback, useState, useEffect } from 'react';
import { useScreenReader } from '../../hooks';
import type { AccessibleMessage } from '../../types';
import { MessageItem } from './MessageItem';

interface AccessibleMessageListProps {
  messages: AccessibleMessage[];
  client?: any;
  typingUsers?: string[];
}

export const AccessibleMessageList: React.FC<AccessibleMessageListProps> = ({ 
  messages, 
  client,
  typingUsers = []
}) => {
  const listRef = useRef<HTMLDivElement>(null);
  const [selectedMessageIndex, setSelectedMessageIndex] = useState<number>(-1);
  const lastMessageIdRef = useRef<string>('');
  const { announce } = useScreenReader();

  // Announce new messages to screen readers
  useEffect(() => {
    const latestMessage = messages[messages.length - 1];
    if (!latestMessage || latestMessage.id === lastMessageIdRef.current) return;

    lastMessageIdRef.current = latestMessage.id || '';

    // Don't announce own messages
    if (latestMessage.user.id !== client?.user?.id) {
      const userName = latestMessage.user.name || 'Unknown user';
      const messageText = latestMessage.text || 'sent an attachment';
      announce(`New message from ${userName}: ${messageText}`, 'polite');
    }
  }, [messages, client, announce]);

  // Keyboard navigation for messages
  const handleKeyDown = useCallback((event: React.KeyboardEvent<HTMLDivElement>) => {
    const messageElements = listRef.current?.querySelectorAll<HTMLDivElement>('.message-item');
    if (!messageElements || messageElements.length === 0) return;

    switch (event.key) {
      case 'ArrowUp':
        event.preventDefault();
        setSelectedMessageIndex(prev => {
          const newIndex = Math.max(0, prev === -1 ? messageElements.length - 1 : prev - 1);
          messageElements[newIndex]?.focus();
          messageElements[newIndex]?.scrollIntoView({ block: 'nearest', behavior: 'smooth' });
          return newIndex;
        });
        break;

      case 'ArrowDown':
        event.preventDefault();
        setSelectedMessageIndex(prev => {
          const newIndex = prev === -1 ? 0 : Math.min(messageElements.length - 1, prev + 1);
          messageElements[newIndex]?.focus();
          messageElements[newIndex]?.scrollIntoView({ block: 'nearest', behavior: 'smooth' });
          return newIndex;
        });
        break;

      case 'Home':
        event.preventDefault();
        setSelectedMessageIndex(0);
        messageElements[0]?.focus();
        messageElements[0]?.scrollIntoView({ block: 'start', behavior: 'smooth' });
        break;

      case 'End':
        event.preventDefault();
        const lastIndex = messageElements.length - 1;
        setSelectedMessageIndex(lastIndex);
        messageElements[lastIndex]?.focus();
        messageElements[lastIndex]?.scrollIntoView({ block: 'end', behavior: 'smooth' });
        break;
    }
  }, []);

  return (
    <div className="accessible-message-list">
      {/* Hidden live region for screen reader announcements */}
      <div 
        aria-live="polite" 
        aria-atomic="false"
        className="sr-only"
        id="message-announcements"
      />

      <div
        ref={listRef}
        className="message-list-container"
        role="log"
        aria-label={`Chat messages, ${messages.length} total. Use arrow keys to navigate.`}
        onKeyDown={handleKeyDown}
        tabIndex={0}
        aria-describedby="navigation-help"
      >
        <div id="navigation-help" className="sr-only">
          Use arrow keys to navigate messages, Enter to select, Home and End to jump to first or last message
        </div>

        {messages.map((message, index) => (
          <MessageItem 
            key={message.id || `message-${index}`}
            message={message} 
            client={client}
            isSelected={index === selectedMessageIndex}
            onSelect={() => setSelectedMessageIndex(index)}
          />
        ))}

        {/* Typing indicator */}
        {typingUsers.length > 0 && (
          <div className="typing-indicator" aria-live="polite" role="status">
            <div className="typing-animation" aria-hidden="true">
              <span></span>
              <span></span>
              <span></span>
            </div>
            <span className="typing-text">
              {typingUsers.length === 1 
                ? `${typingUsers[0]} is typing...`
                : `${typingUsers.join(', ')} are typing...`
              }
            </span>
          </div>
        )}
      </div>
    </div>
  );
};

Individual Message Item Structure

The MessageItem focuses on ensuring a rich semantic context for each entry in the chat log.

Semantic Structure: Each message uses role="article", defining it as a self-contained, independent piece of content.
Logical Association: It uses the aria-labelledby and aria-describedby attributes to explicitly link the message content and timestamp (<time>) back to the author's name, guaranteeing the screen reader announces a complete, coherent unit.
Timestamp Clarity: The timestamp is provided using the semantic <time> element with a machine-readable dateTime attribute, while an aria-label provides a human-friendly reading of the time.
Decorative Images: The avatar images use alt="" and aria-hidden="true", correctly marking them as decorative, since the author's name is already announced via aria-labelledby.

// components/AccessibleChat/MessageItem.tsx
import React, { useRef, useCallback, useMemo, memo } from 'react';
import type { AccessibleMessage } from '../../types';
import { AttachmentComponent } from './AttachmentComponent';
import { EnhancedText } from './EnhancedText';

interface MessageItemProps {
  message: AccessibleMessage;
  client?: any;
  isSelected?: boolean;
  onSelect?: () => void;
  readBy?: string[];
}

export const MessageItem: React.FC<MessageItemProps> = memo(({ 
  message, 
  client, 
  isSelected, 
  onSelect,
  readBy = []
}) => {
  const messageRef = useRef<HTMLDivElement>(null);
  const isOwn = message.user.id === client?.user?.id;

  const createdAt = useMemo(() => 
    message.created_at ? new Date(message.created_at) : null,
    [message.created_at]
  );

  const avatarUrl = useMemo(() => 
    message.user.image || 
    `https://api.dicebear.com/7.x/initials/svg?seed=${encodeURIComponent(message.user.name || message.user.id)}`,
    [message.user.image, message.user.name, message.user.id]
  );

  const handleInteraction = useCallback((e: React.MouseEvent | React.KeyboardEvent) => {
    if ('key' in e && e.key !== 'Enter' && e.key !== ' ') return;
    e.preventDefault();
    onSelect?.();
  }, [onSelect]);

  return (
    <div
      ref={messageRef}
      className={`message-item ${isOwn ? 'own-message' : 'other-message'} ${isSelected ? 'selected' : ''}`}
      role="article"
      aria-labelledby={`message-${message.id}-author`}
      aria-describedby={`message-${message.id}-content message-${message.id}-time`}
      tabIndex={0}
      onClick={handleInteraction}
      onKeyDown={handleInteraction}
    >
      {/* Message author with avatar */}
      <div 
        id={`message-${message.id}-author`}
        className="message-author"
        aria-label={`Message from ${message.user.name || message.user.id}`}
      >
        <img 
          src={avatarUrl}
          alt=""
          className="user-avatar"
          aria-hidden="true"
          width="32"
          height="32"
        />
        <span>{message.user.name || message.user.id}</span>
      </div>

      {/* Message content */}
      <div 
        id={`message-${message.id}-content`}
        className="message-content"
      >
        {message.text && (
          <div className="message-text">
            <EnhancedText text={message.text} messageId={message.id || ''} />
          </div>
        )}

        {/* Attachments */}
        {message.attachments?.map((attachment, index) => (
          <AttachmentComponent 
            key={`${message.id}-attachment-${index}`}
            attachment={attachment}
            messageId={message.id || ''}
          />
        ))}
      </div>

      {/* Message footer with timestamp and read receipts */}
      <div className="message-footer">
        <time 
          id={`message-${message.id}-time`}
          className="message-timestamp"
          dateTime={createdAt?.toISOString() || ''}
          aria-label={`Sent at ${createdAt?.toLocaleString() || 'Unknown time'}`}
        >
          {createdAt?.toLocaleTimeString() || ''}
        </time>

        {isOwn && (
          <div 
            className="message-status"
            aria-label={readBy.length > 0 ? `Read by ${readBy.join(', ')}` : 'Sent'}
          >
            <span className="read-status" aria-hidden="true">
              {readBy.length > 0 ? '✓✓' : '✓'}
            </span>
          </div>
        )}
      </div>
    </div>
  );
});

MessageItem.displayName = 'MessageItem';

Accessible Message Input with File Attachments

The MessageInput is designed to be highly predictable and to announce errors, preventing confusion between keyboard and screen readers immediately.

Form Semantics: The component utilises the native <form> element with clear aria-labels for its overall purpose and provides status feedback on actions (e.g., "Message sent successfully") via the announce() hook.
Error Reporting: When validation fails (e.g., empty message), the error is placed in a container with role="alert", triggering an assertive announcement to notify the user of the failure immediately. The component also sets the aria-invalid attribute on the text area to flag the input itself as having an error.
Accessibility Labelling & Help: The main text area utilises a hidden <label> (sr-only) and is linked to hidden instructions using aria-describedby, which guides keyboard users on shortcuts such as Enter (send) and Shift+Enter (new line).
Character Count: The character count feature utilises aria-live="polite" to notify users when they are approaching the character limit, but only after they pause typing, thereby preventing overwhelming feedback.

// components/AccessibleChat/MessageInput.tsx
import React, { useState, useRef, useCallback } from 'react';
import { useScreenReader } from '../../utils/screenReader';
import './ChatStyles.css';

interface AccessibleMessageInputProps {
  onSubmit: (message: string, attachments?: File[]) => Promise<void>;
  maxLength?: number;
  maxFileSize?: number;
  allowedFileTypes?: string[];
}

interface AttachmentUpload {
  file: File;
  type: 'image' | 'video' | 'audio' | 'file';
}

const DEFAULT_MAX_LENGTH = 1000;
const DEFAULT_MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB

export const AccessibleMessageInput: React.FC<AccessibleMessageInputProps> = ({ 
  onSubmit, 
  maxLength = DEFAULT_MAX_LENGTH,
  maxFileSize = DEFAULT_MAX_FILE_SIZE,
  allowedFileTypes = ['image/*', 'video/*', 'audio/*', '.pdf', '.doc', '.docx']
}) => {
  const [message, setMessage] = useState<string>('');
  const [errors, setErrors] = useState<string[]>([]);
  const [isSubmitting, setIsSubmitting] = useState<boolean>(false);
  const [attachments, setAttachments] = useState<AttachmentUpload[]>([]);

  const inputRef = useRef<HTMLTextAreaElement>(null);
  const fileInputRef = useRef<HTMLInputElement>(null);
  const { announce } = useScreenReader();

  const handleSubmit = useCallback(async (event: React.FormEvent<HTMLFormElement>) => {
    event.preventDefault();
    setErrors([]);

    if (!message.trim() && attachments.length === 0) {
      const error = 'Message or attachment required';
      setErrors([error]);
      inputRef.current?.focus();
      announce(error, 'assertive');
      return;
    }

    try {
      setIsSubmitting(true);
      await onSubmit(message, attachments.map(a => a.file));
      setMessage('');
      setAttachments([]);
      announce('Message sent successfully', 'polite');
    } catch (error) {
      const errorMessage = error instanceof Error ? error.message : 'Failed to send message';
      setErrors([errorMessage]);
      announce(errorMessage, 'assertive');
      inputRef.current?.focus();
    } finally {
      setIsSubmitting(false);
    }
  }, [message, attachments, onSubmit, announce]);

  const handleKeyDown = useCallback((event: React.KeyboardEvent<HTMLTextAreaElement>) => {
    // Enter to send, Shift+Enter for new line
    if (event.key === 'Enter' && !event.shiftKey) {
      event.preventDefault();
      const form = event.currentTarget.closest('form');
      if (form) {
        form.dispatchEvent(new Event('submit', { bubbles: true, cancelable: true }));
      }
    }

    // Escape to clear
    if (event.key === 'Escape') {
      setMessage('');
      setErrors([]);
      announce('Message cleared', 'polite');
    }
  }, [announce]);

  const formatFileSize = (bytes: number): string => {
    if (bytes === 0) return '0 Bytes';
    const k = 1024;
    const sizes = ['Bytes', 'KB', 'MB', 'GB'];
    const i = Math.floor(Math.log(bytes) / Math.log(k));
    return `${parseFloat((bytes / Math.pow(k, i)).toFixed(2))} ${sizes[i]}`;
  };

  const characterCount = message.length;
  const isOverLimit = characterCount > maxLength;
  const isNearLimit = characterCount > maxLength * 0.9;

  return (
    <form 
      className="accessible-message-input" 
      onSubmit={handleSubmit}
      role="form"
      aria-label="Send message"
    >
      {/* Error messages */}
      {errors.length > 0 && (
        <div 
          className="error-messages"
          role="alert"
          aria-live="assertive"
        >
          {errors.map((error, index) => (
            <div key={index} className="error-message">
              {error}
            </div>
          ))}
        </div>
      )}

      {/* Attachment preview */}
      {attachments.length > 0 && (
        <div 
          className="attachment-preview" 
          role="list" 
          aria-label={`${attachments.length} selected attachment${attachments.length !== 1 ? 's' : ''}`}
        >
          {attachments.map((attachment, index) => (
            <div 
              key={`${attachment.file.name}-${index}`}
              className="attachment-item"
              role="listitem"
            >
              <span className="attachment-info">
                <span className="attachment-name">{attachment.file.name}</span>
                <span className="attachment-size">
                  ({formatFileSize(attachment.file.size)})
                </span>
              </span>
              <button
                type="button"
                onClick={() => {
                  setAttachments(prev => {
                    const newAttachments = [...prev];
                    newAttachments.splice(index, 1);
                    announce(`Removed attachment ${attachment.file.name}`, 'polite');
                    return newAttachments;
                  });
                }}
                aria-label={`Remove ${attachment.file.name}`}
                className="remove-attachment"
              >
                ×
              </button>
            </div>
          ))}
        </div>
      )}

      {/* Message textarea */}
      <div className="input-container">
        <label htmlFor="message-input" className="sr-only">
          Type your message
        </label>

        <textarea
          id="message-input"
          ref={inputRef}
          value={message}
          onChange={(e) => setMessage(e.target.value)}
          onKeyDown={handleKeyDown}
          placeholder="Type your message... (Enter to send, Shift+Enter for new line)"
          className={`message-textarea ${isOverLimit ? 'over-limit' : ''}`}
          aria-describedby="message-input-help character-count"
          aria-invalid={errors.length > 0}
          maxLength={maxLength}
          rows={1}
          disabled={isSubmitting}
        />

        <div id="message-input-help" className="sr-only">
          Enter to send message, Shift+Enter for new line, Escape to clear
        </div>
      </div>

      {/* Action buttons */}
      <div className="input-actions">
        <button
          type="submit"
          className="send-button"
          disabled={(!message.trim() && attachments.length === 0) || isSubmitting || isOverLimit}
          aria-label={isSubmitting ? "Sending message..." : "Send message"}
        >
          {isSubmitting ? 'Sending...' : 'Send'}
        </button>

        {/* Hidden file input */}
        <input
          ref={fileInputRef}
          type="file"
          multiple
          accept={allowedFileTypes.join(',')}
          onChange={(e) => {
            const files = Array.from(e.target.files || []);
            // Add files logic here
          }}
          className="sr-only"
          aria-label="Add attachment"
          tabIndex={-1}
          id="file-input"
        />

        <button
          type="button"
          className="attachment-button"
          aria-label="Add attachment"
          onClick={() => fileInputRef.current?.click()}
          disabled={isSubmitting}
        >
          <span aria-hidden="true">📎</span>
        </button>
      </div>

      {/* Character count */}
      <div className="message-info">
        <div 
          id="character-count"
          className={`character-count ${isNearLimit ? 'warning' : ''} ${isOverLimit ? 'error' : ''}`} 
          aria-live="polite"
          aria-label={`${characterCount} of ${maxLength} characters used`}
        >
          {characterCount}/{maxLength}
        </div>
      </div>
    </form>
  );
};

Handling Emojis Accessibly

The EnhancedText.tsx component implements a crucial, advanced pattern to make visual emojis understandable for screen readers.

By default, a screen reader might read an emoji as a cryptic sequence of characters or just "image," which lacks context and obscures the message's true tone. To address this accessibility issue, the component identifies emojis within the text and wraps them in a <span> element. It then applies role="img" along with a descriptive aria-label (gotten from an EMOJI_MAP).

As a result, when a user encounters text like "Great job! 🎉," the screen reader announces "Great job! party popper" instead of an unrecognisable symbol, ensuring universal understanding of the message's content and tone.

// components/AccessibleChat/EnhancedText.tsx
import React, { useMemo, memo } from 'react';
import { EMOJI_MAP } from '../../utils/constants';

interface EnhancedTextProps {
  text: string;
  messageId: string;
}

export const EnhancedText: React.FC<EnhancedTextProps> = memo(({ text, messageId }) => {
  const parts = useMemo(() => {
    // Split text on emoji characters
    return text.split(/([\u{1F600}-\u{1F64F}]|[\u{1F300}-\u{1F5FF}]|[\u{1F680}-\u{1F6FF}]|[\u{1F1E0}-\u{1F1FF}]|[\u{2600}-\u{26FF}]|[\u{2700}-\u{27BF}])/u);
  }, [text]);

  return (
    <>
      {parts.map((part, index) => {
        // If part is an emoji, add aria-label
        if (EMOJI_MAP[part]) {
          return (
            <span key={`${messageId}-emoji-${index}`} role="img" aria-label={EMOJI_MAP[part]}>
              {part}
            </span>
          );
        }
        return <span key={`${messageId}-text-${index}`}>{part}</span>;
      })}
    </>
  );
});

EnhancedText.displayName = 'EnhancedText';

// utils/constants.ts
export const EMOJI_MAP: Record<string, string> = {
  '😀': 'grinning face',
  '😂': 'face with tears of joy',
  '❤️': 'red heart',
  '👍': 'thumbs up',
  '👎': 'thumbs down',
  '🎉': 'party popper',
  '🔥': 'fire',
  '💯': 'hundred points symbol'
};

Below is an image showing the messages sent between two users.

Build Inclusive Video

Meeting Management

The Meeting component manages the entire video call lifecycle, including creating, joining, and displaying the active call interface.

It has four states: Pre-call (for ID input/join/create), Active call (with full video interface), Loading (disables buttons), and Error (displaying accessible messages).

Accessibility as a key focus is achieved through:

Conditional rendering for a simple UI.
Screen reader announcements for the "Generate ID" button.
Properly labelled form inputs.
role="alert" for immediate error feedback and role="note" for supplementary info.

A consistent "Back" button (except during loading).

// components/Meeting/MeetingView.tsx 
// The announcement hook integration is the key logic to keep.
const handleGenerateId = () => {
  const id = `meeting-${Math.random().toString(36).slice(2, 7)}`;
  setMeetingId(id);
  announce(`Generated meeting ID: ${id}`, 'polite'); 
};

// The main JSX returns the form when there is NO active call
return (
  <section className="panel" aria-label="Start or join a meeting">
    <h2 className="panel-title">Start or join a meeting</h2>

    {transcriptionAvailable && (
      <div className="feature-notice" 
        // ARIA: role="note" marks this as supplementary information
        role="note"
      >
        This meeting will support live captions powered by Stream Video closed captions.
      </div>
    )}

    <div className="meeting-form">
      {/* Input Field with Proper Labeling */}
      <label className="field">
        <span>Meeting ID</span>
        <input
          className="input"
          placeholder="Enter meeting ID (or generate one)"
          value={meetingId}
          onChange={e => setMeetingId(e.target.value)}
          // ARIA: Labeling is essential for screen readers
          aria-label="Meeting ID" 
          disabled={isLoading}
        />
      </label>

      <div className="buttons-row">
        {/* Primary Action Button: Dynamic label based on meetingId state */}
        <button
          className="button primary"
          onClick={() => meetingId ? onJoinMeeting(meetingId) : handleGenerateId()}
          disabled={isLoading}
          type="button"
        >
          {/* Dynamic text for loading state and action */}
          {isLoading ? 'Starting...' : (meetingId ? 'Start Meeting' : 'Generate ID')}
        </button>

        {/* Secondary Action Button (Join Meeting) */}
        <button
          className="button secondary"
          onClick={() => onJoinMeeting(meetingId)}
          disabled={!meetingId || isLoading}
          type="button"
        >
          {isLoading ? 'Joining...' : 'Join Meeting'}
        </button>
      </div>
    </div>

    {/* Error Display */}
    {error && (
      <div 
        // ARIA: role="alert" ensures immediate, assertive screen reader announcement
        role="alert" 
        className="error-text"
      >
        {error}
      </div>
    )}
  </section>
);

Custom Accessible Video Controls

The VideoControls implement the Toolbar Accessibility Pattern and dynamic status feedback.

The container uses role="toolbar" and relies on custom JavaScript logic to enable Arrow Key navigation between buttons.
Each button uses an aria-label that dynamically announces the action (e.g., "Turn on microphone") and an aria-pressed attribute to confirm the current state (muted/unmuted). Toggling a button calls the announce() hook for polite confirmation.

// components/AccessibleVideo/VideoControls.tsx
import React, { useState, useRef, useEffect, useCallback } from 'react';
import { useCallStateHooks, type Call } from '@stream-io/video-react-sdk';
import { useScreenReader } from '../../hooks';

interface VideoControlsProps {
  call: Call;
  onToggleFullscreen: () => void;
  onLeaveCall: () => Promise<void>;
  onToggleCaptions: () => void;
  captionsEnabled: boolean;
  captionsSupported: boolean;
}

export const AccessibleVideoControls: React.FC<VideoControlsProps> = ({ 
  call, 
  onToggleFullscreen,
  onLeaveCall,
  onToggleCaptions,
  captionsEnabled,
  captionsSupported
}) => {
  const {
    useCameraState,
    useMicrophoneState,
    useScreenShareState
  } = useCallStateHooks();

  const { camera, isMute: isCameraMuted } = useCameraState();
  const { microphone, isMute: isMicMuted } = useMicrophoneState();
  const { screenShare, isMute: isScreenShareMuted } = useScreenShareState();

  const [isToggling, setIsToggling] = useState({
    mic: false,
    camera: false,
    screen: false
  });

  const controlsRef = useRef<HTMLDivElement>(null);
  const { announce } = useScreenReader();

  // Keyboard navigation between controls
  const navigateControls = useCallback((direction: number) => {
    const buttons = controlsRef.current?.querySelectorAll<HTMLButtonElement>('button:not(:disabled)');
    if (!buttons) return;

    const currentIndex = Array.from(buttons).findIndex(btn => btn === document.activeElement);
    const nextIndex = Math.max(0, Math.min(buttons.length - 1, currentIndex + direction));
    buttons[nextIndex]?.focus();
  }, []);

  useEffect(() => {
    const handleKeyDown = (event: KeyboardEvent) => {
      if (!controlsRef.current?.contains(document.activeElement)) return;

      switch (event.key) {
        case 'ArrowLeft':
        case 'ArrowRight':
          event.preventDefault();
          navigateControls(event.key === 'ArrowLeft' ? -1 : 1);
          break;
        case 'Enter':
        case ' ':
          event.preventDefault();
          (document.activeElement as HTMLButtonElement)?.click();
          break;
      }
    };

    document.addEventListener('keydown', handleKeyDown);
    return () => document.removeEventListener('keydown', handleKeyDown);
  }, [navigateControls]);

  const handleToggleMicrophone = useCallback(async (): Promise<void> => {
    if (isToggling.mic) return;

    try {
      setIsToggling(prev => ({ ...prev, mic: true }));
      await microphone.toggle();

      announce(
        isMicMuted ? 'Microphone turned on' : 'Microphone turned off',
        'polite'
      );
    } catch (error) {
      announce('Failed to toggle microphone', 'assertive');
      console.error('Failed to toggle microphone:', error);
    } finally {
      setIsToggling(prev => ({ ...prev, mic: false }));
    }
  }, [isToggling.mic, microphone, isMicMuted, announce]);

  const handleToggleCamera = useCallback(async (): Promise<void> => {
    if (isToggling.camera) return;

    try {
      setIsToggling(prev => ({ ...prev, camera: true }));
      await camera.toggle();

      announce(
        isCameraMuted ? 'Camera turned on' : 'Camera turned off',
        'polite'
      );
    } catch (error) {
      announce('Failed to toggle camera', 'assertive');
      console.error('Failed to toggle camera:', error);
    } finally {
      setIsToggling(prev => ({ ...prev, camera: false }));
    }
  }, [isToggling.camera, camera, isCameraMuted, announce]);

  const handleToggleScreenShare = useCallback(async (): Promise<void> => {
    if (isToggling.screen) return;

    try {
      setIsToggling(prev => ({ ...prev, screen: true }));
      await screenShare.toggle();

      announce(
        isScreenShareMuted ? 'Screen sharing started' : 'Screen sharing stopped',
        'polite'
      );
    } catch (error) {
      announce('Failed to toggle screen share', 'assertive');
      console.error('Failed to toggle screen share:', error);
    } finally {
      setIsToggling(prev => ({ ...prev, screen: false }));
    }
  }, [isToggling.screen, screenShare, isScreenShareMuted, announce]);

  const handleLeaveCall = useCallback(async (): Promise<void> => {
    const confirmed = window.confirm('Are you sure you want to leave this call?');
    if (!confirmed) return;

    try {
      announce('Leaving the call...', 'polite');
      await onLeaveCall();
    } catch (error) {
      console.error('Error leaving call:', error);
      announce('Error leaving call', 'assertive');
    }
  }, [onLeaveCall, announce]);

  return (
    <div 
      ref={controlsRef}
      className="video-controls"
      role="toolbar"
      aria-label="Video call controls. Use arrow keys to navigate, Enter to activate."
    >
      <div className="primary-controls">
        {/* Microphone control */}
        <button
          className={`control-button mic-control ${isMicMuted ? 'muted' : 'active'}`}
          onClick={handleToggleMicrophone}
          aria-label={isMicMuted ? 'Turn on microphone' : 'Turn off microphone'}
          aria-pressed={!isMicMuted}
          disabled={isToggling.mic}
          type="button"
        >
          <span aria-hidden="true">
            {isToggling.mic ? '⏳' : (isMicMuted ? '🔇' : '🎤')}
          </span>
          <span className="control-text">
            {isToggling.mic ? 'Toggling...' : (isMicMuted ? 'Mic Off' : 'Mic On')}
          </span>
        </button>

        {/* Camera control */}
        <button
          className={`control-button camera-control ${isCameraMuted ? 'muted' : 'active'}`}
          onClick={handleToggleCamera}
          aria-label={isCameraMuted ? 'Turn on camera' : 'Turn off camera'}
          aria-pressed={!isCameraMuted}
          disabled={isToggling.camera}
          type="button"
        >
          <span aria-hidden="true">
            {isToggling.camera ? '⏳' : (isCameraMuted ? '📹' : '📷')}
          </span>
          <span className="control-text">
            {isToggling.camera ? 'Toggling...' : (isCameraMuted ? 'Camera Off' : 'Camera On')}
          </span>
        </button>

        {/* Screen share control */}
        <button
          className={`control-button screen-share-control ${!isScreenShareMuted ? 'active' : ''}`}
          onClick={handleToggleScreenShare}
          aria-label={isScreenShareMuted ? 'Start screen sharing' : 'Stop screen sharing'}
          aria-pressed={!isScreenShareMuted}
          disabled={isToggling.screen}
          type="button"
        >
          <span aria-hidden="true">
            {isToggling.screen ? '⏳' : '🖥️'}
          </span>
          <span className="control-text">
            {isToggling.screen ? 'Toggling...' : (isScreenShareMuted ? 'Share Screen' : 'Stop Sharing')}
          </span>
        </button>

        {/* Leave call button */}
        <button
          className="control-button end-call"
          onClick={handleLeaveCall}
          aria-label="Leave call"
          type="button"
        >
          <span aria-hidden="true">📞</span>
          <span className="control-text">Leave</span>
        </button>
      </div>

      <div className="secondary-controls">
        {/* Fullscreen toggle */}
        <button
          className="control-button fullscreen-button"
          onClick={onToggleFullscreen}
          aria-label="Toggle fullscreen"
          type="button"
        >
          <span aria-hidden="true">⛶</span>
          <span className="control-text">Fullscreen</span>
        </button>

        {/* Captions toggle */}
        <button
          className={`control-button captions-button ${captionsEnabled ? 'active' : ''}`}
          onClick={onToggleCaptions}
          aria-label={
            !captionsSupported 
              ? 'Live captions not supported' 
              : captionsEnabled 
                ? 'Turn off live captions' 
                : 'Turn on live captions'
          }
          aria-pressed={captionsEnabled}
          disabled={!captionsSupported}
          type="button"
        >
          <span aria-hidden="true">
            {!captionsSupported ? '📝❌' : '📝'}
          </span>
          <span className="control-text">
            {!captionsSupported 
              ? 'N/A' 
              : captionsEnabled 
                ? 'Captions On' 
                : 'Captions'
            }
          </span>
        </button>
      </div>
    </div>
  );
};

Live Captions with Stream's Transcription API

The TranscriptDisplay implements the Supplementary Live Region Pattern.

It uses role="complementary" to mark the captions displayed as secondary content related to the video.
The main transcript area uses aria-live="polite" and aria-atomic="false", ensuring that new captions are announced smoothly without requiring the screen reader to read the entire transcript history every time a new word appears.
Each caption includes speaker identification, and auto-scrolling keeps the latest captions visible. Users receive status feedback, informing them whether captions are active, loading, or in error.

// components/AccessibleVideo/TranscriptDisplay.tsx
import React, { useRef, useEffect } from 'react';

export interface TranscriptData {
  sessionId: string;
  text: string;
  userId?: string;
  timestamp: number;
  isFinal: boolean;
  speaker?: string;
}

interface TranscriptDisplayProps {
  transcripts: TranscriptData[];
  enabled: boolean;
  status: 'idle' | 'starting' | 'active' | 'error';
}

export const TranscriptDisplay: React.FC<TranscriptDisplayProps> = ({
  transcripts,
  enabled,
  status
}) => {
  const captionsRef = useRef<HTMLDivElement>(null);

  // Auto-scroll to latest caption
  useEffect(() => {
    if (captionsRef.current && transcripts.length > 0) {
      captionsRef.current.scrollTop = captionsRef.current.scrollHeight;
    }
  }, [transcripts]);

  const getStatusMessage = (): string => {
    switch (status) {
      case 'starting':
        return 'Starting live captions...';
      case 'active':
        return 'Listening for speech... Powered by Stream Video closed captions.';
      case 'error':
        return 'Caption error. Try toggling captions off and on again.';
      default:
        return 'Click the captions button to enable live closed captions.';
    }
  };

  if (!enabled) return null;

  return (
    <div 
      className="live-captions-container"
      role="complementary"
      aria-label="Live captions"
      aria-live="polite"
      aria-atomic="false"
    >
      <div 
        ref={captionsRef}
        className="captions-content"
      >
        {transcripts.length > 0 ? (
          transcripts.map((transcript) => (
            <div 
              key={`${transcript.sessionId}-${transcript.timestamp}`}
              className={`caption-item ${transcript.isFinal ? 'final' : 'interim'}`}
              aria-label={`${transcript.speaker} said: ${transcript.text}`}
            >
              <strong className="caption-speaker" aria-hidden="true">
                {transcript.speaker}:
              </strong>
              <span className="caption-text">
                {' '}{transcript.text}
              </span>
            </div>
          ))
        ) : (
          <div className="caption-placeholder">
            <span className="caption-text">
              {getStatusMessage()}
            </span>
          </div>
        )}
      </div>

      <div className="captions-status" aria-live="polite">
        <span className="sr-only">
          Stream captions status: {status}
        </span>
      </div>
    </div>
  );
};

Caption Implementation in VideoContainer

The VideoContainer component is responsible for setting up and managing the real-time transcription feed, ensuring two primary accessibility points:

Accessible Toggle Feedback: The toggleCaptions function uses the announce() hook to provide immediate feedback on critical state changes. If captions are toggled on, it announces: "Starting live captions..." (polite). If the Stream API reports an error, it announces: "Failed to toggle captions" (assertive).
Real-Time Caption Announcement: The component uses a useEffect hook to listen directly to the Stream SDK's closed caption events (call.on('call.closed_caption', ...)). When a new transcription event arrives, the handler performs three critical steps:
- It associates the caption text with the correct speaker's name.
- It updates the visible TranscriptDisplay component.
- It immediately calls the announce() hook to narrate the speaker and caption text (e.g., "Participant: Welcome to the meeting"), ensuring deaf or hard-of-hearing users following the transcript via a screen reader get timely updates.

// components/AccessibleVideo/VideoContainer.tsx (excerpt)
const toggleCaptions = useCallback(async () => {
  if (!transcriptionSupported) {
    announce('Live captions not supported', 'assertive');
    return;
  }

  if (!call) {
    announce('No active call for captions', 'assertive');
    return;
  }

  try {
    setTranscriptionStatus('starting');

    if (captionsEnabled) {
      await call.stopClosedCaptions();
      announce('Stopping live captions...', 'polite');
    } else {
      // Start captions with English language
      await call.startClosedCaptions({ language: 'en' });
      announce('Starting live captions...', 'polite');
    }
  } catch (error) {
    console.error('Captions toggle error:', error);
    setTranscriptionStatus('error');

    const errorMessage = error instanceof Error ? error.message : 'Failed to toggle captions';
    announce(errorMessage, 'assertive');
  }
}, [call, captionsEnabled, transcriptionSupported, announce]);

// Handle caption events
useEffect(() => {
  if (!call) return;

  const handleClosedCaption = (event: any) => {
    const closedCaption = event.closed_caption || {};
    const captionText = closedCaption.text || '';
    const speakerId = closedCaption.speaker_id || '';

    if (!captionText.trim()) return;

    const participant = participants.find(p => 
      p.userId === speakerId || p.sessionId === speakerId
    );

    const speakerName = participant?.name || 'Participant';

    const newTranscript: TranscriptData = {
      sessionId: speakerId || `session-${Date.now()}`,
      text: captionText.trim(),
      userId: speakerId,
      timestamp: Date.now(),
      isFinal: true,
      speaker: speakerName
    };

    setTranscripts(prev => [...prev, newTranscript].slice(-10));
    announce(`${speakerName}: ${captionText}`, 'polite');
  };

  call.on('call.closed_caption', handleClosedCaption);

  return () => {
    call.off('call.closed_caption', handleClosedCaption);
  };
}, [call, participants, announce]);

Below is an image of two users having video meetings with caption enabled:

Error State Handling

Accessible error announcements are critical for real-time applications. The VideoContainer monitors for connection issues and ensures instant, interruptive feedback. When a connection is lost, the handler triggers an obvious error banner and uses an assertive announcement. When the connection returns, it uses a polite announcement to confirm recovery. The error message container itself uses role="alert" and aria-live="assertive" to ensure maximum visibility for screen readers.

// In VideoContainer.tsx
const [connectionError, setConnectionError] = useState<string | null>(null);

useEffect(() => {
  if (!call) return;

  const handleConnectionError = (event: any) => {
    const errorMessage = 'Connection lost. Attempting to reconnect...';
    setConnectionError(errorMessage);
    announce(errorMessage, 'assertive');
  };

  const handleReconnected = () => {
    setConnectionError(null);
    announce('Connection restored', 'polite');
  };

  call.on('call.session_participant_left', handleConnectionError);
  call.on('call.session_participant_joined', handleReconnected);

  return () => {
    call.off('call.session_participant_left', handleConnectionError);
    call.off('call.session_participant_joined', handleReconnected);
  };
}, [call, announce]);

// Error display component
{connectionError && (
  <div 
    className="error-banner" 
    role="alert" 
    aria-live="assertive"
  >
    <span role="img" aria-label="Error">⚠️</span>
    <span>{connectionError}</span>
  </div>
)}

Testing Accessibility

Keyboard-Only Testing

Keyboard-only testing is the single most critical manual validation step. It verifies that your custom Toolbar and Message List navigation logic works and ensures no user is trapped or confused by missing focus states.

The video below demonstrates the application's keyboard-only usage.

Lighthouse Checks

Lighthouse is an open-source, automated tool developed by Google that audits web page quality.

The Accessibility Score shown in the image confirms the page passes all automated checks for foundational accessibility. A score of 100 validates the following aspects of your code:

ARIA Compliance: All custom roles, states, and labels were implemented correctly.
Color Contrast: Text and background colors meet the strict WCAG 2.1 legibility standards.
Semantic Structure: The correct HTML elements and heading hierarchies were utilized.

Complete Production Code

The code examples in this article are simplified for readability and focus on key accessibility concepts. For the complete, production-ready implementation with all features, check out this repository.

Conclusion: Accessibility and Performance Trade-Offs

This project demonstrates how real-time chat and video applications can be made fully accessible by combining semantic HTML, dynamic ARIA updates, keyboard-first navigation, and custom screen reader announcements. Features like the announcement hook and accessible toolbar patterns ensure that users receive timely feedback and can operate the interface without relying on sight or a mouse.

Building for accessibility requires extra attention from managing focus manually to testing ARIA attributes and ensuring that screen readers interpret rapid updates correctly. However, the result is a more inclusive and dependable experience for every user.

How to Build a Secure React Native Chat App with End-to-End Encryption

Quincy Oghenetejiri — Tue, 14 Oct 2025 15:53:47 +0000

Building secure chat applications goes beyond just sending and receiving messages; true privacy requires end-to-end encryption (E2EE). With E2EE, only the intended participants can read the conversation, while even the service provider remains blind to the content.

In this guide, we'll learn how to implement E2EE in a React Native chat app using Stream's chat infrastructure. We'll combine public-key cryptography with efficient symmetric encryption to achieve strong security.

Technical Prerequisites

Before we begin, ensure you have the following:

Free Stream account
Node.js 14 or higher installed
Basic knowledge of React/React Native and Node.js
Android Studio (for Android development), Xcode (for iOS development), or Expo Go (Real Device)

Solution Architecture

Setting Up Your Development Environment

Backend Setup

We'll start by setting up the backend:

npm init
npm install express dotenv stream-chat nodemon cors

Now, create a .env file there:

STREAM_API_KEY=Your_stream_api_key
STREAM_API_SECRET=Your_stream_api_secret_key
PORT=5000

Next, create an app.js file with the following configuration:

require('dotenv').config();
const express = require('express');
const cors = require('cors');
const app = express();
app.use(cors());
app.use(express.json());

// Health check
app.get('/', (req, res) => {
  res.json({ status: 'ok' });
});

// Auth and Stream routes
const routes = require('./routes');
app.use('/api', routes);

The backend will now run on the configured port:

const PORT = process.env.PORT || 5000;
app.listen(PORT, () => {
  console.log(`Backend server running on port ${PORT}`);
});

The routes.js file defines all the routes required for the application. At the top, the Stream keys are initialized, and the client is configured with an extended timeout to improve reliability and ensure the frontend establishes a stable connection.

const express = require('express');
const { StreamChat } = require('stream-chat');

const router = express.Router();

const apiKey = process.env.STREAM_API_KEY;
const apiSecret = process.env.STREAM_API_SECRET;

// Configure Stream client with increased timeout
const streamServerClient = StreamChat.getInstance(apiKey, apiSecret, {
  timeout: 30000, // 30 seconds instead of default 3 seconds
});

This route allows user registration without encryption and is used solely for testing chat functionality in a non-encrypted environment.

// Registration without encryption key
router.post('/register-simple', async (req, res) => {
  const { userId, name } = req.body;
  if (!userId || !name) return res.status(400).json({ error: 'Missing fields' });
  try {
    await streamServerClient.upsertUser({ id: userId, name });
    res.json({ success: true });
  } catch (err) {
    console.error('Stream upsertUser error:', err);
    res.status(500).json({ error: 'Stream upsertUser failed', details: err.message });
  }
});

For encrypted registration, the route requires a user's public key, which is essential for the encryption process.

router.post('/register', async (req, res) => {
  const { userId, name, publicKey } = req.body;
  if (!userId || !name || !publicKey) return res.status(400).json({ error: 'Missing fields' });
  try {
    await streamServerClient.upsertUser({ id: userId, name, public_key: publicKey });
    res.json({ success: true });
  } catch (err) {
    console.error('Stream upsertUser error:', err);
    res.status(500).json({ error: 'Stream upsertUser failed', details: err.message });
  }
});

The login routes handle user authentication and issue tokens needed to verify users.

// Mock login (no password, just userId)
router.post('/login', async (req, res) => {
  const { userId } = req.body;
  if (!userId) return res.status(400).json({ error: 'Missing userId' });
  // In production, check password or OAuth
  res.json({ success: true });
});

// Issue Stream token
router.post('/stream/token', (req, res) => {
  const { userId } = req.body;
  if (!userId) return res.status(400).json({ error: 'Missing userId' });
  const token = streamServerClient.createToken(userId);
  res.json({ token });
});

These routes fetch all registered users so a logged-in user can choose someone to chat with.

// Get users
router.get('/users', async (req, res) => {
  try {
    const result = await streamServerClient.queryUsers({});

    // Ensure public_key is properly formatted
    const users = result.users.map(u => ({
      id: u.id,
      name: u.name,
      public_key: u.public_key || null,
    }));
    res.json({ users });
  } catch (err) {
    console.error('Error fetching users:', err);
    res.status(500).json({ error: err.message });
  }
});

Frontend Setup

To get started, create a new React Native project using Expo with the navigation (TypeScript) template:

# Create new React Native project
npx create-expo-app --template # and then select Navigation(Typescript)

After running the command above, you will have a fully set up React Native application with React Navigation configured for routing.

The structure of the application will look like this:

my-chat-app/
├── app/
│   ├── _layout.tsx
│   └── index.tsx
├── assets/
├── app.json
├── package.json
├── tsconfig.json
└── node_modules/

Now, let's install these packages, which you will need throughout the tutorial.

What do these dependencies do? The @nobles, expo-crypto, and expo-secure-store packages implement end-to-end encryption. The chat functionalities require stream-chat-expo. React Native's react-native-gesture-handler handles complex gestures (swipes, pans, taps, etc.).

npm install stream-chat-expo @noble/ciphers @noble/curves @noble/hashes expo-crypto expo-secure-store react-native-gesture-handler

To start building our chat app, delete the default app folder that comes with the code, then create an App.tsx file in the root directory and reference it in your package.json as shown below.

{
  "name": "navigation",
  "main": "App.tsx",
  "version": "1.0.0",
}

The App.tsx will work as the root file for our React Native application.

// App.tsx
import React from 'react';
import { NavigationContainer } from '@react-navigation/native';
import { createNativeStackNavigator } from '@react-navigation/native-stack';
import { UserProvider } from './context/UserContext';
import { OverlayProvider } from 'stream-chat-expo';
import { GestureHandlerRootView } from 'react-native-gesture-handler';
import AuthScreen from './screens/AuthScreen';
import UsersScreen from './screens/UsersScreen';
import ChatScreen from './screens/ChatScreen';
import { registerRootComponent } from 'expo';

export type RootStackParamList = {
  Auth: undefined;
  Users: undefined;
  Chat: {
    recipientId: string;
    recipientName: string;
  };
};

const Stack = createNativeStackNavigator<RootStackParamList>();

function App() {
  return (
    <GestureHandlerRootView style={{ flex: 1 }}>
      <NavigationContainer>
        <OverlayProvider>
          <UserProvider>
              <Stack.Navigator
                initialRouteName="Auth"
                screenOptions={{
                  headerStyle: {
                    backgroundColor: '#007AFF',
                  },
                  headerTintColor: '#fff',
                  headerTitleStyle: {
                    fontWeight: 'bold',
                  },
                }}
              >
                <Stack.Screen
                  name="Auth"
                  component={AuthScreen}
                  options={{
                    title: "🔐 Secure Login",
                    headerStyle: {
                      backgroundColor: '#2d7d2d',
                    },
                  }}
                />
                <Stack.Screen
                  name="Users"
                  component={UsersScreen}
                  options={{
                    title: "👥 Select User",
                    headerBackVisible: false, // Prevent going back without proper logout
                  }}
                />
                <Stack.Screen
                  name="Chat"
                  component={ChatScreen}
                  options={({ route }) => ({
                    title: `🔒 ${route.params.recipientName}`,
                    headerStyle: {
                      backgroundColor: '#2d7d2d',
                    },
                  })}
                />
              </Stack.Navigator>
          </UserProvider>
        </OverlayProvider>
      </NavigationContainer>
    </GestureHandlerRootView>
  );
}

export default registerRootComponent(App);

Setting Up the Initial Configuration

First, let's set up the Stream client.

//utils/streamClient.ts
import { StreamChat } from 'stream-chat';
import { STREAM_API_KEY } from '../config/constants';

export const streamClient = StreamChat.getInstance(STREAM_API_KEY);

Next, we'll create a config file containing our API key and backend URL logic.

//config/constants.ts
export const STREAM_API_KEY = 'Your_stream_api_key';
// Helper to get the correct backend URL based on environment
export function getBackendUrl() {
  // Uncomment ONE of the following as needed:

  // 1. For Android emulator:
  // return 'http://10.0.2.2:5000/api';

  // 2. For iOS simulator or web:
  // return 'http://localhost:5000/api';

  // 3. For real device (Android or iOS) on same Wi-Fi as your computer:
  // return 'http://192.168.118.229:5000/api';

  // Default: fallback to IP Address
  return 'http://Your_IP_Address:5000/api';
}

export const BACKEND_URL = getBackendUrl();

Managing User Authentication with Context

To avoid passing user authentication details (like userId and token) through props in every component, we'll use React Context. This allows us to store and share authentication state globally across the app.

// context/UserContext.tsx
import React, { createContext, useContext, useState, ReactNode } from 'react';

interface UserContextType {
  userId: string | null;
  token: string | null;
  setUser: (userId: string, token: string) => void;
  clearUser: () => void;
}

const UserContext = createContext<UserContextType | undefined>(undefined);

export function UserProvider({ children }: { children: ReactNode }) {
  const [userId, setUserId] = useState<string | null>(null);
  const [token, setToken] = useState<string | null>(null);

  const setUser = (newUserId: string, newToken: string) => {
    setUserId(newUserId);
    setToken(newToken);
  };

  const clearUser = () => {
    setUserId(null);
    setToken(null);
  };

  return (
    <UserContext.Provider value={{ userId, token, setUser, clearUser }}>
      {children}
    </UserContext.Provider>
  );
}

export function useUser() {
  const context = useContext(UserContext);
  if (!context) {
    throw new Error('useUser must be used within a UserProvider');
  }
  return context;
}

Creating Screens

Now, create a screens folder. This will hold the main pages of our app:

AuthScreen.tsx: Handles user registration and login
UsersScreen.tsx: Lists all available users and allows logout
ChatScreen.tsx: Handles one-to-one messaging

This is the present structure of our project files:

my-chat-app/
├── App.tsx
├──utils/
│   └── streamClient.ts
├── context/
│   └── UserContext.tsx
├── config/
│   └── constants.ts
├── screens/
│   ├── AuthScreen.tsx
│   ├── UserListScreen.tsx
│   └── ChatScreen.tsx
├── assets/
├── app.json
├── package.json
├── tsconfig.json
└── node_modules/

I will highlight each of these screens below.

1. AuthScreen.tsx (Login & Registration)

//screens/AuthScreen.tsx
import React, { useState } from 'react';
import {
  View,
  Text,
  TextInput,
  TouchableOpacity,
  ActivityIndicator,
  Alert,
  StyleSheet,
  ScrollView
} from 'react-native';
import { NativeStackNavigationProp } from '@react-navigation/native-stack';
import axios from 'axios';
import { streamClient } from '../utils/streamClient';
import { useUser } from '../context/UserContext';
import { BACKEND_URL } from '../config/constants';
import { RootStackParamList } from '../App';

type AuthScreenNavigationProp = NativeStackNavigationProp<RootStackParamList, 'Auth'>;

interface Props {
  navigation: AuthScreenNavigationProp;
}

export default function AuthScreen({ navigation }: Props) {
  const [userId, setUserId] = useState('');
  const [name, setName] = useState('');
  const [loading, setLoading] = useState(false);

  const { setUser } = useUser();

  const handleRegister = async () => {
    if (!userId || !name) {
      Alert.alert('Error', 'Please enter user ID and name');
      return;
    }
    setLoading(true);
    try {
      await axios.post(`${BACKEND_URL}/register-simple`, { userId, name });
      Alert.alert('Success', 'Registration complete! You can now log in.');
    } catch (e) {
      console.log('Registration error:', e);
      Alert.alert('Error', 'Registration failed. Please try again.');
    }
    setLoading(false);
  };

  const handleLogin = async () => {
    if (!userId) {
      Alert.alert('Error', 'Please enter user ID');
      return;
    }

    setLoading(true);
    try {
      await axios.post(`${BACKEND_URL}/login`, { userId });

      const { data } = await axios.post(`${BACKEND_URL}/stream/token`, { userId });

      await streamClient.connectUser(
        { id: userId, name: userId },
        data.token
      );

      setUser(userId, data.token);
      navigation.navigate('Users');
    } catch (error) {
      console.error('Login error:', error);
      Alert.alert('Error', 'Login failed. Please try again.');
    }
    setLoading(false);
  };

  return (
    <ScrollView style={styles.container} contentContainerStyle={styles.contentContainer}>
      <Text style={styles.title}>Chat App</Text>

      <TextInput
        placeholder="User ID"
        value={userId}
        onChangeText={setUserId}
        style={styles.input}
        autoCapitalize="none"
        autoCorrect={false}
        editable={!loading}
      />

      <TextInput
        placeholder="Name (for registration)"
        value={name}
        onChangeText={setName}
        style={styles.input}
        autoCorrect={false}
        editable={!loading}
      />

      {loading ? (
        <View style={styles.loadingContainer}>
          <ActivityIndicator size="large" color="#007AFF" />
          <Text style={styles.loadingText}>Please wait...</Text>
        </View>
      ) : (
        <View style={styles.buttonContainer}>
          <TouchableOpacity style={styles.primaryButton} onPress={handleRegister}>
            <Text style={styles.primaryButtonText}>Register</Text>
          </TouchableOpacity>

          <TouchableOpacity style={styles.primaryButton} onPress={handleLogin}>
            <Text style={styles.primaryButtonText}>Login</Text>
          </TouchableOpacity>
        </View>
      )}
    </ScrollView>
  );
}

2. UsersScreen.tsx (User List and Logout)

// src/screens/UsersScreen.tsx -
import React, { useEffect, useState } from 'react';
import {
  View,
  Text,
  FlatList,
  TouchableOpacity,
  ActivityIndicator,
  Alert,
  StyleSheet
} from 'react-native';
import { NativeStackNavigationProp } from '@react-navigation/native-stack';
import { useUser } from '../context/UserContext';
import { streamClient } from '../utils/streamClient';
import { BACKEND_URL } from '../config/constants';
import { RootStackParamList } from '../App';

type UsersScreenNavigationProp = NativeStackNavigationProp<RootStackParamList, 'Users'>;

interface Props {
  navigation: UsersScreenNavigationProp;
}

interface User {
  id: string;
  name: string;
}

export default function UsersScreen({ navigation }: Props) {
  const [users, setUsers] = useState<User[]>([]);
  const [loading, setLoading] = useState(true);
  const [loggingOut, setLoggingOut] = useState(false);
  const { userId, clearUser } = useUser();

  useEffect(() => {
    fetchUsers();
  }, []);

  const fetchUsers = async () => {
    setLoading(true);
    try {
      const controller = new AbortController();
      const timeout = setTimeout(() => controller.abort(), 10000);

      const res = await fetch(`${BACKEND_URL}/users`, { signal: controller.signal });
      clearTimeout(timeout);

      if (!res.ok) {
        const errorData = await res.json();
        throw new Error(errorData.error || 'Unknown error');
      }

      const data = await res.json();
      setUsers(data.users.filter((u: User) => u.id !== userId));
    } catch (e: any) {
      let message = 'Failed to fetch users';
      if (e.name === 'AbortError') {
        message = 'Request timed out. Please try again.';
      } else if (e.message) {
        message = e.message;
      }
      Alert.alert('Error', message);
    }
    setLoading(false);
  };

  const handleLogout = async () => {
    Alert.alert(
      'Logout',
      'Are you sure you want to logout?',
      [
        { text: 'Cancel', style: 'cancel' },
        {
          text: 'Logout',
          style: 'destructive',
          onPress: async () => {
            try {
              setLoggingOut(true);

              // Disconnect from Stream
              if (streamClient.user) {
                await streamClient.disconnectUser();
              }

              // Clear user and navigate to auth
              clearUser();
              navigation.reset({
                index: 0,
                routes: [{ name: 'Auth' }],
              });
            } catch (error) {
              console.error('Logout error:', error);
              Alert.alert('Error', 'Logout failed');
            } finally {
              setLoggingOut(false);
            }
          }
        }
      ]
    );
  };

  const handleSelectUser = (recipientId: string, recipientName: string) => {
    navigation.navigate('Chat', { recipientId, recipientName });
  };

  return (
    <View style={styles.container}>
      {/* Header Section with User Info and Logout */}
      <View style={styles.headerSection}>
        <View style={styles.userInfoSection}>
          <Text style={styles.welcomeText}>Welcome, {userId}</Text>
        </View>

        <TouchableOpacity
          style={[styles.logoutButton, loggingOut && styles.logoutButtonDisabled]}
          onPress={handleLogout}
          disabled={loggingOut}
        >
          {loggingOut ? (
            <ActivityIndicator size="small" color="#fff" />
          ) : (
            <Text style={styles.logoutButtonText}>🚪 Logout</Text>
          )}
        </TouchableOpacity>
      </View>

      {/* Main Content */}
      <Text style={styles.header}>Select a user to chat with:</Text>

      {loading ? (
        <View style={styles.centered}>
          <ActivityIndicator size="large" color="#007AFF" />
          <Text style={styles.loadingText}>Loading users...</Text>
        </View>
      ) : (
        <FlatList
          data={users}
          keyExtractor={item => item.id}
          renderItem={({ item }) => (
            <TouchableOpacity
              style={styles.userCard}
              onPress={() => handleSelectUser(item.id, item.name)}
              activeOpacity={0.7}
            >
              <View style={styles.userInfo}>
                <Text style={styles.userName}>{item.name}</Text>
                <Text style={styles.userId}>{item.id}</Text>
              </View>
            </TouchableOpacity>
          )}
          ListEmptyComponent={
            <View style={styles.emptyContainer}>
              <Text style={styles.emptyText}>No other users found.</Text>
              <TouchableOpacity style={styles.refreshButton} onPress={fetchUsers}>
                <Text style={styles.refreshButtonText}>🔄 Refresh</Text>
              </TouchableOpacity>
            </View>
          }
          showsVerticalScrollIndicator={false}
        />
      )}
    </View>
  );
}

3. ChatScreen.tsx (Chat Interface)

// ChatScreen.tsx - Plaintext chat (no encryption)
import React, { useEffect, useState, useCallback } from 'react';
import {
  ActivityIndicator,
  View,
  TextInput,
  TouchableOpacity,
  Text,
  StyleSheet,
  Alert,
} from 'react-native';
import { NativeStackNavigationProp } from '@react-navigation/native-stack';
import { RouteProp } from '@react-navigation/native';
import { Chat, Channel, MessageList } from 'stream-chat-expo';
import { streamClient } from '../utils/streamClient';
import { useUser } from '../context/UserContext';
import { RootStackParamList } from '../App';

type ChatScreenNavigationProp = NativeStackNavigationProp<RootStackParamList, 'Chat'>;
type ChatScreenRouteProp = RouteProp<RootStackParamList, 'Chat'>;

interface Props {
  navigation: ChatScreenNavigationProp;
  route: ChatScreenRouteProp;
}

export default function ChatScreen({ navigation, route }: Props) {
  const { recipientId, recipientName } = route.params;
  const { userId, token } = useUser();

  const [channel, setChannel] = useState<any>(null);
  const [loading, setLoading] = useState(true);

  const channelId = [userId, recipientId].sort().join('_');

  useEffect(() => {
    if (!userId || !token) {
      navigation.navigate('Auth');
      return;
    }
    (async () => {
      try {
        if (!streamClient.user) {
          await streamClient.connectUser({ id: userId!, name: userId! }, token!);
        }
        const chatChannel = streamClient.channel('messaging', channelId, {
          members: [userId!, recipientId],
          name: `Chat with ${recipientName}`,
        } as any);
        await chatChannel.watch();
        setChannel(chatChannel);
      } catch (e) {
        console.error('Chat setup failed:', e);
        Alert.alert('Error', 'Failed to open chat');
      } finally {
        setLoading(false);
      }
    })();
  }, [userId, token, recipientId]);

  const handleSendMessage = useCallback(
    async (text: string) => {
      if (!text.trim() || !channel) return;
      try {
        await channel.sendMessage({ text }); // plaintext
      } catch (e) {
        console.error('Send failed:', e);
        Alert.alert('Error', 'Failed to send');
      }
    },
    [channel]
  );

  if (loading || !channel) {
    return (
      <View style={styles.loading}>
        <ActivityIndicator size="large" color="#007AFF" />
        <Text style={styles.loadingText}>Connecting...</Text>
      </View>
    );
  }

  return (
    <Chat client={streamClient}>
      <Channel channel={channel}>
        <View style={styles.header}>
          <Text style={styles.headerText}>{recipientName}</Text>
        </View>
        <MessageList />
        <CustomMessageInput onSendMessage={handleSendMessage} />
      </Channel>
    </Chat>
  );
}

const CustomMessageInput = ({ onSendMessage }: { onSendMessage: (text: string) => void }) => {
  const [inputText, setInputText] = useState('');
  const [sending, setSending] = useState(false);

  const handleSend = async () => {
    if (!inputText.trim() || sending) return;
    setSending(true);
    await onSendMessage(inputText);
    setInputText('');
    setSending(false);
  };

  return (
    <View style={styles.inputContainer}>
      <TextInput
        style={styles.textInput}
        value={inputText}
        onChangeText={setInputText}
        placeholder="Type a message..."
        multiline
        maxLength={2000}
        editable={!sending}
      />
      <TouchableOpacity
        style={[styles.sendButton, (!inputText.trim() || sending) && styles.sendButtonDisabled]}
        onPress={handleSend}
        disabled={!inputText.trim() || sending}
      >
        <Text style={styles.sendButtonText}>🚀</Text>
      </TouchableOpacity>
    </View>
  );
};

Message Security in Stream

When you send a message using Stream, it is encrypted by default at two levels:

In transit (TLS 1.2): Messages are transmitted securely over the internet using TLS, which creates a secure tunnel between your app and Stream's servers. This prevents attackers from snooping on your messages while they travel over Wi-Fi or mobile networks.
At rest (AES-256): Once stored on Stream's servers, messages are encrypted using AES-256, a tough encryption standard also used by banks and governments. This ensures that even if someone gained access to Stream's storage, they couldn't easily read your messages.

Important: This is not end-to-end encryption (E2EE). Stream's default setup still makes messages accessible to Stream's servers. This is necessary for features like moderation, search, and translations. In true E2EE, only the sender and receiver can read the messages, not even the service provider.

Here's an example of a message sent in our current app. Notice that the message data is available on Stream's servers:

Understanding End-to-End Encryption: The Why and How

End-to-end encryption (E2EE) is the gold standard for secure communication, ensuring that only the intended parties can read the exchanged messages. In this React Native chat application, E2EE is implemented using a hybrid cryptographic approach that combines the security of public-key cryptography with the efficiency of symmetric encryption.

The sections below detail the cryptographic foundation, key management, and message flow.

Cryptographic Foundation

Elliptic Curve Cryptography (ECC)

This application uses the secp256k1 elliptic curve, which is employed by Bitcoin and other major cryptocurrencies.

Why secp256k1?

Security: 256-bit keys provide equivalent security to 3072-bit RSA keys
Efficiency: Smaller key sizes result in faster computations and reduced bandwidth
Proven Track Record: It's been extensively tested and analysed by the cryptographic community

//utils/encryption.ts -
import { secp256k1 } from '@noble/curves/secp256k1';
import { getRandomBytes } from 'expo-crypto';
import { gcm } from '@noble/ciphers/aes';
import { sha256 } from '@noble/hashes/sha256';
import { hkdf } from '@noble/hashes/hkdf';
import * as SecureStore from 'expo-secure-store';

React Native lacks built-in crypto APIs, so we rely on:

Polyfills: react-native-get-random-values for Web Crypto API support
Crypto libraries: @noble/curves for ECC operation
Secure storage: expo-secure-store for persisting private keys

Key Generation Process

Each user's cryptographic identity is established through a secure key pair generation process:

export interface KeyPair {
  publicKey: string;  // Hex-encoded public key
  privateKey: string; // Hex-encoded private key
}

The private key is a 256-bit random number, while the public key is derived mathematically from the private key using elliptic curve point multiplication. For maximum compatibility, the uncompressed format (65 bytes) is used.

// Generate a secure key pair using secp256k1 (same curve as Bitcoin)
export async function generateKeyPair(): Promise<KeyPair> {
  try {
    console.log('🔑 Generating new cryptographic key pair...');

    // Generate a random private key using Expo's crypto
    const privateKeyBytes = getRandomBytes(32);

    // Ensure the private key is valid for secp256k1
    let attempts = 0;
    let validPrivateKey = privateKeyBytes;

    while (attempts < 10) {
      try {
        // Check if this private key is valid
        const publicKeyPoint = secp256k1.getPublicKey(validPrivateKey, false);

        // If we get here, the key is valid
        return {
          privateKey: bytesToHex(validPrivateKey),
          publicKey: bytesToHex(publicKeyPoint)
        };
      } catch (error) {
        // Invalid key, generate a new one
        validPrivateKey = getRandomBytes(32);
        attempts++;
      }
    }

    throw new Error('Could not generate valid private key after 10 attempts');
  } catch (error) {
    console.error('Key generation error:', error);
    throw new Error('Failed to generate cryptographic keys');
  }
}

Key Storage Security

Private keys are never stored in plaintext; instead, they are stored using expo-secure-store, which leverages:

iOS: Keychain Services with hardware security module integration when available
Android: EncryptedSharedPreferences with Android Keystore backing

// Store keys securely using Expo SecureStore
export async function storeKeyPair(userId: string, keyPair: KeyPair): Promise<void> {
  try {
    const keyData = JSON.stringify(keyPair);
    await SecureStore.setItemAsync(`crypto_keys_${userId}`, keyData);
    console.log(`✅ Keys securely stored for user: ${userId}`);
  } catch (error) {
    console.error('Key storage error:', error);
    throw new Error('Failed to store encryption keys');
  }
}

This ensures private keys remain safe even if the app's storage is compromised.

Note: Avoid AsyncStorage for keys; it's not secure. Always use expo-secure-store or a native wrapper.

Key Exchange Protocol

Elliptic Curve Diffie-Hellman (ECDH)

The application implements ECDH key exchange to establish shared secrets between communicating parties:

Alice has a key pair (a, A) where a = private key, A = public key
Bob has a key pair (b, B) where b = private key, B = public key
Shared Secret = a × B = b × A (due to elliptic curve mathematics)

Key Derivation Function

The raw ECDH secret isn't used directly. Instead, it's expanded into a strong AES key using HKDF (HMAC-based Key Derivation Function) with SHA-256.

const derivedKey = hkdf(sha256, sharedSecret, salt, info, 32);

This step:

Strengthens entropy
Ensures forward secrecy
Produces a consistent 256-bit key

// Perform ECDH key exchange to derive shared secret
export function deriveSharedSecret(privateKey: string, publicKey: string): Uint8Array {
  try {
    const privateKeyBytes = hexToBytes(privateKey);
    const publicKeyBytes = hexToBytes(publicKey);

    // Perform ECDH
    const sharedPoint = secp256k1.getSharedSecret(privateKeyBytes, publicKeyBytes, false);

    // Use HKDF to derive a proper encryption key from the shared secret
    // This adds forward secrecy and proper key derivation
    const salt = new Uint8Array(32); // Zero salt for simplicity, could be random
    const info = new TextEncoder().encode('ChatAppE2E');

    const derivedKey = hkdf(sha256, sharedPoint.slice(1, 33), salt, info, 32);

    return derivedKey;
  } catch (error) {
    console.error('Shared secret derivation error:', error);
    throw new Error('Failed to derive shared secret');
  }
}

Message Encryption Process

AES-256-GCM Symmetric Encryption

Once the shared key is established, messages are encrypted using AES-256 in Galois/Counter Mode (GCM):

AES-256: Advanced Encryption Standard with 256-bit keys.

GCM Mode: Provides both confidentiality and authenticity.

Encryption flow

For each message, the following process occurs:

Shared Secret Derivation: ECDH between the sender's private key and the recipient's public key
Key Derivation: HKDF transforms the shared secret into an AES key
IV Generation: A random 96-bit initialisation vector is generated
Encryption: AES-GCM encrypts the plaintext message
Authentication: GCM automatically generates a 128-bit authentication tag
Packaging: All cryptographic components are combined into a structured JSON format for transmission

Example JSON output:

{
  "ciphertext": "hex-encoded encrypted data",
  "iv": "hex-encoded initialization vector",
  "authTag": "hex-encoded authentication tag"
}

export interface EncryptedMessage {
  ciphertext: string;  // Hex-encoded encrypted data
  iv: string;          // Hex-encoded initialization vector
  authTag: string;     // Hex-encoded authentication tag
}

// Encrypt a message using AES-GCM
export async function encryptMessage(
  message: string,
  senderPrivateKey: string,
  recipientPublicKey: string
): Promise<string> {
  try {
    console.log('🔒 Starting message encryption...');

    // Derive shared secret
    const sharedSecret = deriveSharedSecret(senderPrivateKey, recipientPublicKey);

    // Generate random IV (12 bytes for GCM)
    const iv = getRandomBytes(12);

    // Convert message to bytes
    const messageBytes = new TextEncoder().encode(message);

    // Create AES-GCM cipher
    const aes = gcm(sharedSecret, iv);

    // Encrypt the message
    const encrypted = aes.encrypt(messageBytes);

    // Extract the ciphertext and auth tag
    const ciphertext = encrypted.slice(0, -16); // All but last 16 bytes
    const authTag = encrypted.slice(-16); // Last 16 bytes

    // Package the result
    const result: EncryptedMessage = {
      ciphertext: bytesToHex(ciphertext),
      iv: bytesToHex(iv),
      authTag: bytesToHex(authTag)
    };

    console.log('✅ Message encrypted successfully');
    return JSON.stringify(result);
  } catch (error) {
    console.error('Encryption error:', error);
    throw new Error('Failed to encrypt message');
  }
}

This guarantees:

Confidentiality: Only intended parties can decrypt
Integrity: Tampering is detected
Authenticity: Sender identity is verifiable
Forward secrecy: Every message gets a fresh key

Message Decryption Process

Decryption Workflow

The reverse process recovers the plaintext:

Parse encrypted JSON
Derive shared secret (ECDH)
Run HKDF → AES key
Validate the auth tag
Decrypt ciphertext → UTF-8 message

// Decrypt a message using AES-GCM
export async function decryptMessage(
  encryptedData: string,
  recipientPrivateKey: string,
  senderPublicKey: string
): Promise<string> {
  try {
    console.log('🔓 Starting message decryption...');

    // Parse encrypted data
    const data: EncryptedMessage = JSON.parse(encryptedData);

    // Derive shared secret
    const sharedSecret = deriveSharedSecret(recipientPrivateKey, senderPublicKey);

    // Convert hex data back to bytes
    const ciphertext = hexToBytes(data.ciphertext);
    const iv = hexToBytes(data.iv);
    const authTag = hexToBytes(data.authTag);

    // Combine ciphertext and auth tag for decryption
    const encryptedWithTag = new Uint8Array(ciphertext.length + authTag.length);
    encryptedWithTag.set(ciphertext, 0);
    encryptedWithTag.set(authTag, ciphertext.length);

    // Create AES-GCM cipher and decrypt
    const aes = gcm(sharedSecret, iv);
    const decrypted = aes.decrypt(encryptedWithTag);

    // Convert back to string
    const decryptedMessage = new TextDecoder().decode(decrypted);
    console.log('✅ Message decrypted successfully');
    return decryptedMessage;
  } catch (error) {
    console.error('Decryption error:', error);
    throw new Error('Failed to decrypt message');
  }
}

Key Management Architecture

Key Lifecycle

Generate: Random key pairs per user

Store: Securely in Keychain / Keystore

Exchange: Public keys shared via backend

Use: Private keys never leave device

Delete: Secure wipe on logout or reset

Key Fingerprinting

Each public key is fingerprinted using SHA-256 to prevent impersonation or tampering.

// Generate a fingerprint for a public key (for verification)
export function generateKeyFingerprint(publicKey: string): string {
  try {
    const pubKeyBytes = hexToBytes(publicKey);
    const hash = sha256(pubKeyBytes);
    return bytesToHex(hash.slice(0, 8)); // First 8 bytes as hex
  } catch (error) {
    console.error('Fingerprint generation error:', error);
    return 'invalid';
  }
}

Fingerprints give users a human-verifiable way to confirm they're communicating with the right person.

Integrating E2EE Into the Chat App

In the typical chat application we built earlier, the text sent between users is visible to the Stream server in plaintext. With E2EE applied, the server only relays encrypted payloads; it never sees message contents.

After implementing the encryption process, coding it, and storing the logic in encryption.ts, the functions are exported into a React Context. This centralizes all end-to-end encryption states and operations, so UI screens can call simple methods without handling cryptographic details or private keys directly.

// context/RealEncryptionContext.tsx
import React, { createContext, useContext, useState, ReactNode } from 'react';
import {
  KeyPair,
  generateKeyPair,
  storeKeyPair,
  loadKeyPair,
  deleteKeyPair,
  encryptMessage,
  decryptMessage,
  isValidPublicKey,
  generateKeyFingerprint,
  checkKeysExist
} from '../utils/encryption';

interface EncryptionContextType {
  keyPair: KeyPair | null;
  isKeysLoaded: boolean;
  generateKeys: (userId: string) => Promise<void>;
  loadKeys: (userId: string) => Promise<boolean>;
  encryptMessage: (message: string, recipientPublicKey: string) => Promise<string>;
  decryptMessage: (encryptedMessage: string, senderPublicKey: string) => Promise<string>;
  getPublicKeyString: () => string | null;
  getKeyFingerprint: () => string | null;
  validatePublicKey: (publicKey: string) => boolean;
  clearKeys: () => void;
  deleteUserKeys: (userId: string) => Promise<void>;
  checkUserHasKeys: (userId: string) => Promise<boolean>;
}

const EncryptionContext = createContext<EncryptionContextType | undefined>(undefined);

export function RealEncryptionProvider({ children }: { children: ReactNode }) {
  // The necessary functions are to be here, which can be found in the complete codebase
  return (
    <EncryptionContext.Provider value={{
      keyPair,
      isKeysLoaded,
      generateKeys,
      loadKeys,
      encryptMessage: encryptMessageWrapper,
      decryptMessage: decryptMessageWrapper,
      getPublicKeyString,
      getKeyFingerprint,
      validatePublicKey,
      clearKeys,
      deleteUserKeys,
      checkUserHasKeys
    }}>
      {children}
    </EncryptionContext.Provider>
  );
}

export function useRealEncryption() {
  const context = useContext(EncryptionContext);
  if (!context) {
    throw new Error('useRealEncryption must be used within a RealEncryptionProvider');
  }
  return context;
}

Wrapping the App in the Encryption Provider

To integrate E2EE into the chat:

Remember that each user has two keys:
- The private key, stored securely on the device.
- The public key, generated during registration and stored in Stream for others.
Wrap your screens in the RealEncryptionProvider.
Use react-native-get-random-values as a polyfill for generating secure random values (must be imported first).

// in App.tsx
import 'react-native-get-random-values'; // Must be first import
import { RealEncryptionProvider } from './context/EncryptionContext';

<RealEncryptionProvider>
  {/* your navigators/screens */}
</RealEncryptionProvider>

User Registration with Keys

After registration, the user logs in using the normal process, after checking for the keys. When registering a user, keys must be generated before connecting to the backend (/register). If keys already exist, they are loaded instead of regenerated.

// AuthScreen.tsx
const handleRegister = async () => {
    if (!userId || !name) {
      Alert.alert('Error', 'Please enter user ID and name');
      return;
    }

    setLoading(true);
    try {
      // Clear any existing keys first
      clearKeys();

      // Check if user already has keys
      const hasExistingKeys = await checkUserHasKeys(userId);

      if (hasExistingKeys) {
        // Load existing keys instead of generating new ones
        const keysLoaded = await loadKeys(userId);
        if (!keysLoaded) {
          throw new Error('Failed to load existing keys');
        }

        Alert.alert(
          'Existing Keys Found',
          'This user already has encryption keys. Using existing keys for registration.',
          [{ text: 'Continue', onPress: () => proceedWithRegistration() }]
        );
        return;
      }

      // Generate new encryption keys for this user
      await generateKeys(userId);
      await proceedWithRegistration();

    } catch (error) {
      console.error('Registration error:', error);
      Alert.alert('Error', 'Registration failed. Please try again.');
    }
    setLoading(false);
  };

Encrypting Messages Before Sending

ChatScreen fetches the partner’s public key, then calls encryptMessage before sending and decryptMessage on receive.

When sending a message, the ChatScreen.tsx fetches the recipient's public key, encrypts the message locally, and sends only the encrypted payload to Stream.

// Add to ChatScreen.tsx
const handleSendMessage = useCallback(async (messageText: string) => {
    if (!messageText.trim() || !channel || !recipientPublicKey) return;

    try {
      console.log('🔒 Encrypting message...');

      // Encrypt the message using real E2E encryption
      const encryptedMessage = await encryptMessage(messageText, recipientPublicKey);

      console.log('📤 Sending encrypted message...');

      // Send the encrypted message
      const messageData = {
        text: encryptedMessage,  // Only encrypted JSON goes to Stream
        encrypted: true,
      };

      const sentMessage = await channel.sendMessage(messageData);

      // Cache the original text for immediate display
      if (sentMessage.message?.id) {
        decryptedMessagesRef.current.set(sentMessage.message.id, messageText);

        // Update the display immediately for the sender
        sentMessage.message.text = messageText;
        sentMessage.message.processed = true;
        processedMessagesRef.current.add(sentMessage.message.id);
      }

      console.log('✅ Message sent and encrypted successfully');
    } catch (error) {
      console.error('❌ Failed to send encrypted message:', error);
      Alert.alert('Error', 'Failed to send message. Please try again.');
    }
  }, [channel, recipientPublicKey, encryptMessage]);

Decrypting Messages Upon Receiving

Messages received from Stream are decrypted before being displayed. If decryption fails, the user sees a placeholder.

// Add to ChatScreen.tsx
const processMessageForDisplay = async (message: any, senderPublicKey: string) => {
    // Add comprehensive null/undefined checks
    if (!message || !message.user || !message.id) {
      console.log('Skipping invalid message:', message);
      return;
    }

    // Skip if already processing or processed
    if (isProcessingRef.current.has(message.id) || processedMessagesRef.current.has(message.id)) {
      return;
    }

    // Ensure message has required properties
    if (!message.text || typeof message.text !== 'string') {
      console.log('Message missing text property:', message);
      processedMessagesRef.current.add(message.id);
      return;
    }

    console.log('🔍 Processing message:', {
      id: message.id,
      userId: message.user?.id,
      encrypted: message.encrypted,
      textLength: message.text.length
    });

    // Mark as processing
    isProcessingRef.current.add(message.id);

    try {
      // If it's an encrypted message, decrypt it
      if (message.encrypted) {
        // Check if we already have a decrypted version cached
        if (decryptedMessagesRef.current.has(message.id)) {
          message.text = decryptedMessagesRef.current.get(message.id)!;
          console.log('✅ Using cached decrypted message');
        } else {
          // Only decrypt if this looks like encrypted JSON data
          const isEncryptedData = message.text.startsWith('{"ciphertext"');

          if (isEncryptedData) {
            let decryptedText: string;

            if (message.user.id === userId) {
              // Our own message - decrypt using recipient's public key
              console.log('🔓 Decrypting own message...');
              decryptedText = await decryptMessage(message.text, senderPublicKey!);
            } else {
              // Message from other person - decrypt using their public key
              console.log('🔓 Decrypting received message...');
              decryptedText = await decryptMessage(message.text, senderPublicKey);
            }

            // Cache the decrypted text
            decryptedMessagesRef.current.set(message.id, decryptedText);
            message.text = decryptedText;
            console.log('✅ Message decrypted and cached successfully');
          } else {
            // This might be a message that was already processed but lost its decrypted state
            message.text = '[🔒 Message needs refresh - reopen chat]';
          }
        }
      }

      message.processed = true;
      processedMessagesRef.current.add(message.id);
    } catch (error) {
      console.error('❌ Failed to decrypt message:', error);
      message.text = '[🔒 Failed to decrypt - may be corrupted]';
      processedMessagesRef.current.add(message.id);
    } finally {
      // Remove from processing set
      isProcessingRef.current.delete(message.id);
    }
  };

App in Action

When two users (Momo12 and Spice12) chat, the messages are displayed differently depending on where they’re viewed:

On their devices: messages appear as normal plaintext
On Stream's servers: only ciphertext is visible, in the following format:

{ "ciphertext":"3d82a8cf09", "Iv":"dda8821c49bd8e945fdac36f", "authTag":"15d6aa9921245c21d765f84f8ec47e99" }

The terminal output below shows the encrypted messages being exchanged between two users during a chat session:

This is the final structure of the app:

my-chat-app/
├── App.tsx
├──utils/
│   ├── encryption.ts
│   └── streamClient.ts
├── context/
│   ├── EncryptionContext.tsx
│   └── UserContext.tsx
├── config/
│   └── constants.ts
├── screens/
│   ├── AuthScreen.tsx
│   ├── UserListScreen.tsx
│   └── ChatScreen.tsx
├── assets/
├── app.json
├── package.json
├── tsconfig.json
└── node_modules/

You can explore the full implementation of the chat app in this repository.

Conclusion

This React Native chat application demonstrates a robust end-to-end encryption (E2EE) design built with well-established cryptographic primitives. The system keeps messages private and secure by using ECDH for key exchange and AES-GCM for encryption, while maintaining ease of use on mobile.

With a zero-knowledge setup, no third party (not even the service provider) can read messages. Keys are stored securely and managed carefully to prevent leaks or misuse.

However, E2EE impacts moderation and observability:

Since payloads are encrypted, the server cannot inspect them.
This disables standard tools such as:
- Block lists (preventing specific words/phrases)
- Advanced filters (domains, emails, regex rules)
- AI moderation (behavioral analysis)
- Pre-send hooks (no access to plaintext)

Instead, developers must adopt new approaches such as client-side filtering, metadata-based heuristics, or opt-in reporting flows.

Despite some trade-offs, end-to-end encryption gives mobile apps bank-level privacy while limiting moderation options—a balance developers must account for when building real-world chat.

Build an AI Meeting Assistant with Stream Video and LLMs

Quincy Oghenetejiri — Mon, 09 Jun 2025 14:15:13 +0000

In today's business environment, virtual meetings consume a significant portion of professionals' workdays. Yet, much of their value evaporates once participants leave the virtual room. Critical insights, decisions, and action items often get lost amid daily tasks, leading to misalignment, missed deadlines, and the dreaded "Wait, what did we decide?" follow-up communications.

AI meeting assistants are systems designed to capture, process, and extract meaningful real-time information from conversations. By combining Stream Video’s robust infrastructure with the powerful language understanding capabilities of Large Language Models (LLMs), organizations can deploy smart meeting solutions that transform how teams collaborate, capture knowledge, and execute decisions.

This article explores how to build effective AI meeting assistants that provide real-time transcription, intelligent summarization, and automated action item extraction without affecting the user experience or disrupting the natural flow of conversation.

What Does Stream Video Offer?

Stream Video provides a comprehensive platform for building video applications, equipped with features essential for creating compelling meeting experiences:

Low-latency video and audio transmission.
Secure and reliable connections across various network conditions.
Recording capabilities with cloud storage.
Customizable layouts and user interfaces.
Access to transcription and captions after meetings.
Extensive APIs for integration with other services.

The platform serves as the foundation upon which you can build advanced AI capabilities, providing access to the meeting content that LLMs will analyze.

Technical Prerequisite

Before we begin, ensure you have the following:

A Stream account with an API key and secret.
Access to an LLM API (e.g., OpenAI, Anthropic, Gemini, etc.).
Node.js and npm/yarn installed.
Basic knowledge of React and Node.js.

Solution Architecture

The integration between Stream Video and LLMs typically follows this flow:

Stream Video captures and transmits the meeting’s audio/video content.
Speech-to-text processing converts spoken language into written transcripts.
The transcript is sent to an LLM via API calls for analysis.
LLM outputs (summaries, action items) are stored and presented to users.

Backend Implementation

Before creating the middleware and connecting it to the client, you'll need to complete a few setup steps.

Set Up Your Server

Start by building your backend.

Create a Node.js project and install dependencies:

mkdir smart-meeting-assistant cd smart-meeting-assistant  cd backend npm init  npm install express stream-chat nodemon dotenv cors axios @google/generative-ai

Create a .env file to store your configuration credentials, including your Stream and LLM provider keys:

STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret GEMINI_API_KEY=your_llm_api_key
LLM_API_URL="https://generativelanguage.googleapis.com/v1beta"
PORT="5000"

Create an index.js file and import the necessary modules:

import express from 'express';
import cors from 'cors';
import dotenv from 'dotenv';
import { GoogleGenerativeAI } from '@google/generative-ai';
import { StreamChat } from 'stream-chat';
import axios from 'axios';
dotenv.config();

const app = express();

// Simple in-memory storage for transcripts { callCid: transcriptText }
let callTranscripts = {};

// Middleware
app.use(cors());
app.use(express.json());

Service Initialization

We initialize the services required for token generation and LLM processing:

// Initialize Gemini
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

// Initialize Stream Chat Client (for token generation)
const serverClient = StreamChat.getInstance(
  process.env.STREAM_API_KEY,
  process.env.STREAM_API_SECRET
);

Server Initialization

This listens for what port the server is running on, which is currently set to run on port 5000:

const port = process.env.PORT || 5000;

app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

API Endpoints

Your server should expose these key endpoints:

1. Video Meeting Creation

POST /api/create-meeting – Generates a Stream video/chat token for a user.

// Stream Video Routes
app.post('/api/create-meeting', async (req, res) => {
  try {
    const { userId } = req.body; // Only need userId to generate a token
    if (!userId) {
      return res.status(400).json({ error: 'userId is required' });
    }

    // Create a token using the StreamChat client
    const expirationTime = Math.floor(Date.now() / 1000) + 3600; // Token valid for 1 hour
    const token = serverClient.createToken(userId, expirationTime);

    res.json({
      token,
      apiKey: process.env.STREAM_API_KEY // Send API key to frontend
    });
  } catch (error) {
    // Use a more specific error message if possible
    console.error('Error generating Stream token:', error);
    res.status(500).json({ error: 'Failed to generate meeting token' });
  }
});

2. Real-Time Transcription Webhook

POST /api/webhook – Handles incoming webhook events from Stream.

This route listens for transcription-ready events, downloads the transcript, parses it, and stores it in memory.

Note: All meeting data is stored in memory and tied to the active session. This approach focuses on showcasing real-time capabilities with Stream Video and LLMs.

Additionally, always verify webhook signatures in production to ensure requests originate from Stream.

app.post('/api/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const event = req.body; // Stream sends event data in the request body

  // NOTE FOR PRODUCTION: Add signature verification for security

  console.log(`Received Stream Event Type: ${event?.type}`);
  console.log('Full Event Body:', JSON.stringify(event, null, 2));

  // Respond quickly to Stream to acknowledge receipt
  res.status(200).send('Event received');

  // Event Handler for transcription.
  if (event?.type === 'call.transcription_ready') {
    const transcriptUrl = event.call_transcription?.url;
    const callCid = event.call_cid; // e.g., "default:test-meeting-2"
    console.log(`Full transcription is ready for call ${callCid}: ${transcriptUrl}`);

    if (transcriptUrl && callCid) {
      // --- Download and Parse Transcript ---
      (async () => {
        try {
          console.log(`Downloading transcript for ${callCid} from ${transcriptUrl}`);
          // Make sure axios is required/imported (it is, at the top)
          const response = await axios.get(transcriptUrl, { responseType: 'text' });
          const jsonlContent = response.data;
          console.log(`Downloaded transcript content length: ${jsonlContent.length}`);

          // Parse the JSONL content
          const lines = jsonlContent.trim().split('\n');
          let fullTranscript = '';
          let speakerTimestamps = {}; // Optional: Track speaker time

          console.log(`Parsing ${lines.length} lines from JSONL...`);
          for (const line of lines) {
            try {
              const speechFragment = JSON.parse(line);
              if (speechFragment.text) {
                const speaker = speechFragment.speaker_id || 'Unknown';
                // Add speaker name if you want it inline
                fullTranscript += `[${speaker}]: ${speechFragment.text} `;
                // Uncomment the below and comment the line above if you do not want the speaker name attached.
                // fullTranscript += speechFragment.text + ' '; // Add space between fragments  

              }
            } catch (parseError) {
              console.error(`Error parsing JSONL line: ${line}`, parseError);
            }
          }
          fullTranscript = fullTranscript.trim();
          console.log(`--- Full Transcript for ${callCid} ---`);
          console.log(fullTranscript);
          console.log(`--- End Transcript ---`);


          // --- Store the fullTranscript ---
          console.log(`Storing transcript for ${callCid}`);
          callTranscripts[callCid] = fullTranscript;



        } catch (downloadError) {
          // Log axios errors more informatively
          if (axios.isAxiosError(downloadError)) {
            console.error(`Axios error downloading transcript from ${transcriptUrl}:`, {
              status: downloadError.response?.status,
              data: downloadError.response?.data,
              message: downloadError.message,
            });
          } else {
            console.error(`Error downloading transcript from ${transcriptUrl}:`, downloadError);
          }
        }
      })(); // Immediately invoke the async function

    } else {
        console.warn('Received call.transcription_ready event but no URL or callCid found.');
    }
  }

});

Exposing/Configuring Your Stream Webhook

To test your webhook locally:

Expose your server using ngrok:

ngrok http 5000

Copy the HTTPS forwarding URL from your terminal.

Configure the webhook in the Stream dashboard:

Log in to your Stream account.
Navigate to your application > "Video & Audio" > "Webhooks".
Paste the URL as: https://<your-ngrok-url>/api/webhook
Enable relevant events like call.created, call.ended, and call.transcription_ready

Note: The ngrok URL changes every time it's restarted.

3. Meeting Note Analyzer

POST /api/analyze-meeting – Analyzes the meeting transcript using Gemini and returns summarized content, highlighting the action items.

// Meeting Note Analyser
app.post('/api/analyze-meeting', async (req, res) => {
  try {
    const { meetingNotes } = req.body;

    const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash" });

    const prompt = `Please analyze the following meeting transcript/notes and provide:
1.  **Concise Summary:** A brief paragraph summarizing the main topic and outcome of the discussion.
2.  **Key Discussion Points:** A bulleted list of the most important topics discussed.
3.  **Action Items:** A numbered list of tasks assigned, including who is responsible if mentioned (e.g., "Action: [Task description] - @[Username/Name]").
4.  **Decisions Made:** A bulleted list of any clear decisions reached during the meeting.

Meeting Transcript/Notes:
---
${meetingNotes}
---

Format the output clearly under these headings.`;

    const result = await model.generateContent(prompt);
    const response = await result.response;

    res.json({ analysis: response.text() });
  } catch (error) {
    console.error('Error analyzing meeting:', error);
    res.status(500).json({ error: 'Failed to analyze meeting notes' });
  }
});

4. Getting the Transcript

GET /api/get-transcript/:callType/:callId – Returns the full transcript for a given meeting ID if available.

This route allows the frontend to retrieve the processed transcript. It takes in two parameters: callType, which is usually set to "default", and callId, also known as the meeting ID. When a request is made from the frontend, the route looks up the transcript in the in-memory store. If found, it returns an object containing the transcript:

{ "transcript": "..." }.

If not found, it returns an error object:

{ "error": "Transcript not found or not ready yet." }

// --- Endpoint to Retrieve Transcript ---
app.get('/api/get-transcript/:callType/:callId', (req, res) => {
  const { callType, callId } = req.params;
  const callCid = `${callType}:${callId}`;

  console.log(`Frontend requested transcript for ${callCid}`);
  const transcript = callTranscripts[callCid];

  if (transcript !== undefined) {
    res.json({ transcript: transcript });
  } else {
    console.log(`Transcript not found for ${callCid}`);
    // Could be that the call ended before transcription was ready,
    // or it wasn't processed yet. Return empty or 404.
    res.status(404).json({ error: 'Transcript not found or not ready yet.' });
  }
});

Frontend Implementation

The frontend of your smart meeting assistant app is built using React, Vite, and the Stream Video SDK. This combination provides a fast development environment and a powerful, real-time video communication layer.

Set Up Your React Application

npm create vite@latest frontend --template react
cd frontend
npm install @stream-io/video-react-sdk
axios @emotion/react @emotion/styled @mui/icons-material @mui/material
npm run dev

Set Up Environment Variables

In your .env file, add the following:

VITE_STREAM_API_KEY=your_stream_api_key
VITE_BACKEND_URL=your_backend_url

Vite Configuration (frontend/vite.config.js)

The vite.config.js file configures the Vite development server and build process.

Plugins:

@vitejs/plugin-react: This essential plugin enables React support in Vite, providing features like Fast Refresh (HMR for React components) and JSX transformation.

Development Server (server options):

port: 3000: Specifies that the Vite development server should run on port 3000.
Proxy (proxy option): This is a crucial part for development when the frontend (Vite server) and backend API run on different ports. This configuration tells Vite's dev server: "Any request that starts with /api (e.g., /api/analyze-meeting, /api/create-meeting) should be forwarded to http://localhost:5000."
- target: 'http://localhost:5000': The URL of your backend server.
- changeOrigin: true: This option is important for virtual hosting environments and sets the Origin header of the request to the target URL. It helps avoid CORS issues or backend configurations that might reject requests based on the origin.

This proxy setup allows the frontend code to make API calls to relative paths like /api/analyze-meeting without worrying about CORS during development, as the browser sends the request to localhost:3000 (Vite server), which then proxies it to localhost:5000 (backend server). In production, a reverse proxy like Nginx or your hosting platform would handle a similar setup.

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react()],
  server: {
    port: 3000,
    proxy: {
      '/api': {
        target: 'http://localhost:5000',
        changeOrigin: true,
      },
    },
  },
})

The Header Component

This is the presentational component for the application's header.

import { AppBar, Toolbar, Typography } from '@mui/material';
import AutoStoriesIcon from '@mui/icons-material/AutoStories';

function Header() {
  return (
    <AppBar position="static">
      <Toolbar>
        <AutoStoriesIcon sx={{ mr: 2 }} />
        <Typography variant="h6" component="div">
          Smart Meeting Assistant
        </Typography>
      </Toolbar>
    </AppBar>
  );
}

export default Header;

The Action Item Display Component

This is a dedicated component used for rendering or showing action items from the analysis text provided by MeetingAnalyzer.jsx.

It takes structured insights and presents them in a clean, user-friendly format—often as a list of tasks or bullet points. This separation of concerns ensures that the logic for extracting action items remains within the analyzer, while this component focuses solely on visualizing the output.

To identify these actions, a parsing function is used, which takes the text as an argument and makes use of Regular Expressions (Regex):

const parseActionItems = (analysisText) => {
  if (!analysisText) return [];

  const lines = analysisText.split('\n');
  const actionItemsSection = [];
  let inActionItemsSection = false;

  for (const line of lines) {
    const trimmedLine = line.trim();
    if (trimmedLine.includes('**Action Items:**')) {
      inActionItemsSection = true;
      continue; // Skip the header line
    }
    // Stop if we hit the next section or empty line after starting
    if (inActionItemsSection && (trimmedLine.startsWith('**') || trimmedLine === '')) {
         if(actionItemsSection.length > 0) break; // Stop only if we've found items
         else continue; // Skip empty lines before items
    }

    if (inActionItemsSection && trimmedLine) {
        // Check if the line looks like an action item (e.g., starts with number. or 'Action:')
        if (/^\d+\.\s/.test(trimmedLine) || trimmedLine.toLowerCase().startsWith('action:')) {
             actionItemsSection.push(trimmedLine.replace(/^\d+\.\s*/, '')); // Remove leading number/dot
        }
    }
  }

  // Fallback if specific section not found, try finding lines starting with 'Action:' anywhere
  if (actionItemsSection.length === 0) {
      for (const line of lines) {
          const trimmedLine = line.trim();
          if (trimmedLine.toLowerCase().startsWith('action:')) {
               actionItemsSection.push(trimmedLine);
          }
      }
  }

  return actionItemsSection;
};

Then the UI is rendered as below:

import React from 'react';
import {
  Box,
  Card,
  CardContent,
  List,
  ListItem,
  ListItemText,
  Typography,
} from '@mui/material';

const ActionItemsDisplay = ({ analysisText }) => {
  const actionItems = parseActionItems(analysisText);

  if (actionItems.length === 0) {
    return null; // Don't render anything if no action items are found
  }

  return (
    <Card sx={{ mt: 3 }}>
      <CardContent>
        <Typography variant="h6" gutterBottom>
          Action Items
        </Typography>
        <List dense>
          {actionItems.map((item, index) => (
            <ListItem key={index} disablePadding>
              <ListItemText primary={`- ${item}`} />
            </ListItem>
          ))}
        </List>
      </CardContent>
    </Card>
  );
};

export default ActionItemsDisplay;

The Meeting Analyser Component

This component provides the UI for users to input or view transcripts, which is received as a prop named initialNotes.

By passing initialNotes as a prop, the component remains decoupled and reusable, enabling dynamic rendering of transcript data as it becomes available after the meeting. This approach also makes it easier to sync with backend services.

Inside MeetingAnalyzer.jsx, the useEffect hook listens for changes to the initialNotes prop. When latestTranscript in App.jsx is updated, this prop changes and updates its internal meetingNotes state, populating its text field with the new transcript.

useEffect(() => {
  if (initialNotes) {
    setMeetingNotes(initialNotes);
  }
}, [initialNotes]);

When clicked, the button “Analyze Meeting Notes” sends the notes/transcript to the backend API for analysis and then displays the analysis.

const handleSubmit = async (e) => {
    e.preventDefault();
    setLoading(true);
    setError('');
    setAnalysis('');

    try {
      const response = await axios.post('/api/analyze-meeting', {
        meetingNotes,
      });
      setAnalysis(response.data.analysis);
    } catch (error) {
      setError('Failed to analyze meeting notes. Please try again.');
      console.error('Error:', error);
    } finally {
      setLoading(false);
    }
  };

Action Items are listed using the ActionItemsDisplay.jsx component, which is integrated into this component.

{analysis && (
        <>
          <Card sx={{ mt: 3 }}>
            <CardContent>
              <Typography variant="h6" gutterBottom>
                Meeting Analysis
              </Typography>
              <Typography
                variant="body1"
                component="pre"
                sx={{
                  whiteSpace: 'pre-wrap',
                  fontFamily: 'inherit',
                }}
              >
                {analysis}
              </Typography>
            </CardContent>
          </Card>

          <ActionItemsDisplay analysisText={analysis} />
        </>
      )}

The Video Call Component

Here's the complete VideoCall.jsx logic broken into key sections:

Initial Setup and Imports

The Stream Video SDK is primarily integrated into the Videocall.jsx component. StreamVideo, StreamCall, and StreamVideoClient are core wrappers needed for client setup.

These core wrappers handle the initialization and context management required to enable real-time video features across your application. StreamVideoClient establishes the connection with Stream’s backend, while StreamCall provides context for managing a specific call instance. StreamVideo wraps the call UI and ensures all child components have access to the necessary call state and configuration. By centralizing these elements in the Videocall.jsx component, the app maintains a clean structure and consistent video behavior across different views or routes.

This SDK provides UI components such as CallControls, CallParticipantsList, SpeakerLayout, and VideoPreview.

These UI components are designed to streamline video call interface development. For example, CallControls offers built-in actions like mute, unmute, and screen sharing, while SpeakerLayout ensures the active speaker is prominently displayed. VideoPreview lets users check their camera feed before joining a call, enhancing the user experience.

The SDK also comes with its own default CSS stylin, which can be customized as needed to match your application’s branding or design system:

import { useState, useEffect } from 'react';
import {
  StreamVideo,
  StreamCall,
  StreamVideoClient,

  CallControls,
  CallParticipantsList,
  SpeakerLayout,
  VideoPreview,
} from '@stream-io/video-react-sdk';
import {
  Box,
  Button,
  TextField,
  Typography,
  CircularProgress,
} from '@mui/material';
import axios from 'axios';

// Import the Stream Video CSS
import '@stream-io/video-react-sdk/dist/css/styles.css';

State Management

The VideoCall.jsx file uses several state variables to manage the application effectively:

const VideoCall = ({ onTranscriptReady }) => {
  const [client, setClient] = useState(null);
  const [call, setCall] = useState(null);
  const [userId, setUserId] = useState('');
  const [meetingId, setMeetingId] = useState('');
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState('');

The fetchTranscriptWithRetry Function

This function fetches the transcript from the backend API. It takes four arguments:

callType: Specifies the type of call (always set to 'default').
callId: The unique meeting ID.
retries: Number of retry attempts.
delayMs: Delay in milliseconds between retries.

Retries are necessary because Stream might not have sent the transcript to the backend when the request is first made. If that happens, an error is logged, but the function will retry (up to three times).

const fetchTranscriptWithRetry = async (callType, callId, retries = 3, delayMs = 30000) => {
  for (let i = 0; i < retries; i++) {
    try {
      console.log(`Attempt ${i + 1} to fetch transcript for ${callType}:${callId}`);
      const response = await axios.get(`/api/get-transcript/${callType}/${callId}`);
      if (response.data?.transcript !== undefined) { // Check if transcript key exists (even if empty string)
        console.log('Transcript fetched successfully.');
        return response.data.transcript;
      }
      // If transcript key is missing but request succeeded, maybe backend error?
      console.warn('Transcript endpoint returned success, but no transcript data found.');
      // Decide if you want to retry in this case or treat as failure
      // For now, let's break and return null if data structure is unexpected
      return null;
    } catch (fetchError) {
      if (axios.isAxiosError(fetchError) && fetchError.response?.status === 404) {
        console.warn(`Transcript not found (Attempt ${i + 1}/${retries}). Retrying in ${delayMs / 1000}s...`);
        if (i < retries - 1) { // Don't wait after the last attempt
          await new Promise(resolve => setTimeout(resolve, delayMs));
        } else {
            console.error(`Transcript not found after ${retries} attempts for ${callType}:${callId}.`);
        }
      } else {
        console.error(`Error fetching transcript (Attempt ${i + 1}/${retries}):`, fetchError);
        // Don't retry on other errors (like server errors, network issues)
        return null; // Indicate failure
      }
    }
  }
  return null; // Indicate failure after all retries
};

Stream SDK Initialization

A StreamVideoClient instance is created using:

apiKey: Retrieved from environment variables.
token: Fetched from your /api/create-meeting backend endpoint.
user: An object like { id: userId }.

const initializeClient = async (userId, token) => {
    try {
      const user = { id: userId }; // Simplified user object

      const videoClient = new StreamVideoClient({
        apiKey: import.meta.env.VITE_STREAM_API_KEY,
        token,
        user
      });

      setClient(videoClient);
      return videoClient;
    } catch (error) {
      console.error('Error initializing client:', error);
      setError('Failed to initialize video client');
      return null;
    }
  };

Join Meeting Function

After the client is initialized, videoClient.call('default', meetingId) retrieves a call object. The join({ create: true }) method joins the call and creates it if it doesn’t already exist.

Once joined, startTranscription enables Stream’s server-side transcription and captions:

const joinMeeting = async (e) => {
    e.preventDefault();
    setLoading(true);
    setError('');

    try {
      // Create or join the call and get the token
      const response = await axios.post('/api/create-meeting', {
        meetingId,
        userId
      });

      const { token } = response.data;

      // Initialize client with the token
      const videoClient = await initializeClient(userId, token);
      if (!videoClient) {
        setLoading(false);
        return;
      }

      // Join the call
      console.log(`Attempting to join call ${meetingId} as user ${userId}`);
      const newCall = videoClient.call('default', meetingId);

      // Add event listener for connection quality changes
      newCall.on('connection.changed', (event) => {
        console.log('[Stream Call Event] Connection quality changed:', event.connection_quality);
      });

      await newCall.join({ create: true });
      console.log(`Successfully joined call ${meetingId}. Call CID: ${newCall.cid}`);

      // Start transcription and closed captions after successfully joining
      try {
        console.log('Attempting to start transcription...');
        await newCall.startTranscription({
          language: 'en',
          enable_closed_captions: true
        });
        console.log('Transcription and closed captions started successfully.');
      } catch (transcriptionError) {
        console.error('Failed to start transcription:', transcriptionError);
        // Optionally set an error state or notify the user
        // setError('Failed to start meeting transcription.');
      }

      setCall(newCall);
    } catch (error) {
      console.error('Error joining meeting:', error);
      // Check if the error object has more details
      if (error.message) {
        setError(`Failed to join meeting: ${error.message}`);
      } else {
        setError('Failed to join meeting. Check console for details.');
      }
    } finally {
      setLoading(false);
    }
  };

The Leave Meeting Function

When the user leaves the meeting:

call.leave() disconnects from the call.
Transcript fetching is triggered using fetchTranscriptWithRetry.
The transcript is passed to the App component via onTranscriptReady.

const leaveMeeting = async () => {
    console.log('Leave Meeting button clicked...');
    const currentMeetingId = meetingId;
    const callType = 'default';

    setLoading(true);
    try {
      if (call) {
        console.log('Attempting call.leave()...');
        await call.leave();
        console.log('Call left successfully via button.');
      }
    } catch (error) {
      console.error('Error during call.leave() via button:', error);
    } finally {
      // Reset UI state immediately
      setCall(null);
      setLoading(false);
      setCurrentCaption('');

      // --- Fetch Transcript after leaving (with retry) ---
      if (currentMeetingId && callType) {
         // Call the retry helper function - runs in background, doesn't block UI reset
        (async () => {
            const transcript = await fetchTranscriptWithRetry(callType, currentMeetingId);
            if (transcript !== null) {
                onTranscriptReady(transcript); // Pass transcript up to App
            } else {
                console.log('Failed to fetch transcript after retries.');
                 // Optionally inform App/user that transcript is unavailable
                 // onTranscriptReady('[Transcript unavailable]');
            }
        })();
      }
    }
  };

Client Disconnection

Ensures a clean disconnect when the component unmounts:

useEffect(() => {
    // Capture the client instance at the time the component mounts
    // (or when client state is first set)
    const clientInstance = client;

    // This cleanup function now runs ONLY when the component unmounts
    return () => {
      console.log('Component unmount cleanup effect running...');
      if (clientInstance) {
        console.log('Unmount Cleanup: Disconnecting user...');
        // Disconnect the user when the component unmounts.
        // The SDK should handle leaving any active call during disconnect.
        clientInstance.disconnectUser().then(() => {
           console.log('Unmount Cleanup: User disconnected successfully.');
        }).catch(disconnectError => {
           console.error('Unmount Cleanup: Error disconnecting user:', disconnectError);
        });
      } else {
         console.log('Unmount Cleanup: No active client instance to disconnect.');
      }
    };

  }, [client]); // Run effect when client instance is set/changes

UI Rendering

Before the Call

Displays form fields for name and meeting ID:

if (!call) {
    return (
      <Box component="form" onSubmit={joinMeeting} sx={{ mt: 3 }}>
        <Typography variant="h6" gutterBottom>
          Join Video Meeting
        </Typography>
        <TextField
          fullWidth
          label="Your Name"
          value={userId}
          onChange={(e) => setUserId(e.target.value)}
          margin="normal"
          required
          disabled={loading}
        />
        <TextField
          fullWidth
          label="Meeting ID"
          value={meetingId}
          onChange={(e) => setMeetingId(e.target.value)}
          margin="normal"
          required
          disabled={loading}
        />
        <Button
          type="submit"
          fullWidth
          variant="contained"
          disabled={loading || !userId || !meetingId}
          sx={{ mt: 2 }}
        >
          {loading ? <CircularProgress size={24} /> : 'Join Meeting'}
        </Button>
        {error && (
          <Typography color="error" sx={{ mt: 2 }}>
            {error}
          </Typography>
        )}
      </Box>
    );
  }

During the Call

Displays the video layout and controls:

return (
    <Box sx={{ mt: 3, height: '60vh', position: 'relative' }}> {/* Added position: relative */}
      <StreamVideo client={client}>
        <StreamCall call={call}>
          <Box sx={{ height: '100%', position: 'relative', display: 'flex', flexDirection: 'column' }}>
            <Box sx={{ flex: 1, minHeight: 0 }}>
              <SpeakerLayout />
            </Box>
            <Box sx={{ p: 2, bgcolor: 'background.paper' }}>
              <CallControls />
            </Box>
            <Button
              variant="contained"
              color="error"
              onClick={leaveMeeting}
              sx={{ position: 'absolute', top: 16, right: 16, zIndex: 1 }}
            >
              Leave Meeting
            </Button>
          </Box>
        </StreamCall>
      </StreamVideo>
    </Box>
  );

Structuring the Main App Component

App.jsx serves as the central hub for passing the transcript between VideoCall and MeetingAnalyzer.

State Management

const [latestTranscript, setLatestTranscript] = useState('');

Callback Mechanism/Function

A callback function, handleTranscriptReady, is defined within App.jsx. This function takes the transcript as an argument and updates the latestTranscript state using setLatestTranscript(transcript).

// Callback function for VideoCall to pass up the transcript
  const handleTranscriptReady = (transcript) => {
    console.log('App.jsx received transcript:', transcript.substring(0, 100) + '...'); // Log snippet
    setLatestTranscript(transcript);
  };

Data Flow

handleTranscriptReady is passed as onTranscriptReady to VideoCall.

latestTranscript is passed as initialNotes to MeetingAnalyzer.

<VideoCall onTranscriptReady={handleTranscriptReady} />
<MeetingAnalyzer initialNotes={latestTranscript} />

The UI

return (
    <ThemeProvider theme={theme}>
      <CssBaseline />
      <Header />
      <Container maxWidth="lg" sx={{ mt: 4, mb: 4 }}>
        <Grid container spacing={3}>
          <Grid item xs={12} md={6}>
            {/* Pass the callback down to VideoCall */}
            <VideoCall onTranscriptReady={handleTranscriptReady} />
          </Grid>
          <Grid item xs={12} md={6}>
            {/* Pass the transcript state down to MeetingAnalyzer */}
            <MeetingAnalyzer initialNotes={latestTranscript} />
          </Grid>
        </Grid>

      </Container>
    </ThemeProvider>

  );

Styling with MUI

The app uses Material-UI (MUI) for styling, which is a set of components that help you build beautiful and responsive user interfaces in React applications.

MUI supports theming out of the box and allows developers to maintain visual consistency while speeding up UI development. MUI also integrates well with responsive layouts and accessibility standards, ensuring the interface works seamlessly across devices and user needs.

In the App.jsx, a custom Material-UI theme is created using createTheme, which allows for a consistent color palette and typography for all MUI components used within the application.

const theme = createTheme({
  palette: {
    mode: 'light',
    primary: {
      main: '#1976d2',
    },
    secondary: {
      main: '#dc004e',
    },
  },
});

Common MUI components used include: Button, TextField, Container, Grid, Typography, CircularProgress, and Box.

These components help create a clean, responsive layout while reducing the need to write custom styles from scratch. For instance, Grid and Box simplify layout management with Flexbox and spacing utilities, while Typography ensures consistent text styling.

The sx prop is used extensively for inline styling and layout control.

Example:

<Container maxWidth="lg" sx={{ mt: 4, mb: 4 }}>

      </Container>

Testing the Application

You must run both the backend and frontend code to test your implementation.

Running the Backend

To run the backend code using nodemon, navigate to the backend folder and execute the following command:

npm run dev

You should see an output similar to the screenshot below in your terminal:

Running the Frontend

To start the frontend, navigate to the frontend folder and use a command similar to the one above. You should see the following interface:

After running the app, the first view lets users input their name and a meeting ID.

Join a Meeting with Two Users

You can test your project by having two users join and interact during the same meeting.

This allows you to validate video and audio functionality, check real-time updates like participant lists, test UI responsiveness during active calls, verify that Stream's transcript is being returned, and ensure that the LLM sends back the analyzed results, which are correctly displayed in the UI.

Let's name the first user Quincy and the meeting ID test-meeting-2 as shown in the image below.

The second user is named Anointed and has the same meeting ID.

Now, here is a video from the meeting:

After the meeting concludes:

Stream generates a transcript.
The backend stores it.
The frontend fetches it.
The user can click “Analyze Meeting Notes” to get insights.

Conclusion

Building effective AI meeting assistants using Stream Video and LLMs offers a huge productivity boost. Teams can operate more efficiently by turning conversations into structured summaries and action points.

The complete source code is available on the GitHub repository.

Now that you know how to combine Stream Video and LLMs to build an AI meeting assistants—what’s next? Try integrating these analyzed action items with task management tools like Trello, Jira, or Asana. To take it a step further, learn how to schedule meetings automatically, make your video experiences accessible, or build a full-featured video conferencing app using our in-depth guides.

How to Implement Real-Time Language Translation in Chat with LLMs

Quincy Oghenetejiri — Wed, 30 Apr 2025 09:12:55 +0000

Real-time language translation has become essential for global applications, communities, and businesses to break down language barriers and foster inclusive environments.

In this article, I’ll explain how to implement real-time language translation in Stream applications using large language models (LLMs).

By the end, you will understand how to:

Authenticate users using a Stream token.
Create a translation middleware service using LLMs.
Integrate this service with Stream Chat.
Implement language preference management on the client side.

Understanding the Problem

The Language Barrier Challenge

When users who speak different languages try to communicate in real-time, the experience often breaks down. Traditional solutions include:

Manual translation: Requiring users to copy/paste text into translation services
Built-in translation buttons: Adding translation options that interrupt the flow of conversation
Separate language channels: Segregating users by language, which reduces cross-cultural interaction

These approaches create friction and slow communication, often resulting in fragmented communities.

Why LLMs for Translation?

LLMs offer several advantages over traditional translation APIs:

Context awareness: LLMs understand conversational context, improving translation accuracy
Cultural nuance: They handle idioms, slang, and culturally specific expressions more effectively
Adaptability: They can follow specific translation instructions when prompted

Technical Prerequisites

Before we begin, ensure you have the following:

A Stream account with an API key and secret
Access to an LLM API (e.g., OpenAI, Anthropic, Gemini)
Node.js and npm/yarn installed
Basic knowledge of React and Node.js

Solution Architecture

Our solution consists of five main components:

Stream Chat SDK: Handles the core chat functionality
Translation Middleware: A Node.js service that intercepts new messages
LLM Service: Performs the actual translation
Caching Layer: Stores previous translations to improve performance
Client Browser: Consumes the translation middleware API

Message Flow

The user authenticates and receives a Stream token
The user sends a message in their preferred language
Stream webhook triggers the translation middleware
Middleware verifies the request and checks the cache for existing translations
If not cached, the LLM API translates the message
Translations are stored in the message metadata
An updated message with translations is sent to recipients
Client displays a message in the user's preferred language

Implementation Guide: Backend

Before diving into how to create the translation middleware and connect it to the client, you need to go through a few setup steps.

Setting Up the Translation Server

Start by building your translation middleware service.

mkdir stream-translation-service
cd stream-translation-service
npm init 
npm install express stream-chat nodemon dotenv cors axios @google/generative-ai

Create a .env File for Configuration. This will store your Stream credentials and the API key for your chosen LLM provider:

STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret
LLM_API_KEY=your_llm_api_key
LLM_API_URL="https://generativelanguage.googleapis.com/v1beta"
PORT="3000"

Next, Create a server.js file to contain your backend logic:

// server.js
const express = require('express');
const { StreamChat } = require('stream-chat');
const axios = require('axios');
const cors = require('cors');
require('dotenv').config();
const { GoogleGenerativeAI } = require('@google/generative-ai');

const app = express();
app.use(express.json());
app.use(cors());

Service Initialization

Initialize both the Stream Chat and Gemini AI services:

// Initialize Gemini AI
const genAI = new GoogleGenerativeAI(process.env.LLM_API_KEY);

// Initialize Stream Chat with configuration
const streamClient = StreamChat.getInstance(
  process.env.STREAM_API_KEY,
  process.env.STREAM_API_SECRET,
  {
    timeout: 10000,
    maxRetries: 3,
    logger: (logLevel, message, extraData) => {
      console.log(message, extraData);
    }
  }
);

Translation Cache System

To optimize performance, implement a caching system for translations:

const translationCache = new Map();

const createCacheKey = (text, sourceLang, targetLang) => {
  return `${text}_${sourceLang}_${targetLang}`.toLowerCase();
};

const setCacheEntry = (text, sourceLang, targetLang, translatedText) => {
  // Cache validation and storage logic
};

const getCacheEntry = (text, sourceLang, targetLang) => {
  // Cache retrieval with 24-hour expiration
};

Core Translation Function

async function translateText(text, sourceLang, targetLang) {
  try {
    // Translation validation and processing
    const model = genAI.getGenerativeModel({ 
      model: 'gemini-2.0-flash'
    });

    const prompt = `Translate this text from ${sourceLang} to ${targetLang}...`;
    // Translation logic and cache management
  } catch (error) {
    console.error('Translation error:', error);
    throw error;
  }
}

API Endpoints

User Management

POST /signup – User registration and channel assignment
POST /login – User authentication
POST /set-language – Language preference setting

Channel Operations

POST /channel – Channel creation and joining
POST /webhook – Real-time message translation

Translation Services

POST /direct-translate – Direct translation requests

System Management

GET /health – System health check
DELETE /delete-messages – Message cleanup for users

Real-Time Message Processing

app.post('/webhook', async (req, res) => {
  try {
    const { type, message } = req.body;
    // Message validation and translation
    // Member language detection
    // Translation distribution
  } catch (error) {
    console.error('Webhook error:', error);
    res.status(500).json({ 
      success: false,
      translations: {},
      error: 'Translation failed' 
    });
  }
});

Server Initialization

Finally, start the server on your configured port:

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Translation service running on port ${PORT}`);
});

This configuration sets up a robust server component that handles real-time chat translation, user management, and message processing—while maintaining performance through caching and solid error handling.

Complete Source Code

You can find the complete source code for this implementation in this repository.

Implementation Guide:Frontend

Backend Setup

The backend should be set up to run on localhost:3000.

Setting up Your React Application

npm create vite@latest stream-chat-front --template react
cd stream-chat-front
npm install axios stream-chat stream-chat-react
npm run dev

Setting Up Environment Variables

In your .env file, add the following:

VITE_STREAM_API_KEY=your_stream_api_key  
VITE_BACKEND_URL=your_backend_url

Authentication Component

This component handles authentication, including both sign-up and login.

import { useState } from 'react';

export function Auth({ onLogin, error, onError }) {
  const [isLogin, setIsLogin] = useState(true);
  const [username, setUsername] = useState('');
  const [language, setLanguage] = useState('en');

  const handleSubmit = async (e) => {
    e.preventDefault();
    try {
      const endpoint = isLogin ? 'login' : 'signup';
      const response = await fetch(`${import.meta.env.VITE_BACKEND_URL}/${endpoint}`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ username, language }),
      });

      const data = await response.json();

      if (!response.ok) {
        throw new Error(data.error || (isLogin ? 'Login failed' : 'Signup failed'));
      }

      onLogin({
        user: data.user,
        token: data.token,
        apiKey: data.apiKey,
        defaultChannel: data.defaultChannel
      });
    } catch (err) {
      console.error(isLogin ? 'Login error:' : 'Signup error:', err);
      if (typeof onError === 'function') {
        onError(err.message);
      }
    }
  };

  return (
    <div className="auth-form">
      <h2>{isLogin ? 'Login' : 'Sign Up'}</h2>
      {error && <div className="error-message">{error}</div>}
      <form onSubmit={handleSubmit}>
        <div className="form-group">
          <label htmlFor="username">Username</label>
          <input
            type="text"
            id="username"
            value={username}
            onChange={(e) => setUsername(e.target.value)}
            required
          />
        </div>
        <div className="form-group">
          <label htmlFor="language">Preferred Language</label>
          <select
            id="language"
            value={language}
            onChange={(e) => setLanguage(e.target.value)}
          >
            <option value="en">English</option>
            <option value="es">Spanish</option>
            <option value="fr">French</option>
          </select>
        </div>
        <button type="submit" className="auth-button">
          {isLogin ? 'Login' : 'Sign Up'}
        </button>
      </form>
      <button
        className="toggle-auth-button"
        onClick={() => setIsLogin(!isLogin)}
      >
        {isLogin ? 'Need an account? Sign Up' : 'Already have an account? Login'}
      </button>
    </div>
  );
}

Language Selector Component

// src/components/LanguageSelector.jsx
import React from 'react';

export const LanguageSelector = ({ currentLanguage, onChange }) => {
  const languageOptions = [
    { value: 'en', label: 'English' },
    { value: 'es', label: 'Spanish' },
    { value: 'fr', label: 'French' },
    { value: 'de', label: 'German' },
    { value: 'ja', label: 'Japanese' },
  ];

  return (
    <div className="language-selector">
      <label htmlFor="language-select">Select language: </label>
      <select
        id="language-select"
        value={currentLanguage}
        onChange={(e) => onChange(e.target.value)}
      >
        {languageOptions.map((lang) => (
          <option key={lang.value} value={lang.value}>
            {lang.label}
          </option>
        ))}
      </select>
    </div>
  );
};

Translated Message Component

// src/components/TranslatedMessage.jsx
import React, { useState, useEffect } from 'react';
import { MessageText, useMessageContext } from 'stream-chat-react';

export const TranslatedMessage = ({ userLanguage, onDelete }) => {
  const { message } = useMessageContext();
  const [translatedText, setTranslatedText] = useState(null);
  const [isDeleting, setIsDeleting] = useState(false);

  useEffect(() => {
    if (message) {
      const translation =
        message.translations?.[userLanguage] ||
        message[`text_${userLanguage}`] ||
        message[`translated_${userLanguage}`];

      setTranslatedText(translation || message.text);
    }
  }, [message, userLanguage]);

  const modifiedMessage = {
    ...message,
    text: translatedText || message.text,
  };

  const handleDelete = async () => {
    if (!onDelete || isDeleting) return;
    try {
      setIsDeleting(true);
      await onDelete(message.id);
    } catch (error) {
      console.error('Failed to delete message:', error);
    } finally {
      setIsDeleting(false);
    }
  };

  return (
    <div className="str-chat__message-translation-wrapper">
      <div className="message-content">
        <MessageText message={modifiedMessage} />
        <button
          className="delete-message-btn"
          onClick={handleDelete}
          disabled={isDeleting}
          title="Delete message"
        >
          {isDeleting ? '...' : '🗑️'}
        </button>
      </div>
    </div>
  );
};

Configuring the Main App Component

Here's the complete App.js logic broken into key sections:

Initial App Structure

// Step 1: The basic app shell
import React from 'react';

function App() {
  return (
    <div className="app">
      <h1>Real-Time Translation Chat</h1>
    </div>
  );
}

export default App;

Initial Setup and Imports

StreamChat—which will be initialized later—is imported in the code below, along with Chat, Channel, ChannelHeader, MessageInput, MessageList, Thread, and Window from the stream-chat-react package.
The Chat and Channel components serve as React context providers. They pass the following to their children via React’s context system:

UI components
Channel state data
Messaging functions

import React, { useState, useEffect } from 'react';
import { StreamChat } from 'stream-chat';
import {
  Chat, Channel, ChannelHeader,
  MessageInput, MessageList,
  Thread, Window
} from 'stream-chat-react';

Key assignment and Stream chat Client Configuration

The StreamChat client abstracts API calls into methods and manages state and real-time events.

To prevent the client from being recreated on every render, the instance is initialized outside the component. This instance also handles connection state and real-time updates.

// Get API key from environment variable
const API_KEY = import.meta.env.VITE_STREAM_API_KEY;
const BACKEND_URL = import.meta.env.VITE_BACKEND_URL;

// Create the client outside of component to prevent recreating it on renders
const chatClient = StreamChat.getInstance(API_KEY);

State Management

The App.js file uses several state variables to manage the application effectively.

function App() {
  const [channel, setChannel] = useState(null);        // Manages active chat channel
  const [user, setUser] = useState(null);             // Stores user information
  const [userLanguage, setUserLanguage] = useState('en'); // Current language
  const [isConnecting, setIsConnecting] = useState(false);
  const [error, setError] = useState(null);
  const [clientReady, setClientReady] = useState(false);

Authentication and CleanUp Handler.

In this section, authentication is handled, and a channel is created using the StreamChat client’s channel() method. The method accepts three arguments:
1.'messaging': The channel type that determines features and permissions.
2.'translation-demo': A unique channel identifier.

A channel data object for additional configuration.

const handleLogin = async (data) => {
    try {
      setIsConnecting(true);

      const { user, token, apiKey } = data;

      // Initialize Stream Chat client
      const chatClient = StreamChat.getInstance(apiKey);

      // Connect user to Stream Chat
      await chatClient.connectUser(
        {
          id: user.id,
          name: user.name,
          language: user.language,
        },
        token
      );

      // Create or join default channel
      const channelName = 'translation-demo';
      const channel = chatClient.channel('messaging', channelName, {
        name: 'Translation Demo Channel',
        members: [user.id],
      });

      await channel.watch();

      setChannel(channel);
      setUser(user);
      setUserLanguage(user.language);
      setClientReady(true);
      setError(null);
    } catch (error) {
      console.error('Login error:', error);
      setError('Login failed. Please try again.');
    } finally {
      setIsConnecting(false);
    }
  };

  const handleLogout = async () => {
    try {
      if (chatClient) {
        await chatClient.disconnectUser();
        setChannel(null);
        setUser(null);
        setClientReady(false);
      }
    } catch (error) {
      console.error('Logout error:', error);
    }
  };
  // Cleanup on unmount
  useEffect(() => {
    return () => {
      if (chatClient.userID) {
        chatClient.disconnectUser();
      }
    };
  }, []);

Language Translation Handler

const handleLanguageChange = async (language) => {
    if (user) {
      try {
        setIsConnecting(true);
        console.log('Requesting translation to:', language);

        const channelMessages = channel.state.messages || [];
        console.log('Messages to translate:', channelMessages.length);

        // Send all messages for translation, even when target is English
        const response = await fetch(`${BACKEND_URL}/set-language`, {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${user.token}`
          },
          body: JSON.stringify({
            userId: user.id,
            language,
            channelId: channel.id,
            messages: channelMessages.map(msg => ({
              id: msg.id,
              text: msg.text,
              // Always send source language for better translation
              language: msg.language || detectLanguage(msg.text)
            }))
          }),
        });

        const result = await response.json();
        console.log('Backend response:', result);

        if (!response.ok) {
          throw new Error(`Backend error: ${result.error || response.statusText}`);
        }

        // Process translations regardless of target language
        if (result.translations && Object.keys(result.translations).length > 0) {
          for (const [messageId, translation] of Object.entries(result.translations)) {
            if (translation && translation.trim()) {
              await channel.updateMessage({
                id: messageId,
                translations: {
                  [language]: translation
                }
              });
            }
          }
        }

        setUserLanguage(language);
        await channel.watch();
      } catch (error) {
        console.error('Translation error:', error);
        setError(`Failed to change language: ${error.message}`);
      } finally {
        setIsConnecting(false);
      }
    }
  };

Message Management.

// Add message deletion handler
  const handleMessageDelete = async (messageId) => {
    try {
      if (!channel || !user) return;

      const response = await fetch(`${BACKEND_URL}/delete-messages`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${user.token}`
        },
        body: JSON.stringify({
          channelId: channel.id,
          channelType: 'messaging',
          messageIds: [messageId],
          userId: user.id
        }),
      });

      if (!response.ok) {
        const error = await response.json();
        throw new Error(error.message || 'Failed to delete message');
      }

      // Refresh channel to update messages
      await channel.watch();
    } catch (error) {
      console.error('Delete message error:', error);
      setError('Failed to delete message. Please try again.');
    }
  };

UI Rendering

The component renders different views based on the application state:
Authentication view – Displayed when no user is logged in.

if (!user) {
  return (
    <div className="auth-container">
      <Auth onLogin={handleLogin} error={error} />
    </div>
  );
}

Loading view – Shown while connecting to the chat service.

if (isConnecting) {
  return (
    <div className="loading-container">
      <div className="loading-indicator">Connecting to chat...</div>
    </div>
  );
}

Main chat interface– Rendered when the application is fully connected.
In the code below, the Chat and Channel components act as context providers:

Chat provides global chat context to all child components, including:
- Connection status
- User information
- Theme settings
Channel provides channel-specific context, including:
- Message list
- Channel state
- Typing indicators
- Message actions

return (
  <div className="app">
    <div className="app-header">
      <h1>Real-Time Translation Chat</h1>
      <LanguageSelector
        currentLanguage={userLanguage}
        onChange={handleLanguageChange}
      />
      <div className="user-controls">
        <div className="user-info">Connected as: {user.name}</div>
        <button onClick={handleLogout}>Logout</button>
      </div>
    </div>

    {channel && (
      <div className="chat-container">
        <Chat client={chatClient}>
          <Channel channel={channel}>
            <Window>
              <ChannelHeader />
              <MessageList
                Message={(props) => (
                  <TranslatedMessage
                    {...props}
                    userLanguage={userLanguage}
                    onDelete={handleMessageDelete}
                  />
                )}
              />
              <MessageInput />
            </Window>
            <Thread />
          </Channel>

     </Chat>
      </div>
    )}
  </div>
);

CSS Styles

Basic CSS Styles

/* Base layout styles */
.app {
  height: 100vh;
  display: flex;
  flex-direction: column;
}

/* Header section */
.app-header {
  padding: 1rem;
  background: #fff;
  border-bottom: 1px solid #eee;
  display: flex;
  justify-content: space-between;
  align-items: center;
}

/* Chat container */
.chat-container {
  flex: 1;
  overflow: hidden;
}

Component Specific Style.

/* Auth container */
.auth-container {
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100vh;
  background: #f5f5f5;
}

/* Language selector */
.language-control {
  select {
    padding: 0.5rem;
    border: 1px solid #ddd;
    border-radius: 4px;
    font-size: 14px;
  }
}

/* User controls */
.user-controls {
  display: flex;
  align-items: center;
  gap: 1rem;
}

Note: You should remove the default styles in src/index.css to ensure your custom styles are applied correctly.

Complete Source Code

You can find the complete source code for this implementation in this repository.

Testing Your Implementation

You must run both the backend and frontend code to test your implementation.

Running the Backend

To run the backend code using nodemon, execute the following command:

npm run dev

You should see an output similar to the screenshot below in your terminal.

Running the Frontend

To start the frontend, use a command similar to the one above. You should see the following interface:

Here's the first view: No user is currently signed in.

Signing Up a New User

Now, let's sign up for a new user named StreamUser123.

Signed-In View

Once signed in, you’ll be redirected to the main chat interface.

Sending Messages

Let's try typing some messages in the chat.

Real-Time Language Switching

A user can switch between different languages in seconds without copying the message to a different endpoint.

Conclusion

Implementing real-time language translation in Stream Chat using LLMs creates a more inclusive and connected chat experience. In this guide, we covered:

Setting up secure authentication with Stream tokens
Building a translation middleware service
Integrating with Stream Chat webhooks
Creating a responsive front end with language preference management

Further Improvements

To take your implementation even further, consider:

Fine-tuning an LLM specifically for your community’s domain and language pairs
Implementing user feedback mechanisms to improve translation quality continuously
Adding role-based access control for different types of users

By leveraging modern LLMs, we can offer more contextually aware and natural translations than traditional translation APIs, significantly enhancing the user experience.

Writing Maintainable Tests in React using the React Testing Library.

Quincy Oghenetejiri — Mon, 07 Nov 2022 20:24:18 +0000

If you want to be sure that your components are functioning correctly for your users, you should build maintainable tests. You want to avoid putting implementation details in your tests as part of this objective so that refactoring of your componentschanges to implementation but not functionalitywon't invalidate them and slow you and your team down.

In this article, I will show you what software testing is all about, the different types of tests you will encounter when writing tests for your react application and how to write maintainable tests using the React testing library.

What is Software Testing?

Business-critical software is tested for accuracy, performance, and quality as part of the organizational process of software development. Software testing is performed to make sure that expected product features and business systems operate as intended.

Software testing can be carried out either manually or automatically.

A team or individual who will manually operate a software product and make sure it operates as intended is in charge of manual software testing. Manual testing is done in person, by clicking through the application or by interacting with the software and APIs with the appropriate tooling. Automated software testing is carried out using a variety of technologies with a range of functions, from isolated code correctness checks to full-fledged simulations of manual testing conducted by humans.

Different types of Software Testing

You can use a variety of software testing approaches to make sure that modifications to your code function as intended.

Unit Testing

Unit tests are basic and carried out near an application's source code. They entail testing each class, component, or module used by your software's distinct methods and functions.

In React, Unit testing is done by testing if a particular component has been rendered into the UI.

Integration Testing

Integration tests ensure that the various components or services used by your application function properly. For instance, it could involve testing database interactions or ensuring that microservices interact as intended.

In React, Integration testing is done by testing the interactions between different components

Functional Testing

The business requirements of an application are the main focus of functional tests. They don't check the system's intermediate states while carrying out an action; they only check the action's outcome.

Because both require a number of components to interact with one another, integration tests and functional tests are occasionally confused.

End-to-end Testing

End-to-end testing simulates user behaviour using the software in the whole context of an application. It checks that different user flows function as intended and can be as straightforward as loading a webpage or login in or somewhat more complicated scenarios checking email notifications, online payments, etc.

Acceptance Testing

Acceptance tests are official testing that determines whether a system complies with business needs. They concentrate on reproducing user actions while executing the complete application during testing. They can, however, take a step further and assess the system's effectiveness, rejecting improvements if particular objectives are not achieved.

Performance Testing

Performance evaluations assess a system's ability to handle a specific workload. These evaluations aid in determining an application's dependability, speed, scalability, and responsiveness. For instance, a performance test can monitor response times when a large number of requests are being processed or can assess how a system responds to a sizable amount of data. It may assess whether a program satisfies performance standards, find bottlenecks, gauge stability during high traffic, and do much more.

Smoke Testing

Smoke tests are simple tests that examine an application's fundamental operation. They are designed to be quickly carried out, and their purpose is to provide you with the confidence that the main system components are functioning as planned.

Why You Should Write Tests for your code in React

Spot ambiguities early.
Write better and more efficient code.
Easier debugging.
Saves time.

Automated Tests in React using the React Testing Library

The React Testing Library is a very lightweight solution for testing React components. It provides light utility functions on top of react-domand react-dom/test-utils, in a way that encourages better testing practices. Its primary guiding principle is:

The more your tests resemble the way your software is used, the more confidence they can give you.

Your tests will therefore deal with actual DOM nodes rather than instances of rendered React components. The features this library offers make it easier to query the DOM in a user-like manner.

Projects created with Create React App have out-of-the-box support for React Testing Library. If that is not the case, you can add it via npm like so:

npm install --save-dev @testing-library/react

React Testing Library works with the following:

Queries.
Matchers.
User Action(Fire Event ).

Queries

The techniques Testing Library provides for locating elements on the page are queries. There are different types of queries (" get ," " find ," and " query"); the distinction is whether the query will throw an error or return a Promise and retry if no element is found. Different searches might be more or less relevant depending on the content of the page you are choosing.

The queries returned from render in React Testing Library are the same as DOM Testing Libraryexcept they have the first argument bound to the document, so instead of getByText(node, 'text')you do getByText('text')

Types of Queries

Single elements:

getBy... , findBy... , queryBy...

Multiple elements:

getAllBy... , findAllBy... , queryAllBy...

The suffixes for all the above-mentioned are:

Role
LabelText
PlaceholderText
Text
DisplayValue
AltText
Title
TestId

Custom Jest Matcher

The @testing-library/jest-dom library provides a set of custom jest matcher that you can use to extend jest. These will make your tests more declarative, and clear to read and maintain.

Some of the popular matcher used are:

toBeInTheDocument
toBeVisible
toBe
toHaveClass
toHaveTextContent

More matcher can be found here

User Action(Fire Event)

Fire Event in the react testing library is similar to event listeners used in DOM manipulation, it gives access to various events such as OnChange, onClick e.t.c

Fire Event takes two parameters:

element: That is the element you want to trigger an event on
target : When an event is dispatched on an element, the event has the subjected element on a property called target. As a convenience, if you provide a target property in the eventProperties (second argument), then those properties will be assigned to the node which is receiving the event.

Unit Testing Using the React Testing Library

Below is an example of unit testing for a header component in a Todo application.

To write a test for a component in React create a folder named __test__ after that add .test.js to the name of the file you want to create.

In the code below, render and screen are being imported from the @testing-library/react, the header component to be tested is also being imported.

Tests in React are written in the test/it method provided by Jest. I prefer it method as its shorter and easier to comprehend.

A test/it method takes in two parameters:

Description of the test you will be writing.
An Asynchronous function.

Within the function

The component to be tested is rendered into the DOM.
Next, the DOM is queried for the headingElement with the text "My Header" using the screen object imported from @testing-library/react, which has several queries, in this case, the getByText query is used to select the headingElement.
Next, an assertion is made using the assert method provided by jest.

//Unit Testing without the describe block method testing a header component
import { render, screen } from '@testing-library/react';
import Header from '../Header';

  it('should render same text passed into title prop', async () => {
        //Components to be tested are rendered into the DOM
    render(<Header title="My Header" />);
        //Query being carried out by the getByText
    const headingElement = screen.getByText(/my header/i);
        //Making use of matcher provided by jest to make an Assertion  
    expect(headingElement).toBeInTheDocument();
  });

You can choose to wrap your test in a describe block or not but it is better to do that as it improves the readability of your test code.

The describe function is similar to the test method as it takes two parameters:

A description.
A function that will run the test.

//Unit Testing with the describe block method
import { render, screen } from '@testing-library/react';
import Header from '../Header';

//The describe method with the test being run by the function in the second parameter
describe("Header",()=>{
  it('should render same text passed into title prop', async () => {
        //Components to be tested is rendered into the DOM
    render(<Header title="My Header" />);
        //Query being carried out by the getByText
    const headingElement = screen.getByText(/my header/i);
        //Making use of matchers provided by jest to make an Assertion  
    expect(headingElement).toBeInTheDocument();
  });
})

To run your test to check if it will pass or fail not cd into the correct directory containing the code you are testing for and paste the code.

npm run test

Your result will be similar to the image below if passed.

Testing Library Hooks

They are four hooks provided by the @testing-library/reactwhich you can use when writing tests :

beforeEach: This hook run before every single test.
beforeAll: This hook runs once before all the tests run.
afterEach: This hook run after each test.
afterAll: This hook runs once only after all the tests have stopped running.

These hooks are called within the test method and they take in a single parameter which is a function.

From the code below you can see the use case of the react testing library hooks:

// Testing Library Hooks
  beforeEach(()=>{
    console.log("RUNNING BEFORE EACH TEST")
  })

  beforeAll(()=>{
    console.log("RAN ONCE BEFORE ALL TESTS")
  })

  afterEach(()=>{
    console.log("RUNNING AFTER EACH TEST")
  })

  afterAll(()=>{
    console.log("RAN ONCE AFTER ALL EACH TEST")
  })

Debugging

You may encounter errors when writing your tests, making use of screen.debug which is a convenience function for pretty-dom allows you to output your errors .

Conclusion

This article has walked you through what testing is all about, different types of tests, why you should test your code and how you should write tests for your react app using the react testing library. You learned how to make queries and assertions when testing your react app.

At this point, you are confident enough to start writing tests for your React App using the react testing library.

Reference

Software Testing in Continuous Delivery | Atlassian

React Testing Library | Testing Library (testing-library.com)

Creating a typewriting-effect in React.js

Quincy Oghenetejiri — Thu, 12 May 2022 06:32:26 +0000

In my previous article, I talked about how to create a typewriter using vanilla JavaScript and I promised to show my react.js folks how to do the same thing using React. Let get started 👇

Installing our react app

Before we start we need to install our react app

npx-create-react-app app

The code above will create a well-structured react app like this:

After this, we need to install a package called typewriter-effect inside the app folder we installed our react app into

npm install typewriter-effect

The code above will install the package and we can check our package.json file to confirm.

The next step involves us creating a JavaScript file inside our src folder to store the words we will be typing out let's name it mockdata.js. In this file we will be creating an array to store a string of words we will be typing out and then export it like the code below:

export const data=[
    "Design. Build.Improve.",
    "Develop. Rebrand. Initiate."
]

Creating our `typewriter.js` file

We will create a component folder in our src folder and then create a typewriter.js file inside the component folder. In the typewriter.js file we will create a React functional component and pass in a prop value text like the code below:

import Typewriter from 'typewriter-effect';

const TypeWriter = ({ texts }) => (
  <Typewriter
    options={{
      strings: texts,
      autoStart: true,
      loop: true,
      deleteSpeed: 70,
      wrapperClassName: 'head_tags'
    }}
  />
); 

export default TypeWriter;

In the code below we needed to import Typewriter from the typewriter-effect unless we won't be able to create the typewriting effect. Also, we passed in an options object which takes in different values as explained below

The Options Object

Now lets define the options we passed above

String: The string option typically refers to the text/string to type out. It accepts a String value.
autoStart: This simply entails whether to start typing or not. It accepts a Boolean value and it works along with the string options in this case we set the value to true.
Loops: This simply entails whether to keep looping or not. It accepts a Boolean value in this case we set the value to true.
deleteSpeed: This simply means the delay between deleting each character. It accepts a Number value.
wrapperClassName: This is simply the class name for the wrapper element. It accepts a String value.

Importing our typewriter

The next step will be to import the function created in our previous JavaScript file to our App.js file.

import TypeWriter from "./typewriter"

After importing we can then use the Typewriter component inside our file like the code below:

<h4><strong><TypeWriter texts={data} /> </strong></h4>

Save and run your react app with the code npm start and you will have a sample like the below:

Conclusion

Congrats!!! You have successfully implemented a typewriter effect using the typewriter-effect package in react.

Creating a Typewriting-effect using Vanilla JavaScript

Quincy Oghenetejiri — Wed, 04 May 2022 17:24:31 +0000

Have you gone to a website and noticed typewriting on the page without you clicking or touching anything, awesome right? yea.

In this article, I will show you how you can create that feature using vanilla JavaScript.

Prerequisites

A basic knowledge of JavaScript.
A basic knowledge of HTML and CSS.
A code editor and a browser
Live Server extension or You can load it manually on your respective browsers.

Getting Started

Well start by creating our HTML file and CSS file, lets call them index.html and index.css. we will create an h1 tag underneath the h1 tag we also need to create a span tag which we will give a class name of txt-type like the code below.

<h1>
        I am Quincy Oghenetejiri A <span class="txt-type" data-wait="3000s" data-words='["Software Developer", "Front End Developer","Backend Developer","Technical Writer"]'></span>
    </h1>

In the span tag, we have two attributes data-wait and data-words. The data-wait is the time we set for the typing speed while the data-words is where we stored the words which we will be typing out and deleting at intervals.

In our index.css file, we will target the class txt-type and give a border-right value to provide that mouse typing effect

.txt-type {
    border-right: 0.2rem solid black;

}

Note: We need to link our index.css file to our HTML file for it to work.

<link rel="stylesheet" href="/index.css">

Our Result Layout 👇

Creating Our JavaScript File

After we have sorted the above we will have to create our index.js file. We also need to link this file for it work.

<script src="./index.js"></script>
</body>
</html>

Inside this file we need to create a class and call it Typewriter

class Typewriter{

}

Inside this class we will create a constructor function, a constructor function is a special function that creates and initializes an object instance of a class. This constructor function will take in three values. It also set default values that will be override in the next function we will be creating inside the class.

constructor(txtElement,words,wait=3000){
        this.txtElement= txtElement
        this.words = words
        this.txt = ""
        this.wordIndex = 0
        this.wait = parseInt(wait,10)
        this.type()
        this.isDeleting = false

    }

After the creation of the constructor function, we will then create another function inside the class Typewriter in this function we create two variables and in the first variable as seen in the code below we used the remainder operator (%) to target the current word length which we will be the typing out and then output it into a second variable to get the current word.

type(){
         //current index of word 
    const current = this.wordIndex % this.words.length
  //Get Full text of current word
    const fullTxt = this.words[current]

    }

The next step involves creating an if and else statement inside the type function to delete and add text characters. The this.txt, which has an initial value of an empty string is now updated.

// Check if deleting 
    if(this.isDeleting){
          //Remove char
        this.txt = fullTxt.substring(0,this.txt.length - 1)
        }

    else{
         //Add Char 
         this.txt = fullTxt.substring(0,this.txt.length + 1)   
        }

Next we will insert the text into an HTML element and set the type speed we want I will be using 300

//Insert txt into element 
        this.txtElement.innerHTML = `<span class="txt">${this.txt}</span>`

        // Initial Type Speed
        let typeSpeed = 300

        if(this.isDeleting){
            typeSpeed /= 2
        }

The next step involves creating an if and else if statement to check if a word is complete to start deleting or not complete to continue typing and also we use the setTimeout function available in JavaScript to sets the timer(a countdown set in milliseconds) which take the value of the typeSpeed variable.

//If word is complete
        if(!this.isDeleting && this.txt === fullTxt){
            //Make Pause at end
            typeSpeed = this.wait
            //set delete to true
            this.isDeleting = true
        }
        else if(this.isDeleting && this.txt === ''){
            this.isDeleting = false
            //move to next word
            this.wordIndex++
            //pause before start typing
            typeSpeed = 500
        }

        setTimeout(()=> this.type(),typeSpeed)

The next step involves initializing our class on DOM load. We'll first of all create an init function which handles our selection of our HTML class and also initializing our class.

//Init App
function init(){
    const txtElement = document.querySelector(".txt-type")
    const words = JSON.parse(txtElement.getAttribute('data-words'))
    const wait = txtElement.getAttribute('data-wait')
    //Init Typewriter
    new Typewriter(txtElement,words,wait)
}

And then we load the function created above in the DOM

document.addEventListener('DOMContentLoaded',init)

And yass that is all 😅.

This is all the JavaScript code we wrote.

class Typewriter{
    constructor(txtElement,words,wait=3000){
        this.txtElement= txtElement
        this.words = words
        this.txt = ""
        this.wordIndex = 0
        this.wait = parseInt(wait,10)
        this.type()
        this.isDeleting = false
    }

    type(){
         //current index of word 
        const current = this.wordIndex % this.words.length
        //Get Full text of current word
        const fullTxt = this.words[current]

            // Check if deleting 
        if(this.isDeleting){
            //Remove char
            this.txt = fullTxt.substring(0,this.txt.length - 1)
        }

        else{
            //Add Char 
            this.txt = fullTxt.substring(0,this.txt.length + 1)   
        }

        //Insert txt into element 
        this.txtElement.innerHTML = `<span class="txt">${this.txt}</span>`

        // Initial Type Speed
        let typeSpeed = 300

        if(this.isDeleting){
            typeSpeed /= 2
        }

        //If word is complete
        if(!this.isDeleting && this.txt === fullTxt){
            //Make Pause at end
            typeSpeed = this.wait
            //set delete to true
            this.isDeleting = true
        }
        else if(this.isDeleting && this.txt === ''){
            this.isDeleting = false
            //move to next word
            this.wordIndex++
            //pause before start typing
            typeSpeed = 500
        }

        setTimeout(()=> this.type(),typeSpeed)

    }

}

//Init on DOM Load
document.addEventListener('DOMContentLoaded',init)

//Init App
function init(){
    const txtElement = document.querySelector(".txt-type")
    const words = JSON.parse(txtElement.getAttribute('data-words'))
    const wait = txtElement.getAttribute('data-wait')
    //Init Typewriter
    new Typewriter(txtElement,words,wait)
}

Conclusion

Congratulations! You have successfully implemented a typewriter using Vanilla JavaScript. In my next article, I will show you guys how to do this with react using a package called a typewriter-effect. Stay tuned and watch this space .

Using Tailwind CSS and SASS to Set Up Your Next.js Project

Quincy Oghenetejiri — Sun, 01 May 2022 19:52:48 +0000

Building a website with Next.js which required the installation of different dependencies can be frustrating. In this article, we will discuss how you can set up your app using Tailwind CSS and SASS.

Intro

Next.js is an open-source web development framework built on top of Node.js that enables server-side rendering and the generation of static webpages in React-based online apps.

Tailwind CSS is a highly configurable, low-level CSS framework that provides you with all of the building blocks you need to create personalized designs without having to struggle to override irritating opinionated styles.

SASS is a Cascading Style Sheets preprocessor programming language that may be interpreted or compiled. The scripting language itself is called SassScript. CSS is entertaining on its own, but stylesheets are becoming larger, more complicated, and more difficult to manage. A preprocessor can aid in this situation. Sass allows you to leverage capabilities that aren't currently available in CSS, such as variables, nesting, mixins, inheritance, and other useful features that make creating CSS more enjoyable. Nesting is one of the features I love using with sass as it gets rid of unnecessary mentioning of classes.

Installing Next.js

Installing next.js can be done with the following code below:

npx-create-next-app

The code above will install the required dependencies and a well-structured next.js app will be set up. To start the project run the code below:

npm run dev

You will see your application running on localhost:3000

Installing Sass

Next.js has built-In Sass Support. But before you can start using sass you need to install sass using the following code below:

npm install sass

Installing Tailwind css and postcss

Now let install tailwindcss into our next.js app with the following code below:

npm install -D tailwindcss postcss autoprefixer

After the installation is complete, run the following code below for the automatic creation of tailwind.config.js and postcss.config.js

npx tailwindcss init

Our tailwind.config.js file needs to look like the code below indicating the respective file types you will be using in your project.

module.exports = {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx}",
    "./components/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

Stylesheet Creation

In this step, we will be creating a style folder inside our next.js app and create a file named index.scss and then paste the following code into the file.

@tailwind base;
@tailwind components;
@tailwind utilities;

After the above step the next step is to import your index.scss file and tailwindcss/tailwindcss to your _app.js file in the pages folder of your next.js app .

import '../styles/index.scss'
import 'tailwindcss/tailwind.css'

function MyApp({ Component, pageProps }) {
  return <Component {...pageProps} />
}

export default MyApp

Congratulations your Next.js app is ready for building .

Conclusion

In this article, I have been able to cover various steps on how you will be able to build a project with Next.js, Tailwind-CSS, and SASS Looking for a fully functional source code check my Github repo

Forem: Quincy Oghenetejiri

Building a Real-Time Security Dashboard with Stream Vision Agents and YOLO11

What We're Building

Technical Prerequisites

Setup & Installation

Backend Setup (Python)

Frontend Setup (React)

System Architecture

The Optimisation Strategy

The Filtering Logic

Building the Security Agent (Backend)

Project Structure

Generating Tokens for Development

Choosing a User ID

The Foundation (Imports & Config)

The Custom Processor

The Agent Assembly

Complete Backend Code (main.py)

The Frontend: Building the Dashboard UI

Environment Setup

Project Structure

Imports & Setup

Configuration

Client & Call Initialisation

The Event Listener (Alert Detection)

Data Flow

Header Section

Video Feed Panel

Event Log Panel

The Complete Source Code(App.tsx)

Running the Application

Start the Python Agent (Backend)

Start the React App (Frontend)

Testing the Security System

Complete Source Code

Conclusion

Build an Accessible Chat and Video App in React

What We're Building

Technologies Used

Technical Prerequisite

Project Architecture

Setup: Project Scaffolding and Backend Settings

Backend Setup (Token Generation)

Frontend Setup

Accessible Core: Utility Hooks

Understanding Semantic HTML and ARIA

Screen Reader Announcement Hook

useFocusManager.ts Hook

useStreamConnection.ts Hook

useAccessibilitySettings.ts Hook

useCallManagement.ts Hook

Building the Application Shell

Authentication Flow

1. AuthView - The Container

2. Signup and Login Form

Forms use strong accessibility patterns for clarity, validation, and immediate error feedback.

Navigation Hub

Building Accessible Chat UI

User Selection - Choosing Who to Chat With

Message List: Semantic Structure with ARIA

Individual Message Item Structure

Accessible Message Input with File Attachments

Handling Emojis Accessibly

Build Inclusive Video

Meeting Management

Custom Accessible Video Controls

Live Captions with Stream's Transcription API

Caption Implementation in VideoContainer

Error State Handling

Testing Accessibility

Keyboard-Only Testing

Lighthouse Checks

Complete Production Code

Conclusion: Accessibility and Performance Trade-Offs

How to Build a Secure React Native Chat App with End-to-End Encryption

Technical Prerequisites

Solution Architecture

Setting Up Your Development Environment

Backend Setup

Frontend Setup

Complete Backend Code (`main.py`)

The Complete Source Code(`App.tsx`)

`useFocusManager.ts` Hook

`useStreamConnection.ts` Hook

`useAccessibilitySettings.ts` Hook

`useCallManagement.ts` Hook