Forem: Emmanuel Aiyenigba

Build an Attendance Tracker for Virtual Classrooms with Stream

Emmanuel Aiyenigba — Thu, 18 Dec 2025 11:39:10 +0000

Introduction

Virtual meetings are now a core part of education, business, and everyday collaboration. As more interactions move online, organizations and individuals need reliable ways to track what goes on in meetings, whether for analytics, attendance, or reporting.

Take a virtual classroom, for example. Teachers need to know how consistently students attend and engage in lessons. While several existing extensions on platforms like Google Meet and Zoom automate this process, building your own attendance tracker gives you control over your meeting data. It allows you to customize a workflow that suits your needs. This is especially powerful when integrated directly into your own video platform.

In this tutorial, we will build an automated attendance tracker for your own video classroom using the Stream Video SDK.

Here’s What We’re Building

Our automated attendance tracker will:

Track when participants join or leave the class.
Calculate the attendance duration for each user at the end of the class session.
Generate a detailed AI summary of the class. The summary will include: overall attendance statistics, engagement level based on duration, notable patterns (e.g, early leavers and participants who joined multiple times), and recommendations for improving attendance.
Send an email containing the PDF of the class attendance report to the teacher at the end of the class.

At the end of the tutorial, here is what our attendance report will look like:

You can reference the GitHub repository for the full code.

Prerequisites

To follow along with this tutorial, you’ll need:

A free Stream account
Some knowledge of React and Node.js
Supabase for storing attendance data
OpenAI for generating a detailed AI summary of the class

We will use Resend for email and PDFkit for PDF generation.

Setting Up Our Application

Now, let’s create our application, set up the Stream environment, and build our classroom.

Creating Our Express Backend

First,let us bootstrap our Express backend

npx express-generator class-backend

Then, you’ll install the Stream Node SDK:

npm install @stream-io/node-sdk

Let’s also install all the other dependencies (Supabase, OpenAI, PDFkit, and Resend) that we’ll need to make our app fully functional:

npm install @supabase/supabase-js openai pdfkit resend

We now have all the necessary dependencies on the backend. Now, let’s initialize Stream.

utils.js

const { StreamClient } = require("@stream-io/node-sdk");

const apiKey = process.env.STREAM_KEY;
const secret = process.env.STREAM_TOKEN;

client = new StreamClient(apiKey, secret);

Next, we will create a file with a /createclass route. (You can choose any name). We want to make a one-time request to this route to create our classroom.

routes/auth.js

const express = require('express');
const auth = express.Router();
const { client } = require('../utils')


const callType = 'default'; 
const callId = 'class_101';

auth.get("/createclass", async (req, res) => {

    const call = client.video.call(callType, callId)
    const data = {
        created_by_id: "1111",
        custom: {
            classroom_name: "psy_101",
            subject: "Psychology"
        }

    }

    await call.create({data}) 

})

Creating and Authenticating Users

Before a person can join the class, they must first be registered as a user, authenticated, and added to the class.
routes/auth.js

auth.post("/auth", async (req, res) => {

    try{

        const {fullName, role, userId } = await req.body;

        const newUser = {
            id: userId,
            role: role === "Student" ? "user" : "admin",
            custom: {
                full_name: fullName,
                userId: `${role} ${userId}`
            }  
        };
        await client.upsertUsers([newUser]);

        const token = await client.generateUserToken({ user_id: userId});


        const call = client.video.call(callType, callId);
        await call.updateCallMembers({
            update_members: [{ 
                user_id: userId, 
                role: role === "Student" ? "user" : "admin"
            }]
        });


        res.json({token: token, userId: userId, callId: callId, callType: callType })
    }catch(err){
        console.error("Error occurred:", err)
    }


});

module.exports = auth;

This route receives the fullName, role, and userId of a user, generates an authentication token, and sends the token, userId, callId, and callType back to the frontend, where they use them to join the class.

Creating a Lobby

To join a classroom, we want the user to enter their full name, role (student or teacher), and ID number in the lobby.
Lobby.jsx


import { useState } from 'react';
import './App.css';
import { useUser } from './UserContext';

const Lobby = ({ onJoin }) => {
  const [fullName, setFullName] = useState('');
  const [role, setRole] = useState('Student');
  const [userId, setUserId] = useState('');
  const [isLoading, setIsLoading] = useState(false);
  const { setUser } = useUser();

  const handleSubmit = async (e) => {
    e.preventDefault();

    if (!fullName.trim()) {
      alert('Please enter your full name');
      return;
    }

    if (!userId.trim()) {
      alert(`Please enter your ${role === 'Student' ? 'student' : 'teacher'} ID`);
      return;
    }

    if (!/^\d+$/.test(userId)) {
      alert('ID must contain numbers only');
      return;
    }

    setIsLoading(true);

    try {
      const payload = {
        fullName,
        role,
        userId
      };

      const response = await fetch('https://9b621ead7.ngrok-free.app/auth', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify(payload),
      });

      if (!response.ok) {
        throw new Error(`Server error: ${response.status}`);
      }

      const data = await response.json();
      console.log("data from backend", data)

      // Store the token, id, and type in the context state
      setUser({
        token: data.token,
        id: data.userId,
        callType: data.callType,
        callId: data.callId
      });

      // Navigate to Call component
      onJoin();
    } catch (error) {
      console.error('Authentication error:', error);
      alert(`Failed to authenticate: ${error.message}`);
    } finally {
      setIsLoading(false);
    }
  };

  return (
    <div className="lobby">
      <h1>Join Classroom</h1>
      <form onSubmit={handleSubmit}>
        <div className="form-group">
          <label htmlFor="fullName">Full Name</label>
          <input
            type="text"
            id="fullName"
            value={fullName}
            onChange={(e) => setFullName(e.target.value)}
            placeholder="Enter your full name"
            required
          />
        </div>

        <div className="form-group">
          <label htmlFor="role">Role</label>
          <select
            id="role"
            value={role}
            onChange={(e) => setRole(e.target.value)}
            required
          >
            <option value="Student">Student</option>
            <option value="Teacher">Teacher</option>
          </select>
        </div>

        <div className="form-group">
          <label htmlFor="userId">
            {role === 'Student' ? 'Student ID' : 'Teacher ID'}
          </label>
          <input
            type="text"
            id="userId"
            value={userId}
            onChange={(e) => setUserId(e.target.value)}
            placeholder={`Enter your ${role === 'Student' ? 'student' : 'teacher'} ID (numbers only)`}
            pattern="\d+"
            required
          />
        </div>

        <button type="submit" disabled={isLoading}>
          {isLoading ? 'Joining...' : 'Join Call'}
        </button>
      </form>
    </div>
  );
};

export default Lobby;

handleSubmit sends the user’s full name, role, and ID to the backend, where they are created and authenticated. token, userId, callId, and callType received from the server are stored inside our UserContext state. Upon successful authentication, onJoin() sends the user to the classroom.

Let’s set up UserContext.

UserContext.jsx

import { createContext, useContext, useState } from 'react';

const UserContext = createContext();

export const useUser = () => {
  const context = useContext(UserContext);
  if (!context) {
    throw new Error('useUser must be used within a UserProvider');
  }
  return context;
};

export const UserProvider = ({ children }) => {
  const [user, setUser] = useState(null);

  return (
    <UserContext.Provider value={{ user, setUser }}>
      {children}
    </UserContext.Provider>
  );
};

Here’s what our classroom lobby looks like:

Joining the Class

Once the user is authenticated, we want to take them into the classroom. Let’s set up the classroom.

Call.jsx

import { useEffect, useState, useRef } from 'react';
import {
  CallControls,
  SpeakerLayout,
  StreamCall,
  StreamTheme,
  StreamVideo,
  StreamVideoClient
} from "@stream-io/video-react-sdk";
import "@stream-io/video-react-sdk/dist/css/styles.css";
import { useUser } from './UserContext';

const apiKey = import.meta.env.VITE_STREAM_KEY;

const Call = () => {
  const { user } = useUser();
  const [client, setClient] = useState(null);
  const [call, setCall] = useState(null);
  const [isLoading, setIsLoading] = useState(true);
  const [error, setError] = useState(null);
  const clientRef = useRef(null);
  const callRef = useRef(null);

  useEffect(() => {
    if (!user) {
      setError('No user data available');
      setIsLoading(false);
      return;
    }
    if(!user?.id || !user?.token){
        setError("User Id or User token not found");
        setIsLoading(false)
        return;
    }

    if (!user.callType || !user.callId) {
      setError('Missing call information');
      setIsLoading(false);
      return;
    }

    if (!apiKey) {
      setError('Missing Stream API key');
      setIsLoading(false);
      return;
    }

    let isMounted = true;

    const initializeCall = async () => {
      try {
        const videoClient = new StreamVideoClient({
          apiKey,
        });

        await videoClient.connectUser({id: user.id}, user.token);

        const videoCall = videoClient.call(user.callType, user.callId);

        await videoCall.join({create: false});

        if (isMounted) {
          clientRef.current = videoClient;
          callRef.current = videoCall;
          setClient(videoClient);
          setCall(videoCall);
          setIsLoading(false);
        } else {
          // If component unmounted during initialization, clean up
          await videoCall.leave().catch((err) => console.error("Failed to leave:", err));
          await videoClient.disconnectUser().catch((err) => console.error("Failed to disconnect:", err));
        }
      } catch (err) {
        console.error('Error initializing call:', err);
        if (isMounted) {
          setError(err.message || 'Failed to initialize call');
          setIsLoading(false);
        }
      }
    };

    initializeCall();

    return () => {
      isMounted = false;

      if (callRef.current) {
        callRef.current.leave().catch((err) => console.error("Failed to leave the call:", err));
      }
      if (clientRef.current) {
        clientRef.current.disconnectUser().catch((err) => console.error("Failed to disconnect:", err));
      }

      clientRef.current = null;
      callRef.current = null;
      setClient(undefined);
      setCall(undefined);
    };
  }, [user?.id, user?.token, apiKey]);

  if (error) {
    return (
      <div className="error-container">
        <h2>Error</h2>
        <p>{error}</p>
      </div>
    );
  }

  if (isLoading || !client || !call) {
    return (
      <div className="loading-container">
        <p>Loading call...</p>
      </div>
    );
  }

  return (
    <StreamVideo client={client}>
    <StreamTheme>
      <StreamCall call={call}>
          <SpeakerLayout />
          <CallControls />
      </StreamCall>
    </StreamTheme>
    </StreamVideo>
  );
};

export default Call;

Once the call component mounts, initializeCall() connects the user to the classroom via the Stream Video SDK.

Inside App.jsx, let’s toggle between the Lobby.jsx and Call.jsx components:

import { useState } from 'react'
import './App.css'
import Call from './Call'
import Lobby from './Lobby'
import { UserProvider } from './UserContext'

function App() {
  const [showCall, setShowCall] = useState(false)

  const handleJoinCall = () => setShowCall(true);

  return (
    <UserProvider>
      {showCall ? (
        <Call />
      ) : (
        <Lobby onJoin={handleJoinCall} />
      )}
    </UserProvider>
  )
}

export default App

Capturing Join and Leave Events

To track class attendance, we need to listen for join and leave events from Stream’s webhook. Additionally, we also want the call_session_ended hook.

Go over to your Stream dashboard and configure your webhook to listen to these events:
call.session_ended, call.session_participant_joined, and call.session_participant_left.

Next, let’s set up our webhooks endpoint.

routes/webhook.js

const express = require('express');
const crypto = require('crypto');
const { supabase } = require(‘../utils’)
const router = express.Router();

// In-memory deduplication store. You should use redis in production
const processedWebhooks = new Set();

// Cleanup old webhook IDs every 24 hours
setInterval(() => {
  if (processedWebhooks.size > 10000) processedWebhooks.clear();
}, 24 * 60 * 60 * 1000);


// Verify webhook signature
const verifySignature = (req, res, next) => {
  const { 'x-signature': signature, 'x-api-key': apiKey } = req.headers;

  if (!signature || !apiKey) {
    console.error(' Missing authentication headers');
    return res.status(401).json({ error: 'Missing authentication headers' });
  }

  if (apiKey !== process.env.STREAM_KEY) {
    console.error(' Invalid API key');
    return res.status(401).json({ error: 'Invalid API key' });
  }

  try {
    const bodyToVerify = req.rawBody || JSON.stringify(req.body);

    const expectedSignature = crypto
      .createHmac('sha256', process.env.STREAM_TOKEN)
      .update(bodyToVerify)
      .digest('hex');

    if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expectedSignature))) {
      console.error('Signature mismatch');
      return res.status(401).json({ error: 'Invalid signature' });
    }

    next();
  } catch (error) {
    console.error(' Signature verification error:', error);
    return res.status(401).json({ error: 'Signature verification failed' });
  }
};

router.post('/webhook',
  express.json({
    verify: (req, res, buf) => {
      req.rawBody = buf.toString('utf8');
    }
  }),
  verifySignature,
  (req, res) => {
  const webhookId = req.headers['x-webhook-id'];
  const webhookAttempt = req.headers['x-webhook-attempt'];
  const { type, ...payload } = req.body;

  // Fast response
  res.status(200).json({ received: true });

  // Deduplication check
  if (processedWebhooks.has(webhookId)) {
    console.log(`Duplicate webhook: ${webhookId}`);
    return;
  }

  processedWebhooks.add(webhookId);

  if (!type) {
    console.error('Missing type field:', req.body);
    return;
  }

  // Process asynchronously
  setImmediate(async () => {
    try {
      await handleEvent(type, payload);
    } catch (error) {
      console.error(`Error processing webhook ${webhookId}:`, error);
    }
  });
});

module.exports = router;

When we receive a webhook event, the verifySignature middleware will verify that the webhook is coming from Stream. Status code 200 is then sent back immediately before processing the event.

The handleEvent function processes the webhook based on the received event.


// Event handlers
const handleEvent = async (type, payload) => {

  switch (type) {
    case 'call.session_ended':
      break;
    case 'call.session_participant_joined':
      const { data: joinData, error: joinError } = await supabase
        .from('participants')
        .insert([{
          call_id: payload.call_cid,
          session_id: payload.session_id,
          user_session_id: payload.participant.user_session_id,
          user_id: payload.participant.user.userId,
          full_name: payload.participant.user.full_name,
          joined_at: payload.participant.joined_at,
          left_at: null,
          duration_seconds: null
        }])
        .select();

      if (joinError) {
        console.error('Error storing join data:', joinError);
      } 
      break;

    case 'call.session_participant_left':

      const { data: sessionData, error: fetchError } = await supabase
        .from('participants')
        .select('*')
        .eq('user_session_id', payload.participant.user_session_id)
        .select();

      if (fetchError) {
        console.error('Error fetching session:', fetchError);
        break;
      }

      if (sessionData && sessionData.length > 0) {
        // Process the most recent session
        const session = sessionData[0];
        const leftAt = new Date(payload.participant.joined_at);
        const joinedAt = new Date(session.joined_at);
        const durationSeconds = payload.participant.duration_seconds

        const { data: updateData, error: updateError } = await supabase
          .from('participants')
          .update({
            left_at: payload.participant.joined_at,
            duration_seconds: durationSeconds
          })
          .eq('user_session_id', payload.participant.user_session_id)
          .select();

        if (updateError) {
          console.error('Error updating leave data:', updateError);
        } 
      }
      break;

    default:
      console.log(`Unhandled event: ${type}`, payload);
  }
};

When a user joins the classroom, Stream sends a call.session_participant_joined event. We extract participant data from this event and save it to our Supabase database.

Similarly, when a user leaves the classroom, we receive a call.session_participant_left event, which updates the record of the user inside our database.

Don’t forget to initialize Supabase inside utils.js:

const supabaseUrl = process.env.SUPABASE_URL
const supabaseKey = process.env.SUPABASE_KEY
const supabase = createClient(supabaseUrl, supabaseKey)

Generating Attendance Report

We want to generate an attendance report when the class ends.

utils.js

const OpenAI = require(‘openai’);

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});


const generateAttendanceSummary = async (callId) => {
  try {
    // Fetch all attendance records for this call
    const { data: attendanceRecords, error: fetchError } = await supabase
      .from('participants')
      .select('*')
      .eq('call_id', callId);

    if (fetchError) {
      console.error('Error fetching attendance data:', fetchError);
      return null;
    }

    if (!attendanceRecords || attendanceRecords.length === 0) {
      console.log('No attendance records found for call:', callId);
      return null;
    }

    // Prepare attendance data for AI
    const attendanceData = attendanceRecords.map(record => {
      const duration = record.duration_seconds || 0;
      const minutes = Math.floor(duration / 60);
      const seconds = duration % 60;

      return {
        name: record.full_name,
        user_id: record.user_id,
        joined_at: new Date(record.joined_at).toLocaleString(),
        left_at: record.left_at ? new Date(record.left_at).toLocaleString() : 'Still in call',
        duration: `${minutes}m ${seconds}s`,
        duration_seconds: duration
      };
    });

    // Calculate total participants and average duration
    const totalParticipants = new Set(attendanceRecords.map(r => r.user_id)).size;
    const totalSessions = attendanceRecords.length;
    const avgDuration = attendanceRecords.reduce((sum, r) => sum + (r.duration_seconds || 0), 0) / attendanceRecords.length;

    // Create AI prompt
    const prompt = `You are an educational assistant analyzing class attendance data. Generate a detailed, professional attendance summary report.

Call ID: ${callId}
Total Unique Participants: ${totalParticipants}
Total Sessions: ${totalSessions}
Average Duration: ${Math.floor(avgDuration / 60)} minutes

Attendance Details:
${JSON.stringify(attendanceData, null, 2)}

Generate a comprehensive summary that includes:
1. Overall attendance statistics
2. Engagement level analysis (based on duration)
3. Notable patterns (e.g., participants who joined multiple times, early leavers, full attendees)
4. Recommendations for improving attendance/engagement

Keep the tone professional and constructive.`;

    // Call OpenAI API
    const completion = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "You are an expert educational analyst providing detailed attendance reports." },
        { role: "user", content: prompt }
      ],
      temperature: 0.7,
      max_tokens: 1000
    });

    const summary = completion.choices[0].message.content;

    // Store summary in database
    const { data: summaryData, error: summaryError } = await supabase
      .from('call_summaries')
      .insert([{
        call_id: callId,
        summary: summary,
        total_participants: totalParticipants,
        total_sessions: totalSessions,
        average_duration_seconds: Math.floor(avgDuration),
        created_at: new Date().toISOString()
      }])
      .select();

    if (summaryError) {
      console.error('Error storing summary:', summaryError);
    } else {
      console.log('Summary stored successfully:', summaryData);
    }

    return {
      summary,
      attendanceData,
      stats: {
        totalParticipants,
        totalSessions,
        avgDuration
      }
    };
  } catch (error) {
    console.error('Error generating attendance summary:', error);
    return null;
  }
};

First, we fetch all the attendance records for the class, then we calculate the total number of participants and the average duration. After that, OpenAI generates a detailed summary of the attendance for us.

Creating a PDF From the Attendance Summary

Now that we have the attendance summary, let’s create a PDF version of it.

utils.js

const PDFDocument = require('pdfkit');

const generatePDF = async (callId, summary, attendanceData, stats) => {
  return new Promise((resolve, reject) => {
    try {
      // Create PDF directory if it doesn't exist
      const pdfDir = path.join(__dirname, '../pdfs');
      if (!fs.existsSync(pdfDir)) {
        fs.mkdirSync(pdfDir, { recursive: true });
      }

      const fileName = `attendance-${callId}-${Date.now()}.pdf`;
      const filePath = path.join(pdfDir, fileName);

      const doc = new PDFDocument({ margin: 50 });
      const stream = fs.createWriteStream(filePath);

      doc.pipe(stream);

      // Header
      doc.fontSize(24).font('Helvetica-Bold').text('Attendance Summary Report', { align: 'center' });
      doc.moveDown();

      // Class details
      doc.fontSize(12).font('Helvetica');
      doc.text(`Call ID: ${callId}`);
      doc.text(`Generated: ${new Date().toLocaleString()}`);
      doc.moveDown();

      // Attendance statistics
      doc.fontSize(16).font('Helvetica-Bold').text('Statistics');
      doc.fontSize(11).font('Helvetica');
      doc.text(`Total Unique Participants: ${stats.totalParticipants}`);
      doc.text(`Total Sessions: ${stats.totalSessions}`);
      doc.text(`Average Duration: ${Math.floor(stats.avgDuration / 60)} minutes`);
      doc.moveDown();

      // AI Summary
      doc.fontSize(16).font('Helvetica-Bold').text('AI Analysis');
      doc.fontSize(10).font('Helvetica');
      doc.text(summary, { align: 'justify' });
      doc.moveDown();

      // Attendance Details Table
      doc.fontSize(16).font('Helvetica-Bold').text('Detailed Attendance');
      doc.moveDown(0.5);

      doc.fontSize(9).font('Helvetica');
      attendanceData.forEach((record, index) => {
        if (doc.y > 700) {
          doc.addPage();
        }

        doc.fontSize(10).font('Helvetica-Bold').text(`${index + 1}. ${record.name}`);
        doc.fontSize(9).font('Helvetica');
        doc.text(`   User ID: ${record.user_id}`);
        doc.text(`   Joined: ${record.joined_at}`);
        doc.text(`   Left: ${record.left_at}`);
        doc.text(`   Duration: ${record.duration}`);
        doc.moveDown(0.5);
      });

      // Footer
      doc.fontSize(8).text('Generated by AI-Powered Attendance System using Stream', {
        align: 'center'
      });

      doc.end();

      stream.on('finish', () => {
        resolve(filePath);
      });

      stream.on('error', (error) => {
        reject(error);
      });
    } catch (error) {
      reject(error);
    }
  });
};

Sending Email With PDF Attached

We also want to send an email using Resend with the attendance report PDF attached to the teacher when the class ends.
utils.js

const { Resend } = require('resend');

const sendEmailWithPDF = async (callId, summary, pdfPath) => {
  try {
    const pdfBuffer = fs.readFileSync(pdfPath);
    const pdfBase64 = pdfBuffer.toString('base64');
    const fileName = path.basename(pdfPath);

    const { data, error } = await resend.emails.send({
      from: 'Attendance System <onboarding@resend.dev>',
      to: 'loyaltysamuel001@gmail.com', // this should be the email of the class teacher from your database
      subject: `Attendance Summary - Call ${callId}`,
      html: `
        <h2>Attendance Summary Report</h2>
        <p>The class session has ended. Please find the detailed attendance summary attached.</p>
        <h3>Quick Overview:</h3>
        <div style="background-color: #f5f5f5; padding: 15px; border-radius: 5px;">
          ${summary.split('\n').slice(0, 5).join('<br/>')}
        </div>

        <p><strong>Full report is attached as PDF.</strong></p>

        <hr/>
        <p style="color: #666; font-size: 12px;">Generated by AI-Powered Attendance System</p>
      `,
      attachments: [
        {
          filename: fileName,
          content: pdfBase64
        }
      ]
    });

    if (error) {
      console.error('Error sending email:', error);
      return false;
    }

    console.log('Email sent successfully:', data);

    // Clean up PDF file after sending (optional)
    fs.unlinkSync(pdfPath);
    console.log('PDF file cleaned up:', pdfPath);

    return true;
  } catch (error) {
    console.error('Error in email sending process:', error);
    return false;
  }
};

module.exports = { generatePDF, sendEmailWithPDF, generateAttendanceSummary }

Finally, let’s call generateAttendanceSummary, generatePDF, and sendEmailWithPDF functions whenever we receive the call.session_ended webhook event.
routes/webhook.js

const { generatePDF, sendEmailWithPDF, generateAttendanceSummary } = require("../utils");


const handleEvent = async (type, payload) => {

  switch (type) {
    case 'call.session_ended':

      // Generate AI attendance summary
      const callId = payload.call_cid;

      const result = await generateAttendanceSummary(callId);

      if (result) { 

        try {
          // Generate PDF
          const pdfPath = await generatePDF(
            callId,
            result.summary,
            result.attendanceData,
            result.stats
          );

          // Send email with PDF attachment
          const emailSent = await sendEmailWithPDF(callId, result.summary, pdfPath);

          if (emailSent) {
            console.log('Attendance summary email sent successfully!');
          } else {
            console.log('Failed to send attendance summary email');
          }
        } catch (error) {
          console.error('Error in PDF/Email process:', error);
        }
      } else {
        console.log('Failed to generate attendance summary');
      }
      break;
}

//...

Run the code and see the result. We now have a fully functional attendance tracker for our video classroom. 🎉

Summary

In this tutorial, we built a video classroom with automated attendance tracking using the Stream Video SDK, OpenAI, and Supabase.

We started by creating a functional classroom with the Node.js Video SDK, and then we utilized Stream’s webhook to track participants' real-time join and leave events. After that, we used OpenAI to generate a detailed attendance report. Finally, the report was converted into a PDF and sent to the teacher at the end of each session.

Want to keep building? Check out these tutorials next:

I built a habit tracker app with Xano

Emmanuel Aiyenigba — Sun, 14 Dec 2025 00:30:08 +0000

This is a submission for the Xano AI-Powered Backend Challenge: Full-Stack, AI-First Application

What I Built

Habit tracker, the app I built, helps people log their daily routines and track completion. It also allows you to invite an accountability partner to keep you accountable with fulfilling your habits.

Demo

Live app (via Xano static hosting): https://default-dev-8050f4-xn9g-tt4h-4ei5.n7e.xano.io/

Full code on GitHub: https://github.com/EmmanuelTheCoder/Lite-habit-tracker-xano

Test Login

Email: tester@gmail.com
password:12345678

The AI Prompt I Used

Original prompt for my backend

Create a habit tracking backend with accountability features:

Database tables:
- Users (authentication enabled)
- Habits (linked to users): name, description, category, frequency, target_days array, is_active, is_public
- Habit_Logs: habit_id, user_id, log_date (unique together), completed boolean, notes, mood
- Accountability_Partners: user_id, partner_id, status (pending/accepted)
- Encouragements: messages between partners
- Achievements: track user milestones
Core API endpoints:
- Auth: signup, login
- Habits: CRUD operations, get user's habits
- Logs: POST /habits/{id}/log (mark complete/incomplete), GET /habits/{id}/logs (history)
- Stats: GET /habits/{id}/stats (current streak, completion rate, total completions)
- Dashboard: GET /dashboard (today's habits, overall stats)
- Partners: send/accept requests, view partner progress
- Encouragements: send/receive messages
Business logic:
- Calculate current streak (consecutive days completed)
- Calculate completion rate (percentage over time period)
- Check if habit is due today based on target_days
- Prevent duplicate logs for same date
- Auto-award achievements (7-day streak, 30-day streak, etc.)
Security:
- All endpoints require authentication
- Users can only access their own data
- Partners can only see habits marked as public

Use proper indexing and data validation.

How I Refined the AI-Generated Code

Didn't really have to do much work because my prompt was very detailed, and luckily, the Co-pilot and the Xanoscript extension knew how to execute it well.

All I had to do was refine the user table, and add security.create_auth_token to my login and signup endpoints.

My Experience with Xano

I find Xano to be a simple way to build full-stack applications. The fact that it could host my static frontend with just a click of the button on VS Code made it even more interesting for me. I also appreciate that I could set up authentication that easily.
As for challenges, I think that not being able to inspect a variable or add a breakpoint to my debugging can be time-consuming for robust applications; thankfully, my application was just something simple so I had less bugs to deal with.

Build a Calorie Estimation App With Vision Agents

Emmanuel Aiyenigba — Mon, 17 Nov 2025 19:09:33 +0000

Introduction

Knowing the calorie count in the food you consume daily can help you maintain a healthy diet. But, manual calorie counting can be difficult, especially for foods without labels. With computer vision, we can now automate calorie estimation for any food.

In this tutorial, we’ll build an agent that estimates and tracks your total calorie consumption throughout the day. We will leverage computer vision through the Vision Agents SDK for this. By the end, you’ll be able to point your camera at a portion of food, and the agent will estimate portion size and calories.

Vision Agents (explore the code on GitHub) is a framework that enables developers to build real-time voice and video AI applications using any LLM and video edge network. The Vision Agents framework combines a series of other frameworks under the hood (e.g., processor, LLM, video edge network, etc.) to help you build real-time AI applications in simple steps.

Let’s get started.

Here’s What We’re Building

In our calorie estimation app, the agent will:

Analyze the food on camera and identify it.
Estimate the portion size based on the area of the bounding box.
Get the food’s calorie information via the USDA (United States Department of Agriculture) FoodData API and provide the user with the nutritional information.
Store the calorie information in an Excel sheet. The user can later reference it to add up their total calorie consumption for the day.

At the end of the tutorial, here is what our calories estimation app will look like:

You can reference GitHub for the full code.

Prerequisites

To follow along with this tutorial, you will need:

A free Stream account. We’ll use Stream as our video edge network.
Some knowledge of Python.
uv package installer.

Setting Up Your Application

Let’s start by initializing our application using uv:

uv init

Add the packages that we need:

uv add "vision-agents[getstream, cartesia, deepgram, openai]" python-dotenv httpx ultralytics openpyxl

We’ll use Stream as our video edge network, Cartesia for text-to-speech, Deepgram for speech-to-text, OpenAI as our LLM, Ultralytics for object detection, httpx for HTTP requests, and openpyxl for Excel sheets.

Next step, in our main.py file, let’s import the necessary packages for our project:

import asyncio
import logging
import sys
import os
from uuid import uuid4
from typing import Dict, List, Any
import httpx
from dotenv import load_dotenv
from vision_agents.plugins import getstream, openai, deepgram, cartesia
from vision_agents.core import User, Agent
from datetime import datetime
from openpyxl import Workbook, load_workbook

We’ll come back to this file shortly to implement our agent. In the meantime, let’s proceed with implementing the processor.

Processing Video Frames for Agent Analysis

For our agent to detect what food we are pointing the camera at, we need to implement a processor. Processors in Vision Agents analyze and transform audio and video streams in real-time and pass the frames to the agent for further use.

In our case, we want to process video frames as PIL images, extract food detections, and then use the detections in our agent to determine the nutritional information of the food.

Let’s create a food_detection.py file for this:

food_detection.py

from PIL import Image
from vision_agents.core.processors import AudioVideoProcessor, ImageProcessorMixin
from ultralytics import YOLO
import logging
import time
from typing import List, Dict, Any, Optional

logger = logging.getLogger(__name__)


class FoodDetectionProcessor(AudioVideoProcessor, ImageProcessorMixin):
    """
    Detects food items in video frames using YOLO.
    """

    def __init__(
        self,
        model_path: str = "yolov8n.pt",
        conf_threshold: float = 0.2,
        interval: int = 1,
        detection_timeout: float = 3.0,
    ):
        """
        Initialize food detection processor.

        Args:
            model_path: Path to YOLO model (default: yolov8n.pt)
            conf_threshold: Confidence threshold for detections
            interval: Process every N seconds (default: 1)
            detection_timeout: Keep last valid detections for this many seconds (default: 3.0)
        """
        super().__init__(
            interval=interval,
            receive_audio=False,
            receive_video=True,
            send_audio=False,
            send_video=False,
        )

        logger.info(f"Initializing FoodDetectionProcessor")
        logger.info(f"  Model: {model_path}")
        logger.info(f"  Confidence threshold: {conf_threshold}")
        logger.info(f"  Processing interval: {interval}s")
        logger.info(f"  Detection timeout: {detection_timeout}s")

        self.model = YOLO(model_path)
        self.conf_threshold = conf_threshold

        # COCO food-related classes
        self.food_classes = [
            40,  # wine glass
            41,  # cup
            42,  # fork
            43,  # knife
            44,  # spoon
            45,  # bowl
            46,  # banana
            47,  # apple
            48,  # sandwich
            49,  # orange
            50,  # broccoli
            51,  # carrot
            52,  # hot dog
            53,  # pizza
            54,  # donut
            55,  # cake
            59,  # potted plant
            61,  # dining table
        ]

        self.latest_detections: List[Dict[str, Any]] = []
        self.last_detection_time: float = 0
        self.detection_timeout = detection_timeout
        self.frame_count = 0

        logger.info("Food detection model loaded successfully")

    async def process_image(
        self,
        image: Image.Image,
        user_id: str,
        metadata: Optional[dict] = None,
    ):
        """
        Process individual video frames as PIL Images.

        This method is called automatically by the agent for each video frame.
        """
        if image.size[0] == 0 or image.size[1] == 0:
            logger.warning("Skipping invalid frame (zero size)")
            return

        if not self.should_process():
            logger.debug("Skipping frame due to processing interval")
            return

        self.frame_count += 1
        logger.debug(f" Processing frame #{self.frame_count} from user {user_id} (size: {image.size})")

        try:
            # Run YOLO detection
            results = self.model(
                image,
                conf=self.conf_threshold,
                classes=self.food_classes,
                verbose=False,
            )

            detections = self._extract_detections(results)

            if detections:
                self.latest_detections = detections
                self.last_detection_time = time.time()
                detected_items = [
                    f"{d['class_name']} ({d['confidence']:.2f})" for d in detections
                ]
                logger.info(f" Detected {len(detections)} items: {', '.join(detected_items)}")
            else:
                logger.debug("No food items detected in this frame")

        except Exception as e:
            logger.error(f"Error processing frame: {e}", exc_info=True)

    def _extract_detections(self, results) -> List[Dict[str, Any]]:
        """Extract detection information from YOLO results."""
        detections = []

        if not results or len(results) == 0:
            return detections

        result = results[0]
        if result.boxes is None or len(result.boxes) == 0:
            return detections

        boxes = result.boxes
        for i in range(len(boxes)):
            box = boxes.xyxy[i].cpu().numpy()
            conf = float(boxes.conf[i].cpu().numpy())
            cls_id = int(boxes.cls[i].cpu().numpy())
            class_name = result.names.get(cls_id, f"class_{cls_id}")

            bbox = [float(x) for x in box]
            width = bbox[2] - bbox[0]
            height = bbox[3] - bbox[1]
            area = width * height

            detections.append({
                "bbox": bbox,
                "bbox_area": area,
                "confidence": conf,
                "class_id": cls_id,
                "class_name": class_name,
            })

        return detections

    def get_latest_detections(self) -> List[Dict[str, Any]]:
        """
        Get the most recent food detections, but only if they are not stale.
        """
        if time.time() - self.last_detection_time <= self.detection_timeout:
            return self.latest_detections.copy()
        return []  # Return empty if detections are stale

    def reset(self):
        """Reset detection state at the start of a new session."""
        self.latest_detections = []
        self.last_detection_time = 0
        self.frame_count = 0
        logger.info(" FoodDetectionProcessor state reset")

Our FoodDetectionProcessor implements AudioVideoProcessor and ImageProcessorMixin to receive and process video frames in real-time. The processor receives video frames from Stream and uses the process_image() method to process video frames as PIL images every second.

Our processor class runs the YOLOv8 object detection model and uses the food classes of the COCO dataset to detect food objects in the frames.

Note: The food classes of the COCO dataset are limited, so our processor can only detect a few food objects (e.g, apple, banana, sandwich, broccoli, etc). For production purposes, you should choose a dataset with a lot more food-specific annotations than COCO.

Once food is detected, the bounding box coordinates, bounding box area, confidence scores, class IDs, and class names are extracted. We will need this data later for our agent.

get_latest_detections() returns recent detections if they are not stale, and reset() clears the detection history for a new session.

Now that we have our food detection processor working, let’s go back to the main.py file and build our nutrition analysis pipeline.

Building the Nutrition Analysis Pipeline

For our calorie estimation app to function well, we need to integrate nutrition analysis into our pipeline.

Once food is detected, we want to estimate the portion size and then search for calorie information from the USDA FoodData Central API.

Let’s go ahead and do that.

main.py


# USDA FoodData Central API configuration
USDA_API_KEY = os.getenv("USDA_API_KEY")  # Get your free key at https://fdc.nal.usda.gov/api-key-signup.html
USDA_API_BASE = "https://api.nal.usda.gov/fdc/v1"



async def search_food_calories(food_name: str, portion_grams: float = 100.0) -> Dict[str, Any]:
    """
    Search for food calorie information using the USDA FoodData Central API.
    """
    try:
        async with httpx.AsyncClient() as client:
            search_url = f"{USDA_API_BASE}/foods/search"
            params = {
                "api_key": USDA_API_KEY,
                "query": food_name,
                "pageSize": 1,
                "dataType": ["Survey (FNDDS)", "Foundation", "SR Legacy"]
            }

            response = await client.get(search_url, params=params, timeout=10.0)
            response.raise_for_status()
            data = response.json()

            if not data.get("foods"):
                logger.warning(f"No nutritional data found for: {food_name}")
                return {
                    "food_name": food_name,
                    "calories": "Unknown",
                    "error": "Food not found in database"
                }

            food_item = data["foods"][0]
            nutrients = {n["nutrientName"]: n["value"] for n in food_item.get("foodNutrients", [])}

            calories_per_100g = nutrients.get("Energy", 0)
            calories_for_portion = (calories_per_100g * portion_grams) / 100

            result = {
                "food_name": food_item.get("description", food_name),
                "portion_grams": portion_grams,
                "calories": round(calories_for_portion, 1),
                "protein_g": round(nutrients.get("Protein", 0) * portion_grams / 100, 1),
                "carbs_g": round(nutrients.get("Carbohydrate, by difference", 0) * portion_grams / 100, 1),
                "fat_g": round(nutrients.get("Total lipid (fat)", 0) * portion_grams / 100, 1),
            }

            logger.info(f"Found nutritional data for {food_name}: {result}")
            return result

    except Exception as e:
        logger.error(f"Error fetching calorie data for {food_name}: {e}")
        return {
            "food_name": food_name,
            "calories": "Unknown",
            "error": str(e)
        }


def estimate_portion_size(bbox_area: float, reference_object: str = "plate") -> Dict[str, Any]:
    """
    Estimate portion size based on bounding box area.
    """
    portion_multiplier = bbox_area / 50000  # Normalized to standard plate
    estimated_weight_grams = portion_multiplier * 150
    estimated_volume_ml = portion_multiplier * 200

    size_description = "small"
    if portion_multiplier > 1.5:
        size_description = "large"
    elif portion_multiplier > 0.8:
        size_description = "medium"

    result = {
        "estimated_weight_grams": round(estimated_weight_grams, 1),
        "estimated_volume_ml": round(estimated_volume_ml, 1),
        "size_description": size_description,
        "reference": reference_object
    }

    logger.info(f"Estimated portion size: {result}")
    return result


async def analyze_food_and_calories(food_detections: List[Dict[str, Any]]) -> str:
    """
    Analyze detected foods and calculate total calories.
    """
    if not food_detections:
        return "No food items detected in the frame."

    results = []
    total_calories = 0

    for detection in food_detections:
        food_name = detection.get("class_name", "unknown food")
        bbox = detection.get("bbox", [0, 0, 0, 0])
        confidence = detection.get("confidence", 0)

        bbox_area = (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
        portion_info = estimate_portion_size(bbox_area)
        calorie_info = await search_food_calories(food_name, portion_info["estimated_weight_grams"])

        # log calorie information to excel spreadsheet
        log_food_to_excel(calorie_info, user_id="demo_user")

        if isinstance(calorie_info.get("calories"), (int, float)):
            total_calories += calorie_info["calories"]

        results.append({
            "food": food_name,
            "confidence": f"{confidence * 100:.1f}%",
            "portion": portion_info,
            "nutrition": calorie_info
        })

    response_parts = [f"I detected {len(food_detections)} food item(s):\n"]

    for i, result in enumerate(results, 1):
        nutrition = result["nutrition"]
        portion = result["portion"]

        response_parts.append(
            f"{i}. {result['food'].title()} "
            f"({portion['size_description']} portion, ~{portion['estimated_weight_grams']}g): "
        )

        if isinstance(nutrition.get("calories"), (int, float)):
            response_parts.append(
                f"{nutrition['calories']} calories, "
                f"{nutrition.get('protein_g', 0)}g protein, "
                f"{nutrition.get('carbs_g', 0)}g carbs, "
                f"{nutrition.get('fat_g', 0)}g fat\n"
            )
        else:
            response_parts.append("Nutritional data unavailable\n")

    if total_calories > 0:
        response_parts.append(f"\nTotal estimated calories: {round(total_calories)} kcal")

    return "".join(response_parts)

In the above code, we have three functions that essentially fetch nutritional data, estimate the food portion using its bounding box area, and implement the food and calorie analysis pipeline, respectively.

search_food_calories() fetches nutritional data from the USDA FoodData Central API for the detected food and returns a structured dictionary with dietary values.
estimate_portion_size() estimates the weight, volume, and portion size category of the detected food. It uses the bounding box area to estimate portion size.
analyze_food_and_calories() calls both functions and returns a readable summary describing the detected food, estimated portion, and calorie information.

Implementing Our Calorie Estimation Agent

Now that we have all the necessary functions and processor in place, it’s time to put everything together and bring our Agent to life.

Let’s set up the video application and implement our Agent.

main.py

from food_detection import FoodDetectionProcessor



async def start_agent() -> None:
    """Initialize and start the video AI agent with food detection."""
    try:
        logger.info("=" * 60)
        logger.info("Starting Food Detection Agent")
        logger.info("=" * 60)

        food_processor = FoodDetectionProcessor(
            model_path="yolov8n.pt",
            conf_threshold=0.2,
            interval=1
        )

        llm = openai.LLM(model="gpt-4o-mini")

        @llm.register_function(
            description="Analyzes food items currently visible in the video feed and provides detailed nutritional information including calories, portion size, and macronutrient"
        )
        async def analyze_food() -> str:
            """
            Analyzes detected food items from the video feed.
            Waits up to 1.5 seconds for a detection if none is immediately available.
            """
            try:
                for _ in range(15):  # 15 * 0.1s = 1.5s max wait
                    detections = food_processor.get_latest_detections()
                    if detections:
                        items = [f"{d['class_name']} ({d['confidence']:.2f})" for d in detections]
                        logger.info(f" [TOOL] Detected: {', '.join(items)}")
                        result = await analyze_food_and_calories(detections)
                        logger.info(f"Analysis result: {result}")
                        return result
                    await asyncio.sleep(0.1)

                logger.warning("[TOOL] No detections found after 1.5s wait")
                return "I'm still not seeing any food clearly. Please point your camera at some food and ensure it's well-lit!"

            except Exception as e:
                logger.error(f"Error in analyze_food tool: {e}", exc_info=True)
                return f"Sorry, I had trouble analyzing the food. Please try again."

        agent = Agent(
            edge=getstream.Edge(),
            agent_user=User(name="Food Detection Assistant", id="agent"),
            instructions=(
                "You're a friendly food portion and calorie estimator. "
                "When you detect food items in the video, use the analyze_food tool to get detailed "
                "nutritional information including portion size and calories. "
                "Present the information in a natural, conversational way. "
                "Be encouraging and helpful. Keep responses concise and easy to understand."
            ),
            llm=llm,
            stt=deepgram.STT(),
            tts=cartesia.TTS(),
            processors=[food_processor]
        )


        if agent.image_processors:
            logger.info(f"  Image processor type: {type(agent.image_processors[0]).__name__}")
        logger.info("=" * 60)

        call_id = str(uuid4())

        await agent.create_user()
        call = agent.edge.client.video.call("default", call_id)
        await agent.edge.open_demo(call)

        await asyncio.sleep(5)
        session = await agent.join(call)

        with session:
            food_processor.reset()
            frames_seen = False
              # Wait for first frame to be processed (up to 10 seconds)
            for _ in range(50):  # 50 * 0.2s = 10s max
                if food_processor.frame_count > 0:
                    logger.info("Video stream confirmed. Ready for food detection!")
                    frames_seen = True
                    break
                await asyncio.sleep(0.5)
            if not frames_seen:
                logger.warning(" No video frames after 10s. Proceeding anyway.")


            await agent.llm.simple_response(
                "Hello! I'm your food detection assistant. "
                "Point your camera at some food and ask me what I see!"
            )

            await agent.finish()

    except KeyboardInterrupt:
        logger.info("Agent stopped by user")
    except Exception as e:
        logger.error(f"Error running agent: {e}", exc_info=True)
        raise
    finally:
        logger.info("Agent shutting down...")


def main() -> None:
    """Main entry point."""
    try:
        asyncio.run(start_agent())
    except KeyboardInterrupt:
        logger.info("Application interrupted by user")
        sys.exit(0)
    except Exception as e:
        logger.error(f"Application failed: {e}")
        sys.exit(1)


if __name__ == "__main__":
    main()

We initialized a FoodDetectionProcessor instance to detect food items in video frames using YOLOv8. The analyze_food function is registered with the LLM. The function waits for about 1.5 seconds for the processor to detect food items in the video feed. It then calls analyze_food_and_calories() to compute nutritional information.

Our agent is configured with LLM, STT, TTS, and our FoodDetectionProcessor. Once the processor identifies a food item in the video, it passes the information to the Agent, which then calls analyze_food to get the nutritional information for the food. We’re using Stream as our video edge client because its low-latency streaming delivers high-quality video performance, perfect for real-time food detection.

This is what our .env file looks like:

# Stream credentials - Get yours at: https://getstream.io/dashboard/
STREAM_API_KEY=your-stream-key
STREAM_API_SECRET=your-stream-secret

OPENAI_API_KEY=your-openai-key


USDA_API_KEY=your-USDA-key
DEEPGRAM_API_KEY=your-deepgram-key
CARTESIA_API_KEY=your-cartesia-key

At this point, our Calorie Estimation App is nearly ready to go! 🎉 You can test it by running the command below:

uv run main.py

Next, let’s add one final touch. We will log the calorie information in an Excel file for easy reference by the user.

Logging Calorie Information in an Excel File

For record keeping and easy referencing, let’s log the food calorie information in an Excel file.

def log_food_to_excel(food_result: Dict[str, Any], user_id: str = "user"):
    """
    Append the calorie analysis result to an Excel (.xlsx) log file.

    """
    file_exists = os.path.isfile(FOOD_LOG_FILE)

    if file_exists:
        wb = load_workbook(FOOD_LOG_FILE)
        ws = wb.active
    else:
        wb = Workbook()
        ws = wb.active
        ws.append([
            "Timestamp",
            "User ID",
            "Food Name",
            "Portion (g)",
            "Calories",
            "Protein (g)",
            "Carbs (g)",
            "Fat (g)",
            "Error"
        ])
        # Optional: auto-fit columns (basic)
        for col in ws.columns:
            max_length = 0
            column = col[0].column_letter
            for cell in col:
                try:
                    if len(str(cell.value)) > max_length:
                        max_length = len(str(cell.value))
                except:
                    pass
            adjusted_width = min(max_length + 2, 50)
            ws.column_dimensions[column].width = adjusted_width

    # Append data row
    ws.append([
        datetime.now().isoformat(),
        user_id,
        food_result.get("food_name", "Unknown"),
        food_result.get("portion_grams", "N/A"),
        food_result.get("calories", "Unknown"),
        food_result.get("protein_g", "N/A"),
        food_result.get("carbs_g", "N/A"),
        food_result.get("fat_g", "N/A"),
        food_result.get("error", "")
    ])

    # Save file
    wb.save(FOOD_LOG_FILE)

We are using openpxyl to create or update an Excel sheet with the calorie information of the food.

Let’s call the log_food_to_excel function inside analyze_food_and_calories().

 async def analyze_food_and_calories(food_detections: List[Dict[str, Any]]) -> str:
        calorie_info = await search_food_calories(food_name, portion_info["estimated_weight_grams"])

        # log calorie information to excel spreadsheet
        log_food_to_excel(calorie_info, user_id="demo_user")

For our demonstration purposes, openpyxl works perfectly fine. However, in a production environment, it is better to use a real database, such as Supabase, to store the calorie information and export to an Excel sheet when needed. This is because openpyxl does not support true concurrent writes, which can cause issues when multiple users attempt to log data simultaneously.

Here’s how you could implement this using Supabase:

In your Supabase SQL Editor, run the query below:

CREATE TABLE food_log (
  id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  created_at TIMESTAMPTZ DEFAULT NOW(),
  user_id TEXT NOT NULL,
  food_name TEXT,
  portion_grams REAL,
  calories REAL,
  protein_g REAL,
  carbs_g REAL,
  fat_g REAL,
  error TEXT
);

This creates a food_log table with the necessary columns.

Next, install Supabase:

uv add supabase

Then, import it into your project:

from supabase import create_client, Client

Add your Supabase credentials to the .env file:

SUPABASE_URL=your-project-url.supabase.co
SUPABASE_ANON_KEY=your-anon-key

Add a function to save calorie information in Supabase:


def log_food_to_supabase(food_result: Dict[str, Any], user_id: str = "demo_user"):
    """
    Log food analysis to Supabase database.
    """
    try:
        supabase: Client = create_client(
            os.getenv("SUPABASE_URL"),
            os.getenv("SUPABASE_ANON_KEY")
        )

        data = {
            "user_id": user_id,
            "food_name": food_result.get("food_name", "Unknown"),
            "portion_grams": food_result.get("portion_grams"),
            "calories": food_result.get("calories") if isinstance(food_result.get("calories"), (int, float)) else None,
            "protein_g": food_result.get("protein_g"),
            "carbs_g": food_result.get("carbs_g"),
            "fat_g": food_result.get("fat_g"),
            "error": food_result.get("error", "")
        }

        response = supabase.table("food_log").insert(data).execute()
        logger.info(f"Food logged to Supabase: {response.data[0]['id']}")
    except Exception as e:
        logger.error(f" Failed to log to Supabase: {e}")

Call the function inside the analyze_food_and_calories function, right after calorie_info:

async def analyze_food_and_calories(food_detections: List[Dict[str, Any]]) -> str:

        calorie_info = await search_food_calories(food_name, portion_info["estimated_weight_grams"])

        # log calorie information to Supabase
    log_food_to_supabase(calorie_info, user_id="demo_user")  # use real user_id

You can then provide a download link on demand via Supabase Edge Function or client-side export.

Summary

In this tutorial, we built a calorie estimation and tracker app using Vision Agents. We started by implementing a processor to receive frames from the Stream video edge network, detect food objects, and then pass them to the agent.

Next, we built our nutrition analysis pipeline so that once food is detected in the camera, we can look up the calorie information from the USDA FoodData Central API. The agent retrieves the calorie information of the food by calling the function registered to it.

Finally, we added an Excel sheet to store the calorie information for reference.

🛠️ Want to keep building with Vision Agents? Explore these tutorials next:

Building a RAG-Powered AI Customer Support Chatbot with Stream and OpenAI

Emmanuel Aiyenigba — Wed, 16 Apr 2025 12:45:41 +0000

Large Language Models, although trained on large amounts of data, have limited domain-specific knowledge. This limitation makes them less effective for customer support chatbots or similar apps that require specific data.

Retrieval Augmented Generation (RAG) solves this concern by allowing LLMs access to external knowledge sources for a more accurate response generation. This means that with RAG, you can have an LLM that receives relevant information from your knowledge base as context during the generation process to respond accurately to questions and inquiries from your customers, giving you an automated customer support system.

Don’t get me wrong—RAG is not the only way to enhance an LLM's capabilities. Another way is fine-tuning. However, RAG is more suitable for solutions that require sizeable external knowledge bases or frequently changing data, such as those in customer support systems.

This tutorial will guide you through building a RAG-powered customer support chatbot using Stream, OpenAI’s GPT-4, and Supabase’s pgvector. Stream empowers developers to seamlessly build scalable in-app chat, video, and feeds. Its powerful APIs, SDKs, and AI integrations make building your app faster.

This tutorial will cover:

Creating and storing vector embeddings from our knowledge base using Supabase’s pgvector feature. pgvector provides an efficient way to store and query vector embeddings. Knowledge base refers to the external knowledge source your LLM will depend on as a helpful context.
Embedding customers’ chat and performing a similarity search on the stored knowledge base embeddings.
Setting up Stream and building our RAG-powered chatbot.

Prerequisite

To follow along, you need:

A free Stream account.
An OpenAI account.
A Supabase account.

Vector Database Set-Up

We will begin by setting up a vector database using Supabase to store our vector embeddings and perform similarity searches. The Stream React docs will serve as our knowledge base for this tutorial.

Go to your Supabase SQL Editor, and let’s enable pgvector, create a vector table, and write a function that performs a similarity search on our knowledge base.

-- Enable pgvector
create extension if not exists vector;

-- Create documents table
create table if not exists documents (
    id bigserial primary KEY,
    content TEXT,
    embedding vector(1536)
);

-- Create match document function
create or replace function match_documents (
  query_embedding vector(1536),
  match_threshold float,
  match_count int
)
returns table (
  id bigint,
  content text,
  similarity float
)
language plpgsql
as $$
begin
  return query
  select
    documents.id,
    documents.content,
    1 - (documents.embedding <=> query_embedding) as similarity
  from documents
  where 1 - (documents.embedding <=> query_embedding) > match_threshold
  order by similarity desc
  limit match_count;
end;
$$;

Run the code above to execute your query. This creates a table named documents with three (3) columns: id, content, and embedding. The content column will store the actual text content of our knowledge base, while the embedding column will store the vector embedding of the knowledge base. We also created a match_documents function that performs similarity searches on the documents table.

Next, let's create an Express server and write a function to create vector embedding from our knowledge base and store them.

Create and Store Vector Embeddings

Create an Express server and install all the dependencies we need using the following command:


mkdir rag-support-bot
cd rag-support-bot
npx express-generator rag-server
cd rag-server
npm install @langchain/community @langchain/core langchain @supabase/supabase-js cheerio dotenv openai stream-chat

I set my server to run on port 8000 (you can use any port you prefer). We will see these dependencies in action shortly.

Create an embed.js route in your server.

A POST request to this route will trigger the creation of vector embeddings from our knowledge base. This is a one-time operation, necessary only when the knowledge base is initially set up or subsequently modified.

routes/embed.js

const express = require('express')
const embed = express.Router().use(express.json(), express.urlencoded({ extended: false }));
const { RecursiveCharacterTextSplitter } = require('langchain/text_splitter');
const { CheerioWebBaseLoader } = require('@langchain/community/document_loaders/web/cheerio');

const { supabase, openai } = require("../utils");

async function storeEmbeddings() {
    const loader = new CheerioWebBaseLoader(
      'https://getstream.io/chat/docs/react/'
    );
    const docs = await loader.load();

    const textSplitter = new RecursiveCharacterTextSplitter({
      chunkSize: 1000,
      chunkOverlap: 200,
    });

    const chunks = await textSplitter.splitDocuments(docs);

    const promises = chunks.map(async (chunk) => {
      const cleanChunk = chunk.pageContent.replace(/\n/g, ' ');

      const embeddingResponse = await openai.embeddings.create({
        model: 'text-embedding-3-small',
        input: cleanChunk,
      });

      const [{ embedding }] = embeddingResponse.data;

      const { error } = await supabase.from('documents').insert({
        content: cleanChunk,
        embedding,
      });

      if (error) {
        throw error;
      }
    });

    await Promise.all(promises);
  }

embed.post("/embed", async (req, res) => {

    try {
        await storeEmbeddings();
        res.status(200).json({ message: 'Successfully Embedded' });
    }catch (error) {
        res.status(500).json({
            message: 'Error occurred',
        });
    }

});

module.exports = embed;

The storeEmbeddings function loads the content of our knowledge base using CheerioWebBaseLoader. It then splits the content into smaller chunks with RecursiveCharacterTextSplitter and generates vector embeddings of each chunk using the OpenAI embeddings API. The chunks and their corresponding embeddings are stored inside our Supabase vector database.

The utils.js file initializes Supabase and OpenAI clients:

utils.js


const OpenAI = require('openai');
const { createClient } = require('@supabase/supabase-js');

const supabase_url = process.env.SUPABASE_URL;
const supabase_key = process.env.SUPABASE_KEY;
const openaisecret = process.env.OPENAI_API_KEY;

const supabase = createClient(supabase_url, supabase_key);

const openai = new OpenAI({
  apiKey: openaisecret,
});

module.exports = {supabase, openai };

Make a POST request to the /embed endpoint to trigger the embedding process.

Once complete, this is what the embeddings look like inside Supabase:

Next, let’s set up the route to authenticate users in Stream.

Authenticating Customers With Stream.

Create an auth.js route in your Express app.

routes/auth.js


const express = require('express');
const auth = express.Router().use(express.json(), express.urlencoded({ extended: false }));
const { StreamChat } = require("stream-chat")

const chatServer = StreamChat.getInstance(
  process.env.STREAM_KEY,
  process.env.STREAM_SECRET
)

auth.post('/auth', async (req, res) => {
const { customerId } = await req.body;

await chatServer.upsertUser({  
  id: customerId,   
  role: 'admin'
});  

await chatServer.upsertUser({
  id: 'ai-support-bot', 
  name: "AI Support Bot",
  role: 'user'
});

try {
  const token = chatServer.createToken(customerId);
  res.json({ token })
} catch (error) {
  res.status(500).json({error: error})
}

});
module.exports = auth;

This route receives the customerId from the frontend, then creates or updates users in Stream, generates a token for the user/customer, and sends the token to the frontend. This token will be used later to connect the customer to the channel.

We need to create one more route to handle AI responses.

AI Responses

Let’s create a response.js route in our Express app that will receive customers’ questions and generate AI responses based on context from the knowledge base.

routes/response.js


const express = require("express");
const response = express.Router().use(express.json(), express.urlencoded( { extended: false }))
const { supabase, openai } = require("../utils");

const handleQuery = async (query) => {
    const input = query.replace(/\n/g, ' ');

    const embeddingResponse = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input,
    });

    const [{ embedding }] = embeddingResponse.data;

    const { data: documents, error } = await supabase.rpc('match_documents', {
      query_embedding: embedding,
      match_threshold: 0.5,
      match_count: 10,
    });

    if (error) throw error;

    let contextText = '';

    contextText += documents
      .map((document) => `${document.content.trim()}---\n`)
      .join('');

    const messages = [
      {
        role: 'system',
        content: `You are a customer support bot for Stream, only ever answer
      truthfully and be as helpful as you can!`,
      },
      {
        role: 'user',
        content: `Context sections: "${contextText}" Question: "${query}"`,
      },
    ];

    const completion = await openai.chat.completions.create({
      messages,
      model: 'gpt-4,
      temperature: 0.8,
    });

    return completion.choices[0].message.content;
  }

response.post("/response", async (req, res) => {
    const { message } = req.body;
    const answer = await handleQuery(message);

    res.json({answer})

})

module.exports = response;

Here, the handleQuery function receives the customer’s query and generates an embedding. The embedding is used to perform a similarity search within the vector database. The content retrieved is provided as context to the OpenAI chat completion endpoint alongside the customer’s original query.
The generated response from OpenAI is sent to the front end.

We can now build the chat user interface with our backend development complete.

Building the Chat UI

Let's go ahead and set up React (using Vite) for our front end.

cd rag-support-bot
npm create vite@latest rag-client -- --template react
npm install

Let’s also install the Stream Client:

npm install stream-chat-react stream-chat

Let’s create a Chat component.

src/Chat.jsx

import React, {useState, useEffect} from 'react';
import {Channel, ChannelHeader, MessageList, MessageInput, Window, useChannelStateContext, Chat } from 'stream-chat-react';
import { StreamChat } from 'stream-chat';

const chatClient = StreamChat.getInstance(import.meta.env.VITE_STREAM_KEY);

const CustomMessageInput = () => {
  const { channel } = useChannelStateContext();

  const submitMessage = async (message) => {
    // send user's message

    try{
      const userText = await channel.sendMessage({
        text: message.text,
      })

      // typing indicator for the AI bot
      await channel.sendEvent({
        type: 'typing.start',
        user: {id: 'ai-support-bot'},
      });

      //send customer chat to the backend.
      const sendTexttoAI = await fetch("http://localhost:8000/response", {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({message: message.text }),

      })
      const data = await sendTexttoAI.json();

      // stop AI typing indicator
      await channel.sendEvent({
        type: 'typing.stop',
        user: { id: 'ai-support-bot'},
      })

      //send response from AI

      await channel.sendMessage({
        text: data.answer,
        user: {
          id: 'ai-support-bot',
          name: 'AI Support Bot'
        }
      });

    } catch (error) {
      console.error('Error message', error);
      await channel.sendMessage({
        text: "I could not process your query. Please try again shortly.",
        user: {
          id: "ai-support-bot",
          name: "AI Support Bot"
        },
      });
    }

  }

  return (
    <div className='relative'>
      <MessageInput overrideSubmitHandler={submitMessage}  />
    </div>
  )
}

export default function ChatComponent() {
  const [channel, setChannel] = useState(null);
  const customerId = `cus-${Math.random().toString(20).substring(5)}`

  useEffect(() => {
    const initialize = async () => {
      try{

        const getToken = await fetch("http://localhost:8000/auth", {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
          }, 
          body: JSON.stringify({customerId})
        });

        const { token } = await getToken.json()

        await chatClient.connectUser(
          {
            id: customerId,
            name: "Customer"
          },
          token
        )

        //create or join support channel
        const channel = chatClient.channel('messaging', 'support', {
          name: "Customer Support",
          members: [customerId, 'ai-support-bot']
        });
        await channel.watch()
        setChannel(channel);

      } catch (error) {
        console.error("Chat initialization error", error);
      }
    }

    initialize()

  }, [])

  return (
    <div>
      <Chat client={chatClient}>
        <Channel channel={channel}>
          <Window>
            <ChannelHeader />
            <MessageList />
            <CustomMessageInput />
          </Window>
        </Channel>

      </Chat>
    </div>
  )
}

The CustomMessageInput component has a submitMessage function that sends customers’ queries to OpenAI at the backend, receives an answer, and adds it to the channel messages.

ChatComponent initializes the Stream Client and creates a “support” messaging channel where a customer and the AI Support Bot can interact after authenticating the customer.

Let’s add this component to the main page:

src/App.jsx

import './App.css';
import "stream-chat-react/dist/css/v2/index.css";
import ChatComponent from './Chat';

function App() {

  return <ChatComponent />
}

export default App

Our Chat app is ready. Let’s ask a few questions and see how the AI responds.

Conclusion

The customer support chatbot application we built in this tutorial can be extended with escalation triggers, enabling customers to request human support when necessary. You could also make the app multi-user to support collaboration for teammates.

Remember, there are many ways of building a RAG system using different technologies. While exploring them, it is good to consider the potential of function calling when your application involves predefined operations.

This tutorial demonstrated how Stream empowers you to build scalable chat applications, eliminating development complexities efficiently. Its ready-to-use components enable you to move from development to market quickly.

Recent Security Vulnerability Detected in Clerk - Should You Roll Your Own Auth?

Emmanuel Aiyenigba — Wed, 17 Jan 2024 13:01:42 +0000

January 12th, 2024, Clerk, an authentication and user management solution for web and mobile applications, notified the public about a critical security vulnerability discovered in the @clerk/nextjs SDK. The vulnerability allowed malicious actors access to act on behalf of other users.

After this information was made public on X (formerly known as Twitter), developers reacted to the news from different angles.

// Detect dark theme var iframe = document.getElementById('tweet-1745944860316856346-457'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1745944860316856346&theme=dark" }

“Roll your own auth” is the most common reaction.

The incident with Clerk is one of the reported recent security incidences in the tech space in the last few weeks. On January 7th Resend reported a security incident where attackers gained access to customer data like emails sent, contacts, domain, etc., due to a leaked environment variable of Resend’s database API key. Sometimes in December last year, MongoDB also reported an unauthorized actor gaining access to systems with customer information (names, phone numbers, email addresses, and a customer’s logs)

With the recent Clerk incident, some developers are worried about the frequent occurrence of these security breaches.

Details of the Clerk’s security vulnerability

Clerk discovered a security vulnerability in their Next.js SDK during an internal audit. The vulnerability impacted only applications that use Next.js backend with @clerk/nextjs versions 4.7.0 - 4.29.2. Specifically, applications that call auth() in App Router and getAuth() in Page Router.

Clerk’s post-mortem is not very transparent about the details of the vulnerability. They claim they are unaware of any exploit until they access their server logs. They also said, “Detailed instructions for inspecting logs for an attack will be made available to impacted customers who request them, but will not be published publicly.”

Customers are to upgrade their Next SDK to version 4.29.3 immediately. The company also announced that cloud providers like Vercel, Netlify, and Cloudflare have deployed mitigations at the network layer to keep customers safe.

Should you roll out your own Auth?

“Roll your own auth” is a popular saying among developers whenever an auth provider performs inefficiently or a security vulnerability is discovered.

Auth companies with millions of dollars in funding still face security incidences. Chances are that a small team will face an even bigger problem securing their auth pipeline if they decide to roll out one. Building and maintaining authentication systems is a tough job requiring a lot of time and effort. Small companies do not have that much time to spare as they are more focused on implementing the details of their actual product - as it should be! Rolling your own auth for companies means that development time is diverted away from building the actual product to building and maintaining an auth system.

Auth companies do regular internal audits and even pay security experts to try and exploit their systems. This is not something that small teams (and some large teams) can afford because they do not have the resources for it. Plus, doing so is a diversion from their core therefore a waste of dev time.

The following are why you should not roll out your own auth:

Development and maintenance: Building and maintaining your own auth system requires significant developer resources. Small teams do not have the resources to spare to maintain a system that is not a part of their core product offerings.
In-house expertise: Having your own auth system requires in-house expertise to secure the system. It is costly to hire security experts.
Building what already exists: Re-inventing the wheel is a redundant process. There are auth services with proven features and functionalities that you can use.
Scalability concerns: As your user base grows, you will worry about scaling your auth system instead of prioritizing serving your users.
Diversion from your core application: If auth service is not part of your product offering, building one would be a diversion from your application core and a waste of development time and resources.

If you feel less confident about a particular auth provider, you can try out others to find the one that meets your needs. Singlebase, Supabase, Okta, and Auth0 are powerful auth providers you can pick from.

More than just an auth provider, Singlebase also offers unlimited usage for Vector Datastore, Document Datastore, Search, and Storage, making it a complete solution for backend needs.

Conclusion

Maintaining an auth system requires a dedicated effort and is not something you do on this side if security is a priority for your product. For this reason, you shouldn’t roll out your own auth if you cannot afford a dedicated in-house team to serve that purpose.

Rolling out your own auth without in-house experts would more often than not end in a disaster. Building and maintaining a secure auth system is a tedious job requiring the expertise of a dedicated team. Do not roll out your own auth, unless you absolutely need to and have the resources to build, maintain, and secure it.

Singlebase is an auth provider that offers great features and industry-standard security for a small fee.

An ML Engineer Review on Deepchecks LLM Evaluation Solution

Emmanuel Aiyenigba — Tue, 16 Jan 2024 13:32:18 +0000

Introduction

Validating LLM-based applications from deployment to production helps ensure that models perform optimally and flag potential issues on time. LLM evaluation allows developers and product managers to monitor, safeguard, and analyze the performance and capabilities of AI systems.

As an ML engineer, I know for a fact that LLM-based applications are becoming increasingly complex, which makes prioritizing continuous LLM evaluation using efficient tools the best way to understand the capabilities of my models and ensure optimal performance for my application.

Among the diverse LLM evaluation tools available, I find Deepchecks LLM Evaluation System to stand out through its unique way of ensuring model accuracy and safety. It thoroughly analyses for bias, toxicity, and privacy leaks (PII). Sometimes, LLMs may output sensitive personal information from seemingly innocuous inputs. As a machine learning engineer, It is important to protect my pipeline against PII leaks. Deepchecks’ engine helps me do that by continuously monitoring my models for deviation and anomalies and safeguards my models from generating harmful content.

I couldn’t say no to Deepchecks when they asked for my expert review of their LLM Evaluation Solution a couple of months after they launched it. The tool felt timely as it tackles a crucial challenge in the LLM space. Plus, who am I to turn down flattery?

When I tried out the Deepchecks LLM Evaluation solution with real data, I noticed that it helped reduce risk in my LLM pipeline by flagging potential issues and safeguarding my model from hallucination. I am going to walk you through how the tool works and give my honest expert review about it based on my experience.

First off, let me show you some benefits of LLM evaluation.

Benefits of LLM Evaluation

LLM evaluation helps you understand the strengths and weaknesses of your models and monitor them to ensure that their performance is optimal at all times. It offers a wide range of benefits for LLM-based applications.

The following are some benefits of LLM evaluation:

Improve model accuracy: Evaluation allows you to assess how well the LLM understands prompts and how you can help it improve. The evaluation engine analyzes the model and provides calculated values for accuracy and coherence. Developers can leverage these metrics to improve the model.
Optimize performance: Evaluation identifies and highlights the weak areas of your LLM. Identifying where your model underperforms will help you know where to optimize performance.
Flag potential pitfalls: LLM evaluation systems like Deepchecks allow you to monitor your model in real-time to detect anomalies before they affect users. It analyzes for biases and harmful stereotypes throughout the lifecycle of your application.
Prevent model hallucination: Large language models may generate factually misleading and potentially harmful output. This is known as LLM hallucination. Continuous evaluation can help safeguard your LLM from hallucination.
Enhance efficiency: Evaluating your LLM can help boost efficiency by identifying the areas where it is inefficient. Evaluation can help you reduce operational costs for your LLM-based apps by identifying situations where your model performs well with less data.
Prevent harmful output: Evaluation can identify and block harmful content generated by the model.

Understanding Deepchecks LLM Evaluation

Before I proceed to evaluate real-world data with Deepchecks LLM Evaluation Solution, let me first help you understand what Deepchecks LLM Evaluation is all about.

Deepchecks LLM Evaluation offers solutions that optimize LLM pipelines, help you understand how your LLM performs, discover pitfalls, and prevent model hallucinations. It allows LLM-based applications to monitor, safeguard, and validate their models.

The system analyzes uploaded data, measures various aspects of LLM interactions (like completeness, coherence, relevance, toxicity, fluency, etc), calculates the average values for each property, provides the overall score for your data, and highlights weak segments. Deepchecks LLM Evaluation also helps you understand users' thoughts by grouping related user inputs into topics. This way, you not only know what your users are interested in, but you also identify the topics where the response of your LLM is below optimal.

Deepchecks provides an estimated annotation for each interaction. Annotation helps you understand how good or bad the response to an input is. Although you can manually annotate your data, it is efficient to allow Deepchecks Evaluation to auto-annotate. You can review/edit the estimated annotations and customize the annotation configuration file.

The following are the key highlights of Deepchecks LLM Evaluation:

Dual focus: The Evaluation system is dual focus, allowing you to analyze for safety and accuracy. It checks for toxicity, bias, and personal information leaks in model output. The system also evaluates the quality of the responses by calculating the average values of properties like Fluency, Coherence, Completeness, Grounded in Context, Avoided Answer, etc.
Flexible testing and version comparison: Deepchecks LLM Evaluation allows you to combine automated tests with manual verification for a better result. Compare multiple versions of your data to help identify weak areas and improve the overall performance of your LLM.
Continuous monitoring for deviation and anomalies: The system ensures the optimal performance of your LLM pipeline by continuously monitoring it for deviation and anomalies.
Continuous evaluation throughout the lifecycle: Deepchecks lets you evaluate your data at various phases - testing, staging, and production - so that you can catch issues early.

Evaluating Sample Data with Deepchecks LLM Evaluation System

In this section, I will dive into how the Deepchecks LLM evaluation solution works. I want you to follow along. Together, we will evaluate real-world data and explore all the features in the tool to understand how it validates LLM-based pipelines.

Account and setup

Create an account
Create new application. Add application name, version, and type.

Set Data Source to Golden Set.

Golden Set data is essentially a benchmark dataset used to evaluate and compare the performance of different versions of your model. It will help me gauge how well my model understands user inputs. Golden Set data should represent the data distribution your application will encounter in production to have a dependable yardstick to compare experiments.

Production data is the data encountered in production. Uploading production data will allow Deepchecks’ engine to automatically evaluate the user input and output of your model in production to help you understand the efficiency and accuracy of your pipeline.

Uploading data

I am using my ChatGPT data as my Golden Set.

Follow the steps below to download your ChatGPT data:

On the ChatGPT page, click on your name and select Settings.

Navigate to Data controls and click Export to export your data.

After downloading and unzipping your ChatGPT data, convert conversations.json, one of the ChatGPT files downloaded, to a CSV file. Also, format your CSV data to at least reflect the mandatory structure of Deepchecks.

Below is the file structure the Deepchecks engine expects:

user_interaction_id: This is an optional column. The user_interaction_id is used for identifying interactions across different versions and when updating annotation. It should be unique.
input: This is a mandatory column. It represents user inputs to the model.
output: Output generated by the model to the user. This is a mandatory column.
information_retrieval: Information used as context to understand your current request. This is an optional column.
full_prompt: The full prompt in the current interaction. This is an optional column.
annotation: Rate the model’s output - good, bad, or unknown.

Upload your data after formatting it to the above structure. The Deepchecks engine will automatically evaluate it. You can also upload your data using the Deepchecks Python SDK.

This is the structure of my data after formatting it.

I am making use of the UI upload to upload my data.

Annotation

Let me break down the evaluation result. I will begin with annotation.

You will notice that I did not provide annotations in my sample. This is because I want the Deepchecks system to do it for me. Deepchecks recommends having a baseline annotated by experts. There are two ways to do this:

Provide initial annotations manually done by experts in your sample. Manual annotation is not scalable - it requires a lot of time and effort.
Get estimated annotations from the system and do a human review of it. This is a great way to annotate samples.

In my case, I did the latter. Let’s take a look at the estimated annotations provided by Deepchecks.

The image above is the annotation score for my 104 interaction data. 77% of the interaction was annotated good, 11% annotated bad, and 13% not annotated.

Next, let’s go over to the Data page to see how each interaction is annotated and do a human review.

I can review each interaction and edit the annotation. Click on the annotation sign to change the annotation.

Note: Full colored thumbs up/thumbs down denotes manual annotation while the outline colored sign denotes estimated annotation (annotation suggested by the system)

I can also customize the estimated annotation rules to improve score quality. To customize the estimated annotation rules, go to the Annotations Config page, download the current YAML config file, edit it, and upload a new version.

This is what the current configuration looks like

How you can customize the current configuration:

Add or remove properties
Change the thresholds. Thresholds are used to flag anomalies.
Modify the sample evaluation prompt for unclassified data.
Change the similarity algorithm and its location.

Properties

Deepchecks LLM Evaluation has over 34 built-in properties. Properties measure aspects of your interactions like completeness, coherence, relevance, toxicity, etc. These properties determine estimated annotation.

Built-in, Custom, and LLM properties are the types of properties on Deepchecks LLM Evaluation.

The built-in properties are calculated by Deepchecks’ system using specially trained NLP models and algorithms. Built-in properties include Invalid Links, Reading Ease, Toxicity, Fluency, Formality, Sentiment, Avoided Answer, Grounded in Context, Relevance and Retrieval Relevance.

Custom properties are values provided by user alongside other fields in their data sample. I can add new custom properties if I want by going to the Custom Properties page.

LLM properties evaluate the quality of interactions using LLMs.

Click Add property filter on the Data page to see the list of all properties and to filter properties.

The Overview page shows the values of calculated properties. Below are the calculated properties for my sample.

Breakdown of some of the calculated properties:

Relevance: This property shows the relevance of the output to the input. 0 is the lowest range (not relevant) and 1 is the highest (very relevant). The relevance value of my sample is 0.87.
Grounded in Context: This property shows how well the output relates to the conversation context. 0 is the lowest range (not grounded in context) and 1 is the highest (grounded in context). The Grounded in Context value of my sample is 0.35.
Avoided Answer: The Avoided Answer value is the calculated probability (0 to 1) of the LLM intentionally or unintentionally avoiding the user’s questions.
Toxicity: The Toxicity value for my sample is 0. It means that my model did not generate harmful or offensive outputs.
Fluency: The Fluency value for my model is 0.91. It means that the input texts are well-written. 1 is the highest fluency value.
Sentiment: Sentiment ranges from -1 to 1. It is the measure of the emotions expressed in a text.

Topics

Topics help understand what users are asking about and identify what topics the model underperforms.

The topics in my sample:

I can filter through interactions associated with a certain topic. Below are some of the interactions associated with Subscription management and Server-Driven UI topics.

Segments

Deepchecks helps identify weak segments within data. With Segments, I can easily see areas where my LLM underperforms.

Below are the weak segments in my sample.

Versions

I can add new versions of my data for comparison. Comparison can help improve the overall performance of your model.

Samples Generation

I can increase my Golden Set coverage by generating automatic user inputs. To do this, click Generate Data in the dashboard, and add the guidelines and application description. Including a relevant link or file will help boost context.

Conclusion

Continuous evaluation of your LLM pipeline will help your LLM-based application perform optimally. By continuously validating your models throughout their lifecycle, you can catch issues like deviation and anomalies early, prevent harmful output, and comply with AI regulations.

The Deepchecks Evaluation system is a quality tool to monitor your LLM pipeline and identify areas where your model underperforms so that you can optimize and deliver quality to your users. You can monitor the performance of your LLM from the dashboard and gain insight into how well your model understands and responds to user inputs. The system scores your data to help you understand the overall performance of your pipeline and how you can improve it.

The benefits of continuous LLM evaluation are enormous. Product owners can deliver quality to their users and ensure their models conform to industry ethics by continuously monitoring and validating their pipeline from testing to production using Deepchecks LLM Evaluation.

I strongly recommend Deepchecks LLM Evaluation Solution for ML engineers, managers, and product owners looking to optimize their LLM pipeline.

Top Firebase Alternatives

Emmanuel Aiyenigba — Tue, 10 Oct 2023 10:13:28 +0000

Introduction

In today’s fast-paced world of software development, startups want to build performant and scalable applications fast and with a small team. This quest has led to the demand for cloud service models that can take care of the backend complexities so that developers can focus more on writing and maintaining the frontend.

For early-stage startups that want to move fast, it is time-consuming and capital-intensive to hire experts to build their backend from the ground up, not to mention the complexities of building and maintaining backend infrastructure and their services - authentication, datastore, storage, analytics search, real-time event streaming, hosting, etc. For this and many other reasons, development teams look to Backend-as-a-Service (BaaS) vendors to provide effective, efficient, performant, and scalable software to handle the backend and backend-related services so that developers can build faster by having less tasks to focus on and also reduce the time to market significantly.

Firebase is a cloud-based Backend-as-a-Service (BaaS) that helps accelerate the development of applications. It presents several built-in services to build web and mobile applications fast. However, its numerous limitations, some of which are deal-breakers, have led development teams to explore alternative BaaS providers.

This article will delve into the limitations of Firebase, explore a few existing alternatives and provide guidance on selecting a superior Backend-as-a-Service solution.

Benefits of Backend-as-a-Service (BaaS)

Backend-as-a-Service has many benefits for developers. The way the web and mobile applications are built is changing as demands for apps and features are increasing daily. Development teams want to move fast and scale together. BaaS makes this possible. The following are some of the benefits of BaaS:

Development speed: BaaS helps you build fast by taking care of the entire backend infrastructure for you so that you do spend valuable time figuring out complexities and lets you focus on fewer tasks - only the frontend. This way, the time from building to market is reduced.
Cross-platform compatibility: with BaaS, cross-platform compatibility becomes easier because you can integrate your web and mobile applications under a single backend designed to scale by default.
Scalability: Most BaaS providers design their platforms to scale by default. Your backend scales up as the load and traffic increase and scales down as they decrease. You don’t have to worry about managing scalability yourself.
Cost: BaaS reduces the cost of building, hosting, maintaining, and updating your backend by yourself. It eliminates the need to hire experts to build and maintain your servers in-house. It also reduces the maintenance cost for backend services like storage, infrastructure provisioning, and database management.
Server availability: BaaS providers typically have global server availability, ensuring that your apps are available globally to users always.
Security: BaaS providers invest heavily in security and conform to best security practices to protect sensitive user data. On top of that, they allow you to set permission regarding who can read or write to your application.

Firebase; the downsides

Firebase offers several services out-of-the-box, like authentication, analytics, crashlytics, firestore, performance monitoring and cloud function, etc. It also has a large community of users. However, it is not always the easiest, fastest, and most flexible option available due to its limitations. Some of these limitations of Firebase can be a deal breaker for development teams. Let’s take a look at some of the downsides.

Challenges with data storage: Firebase uses a real-time database as its primary store. However, the database has less querying capabilities. You cannot do a full-text search or query multiple keys at once in Firebase. It also doesn’t help you filter your data efficiently since the filtered data is returned unordered. The limitation in the query capabilities affects the efficiency of data access and may slow down performance.
Restriction in data migration: Firebase promotes platform dependence by restricting data migration, making it difficult to move your data out when you want. Every developer wants a BaaS provider that allows them the flexibility to move their data out whenever and without restriction.
Built to lock you in. Vendor lock-in is the major downside of Firebase. It wants to keep you dependent on the platform by making it difficult for you to move away. You rely heavily on the services and updates provided by Google. It can be restrictive at a certain point as you don’t have any control over what goes on behind the scenes.
Pricing: The Firebase free tier is limited, making it suitable for only small applications. Pricing becomes very expensive as your app grows. Firebase’s pricing model is complex and hard to predict since the upgrade plan is pay-as-you-go.
Security: Firebase’s security rules work by comparing a pattern to database paths and applying customized conditions to grant access to data at these paths. However, building large-scale apps on top of them can be difficult because the rules are limited. The Firebase real-time database security rules leverage JSON in the rule definition. The JSON file can grow quickly and become excessively large, making it difficult to manage, as any mistake in modification can potentially affect the entire application.

Firebase shines more with smaller applications. For large and complex applications, you should consider using other BaaS. Let’s delve into the alternatives to Firebase.

Top Firebase Alternatives

Many Firebase alternatives exists, but we will only look at a few. We will discuss features, benefits, and cons of each platform, as well as their pricing option and developer support. Whether you are looking for a BaaS platform that is more flexible, scalable, secure, and affordable than Firebase, this list will help you find the right one.

SinglebaseCloud

SinglebaseCloud is an AI-powered cloud-based low-code Backend-as-a-Service that helps you build web and mobile applications faster at scale. SinglebaseCloud provides features like datastore, authentication, storage, search, analytics, etc. through SDKs, GraphQL, SQL, and REST to build apps faster and better. SinglebaseCloud is the first BaaS provider that offers Vector Database and Vector Search out-of-the-box.

It addresses and resolves the bottlenecks and constraints found with Firebase. Vector database, advanced querying and indexing capabilities, AI-powered vector search, and other features impossible in Firebase are now possible with SinglebaseCloud out-of-the-box. Think of SinglebaseCloud as the alternative that resembles Firebase the closest but with much more flexibility, scalability, and features. If you love Firebase but are concerned about its rigidity and numerous limitations, SinglebaseCloud may be the right alternative for you. SinglebaseCloud’s NoSQL document database is equipped with SQL capabilities to help you query data quickly.

SinglebaseCloud offers a NoSQL document database that allows you to store, manage, and use unstructured data efficiently. It takes care of all of the server-side complexities for you so that you can focus on your frontend

Features & Pros

Vector database: SinglebaseCloud’s vector database can store and manage vector data efficiently. With a vector database, you can index and store vector embeddings for fast data retrieval and similarity search. It enables you to equip large-language models (LLMs) with long-term memory.
Vector search: Vector search or Similarity search helps you find similar vectors by calculating the similarities (closeness and distance). With vector search, you can search and retrieve information from a vector database based on meaning or context instead of keywords as in other databases. Vector search helps you unlock intelligent search results. SinglebaseCloud provides these features out of the box.
NoSQL datastore: SinglebaseCloud has an efficient document-based NoSQL datastore to help you manage your text data. Its datastore is equipped with advanced matching query and SQL capabilities to enable you to query for data quickly.
Seamless integration with your applications: SinglebaseCloud’s services and features need no extra configuration to make them work - they work by default. It can integrate seamlessly with your app’s existing functionalities without creating conflicts.
Fast and intuitive: SinglebaseCloud is designed to be fast and intuitive. Because it is AI-powered, the database doesn’t require explicit instructions for every task; it possesses an innate sense of intuition to respond to your queries and requests. It also has fast querying and indexing capabilities.
Performant and scalable: Your SinglebaseCloud backend can scale up as the load and traffic increase and scale down at less traffic by default, making your app performant regardless of the traffic. Its NoSQL document database allows you to store and manage unstructured data efficiently.
Efficient storage system: SinglebaseCloud provides an efficient storage service to handle your app’s files and media. Its robust image processing capabilities let you store, manipulate, and deliver images at scale. For optimal performance, SinglebaseCloud allows you to cache and automatically resize images. It has a secure and flexible storage system with features like access controls, automatic backups, and versioning, making it easier to serve images.
Seamless authentication experience: SinglebaseCloud has industry-standard authentication protocols to secure your application and protect user data. Multi-factor authentication and social login options are also available.
Insightful analytics and efficient search capabilities: SinglebaseCloud’s analytics help you make data-driven decisions. Its customizable indexing, filtering, and relevance ranking features help your application unlock fast and accurate search results.
Predictable pricing and developer support: SinglebaseCloud has predictable pricing and great developer support to keep you going. Unlike other BaaS providers, SinglebaseCloud’s prices will be flat for your tier, so you should not expect any fluctuations in cost.

Cons

SinglebaseCloud is new, so the community around it is still small. You may get little or no peer support while building.
It uses a NoSQL document database. If you prefer relational databases, this can be an issue for you
It is not open-source, and you cannot self-host.

Supabase

Supabase is an open-source BaaS platform that provides tools and services for building fast, scalable, and secure applications. It leverages PostgreSQL as its database and offers a range of features and functionalities to enable the development of full-fledged modern applications. Authentication, instant APIs, storage, analytics, vector embeddings, and edge functions are all provided by Supabase out of the box.

Features & Pros

Realtime database: Supabase offers a real time PostgreSQL database, allowing you to store and retrieve data in real time, making it suitable for applications that require instant updates.
Flexible services: Supabase is super flexible and extensible - you can use any of their services as a standalone. For example, you can use the Supabase storage alone and then use another product’s datastore - it gives you that much flexibility.
Industry-standard security: Supabase takes security very seriously. It allows you to define granular access control. Its powerful Postgres Row Level Security (RLS) allows to create policies to control access to your data. RLS ensures that users only access the data they are authorized for. You can enable RLS for any table in your database.
Real time event triggers: Developers can set up event triggers that respond to changes in the database. When an event is triggered, Supabase can execute serverless functions enabling you to automate tasks or send notifications in real time.

Cons

Learning curve: Supabase has a steep learning curve. Developers who are new to PostgreSQL may find it hard to navigate
It only supports PostgreSQL database. If you prefer a non-relational database, then Supabase may not be the right tool.
Scalability: If self-hosting, you may have to manage scalability concerns as your app grows.
Limited Ecosystem: Supabase is a relatively new platform compared to more established Backend-as-a-Service providers. It may have a smaller ecosystem of third-party integrations, libraries, and plugins compared to more mature alternatives.

Appwrite

Appwrite is an open-source BaaS platform built by hundreds of developers. It enables you to spin up a backend easily, build any application at scale with any coding languages, and host it wherever you want. It is a great Firebase alternative with core functionalities like authentication, database, storage, function, and real-time events.

Features & Pros

Database services: Appwrite’s NoSQL database lets you store and retrieve structured data. You can organize your data into databases, collections, and documents so that you can query with ease. It also supports relationships for efficient data modeling.
Functions: Appwrite’s functions let you extend Appwrite with code snippets and build anything you want. Functions are triggered by webhooks, HTTP requests, SDK options, and scheduled executions.
Robust authentication: Appwrite offers user authentication and identity management services, allowing developers to implement user registration, login, and access control. It supports various authentication methods, including email/password, OAuth, and custom JWT.
Cloud Functions: Appwrite supports serverless computing through its Cloud Functions feature. Developers can write and deploy serverless functions that run in response to HTTP requests or database events, extending the functionality of their applications.

Cons

Learning curve: It may take a while to familiarize yourself with the Appwrite platform.
Customization complexity: Sometimes, you may need to customize your backend, which can lead to extensive custom coding and more development efforts.
Infrastructure management: If you self-host Appwrite, you will be responsible for managing, updating, and scaling your server. It may not be a good idea if you are looking for a fully managed BaaS solution.

Parse Server

Parse Server is an open-source backend framework that enables developers to build scalable and customizable server-side logic for web and mobile applications. Initially developed by Facebook as a Backend-as-a-Service (BaaS) platform, Parse Server is now maintained by an open-source community.

Features & Pros

Database Agnostic: Parse Server is designed to be database-agnostic, allowing developers to choose the database that best suits their project's needs. It supports databases like MongoDB, PostgreSQL, and others, providing flexibility in data storage.
Data Storage and Querying: Parse Server offers a powerful data storage system with support for classes and objects, making it easy to organize and query data. Developers can perform complex queries and filtering operations, making it suitable for various application types.
Real-Time Features: Parse Server includes real-time features that allow developers to build applications with live updates and collaboration. It provides a real-time database and live queries, enabling applications to push data updates to clients in real time.
Authentication and User Management: Parse Server provides robust authentication and user management services, allowing developers to implement user registration, login, password reset, and access control. It supports various authentication methods, including email/password, social logins, and custom authentication.

Cons

As of the time of writing this article, one major con of Parse Server is that it is not actively maintained, which can be a deal-breaker.
Self-hosting complexity: Setting up and maintaining Parse can be complex for developers without experience in server administration

Conclusion

In your quest to find the ideal Backend-as-a-Service (BaaS) platform, it's essential to consider a solution that not only accelerates development but also offers the flexibility and scalability required for your projects. While Firebase has garnered recognition for its out-of-the-box services and vibrant community, it's vital to acknowledge its limitations, especially when embarking on larger and more complex applications. The alternatives provided in this article can help you build a better, faster, and more scalable backend for your applications.

Building a Note taking app with Storyblok and Next.js

Emmanuel Aiyenigba — Tue, 05 Sep 2023 09:12:18 +0000

Introduction to Headless CMS

Re-inventing the wheel has always been tedious, and it prolongs the distance between idea and production. Headless CMS helps developers, marketing teams and other creative professionals move from idea to production with speed because you don’t have to worry about anything server related since it is taken care of already. A headless CMS can connect to any frontend (hence the name headless). It gives you the liberty to use any frontend library/framework of choice, unlike the traditional CMSs which are coupled thereby constraining you to a particular frontend.

A headless CMS primarily acts as a content repository where you can store data and retrieve it easily through a content delivery API which could be a GraphQL API or a REST API. The data retrieved via the API can be displayed on any device of your choice. This will enable you to deliver content seamlessly to any device - web, mobile app, IoT, electronic billboards, etc.

Why Storyblok

In choosing a headless CMS platform, there are certain things to consider: Scalability, freedom & flexibility, developer experience, omnichannel publishing, performance, security, etc. Storyblok is cut out for this with its built-in features optimized to meet your product needs. Storyblok provides you the flexibility of building with tech stacks that you are familiar with because it supports all frameworks - you do not need to use any technology that is unfamiliar to you. Its powerful SDKs help you connect and build seamlessly. Later in this article, we shall explore one of the SDKs, the React SDK, in building a Note app.

Furthermore, with Storyblok's real-time visual editor you can see a preview when creating or editing content. You would also have the liberty of exploring the components of your page. This feature can be enabled and disabled per your need. Other benefits of Storyblok include fast performance, scalability, easy team collaboration, built-in features, etc.

Next up in this article, we will learn how to tap into the power of Storyblok to build an application - a Note taking app.

Requirements

To follow this article you need:

Node and npm installed on your machine
Next.js 12.
A Storyblok account
Any code editor of your choice

Now, let’s set up our Storyblok project and begin building.

Setting Up and Configuring Our Storyblok Project

Before creating a new space on Storyblok, it is important to define some terms you’d come across in this article.

Space: A space is a content repository. You can create as many spaces as you want. Ideally, you’d need just a space per project. A space holds everything for your project - folder, assets, block, stories, etc.
Story: Stories are content entries that utilize any selected template as their content type.
Folder: A folder is where you keep related stories
Block: Blocks are what makes up stories. They are modular components put together to form content in its entirety. There are 3 block types in Storyblok: Nestable block, Content-Type block, and Universal block.

You will see them in action in a bit. What we need to do now is to create a space and set up what we’d need for our application.

Create a new space and give it a name. Here we gave it the name Note Application and selected the server location as EU.

Creating Blocks

Let’s go over and create some blocks that we’d be needing for our application.

You should remove all the default blocks except the page block because we would not need them. Let's create some fresh blocks 😉

In addition to the page block, we need to create 2 more blocks:

Note: The Note block will take 3 fields: title, body and created_at

Note-highlights: this will be a Nestable block and takes a field name note of type Multi-Options. We want Note-highlights to display a few notes on the homepage - each note will be clickable. It will take you to the entire note when you click.

Click on the note field then configure the source for the Multi-Option field to point to Stories. This is where we want to read the data from. Make the path to folder of story to be notes/ and also restrict content-type to the note block. We will create the note folder shortly.

Our blocks are set. Now let’s create the notes folder which will hold all our notes and also make the Home story display the Note-highlights page.

Creating the Notes Folder

Navigate inside Content > click Create new > select Folder. Give it a name of Notes and make the Content type Note (we want it to use the Note block we created earlier as its content type).

Setting Up The Homepage

When you navigate into Content, you’d observe that Storyblok gives us a Homepage by default. We will make this Homepage display the Note-highlights page.

Let’s add the Note-highlights page to Home

You’d observe that our development server is pointing to https://quickstart.me.storyblok.com . We need to set up an HTTPS proxy for our dev server and make port 3010 the port for our application. The URL to access our application will be https://localhost:3010/. To do this, go to Settings > Visual Editor then change the URL of the local environment to https://localhost:3010/

Everything is set here in Storyblok. Now, let’s build the frontend of our application with Next.js.

Creating A Next.js Project

npx create-next-app note-app

Go ahead and install Storyblok’s React SDK. Also, we need to install Storyblok rich text renderer.

npm install @storyblok/react storyblok-rich-text-react-renderer

Connecting Our Application To Storyblok

Let’s connect our Next.js application to Storyblok

pages/_app.js


import {storyblokInit, apiPlugin} from "@storyblok/react"
import Page from "../components/Page"
import Note from "../components/Note"
import NoteHighlights from "../components/NoteHighlights"

const components = {
  page: Page,
  note: Note,
  "note-highlights": NoteHighlights,

}

storyblokInit({
  accessToken: "Your-access-token",
  components,
  use: [apiPlugin]
})

export default function App({ Component, pageProps }) {
  return <Component {...pageProps} />
}

The storyblokInit function initializes Storyblok and connects to it via your accessToken. It makes use of the apiPlugin. We added the Storyblok components that we will be utilizing in our app. We will create the components inside our project shortly.

Replace the value of accessToken with your preview token found inside Settings > Access Tokens.

Fetching Data From Storyblok

At this point, we want to set up our landing page (otherwise called homepage). To do this, we have to fetch data from our Storyblok content repository

pages/index.js

import Head from 'next/head'
import {StoryblokComponent, getStoryblokApi, useStoryblokState} from "@storyblok/react"

export default function Home({story}) {
story = useStoryblokState(story, {
    resolveRelations: ["note-highlights.note"],
  });
  return (
    <div>
      <Head>
        <title>Note app</title>
        <meta name="description" content="Generated by create next app" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <link rel="icon" href="/favicon.ico" />
      </Head>
      <div>

        <StoryblokComponent blok={story.content} />

      </div>
    </div>
  )
}

export async function getStaticProps(){
  let slug = "home";

  let sbParams = {
    version: "draft", // it can also be "published"
    resolve_relations: ["note-highlights.note"]
  }

  const storyblokApi = getStoryblokApi()

  let {data} = await storyblokApi.get(`cdn/stories/${slug}`, sbParams)
  return {
    props: {
      story: data ? data.story : false,
      key: data ? data.story.id : false
    },
    revalidate: 3600, 
  }
}

StoryblokComponent sets up a page for our Storyblok components and getStoryblokApi fetches data from Storyblok through your API endpoint.

useStoryblokState is a custom hook that connects the current page to Storyblok’s real-time visual editor.

We want to use the Next.js Static Site Rendering feature hence the getStaticProps function we called our API in. Our API request takes in two parameters: version, where you want Storyblok to read the data from (in this case from your Draft) and resolve_relations which gets all the details related to the note-highlights.note block.

revalidate allows us to update our page (in this case our homepage) without rebuilding our entire site. We are revalidating every 3600 milliseconds (1 hour).

Dynamic Routing

We are going to create a catch-all route to capture all the individual notes.

Create a [...slug].js file inside the pages directory

[...slug].js

import React from 'react';
import Head from "next/head"

import { StoryblokComponent, getStoryblokApi } from '@storyblok/react';

const Page = ({story}) => {
    return (
        <div>
            <Head>
                <link rel="icon" href="/favicon.ico" />
                </Head>

                <StoryblokComponent blok={story.content} />

        </div>
    );
}
export default Page

export async function getStaticProps({params}) {
    let slug = params.slug ? params.slug.join("/") : "home"
    let sbParams = {
        version: "draft",
        // resolve_relations: ["note-highlights.note"]
    };

    const storyblokApi = getStoryblokApi()
    let {data} = await storyblokApi.get(`cdn/stories/${slug}`, sbParams)
    return {
        props: {
            story: data ? data.story : false,
            key: data ? data.story.id : false
        },
        revalidate: 3600
    };
}

export async function getStaticPaths() {
    let sbParams = {
        version: "draft",

    };
    const storyblokApi = getStoryblokApi()
    let {data} = await storyblokApi.get(`cdn/links/`, sbParams)
    let paths = [];

    Object.keys(data.links).forEach((linkKey) => {

        if (data.links[linkKey].is_folder || data.links[linkKey].slug === "home") {
            return;
        }
        const slug = data.links[linkKey].slug;
        let splittedSlug = slug.split("/");
        paths.push({ params: { slug: splittedSlug } });
    });
    console.log(paths)

    return {
        paths,
        fallback: false
    };
}

getStaticPaths gets the path for all our notes so that we can route them dynamically. Note that we specified that it should not get the paths of folders and the homepage.

Building Our Components

Recall that we added the Pages, Note, and Note-Highlights components to our Storyblok connection. Now, let’s build them. You should create these components inside the root of your project.

Pages.js

import React from 'react'
import { StoryblokComponent, storyblokEditable } from '@storyblok/react'

function Page({blok}) {
  return (
    <main {...storyblokEditable(blok)}>
      {blok.body.map((nestedBlok) => (
          <StoryblokComponent blok={nestedBlok} key={nestedBlok._uid}/>
      ))}
    </main>
  )
}

export default Page

StoryblokEditable lets you edit this page inside Storyblok’s real-time Visual Editor.

Note-Highlights.js

import React from 'react'
import { storyblokEditable } from '@storyblok/react'
import { render } from 'storyblok-rich-text-react-renderer';
import Link from "next/link";

function NoteHighlights({blok}) {
  return (
    <div {...storyblokEditable(blok)} key={blok._uid}>
      <h2 className='my-header'> My Notes</h2>
      {blok.note.map(note => {
        return(
          <div key={note.id} className="note-highlight">
            {/* Link to full note */}

            <Link href={`/${note.full_slug}`} key={note.id}>
            <h2>{note.content.title}</h2>
            </Link>

            <div className="line-clamp-4">
              {render(note.content.body)}
            </div>

            <Link href={`/${note.full_slug}`}><p className="note-link">Read more</p></Link>
          </div>
        )
      })}
    </div>
  )
}

export default NoteHighlights

This page displays some of our notes. We used line-clamp which is a Tailwind CSS property to display only a few lines of a paragraph. The link will take us to the full_slug (where you can see the entire note).

Note.js

import React from 'react'
import { storyblokEditable } from '@storyblok/react'
import {render} from "storyblok-rich-text-react-renderer"
import Link from 'next/link'

function Note({blok}) {
  return (
    <div {...storyblokEditable(blok)} key={blok._uid}>
      <div>
        <div className="note-container"> 
        <h2>{blok.title}</h2>
        <div className='render-note'>
          {render(blok.body)}

        </div>
        </div> 
          <p className='createdAt'>{blok.created_at}</p>
      </div>
      <button className='bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded ml-2 mb-5'>
      <Link href="/">
        Back

      </Link>
      </button>
    </div>
  )
}

export default Note

Let’s go over and add some notes inside Storyblok.

Navigate to Content > go inside the Note folder we created earlier > click Create new > select Story.

Give it a name make the Content type Note. Click Create when you are done.

Add a few more notes and let’s take a look at our application.

Voila! Our application looks great. If you click the Read more text on any note, it takes you to the full slug of the note - where you’d see the entire note.

Summary

Using the Storyblok Headless CMS, you can move from idea to production in a short time. Storyblok connects with all frameworks and therefore helps you build seamlessly. The real-time Visual Editor enables you to make live edits that immediately reflect in your application. Storyblok has great documentation to get you started with headless CMS development.

Do you need help creating technical content for your developer audience? Reach out and let’s work together.

Shameless Plug

If you find my content valuable and engaging, consider sharing it with your network and follow me here and on Twitter. It would mean a lot to me.

I just launched my newsletter where I’ll be sharing carefully curated technical articles and development news. If you enjoy learning about Open source software, Web engineering, Software engineering, Cloud & DevOps and other aspects of product engineering, please join my readers’ list.

Emmanuel’s Substack | Emmanuel Aiyenigba | Substack

Carefully curated engineering articles, open source software updates and development news worth knowing about. Click to read Emmanuel’s Substack, by Emmanuel Aiyenigba, a Substack publication. Launched 2 years ago.

emmanuelthecoder.substack.com

VS Code Settings Sync: Basics, Troubleshooting, and More

Emmanuel Aiyenigba — Tue, 22 Aug 2023 12:53:44 +0000

Introduction

The introduction of Settings Sync as a built-in functionality lets you share your VS Code configurations across multiple machines and not worry about installing dependencies afresh or environment configuration when working on another machine or with your team.

This was added into version 1.48.0 released in July 2020. Previous versions had no such functionality built in. Before the release of version 1.48.0, if you needed to synchronize settings across your machines, you would have to install an extension from the marketplace.

This post will cover everything you need to know to get started with Settings Sync in Visual Studio Code. You’ll learn how to take advantage of Settings Sync to boost productivity. At the end of this post, you’ll understand the nitty-gritty of Settings Sync and gain the necessary knowledge needed to begin using it. Given these points, let’s dive into Settings Sync in detail.

What Is Settings Sync on VS Code?

Setting Sync is a VS Code built-in functionality that allows you to share and use your preferences across multiple machines. You can sync preferences such as Settings, Keyboard Shortcuts, User Snippets, Extensions, and UI State. VS Code Settings Sync is not the same as the Settings Sync extension built by Shan Khan, which stores your settings in a private Gist on GitHub. The VS Code Settings Sync stores your synced settings on the cloud.

It is recommended that you use the built-in Settings Sync and not the Settings Sync extension. This is because you can do so much with the former, plus it is maintained by the Visual Studio Code team so you can expect it to get constant improvements to better serve your development needs.

You can choose to sync settings to either your GitHub or Microsoft account. The Microsoft account could be either your Outlook or Azure account. It is equally important to note that VS Code does not support GitHub Enterprise accounts for Settings Sync; use either a personal account or an organization account. Without delay, let’s go into how you can sync your VS Code settings.

Importance of Settings Sync

Undoubtedly, VS Code Settings Sync has a lot of importance for individuals and teams working on the same project. Individuals who prefer to use more than one machine for work can do so while maintaining the same settings across their machines.

This is to say that if you are using machine A for work now and will be working with machine B later (probably because you’re in another location and didn’t take machine A with you), if you have Settings Sync turned on in machine A, you can merge your settings preferences that you have to your account on the cloud by signing in to the account (GitHub or Microsoft) your data was synced to. This way, you don’t need to install your favorite extensions, configure your choice development environment, or organize your UI State on your current machine.

Because of Settings Sync, teams scattered across the world but working on the same project can use the same settings on their machines at the workspace level by creating a .vscode folder with a setting.json file containing the settings you want to share with your team.

Other Important Features of Settings Sync

Settings Sync gives you the flexibility of working from any of your machines while maintaining your preferred VS Code environment.
It saves you the stress of reinstalling your favorite setup if you lose your machine or have to work with another machine.
You and your team can share the same work environment. Sounds cool, right?

Your settings are protected. VS Code doesn’t jeopardize your data privacy; you can always turn off Settings Sync and have your data wiped out of the cloud.
You don’t need to install a marketplace extension to synchronize settings across your machines.
With Settings Sync, you get to choose your preferences (the things you want to sync) and leave out the things you don’t want to sync.
Furthermore, VS Code always stores a local and remote backup. In case something goes wrong, you can restore your data.

How Do I Sync My VS Code Settings?

Open an instance of VS Code then click on the Manage gear.
Select Turn on Settings Sync.

Select your preferences (the data you want to sync).

Note: The following UI State options are currently available for syncing (more might be added in the future): Panel entries, Activity Bar entries, Display Language, Views layout and visibility, Recently used commands, and Do not show notification again.

After selecting your preferences, click Sign in & Turn on. You will be asked to choose between your GitHub or Microsoft account for signing in.

After you select the account to sign in, an authentication screen will pop up. Settings Sync turns on after you sign in. At this point, you have Settings Sync enabled and your preferences activities will be synced in the background from time to time.

Now that you’re signed in and synced, you can manage your synchronization preferences at will. To do that, click the Manage gear > Setting Sync is On > select Settings Sync: Configure. VS Code gives you the flexibility to sync your data to a different account any time you want. This is to say that you can change your sync account from GitHub to Microsoft and vice versa at will. To do this, turn off then turn on Settings Sync again, then select a different account (GitHub or Microsoft).

Syncing Options

When you turn on Settings Sync on one machine, signing in to the account that was synced to and turning on Settings Sync on another machine will automatically present to you a dialog. You have the option to Merge, Replace Local, or Merge Manually.

Merge: This option merges the settings on your local machine (local settings) with the settings on the cloud (remote settings). It combines the settings of your local and cloud environments for use on your current machine.
Replace Local: Selecting this option replaces your local settings with your remote settings. You should be sure you don’t need any of your local settings before selecting this option; otherwise you should merge.
Merge Manually: This option allows you to select the settings you want to merge. You can choose your preferences manually then merge them. You’ll have the liberty to go through each entry to choose the settings you want to merge. It shows you a few JSON files that you can edit to meet your preferences: settings.json (Settings), keybindings.json (Keyboard Shortcuts), typescript.json (User Snippet), and extensions.json (Extensions).

Data Privacy

Without a doubt, your data is protected on the cloud and lives under your rules. Therefore, you can decide to clear it from the cloud any time you like. You alone have access to your data through the account(s) you synced to. It is recommended you clear your data when you turn off Settings Sync. Sometimes you don’t want certain data to persist on the cloud when you turn off Settings Sync, probably due to its sensitivity. So you can clear it out of the cloud. For example, you might not want certain data in your UI State, such as activity bar entries that give indicators like the number of outgoing Git changes, to persist on the cloud once you turn off sync.

Some extensions store important data such as Git history, which gives a visual of the Git log and allows you to compare branches and commits. It’s good to wipe out your data when you turn off sync. If you decide to turn on sync again, your data is synced afresh.

To turn off Settings Sync and clear out data, go to the Manage gear, click on Setting Sync is On, then select Setting Sync: Turn Off and select the checkbox to wipe out your cloud data. With this in mind, let’s take a look at the levels at which Settings Sync occurs.

Levels at Which Settings Sync Occurs

At this point, it’s important to state that Settings Sync in VS Code occurs at different levels and later settings override earlier settings. According to the VS Code official documentation, Settings Sync occurs at the following levels.

Default settings: Default settings come with a newly downloaded Visual Studio Code.
User settings: Global settings that override the default settings and apply to all VS Code windows.
Remote settings: This applies to a remote machine - utilizing cloud development environments to manage your projects. You can sync settings to a remote machine. These settings can be used across your team. Remote teams can collaborate better by storing Sync Settings in a virtual machine, to be shared by members of the team at the workspace level.
Workspace settings: These settings apply at the workspace level. It overrides the previous settings. This is useful for helping developers collaborate on a project by using the same setting and without globally overriding their personal preferences.
Workspace folder settings: This applies to a specific folder in a multi-root workspace

Resolving Synchronization Conflicts

It’s important to realize that conflict may happen when syncing settings between more than one machine. There will be sync conflict when changes occur when a machine is offline. This is because the changes can not be uploaded to the remote environment if the local machine is offline (without an internet connection). To point out, when conflicts occur, you can either Accept Local, Accept Remote, or Show Conflicts.

Accept Local will replace remote settings with local settings.
Accept Remote will replace local settings with remote settings.
Show Conflicts will open a “diff” editor where you can preview the local and remote settings and then choose either to accept local or accept remote.

Tracking Sync Activities

Another key point to note is that VS Code tracks sync data by default and stores local and remote backups of your preferences, which allows you to undo an action or restore your data if anything goes wrong. A local backup folder is created to track your local changes. The local backup is automatically deleted after 30 days, while for the remote backup, only the latest 20 versions of each individual resource are retained.

The machines synchronizing your preferences are tracked and given a default name based on their platform and the type of VS Code they’re using (Insiders or Stable). You can turn off preferences sync for any of the synced machines at any time. To see sync activities and sync machines, click the Manage gear > Setting Sync is On > select Settings Sync: Show Synced Data.

Troubleshooting Settings Sync

Undeniably, you can sometimes run into issues when using Settings Sync. The following is how to troubleshoot a few common issues.

Error while turning on Settings Sync. No authentication providers.
This error usually occurs in Linux. This error happened because org.freedesktop.secrets was not provided by any .service files. Gnome provides org.freedesktop.secrets in many desktop environments, so to resolve this issue, install gnome-keyring, libsecret, and libgnome-keyring.
Turning on Settings Sync immediately gets a request failed error.
This happens on rare occasions when turning on Settings Sync for the first time. Another attempt to turn on Settings Sync should succeed.

Conclusion

In conclusion, VS Code Settings Sync allows you to synchronize your favorite preferences such as Settings, Keyboard Shortcuts, User Snippets, Extensions, and UI State and reuse them on any of your machines. With this, you don’t have to worry about reconfiguring your favorite environment to your new machine. Remote teams can collaborate more effectively by using Settings Sync. And your data lives under your rules; therefore, you can turn off Settings Sync and wipe your data from the cloud when you feel it’s necessary to do so.

Do you need help creating technical content for your developer audience? Reach out and let’s work together.

Shameless Plug

If you find my content valuable and engaging, consider sharing it with your network and follow me here and on Twitter. It would mean a lot to me.

Emmanuel’s Substack | Emmanuel Aiyenigba | Substack

emmanuelthecoder.substack.com

Using Large Language Models inside your database with MindsDB

Emmanuel Aiyenigba — Mon, 14 Aug 2023 13:15:46 +0000

Introduction

Large Language Models (LLMs) are AI algorithms trained on a large dataset through deep learning techniques to understand, predict and generate new content. LLMs are the forces that power Natural Language Processing (NLP), allowing machines to understand and respond to natural languages in text or voice data.

The versatility of LLMs depends on the volume of data they are trained on. Usually, a model may be considered a large language model if it has at least 1 billion data parameters. GPT-4 has over 1.7 trillion training data parameters, GPT-3 has 175 billion, and PaLM-2, the LLM that powers Bard, has over 360 billion parameters. From their parameters, one can tell which model is the most versatile.

Now, imagine if you can deploy these highly trained models in your database to get insights, make predictions, understand your users, auto-generate content, and more. MindsDB makes this possible! MindsDB is an open-source AI database middleware that allows you to supercharge your databases by integrating various machine learning (ML) engines.

In this article, you will learn how to supercharge your database by deploying LLMs into it using MindsDB.

This article will cover all you need to know about using LLMs in your database with MindsDB. At the end of this article, you will understand how LLMs can supercharge your database and help you perform NLP activities like text/image generation, word processing, behavior prediction, etc. You will also gain the necessary knowledge to deploy LLMs in your database. Given these points, let’s dive into the details by first learning what MindsDB is about.

What is MindsDB

MindsDB is an open-source AI database middleware that manages and connects AI modules with enterprise databases. It is a complete solution for deploying LLMs using various ML engines, carrying out vector operations, time series forecasting, and classical ML, all inside your database.

MindsDB allows you to create AI models off ML engines like OpenAI, Hugging Face, Pytorch, etc. These models are abstracted as generative AI tables. The models are assigned a prompt template that tells the AI what you want to do and how you want it done. For example, you can create a model off OpenAI’s GPT-4 LLM to auto-respond to users’ reviews of your product based on the type - good, bad, or neutral. After creating the model, you will give it a prompt template for it to know what to do, in this case, respond to reviews based on the type and how to do it - you may want the model to respond to bad reviews with an apology, good reviews with a thank you note and neutral review with a promise to do better.

MindsDB supports several databases so that you can connect your database quickly and start carrying out NLP tasks easily. MySQL, PostgreSQL, Oracle, SQLite, QuestDB, CouchDB, MongoDB, AWS DynamoDB, and many other databases are supported. It also supports several applications like Twitter, Slack, GitHub, YouTube, etc., allowing you to supercharge these applications by bringing in-database ML to them.

Let's quickly highlight the advantages of deploying and using LLMs in your database. After that, we will do a practical dive-in on deploying the GPT and DALL-E models inside your database.

Advantages of using LLMs in your database

There are so many advantages to using LLMs in your database. Here are a few:

Making insightful predictions: Using Large language Models within your database can help you make insightful predictions from previous data. For example, you can predict the future price change of a product based on available attributes like price changes in the past.
Content generation and language interpretation: LLMs allow you to carry out natural language processing like text generation, text summarization, content analyses, language interpretation, and more.
Automatically answering questions: Users and visitors on your website can get their questions automatically answered by AI models that have been trained on your product. It prevents users from waiting several hours for support staff to respond to simple questions like “What is your pricing model?”
Improve the security of your database: LLMs can be used to improve the security of your database by generating synthetic data to test the security of your database.

The advantages of using LLMs in your database are enormous. Given these points, it’s time for some practical. Let’s learn how to use the GPT and DALL-E models in our database with MindsDB.

Getting started with using LLMs in your database

We will deploy and use OpenAI’s GPT and DALL-E LLMs in our database. We will use the GPT model to analyze and respond to reviews. The response will be based on the review type - we would respond with a thank you note for a positive review, an apology for a negative review, and a promise to do better for a neutral review.
For the DALL-E model, we will be using it to generate realistic images from a text description. We can tell the model to generate whatever images we want from our input text. Let’s get right in.

Pre-requisite

You need a MindsDB cloud account to follow this tutorial. You can also set up MindsDB via Docker or pip if you prefer to operate a local instance. A MindsDB cloud account is recommended for this tutorial (it’s easy to setup, and no configurations necessary).

Connecting a database

MindsDB only processes your data but doesn’t store it. You’d need to connect your database to MindsDB to store generated data. For this example, we would connect and use a public MySQL database as our database.

CREATE DATABASE review_response
WITH ENGINE = 'mysql',
PARAMETERS = {
    "user": "user",
    "password": "MindsDBUser123!",
    "host": "db-demo-data.cwoyhfn6bzs0.us-east-1.rds.amazonaws.com",
    "port": "3306",
    "database": "public"
};

We gave our database the name review_response. Let’s take a look at the list of tables inside our database. To do this, run the following SQL query.

SHOW FULL TABLES FROM review_response;

There are 18 tables inside this database. We will be using only the amazon_reviews table. Let’s take a look at some of its content.

SELECT * FROM review_response.amazon_reviews
limit 3

We are selecting the first 3 product reviews.

product_name	review
All-New Fire HD 8 Tablet, 8 HD Display, Wi-Fi, 16 GB - Includes Special Offers, Magenta	Late gift for my grandson. He is very happy with it. Easy for him (9yo ).
All-New Fire HD 8 Tablet, 8 HD Display, Wi-Fi, 16 GB - Includes Special Offers, Magenta	I'm not super thrilled with the proprietary OS on this unit, but it does work okay and does what I need it to do. Appearance is very nice, price is very good and I can't complain too much - just wish it were easier (or at least more obvious) to port new apps onto it. For now, it helps me see things that are too small on my phone while I'm traveling. I'm a happy buyer.
All-New Fire HD 8 Tablet, 8 HD Display, Wi-Fi, 16 GB - Includes Special Offers, Magenta	I purchased this Kindle Fire HD 8 was purchased for use by 5 and 8 yer old grandchildren. They basically use it to play Amazon games that you download.

Generating response based on review type using OpenAI’s GPT LLM

Let’s create a model that will analyze reviews and generate a response based on the review type.

CREATE MODEL review_response_model
PREDICT response
USING
    engine = 'openai',
    prompt_template = 'Give a short response to customers based on the review type.
    "I love this product": "positive"
    "It is not good" : "negative"
    "The product is fair": "neutral"
    If the review is negative, respond with an apology and promise to make it better.
    If the review is positive, respond with a thank you and ask them to refer us to their friends.
    If the review is neutral, respond with a promise to make it better next time they buy from us.
    {{review}}';
        api_key = 'YOUR_OPENAI_API_KEY';

We have created our model that uses the OpenAI’s GPT engine and would give a response for every review based on our prompt_template. The prompt_template tells the AI model what we want to do and how we want it done. We can begin using this model after it's done generating (it takes only a few seconds to generate).

Note
By default, MindsDB uses the gpt-3.5-turbo model but you can use the gpt-4 model by passing it to the model-name parameter under the USING clause.

Run the following command to check the status of the model we just created:

DESCRIBE review_response_model

The status of our model is complete which means it’s done generating and we can start using it. Do not forget to add your Open AI api_key to the list of parameters when creating a model.

For convenience, create a MindsDB ML engine to hold your API key so you don’t have to type it in every time you create a model.

CREATE ML_ENGINE openai2
FROM openai
USING
    api_key = 'YOUR_OPENAI_API_KEY';

Note
If you are on MindsDB cloud you can use the API key provided my MindsDB by confirming your email address. However, MindsDB Pro users are required to provide an OpenAI API key.

Let’s query the model with some synthetic data and see the response.

SELECT review, response
FROM review_response_model
WHERE review = "I love this product so much. I am definitely going to buy more"

Output:

review	response
I love this product so much. I am definitely going to buy more	Thank you for your positive feedback! We're thrilled to hear that you love our product. If you have any friends or family who might also enjoy it, we would greatly appreciate it if you could refer them to us. We look forward to serving you again in the future!

The response from our AI model looks great. Let’s try it out again, this time with some negative comment

SELECT review, response
FROM review_response_model
WHERE review = "I hate this product. It didn't meet the description. You guys are fraud"

Output:

review	response
I hate this product. It didn’t meet the description. You guys are fraud	I'm sorry to hear that you're unhappy with our product. We apologize for any inconvenience caused. We strive to provide the best experience for our customers and we would like to make it right. Please reach out to our customer support team so we can assist you further. Thank you for bringing this to our attention.

Great response from our AI model. Now, let’s make a batch prediction with the reviews in the amazon_reviews table.

SELECT input.review, output.response
FROM review_response.amazon_reviews AS input
JOIN review_response_model AS output
LIMIT 3;

Output:

review	response
Late gift for my grandson. He is very happy with it. Easy for him (9yo ).	Thank you for your positive review! We're glad to hear that your grandson is happy with the gift. If you have any friends or family who might also enjoy our products, we would greatly appreciate it if you could refer them to us. Thank you again for your support!
I'm not super thrilled with the proprietary OS on this unit, but it does work okay and does what I need it to do. Appearance is very nice, price is very good and I can't complain too much - just wish it were easier (or at least more obvious) to port new apps onto it. For now, it helps me see things that are too small on my phone while I'm traveling. I'm a happy buyer.	Based on the review provided, the sentiment is neutral. Response: Thank you for your feedback! We appreciate your support and are glad to hear that our product meets your needs. We understand your concerns about the proprietary OS and we will definitely take your feedback into consideration for future improvements. If you have any suggestions on how we can make it easier to port new apps, please let us know. We value your satisfaction and look forward to serving you even better next time.
I purchased this Kindle Fire HD 8 was purchased for use by 5 and 8 years old grandchildren. They basically use it to play Amazon games that you download.	Thank you for your positive review! We're glad to hear that your grandchildren are enjoying the Kindle Fire HD 8 for their gaming needs. If you have any friends or family who might also benefit from this product, we would greatly appreciate it if you could refer them to us. Thank you again for your support!

Writing good prompts is vital to generating accurate content. AI models use your prompts to respond. Therefore, poorly written prompts equals inaccurate responses.

Next, we will learn how to generate images using OpenAI’s DALL-E model.

Generating images using the DALL-E LLM

CREATE MODEL image_generation_model
PREDICT img_url
USING
   engine = 'openai',
   mode = 'image',
   prompt_template = '{{text}}, 4K | detailed and realistic portrait about money and finance |  natural lighting | different currencies';

Let’s generate an image using text. We are using the text “make money online” as our prompt.

SELECT * 
FROM image_generation_model 
WHERE text = 'make money online';

Output:

You can also generate batch images from texts inside your database.

Conclusion

Using Large Language Models in your database can help improve your product by helping you gain insights from data, make relevant predictions, understand user behavior, and generate contextually relevant human-like content. MindsDB allows you to build AI applications fast by simplifying the processes of using ML models inside your database. The models are designed to be production ready by default without the need for an ML Ops flow. MindsDB acts as a middleware for connecting your database with popular AI frameworks. It supports many ML engines, databases, and applications by default, making the process of building and deploying ML models easy. To see what’s possible for yourself, create a MindsDB free account and explore.

Do you need help creating technical content for your developer audience? Reach out and let’s work together.

Shameless Plug

If you find my content valuable and engaging, consider sharing it with your network and follow me here and on Twitter. It would mean a lot to me.

Emmanuel’s Substack | Emmanuel Aiyenigba | Substack

emmanuelthecoder.substack.com

How to automatically mark a conversation as "answered" in TalkJS

Emmanuel Aiyenigba — Wed, 05 Apr 2023 15:25:32 +0000

In a field where you have to respond to messages from your customers, you would want to remove answered chats from your inbox or maybe add a sign to indicate that you have responded to this customer. Doing so helps you stay organized and prioritized. TalkJS allows you to add custom data to every message. This custom data can help filter conversations in your inbox.

This article aims to teach you how to automatically mark a conversation as “answered" and filter out answered conversations from your inbox or indicate with a label if it is answered. A good use case for this is in the field of customer service, where customers can create support tickets regarding any issues they might have or inquiries they might want to make about your product. Support wants to know which messages they have answered to stay prioritized.

Pre-requisites

To follow along, you need:

A free TalkJS account.
Familiarity with JavaScript
Any code editor of your choice

Creating a conversation

Let’s add TalkJS to our application and start a conversation between a Support staff and two customers. Smith and John are customers chatting with Lynda, a support staff. Smith has a payment issue to resolve, while John wants to inquire about a product.

index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Auto mark conversation as read</title>
</head>
<body>
<script>
(function(t,a,l,k,j,s){
s=a.createElement('script');s.async=1;s.src="https://cdn.talkjs.com/talk.js";a.head.appendChild(s)
;k=t.Promise;t.Talk={v:3,ready:{then:function(f){if(k)return new k(function(r,e){l.push([f,r,e])});l
.push([f])},catch:function(){return k&&new k()},c:l}};})(window,document,[]);
</script>

    <div id="talkjs-container" style="width: 90%; margin: 30px; height: 500px">
      <i>Loading chat...</i>

    </div>

</body>
</html>

Let’s initialize TalkJS inside a JavaScript file.


Talk.ready.then(function() {

        const support = new Talk.User({
            id: '123456',
            name: 'Lynda from Support',
            email: 'lynda@test.app',
            photoUrl: 'https://talkjs.com/images/avatar-1.jpg',
            welcomeMessage: 'Hi! I am Lynda from support, how may I be of   help!',
            role: 'support'
        });

    window.talkSession = new Talk.Session({
        appId: 'YOUR_APP_ID',
        me: support,
    });
    const customer1 = new Talk.User({
        id: '654321',
      name: 'Smith',
      email: 'smith@example.com',
      photoUrl: 'https://talkjs.com/images/avatar-5.jpg',
      welcomeMessage: 'Hey, I have some troubles with...',
      role: 'customer'
    });

    const customer2 = new Talk.User({
      id: '1000001',
      name: 'John',
      email: 'john@example.com',
      photoUrl: 'https://talkjs.com/images/avatar-4.jpg',
      welcomeMessage: 'Hey, I want to make an inquiry...',
      role: 'customer'
    });

Replace YOUR_APP_ID with the App ID on your TalkJS dashboard. We added the customer and support roles we shall create in our dashboard shortly. The role attribute will help us determine whether a message is from support or a customer.

Creating a custom role and theme

Go to your TalkJS dashboard and create the two roles we assigned above.

At some point, we would want to display a label in the conversation list indicating whether a conversation is answered. To do this, we need to create a custom theme because we don’t want to modify the default theme.

Let’s create a custom theme and set our roles to use it. We will give it the name custom.

Set the customer and support roles to use the created custom theme.

Starting a conversation between support and customers

Let’s start a chat between our imaginary support staff, Lynda and Smith. And another chat between Lynda and John.

const conversation = talkSession.getOrCreateConversation(
        Talk.oneOnOneId(support, customer1)
        );
    conversation.setAttributes({
        subject: "Payment Issues"
    })
    conversation.setParticipant(support);
    conversation.setParticipant(customer1);

    const otherConversation =   talkSession.getOrCreateConversation(Talk.oneOnOneId(support, customer2));
    otherConversation.setAttributes({ subject: "Inquiry"});

    otherConversation.setParticipant(support);
    otherConversation.setParticipant(customer2);

The subject of the conversation between Lynda (support) and Smith (customer1) is “Payment Issues” while John (customer2) wants to make an “Inquiry.”

Create an inbox for the conversation between Lynda and Smith.

    const inbox = talkSession.createInbox({ selected: conversation});
    inbox.mount(document.getElementById('talkjs-container'));

Run your program! You should see something like this.

To display the conversation between Lynd and John, replace:

    const inbox = talkSession.createInbox({ selected: conversation});

With

    const inbox = talkSession.createInbox({ selected: otherConversation});

Now you have a conversation with John set up.

Listening to messages via webhooks

To know whether a message is from a customer or a support staff, we need to listen to it via the TalkJS webhooks. Integrate ngrok to test the webhooks locally. ngrok enables you to expose your local development server to the internet. Follow this guide to set up ngrok on your local machine.

Let’s spin up an Express.js server to use webhooks.

    mkdir chat-backend
    cd chat-backend
    npm install express axios

We need axios to make API calls.

Create an index.js file inside your Express project and add the following code

    const express = require('express');
    const app = express();
    const http = require('http');
    const server = http.createServer(app);

    app.use(express.json())

    app.post('/talkjs', (req, res) => {
      console.log(req.body)
      res.status(200).send('OK')
    });

    server.listen(8080, () => {
      console.log('listening on *:8080');
    });

“/talkjs” is the route that will receive the webhooks request. Go ahead and run ngrok on your machine.

Go to your TalkJS dashboard, and add the forwarding address from ngrok as your webhook URL. Don’t forget to add the “/talkjs” suffix. You should also enable message.sent as that is the only hook we need.

Now, we can receive sent messages in our backend via the webhook. Let’s test it out!

Lynda (support) sends John (customer2) a message.

If you navigate to the ngrok address at localhost:4040 you will get the message object.

You can also see this message object in the console of your server running on port 8080 (or whatever port you specified). We will update the message data from the backend via TalkJS REST API

Updating a conversation via REST API

Whenever the backend receives the message object, we want to update it to include custom data. Ensure to do this on the backend so as not to expose your application’s secret token - that would be a big security risk.

If the sender is a customer, we want to add a custom property {answered: “false”} to the conversation (custom property can also have string values hence the boolean in a string). But if the sender is Lynda, the support staff, we want to add a custom property {answered: “true”}.

const express = require('express');
const app = express();
const http = require('http');
const server = http.createServer(app);
const axios = require("axios");
require('dotenv').config()


app.use(express.json())




const token = process.env.SECRET_TOKEN
const appId = process.env.APP_ID


async function UpdateConversationforSupport(conversationId){


const supportData= {
  custom: {
    answered: "true"
  }
}


const config = {
  headers : {
    Authorization: `Bearer ${token}`
  }
}


  axios.put(
    `https://api.talkjs.com/v1/${appId}/conversation/${conversationId}`,
    supportData,
    config


  )

}




async function UpdateConversationforCustomer(conversationId){


  const customerData= {
    custom: {
      answered: "false"
    }
  }

  const config = {
    headers : {
      Authorization: `Bearer ${token}`
    }
  }

    axios.put(
      `https://api.talkjs.com/v1/${appId}/conversation/${conversationId}`,
      customerData,
      config

    )  
}


app.post('/talkjs', (req, res) => {
    console.log(req.body)
    res.status(200).send('OK')


    const {role} = req.body.data.sender
    const {conversationId} = req.body.data.message


    if(role == "support"){
      UpdateConversationforSupport(conversationId)
    }else{
      UpdateConversationforCustomer(conversationId)
    }
 });




server.listen(8080, () => {
  console.log('listening on *:8080');
});

Ensure to put your application’s secret key and token inside a .env file.

A custom data will be added to every message based on the sender's role. Let’s try it out.

Lynda sends another chat to John

If you go to our TalkJS dashboard, click on Activity, scroll down, and select the conversation.

You can see that the custom data was added to the conversation.

Filtering Inbox based custom data

As a support staff, you’d want to stay organized. You want every answered conversation to be automatically filtered out of your inbox so that you can easily prioritize the unanswered conversations.

To demonstrate the TalkJS FeedFilter method for filtering our inbox, Smith and John (our customers) will each send Lynda a message.
You can go to another page of your application, set it up and send a message as John and another as Smith.

To display both conversations in your inbox, replace:

    const inbox = talkSession.createInbox({ selected: otherConversation});

With

const inbox = talkSession.createInbox();

Now, we can see the messages from Smith and John.

Let’s make Lynda reply to John but not Smith.

We now have one answered conversation and one unanswered conversation. Let’s use the FeedFilter method to remove the answered conversation from the inbox.

 inbox.setFeedFilter({ custom: { answered: ["==", "false"] } })

Any conversation with custom property {answered: “true”} is removed from your inbox.

The conversation with John has been filtered out of the list because it is answered.

Adding a label to answered conversation

Alternatively, you may want to add a label to every answered conversation instead of filtering it out of your inbox. Let’s add “✔✔” to every answered conversation.
To do this, we need to go inside our TalkJS dashboard to edit the ConversationListItem subcomponent of our custom theme.

We have access to the custom data inside the ConversationListItem subcomponent. Let’s use it to display the “✔✔” label on answered conversations.

Scroll down to line 62. The next line after the following code

<span class="timestamp"><TimeAgo timestamp="{{lastMessage.timestamp}}"/></span>

Add

<span t:if="{{conversation.custom.answered == 'true'}}">✔✔ </span>

The above code adds “✔✔” to every answered conversation.

To see the effect, remove the FeedFilter to display all conversations inside the inbox.

Remove (or comment out)

 inbox.setFeedFilter({ custom: { answered: ["==", "false"] } })

Now we have the double check mark beside John’s conversation (because it is answered).

The check mark seems too close to the timestamp. Let's add some padding to it.
Scroll down to the styling, locate the .timestamp class and add a padding-right property.

.timestamp {
  padding-right: 10px;
}

Take a look!

Summary

TalkJS lets you listen to incoming messages via its webhooks. You can also add custom data to conversations. This custom data can be useful for conditionally filtering out conversations from your inbox using the FeedFilter method.

Automatically marking a conversation as answered can be helpful in customer service. Support staff stays organized and prioritized while serving their customers.

Making Something Waspy: A Review Of Wasp

Emmanuel Aiyenigba — Tue, 10 Jan 2023 16:07:53 +0000

A Short Story About The Reviewer

Developers all over the world are constantly in search of technologies that can make their work easier, faster, and better. They do not care much about what these technologies are made from and only a few bother to check how it works under the hood. When I started my journey as a developer some 5 years ago, like almost everyone else, I was searching for tools that can make my journey easy and fast. I wanted to be able to do more with less, this made me constantly search for tools and technologies that can help me build UIs better as a frontend developer (I started my tech journey as a frontend developer), something outside of the conventional HTML, CSS, and vanilla JavaScript approach. I found React to be what I was looking for at the time.

A while after, I decided to further my journey and become a full-stack developer. Since I was in love with JavaScript and Typescript, Node.js became my go-to tool for building APIs.

Connecting the frontend to the backend every time I needed to build a full-stack application was a real struggle. An even bigger struggle was having to deal with boilerplates and deployment configs.

I wanted a tool that can do the heavy lifting for me so that I can focus more on building the actual application. This led me to discover Wasp, a DSL for building full-stack web applications.

How I Came Across Wasp

It was near mid-October in the afternoon when a thought flashed through my mind “oh, Emmanuel! Hacktoberfest is on and you are yet to look for an open-source project to contribute to.” I love contributing to open-source and helping developer communities (I think this is where my passion lies - helping developers). After realizing that I am yet to pick open-source projects to contribute to, I set out to look for projects for Hacktoberfest.

I picked a few of them on the hacktoberswag.com website. Precisely three: pusher.js, refine.dev, and MindsDB. You are probably asking Oh, you did not pick Wasp? The thing was, Wasp wasn’t listed on that web page and I didn’t know if any tool by that name existed.

How did you discover Wasp?

I wish I could remember what took me to YCombinator's website on the 10th of October, 2022. That was when I first heard about Wasp and another language called DarkLang. After I learned about Wasp, I was intrigued and curious to know how it works, which led me to join the discord server the next day.

At this time, I had a number of open source projects to select from for Hacktoberfest: Pusher, Refine, MindsDB, Wasp, and DarkLang.

Why did I pick Wasp?

What intrigued me about Wasp is how it simplifies building full-stack web applications. It let’s you use React, Node.js and Prisma to build full-stack web applications - you don’t need to learn anything you’re not already familiar with. Writing deployment configs and boilerplate code was a pain point for me as a full-stack engineer, but seeing a tool that solves this pain caught my eye, so I felt it deserved my attention more. By the 14th of October, I had started contributing to the Wasp project. These contributions continued after Hacktoberfest. This is due to two reasons: the project was interesting because it addresses a pain point I was having, so it makes sense to contribute to it, and the warmth received from the Wasp team was super great. The team was very welcoming and open to receiving feedback from the community.

What I Built During The Wasp Hackathon

I may have forgotten to mention that when I started contributing to Wasp, the project was still at Alpha. As you’d guess, it was quite buggy and wasn’t fit for use in a production application. About a month later, Wasp beta was launched. This launch fixed a lot of bugs, added new features, and provided support for tools that developers need. At the end of the launch week, Wasp betathon was announced - a hackathon for building any project with Wasp for 10 days.

I built BetaNotes, an application for taking notes during the Wasp hackathon. This was my first time building a real project with Wasp. The application can do what any other notes app can do. Users are properly authenticated and authorization to make, edit and delete notes upon successful authentication. User records are well managed. All these were done with Wasp in only a few hours - I joined the hackathon less than 48 hours before the submission deadline. Wasp made it look easy because it handled the hard parts for me. For example, username/password authentication took less than 7 lines of code to implement.

You can find the code for this application on GitHub.

What I Think About The Future Of Wasp

Of a truth, Wasp is not there yet. There is still a lot of work to be done to make it better. For example, server-side rendering (SSR), static site generation (SSG), framework agnosticism, etc. are not a part of Wasp yet. However, I have no doubt that Wasp has a very bright future in the web application development domain. It is tactically solving problems that have been overlooked and left unsolved in the web domain for many years. Developers tend to spend a lot of time on configurations, connecting clients to servers, writing boilerplate code, setting up deployment configs and databases, etc. All these eat up development time and slow down the delivery of applications to the finish line.
I believe that in a few years from now, Wasp will be the go-to tool for many developers building full-stack web applications. When this happens, you can revert to this article and call me a Prophet of tech tools 😆😉. I am glad to have been a part of this project, and I look forward to making more contributions as an open-source lover or as a Wasp team member in the future (this shows how much I believe in the Wasp project).