Forem: Chris McKenzie

Real-Time Face Tracking in the Browser with MediaPipe

Chris McKenzie — Mon, 21 Jul 2025 16:06:31 +0000

Google MediaPipe

Google MediaPipe is a suite of libraries and tools that make it very simple to drop ML into apps — supporting vision, text and audio tasks — without needing to be an ML expert or spin up expensive cloud infrastructure. It runs fast, on-device, and gives you tools to build interactive experiences that work in the real world.

The standout feature: on-device inference with support for Android, iOS, Python, and the web. That means no round-trips, no user data sent to servers, and minimal latency.

MediaPipe supports a wide range of use-cases, such as LLM inference, Object detection, gesture recognition, and much more. This would be an epic if I tried to demo the entire API. To keep it focused, I’ll demo the Face Landmark detection. For more, see the docs

Face Landmark

MediaPipe’s Face Landmarker lets you track 3D face landmarks and expressions in real time — whether from single frames or live video. You get blendshape scores for expressions, 3D points for facial geometry, and matrices for applying effects. Great for filters, avatars, or anything that takes facial input.

This demo uses the BlazeFace (short-range) model, which is optimized for selfie cameras.

At the time of writing, BlazeFace (short-range) is the only model available for this task, but BlazeFace (full-range) and BlazeFace Sparse (full-range) are coming soon, and may be worth checking out if the BlazeFace (short-range) doesn’t work for your use case.

Demo

Try the working demo here:
👉 https://monkey-ears-filter.vercel.app

or run locally:

Clone repo git clone git@github.com:kenzic/monkey-ears-filter.git && cd monkey-ears-filter

Start server npm run start → Then open http://127.0.0.1:3030/

In the monkey-ears-filter/public folder, the core logic is in filter.js.

What the code does

Imports & Constants

Pulls in FaceLandmarker & FilesetResolver from MediaPipe’s vision tasks.
Defines landmark indices for outer eyes and ears.

DOM References

Gets video and canvas elements and 2D context, setting up an overlay on the webcam feed.

Ear Image Loader

makeEar() returns a new transparent ear image each call

Webcam Setup (setupCamera)

Requests user media, attaches it to the video element, and waits for metadata before proceeding.

Model Initialization (loadFaceLandmarker)

Loads WASM runtime via FilesetResolver.forVisionTasks(…).
Creates a FaceLandmarker in LIVE_STREAM mode: GPU delegate, up to 2 faces, outputs blendshapes .

Tilt Calculation (getRollAngle)

Computes head tilt using atan2 of two eye landmarks.

Ear Overlay (drawEars)

Positions, rotates, and draws mirrored ear images at detected ear landmark positions.

Render Loop (render)

Matches canvas size to video, runs faceLandmarker.detectForVideo(…), overlays video + ear images, and loops via requestAnimationFrame.

Startup (main)

On button click: hides UI, starts camera, loads model, and begins the render loop.

The main components of this app: loading model, handling the overlay, and rendering it to the screen.

🧠 Model Initialization

This part initializes WebAssembly and the face landmarking model:

async function loadFaceLandmarker() {
  const filesetResolver = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3/wasm"
  );
  faceLandmarker = await FaceLandmarker.createFromOptions(filesetResolver, {
    baseOptions: {
      modelAssetPath: "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task",
      delegate: "GPU",
    },
    outputFaceBlendshapes: true,
    runningMode: "LIVE_STREAM",
    numFaces: 2,
  });
}

FilesetResolver.forVisionTasks(…) downloads MediaPipe’s WASM runtime optimized for your browser.

createFromOptions(…) sets:

baseOptions.delegate: “GPU” → uses WebGL/WebGPU to accelerate inferences.
outputFaceBlendshapes: true → returns blendshape coefficients for expression data.
runningMode: “LIVE_STREAM” → asynchronous video mode for real-time streams.
numFaces: 2 → allows up to two faces to be tracked.

Result: faceLandmarker is now a live-stream-ready tracker with GPU acceleration, expressive blendshape output, and capable of handling up to two faces.

🎧 Ear Overlay

function drawEars(landmarks) {
  const img = makeEar(); // fresh Image each frame
  const left = landmarks[LANDMARKS.LEFT_EAR];
  const right = landmarks[LANDMARKS.RIGHT_EAR];
  const angle = getRollAngle(landmarks); // gets head tilt
  const w = 100, h = 100;
  ctx.save();
  ctx.translate(left.x * canvas.width - 26, left.y * canvas.height - 10);
  ctx.rotate(angle);
  ctx.drawImage(img, -w/2, -h/2, w, h);
  ctx.restore();
  ctx.save();
  ctx.translate(right.x * canvas.width + 26, right.y * canvas.height - 10);
  ctx.rotate(angle);
  ctx.scale(-1, 1); // mirror image
  ctx.drawImage(img, -w/2, -h/2, w, h);
  ctx.restore();
}

Positioning: landmarks are normalized ([0,1]), so we multiply by canvas dimensions.
Tilt rotation: angle (via atan2) tilts the ears to match your head roll.
Mirroring right ear: we invert with scale(-1,1) so ears attach with correct orientation.

function getRollAngle(landmarks) {
  const A = landmarks[LANDMARKS.RIGHT_EYE_OUTER];
  const B = landmarks[LANDMARKS.LEFT_EYE_OUTER];
  return Math.atan2(B.y - A.y, B.x - A.x);
}

🎯 Render Loop

async function render() {
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;
  const res = await faceLandmarker.detectForVideo(
    video, performance.now()
  );
  ctx.clearRect(0,0,canvas.width,canvas.height);
  ctx.drawImage(video, 0,0);
  if (res.faceLandmarks?.length) {
    drawEars(res.faceLandmarks[0]);
  }
  requestAnimationFrame(render);
}

Sizing: ensures canvas matches video.
Inference: uses detectForVideo(), feeding current frame + timestamp for the live-stream model.
Drawing: clears canvas, redraws video, and overlays ears if at least one face is detected.
Looping: requestAnimationFrame(render) makes it go again, achieving real-time performance.

Final Thoughts

MediaPipe doesn’t get much hype, but it’s one of the most practical tools for real-time ML on-device. No servers. No latency. No user data handoffs just to track a face. That’s a big deal for privacy, performance, and reliability.

You can still use whatever backend stack you like — but with MediaPipe, you don’t have to. This changes the equation: local-first ML is finally viable. This demo barely scratches the surface. MediaPipe supports gesture detection, object tracking, and even early LLM integration. If you’re building interfaces that respond to people in real time, this deserves a place in your toolbox.

To stay connected and share your journey, feel free to reach out through the following channels:

👨‍💼 LinkedIn: Join me for more insights into AI development and tech innovations.
🤖 JavaScript + AI: Join the JavaScript and AI group and share what you’re working on.
💻 GitHub: Explore my projects and contribute to ongoing work.
📚 Medium: Follow my articles for more in-depth discussions on the intersection of JavaScript and AI.

Bringing AI to the Browser: Transform Your Web App with On-Device Models

Chris McKenzie — Sun, 20 Jul 2025 01:39:02 +0000

TL;DR

Tired of bleeding money on AI API costs? Here’s the fix: run AI models directly in your users’ browsers. I’ve built Browser.AI, a prototype that lets you tap into on-device AI through a simple window.ai API — think localStorage, but for AI. No more API fees, better privacy, and faster performance. Users get control over their data and which models to run, while you get to scale your AI features without scaling costs. Best part? It works offline and isn’t locked to any specific vendor. Check out the prototype, contribute to the W3C proposal, and let’s make AI more accessible for developers everywhere.

Key points:

🚫 No more API costs
🔒 Privacy-first (data stays on device)
⚡ Better performance (no server roundtrips)
🔌 Works offline
🎮 Users control their models
💻 Simple window.ai API

Intro

You’ve built something awesome — an AI-powered app for your startup or passion project. At first, you lean on OpenAI’s API to get it out the door quickly, and things go well. You gain users, feedback is positive, and it feels like you’re on the right track. But then reality hits: the API costs are ballooning, and each new user brings you closer to burning through your budget. Sound familiar?

This is the problem every small team building AI apps eventually faces: powerful AI capabilities come with a steep price tag. You can’t keep paying these rising costs without killing your runway. But what’s the alternative? Do you compromise on features, slow down your roadmap, or drop out of the AI race altogether?

No. There’s a smarter solution — on-device AI models.

By shifting AI workloads directly onto your users’ devices, you can eliminate those crushing API costs, speed up your app, and give users more control over their data. Best of all, this approach makes your product more scalable, with zero trade-offs in performance. You keep your hard-earned dollars, your users keep their privacy, and everyone wins.

In this article, I’m going to break down the why and how of on-device AI, walk you through a prototype I built called Browser.AI, and show you how this approach is set to transform the way developers integrate AI into their web apps.

The Big Idea

Imagine tapping into powerful AI models directly from the browser, using an API as familiar as **window.fetch** or **window.localStorage**. No more server-side processing, no more exorbitant API fees. Just seamless, on-device AI that you control.

Here’s what that could mean for you:

Unified API Access: A standard window.ai API that works across all browsers.
Model Flexibility: Support for open-source models, so you’re not locked into any vendor.
User Empowerment: Users can choose which models they enable, enhancing privacy and control.

Just as the MediaDevices API provides you access to hardware like the camera or microphone, the window.ai API could provide access to AI models installed on the user's device.

Who Benefits

This approach benefits anyone building AI-powered apps, from indie developers to large engineering teams, along with their users. Here’s how:

Developers Get:

Cost Efficiency: Say goodbye to skyrocketing API bills. By running AI models directly in the user’s browser, you’re offloading processing to their hardware — at zero extra cost to you.
Improved Performance: No more latency from server round-trips. On-device AI can make your app snappier, providing a better user experience.
Scalability: With the heavy lifting done on the client side, scaling your app to more users doesn’t mean increased server costs.
Flexibility and Control: You’re not tied to a specific AI provider. Use open-source models and switch them out as you see fit.

Users Get:

Enhanced Privacy: Data stays on their device, reducing concerns about data being sent to third parties.
Offline Functionality: Features work even without an internet connection — think editing a document on a plane with full AI assistance.
Control Over Resources: Users can decide which models to enable, giving them more say in their experience.

You cut costs, and ditch third-party dependencies. Users get data control and a smoother experience.

How It Works

Think of this like accessing speech synthesis or camera features through browser APIs, but for AI models. With processors now optimized for on-device AI, local models like LLMs and CNNs can power features directly on users’ devices, much like Apple, Microsoft, and Google are already doing.

Browser-Level Integration: The idea is for an API, like window.ai, built right into the browser to access AI models on the user’s device.
Model Management: Users manage which AI models are installed and control site access, similar to camera or microphone permissions.
Cross-Browser Compatibility: A standardized API means developers write code once, and it works across all browsers, regardless of the specific AI model in use.

Technical Details

JavaScript API: The window.ai API provides methods for permissions, model info, and creating sessions.
Permissions: Before accessing a model, your app requests permission from the user.
Sessions: Once permission is granted, you establish a session with the model to perform tasks like text completion or embeddings.

When This Could Happen

Progress is already underway — I’ve submitted a proposal to the W3C, and if this idea resonates, I’d love your feedback. Google Chrome is also exploring this space, but I see some gaps in their approach, which I’ll get into later.

Alternatives

Before we dive into Browser.AI, let’s look at existing options and where I see their limitations:

window.ai Extension

I really like this project — they’re doing a lot of smart things, especially in giving users control over which models they use. Definitely worth checking out.

Pros:

Gives users control over which models to use.
Supports both local and remote models.

Cons:

Adoption Barrier: Users need to install an extension — not ideal for mass adoption.
Technical Complexity: Geared towards users comfortable with setting up API keys and local models.
Mobile Limitations: Extensions aren’t available on mobile browsers.

Chrome window.ai Implementation

What It Is: An experimental API built into Chrome, offering basic AI features.

Pros:

Integrated directly into the browser — no extension required.

Cons:

Limited Models: Uses Google’s own models, which tend to perform poorly compared to alternatives.
Ecosystem Fragmentation: Risks each browser supporting only their preferred models.
Lack of Flexibility: Limited control for developers and users over which models to use.

transformers.js

What It Is: A JavaScript library that runs AI models in the browser using WebAssembly.

Pros:

Runs entirely in the browser — no server required.
Leverages open-source models.

Cons:

Redundant Downloads: Each website must package and serve the model, leading to duplicate downloads.
Performance Overhead: Large models can impact load times and performance.
Complex Integration: Managing models and performance optimization can be challenging.

While I do think there are downsides, it’s still a great project. If you’re interested in experimenting with this approach, you can check out my article, Run Models in the Browser With Transformers.js

Introducing Browser.AI

The future of AI on the web isn’t about relying on costly third-party services. It’s about putting power back in the hands of developers and users. That’s where Browser.AI comes in.

It’s designed around three key principles:

Keep it simple: An intuitive API that feels natural for JavaScript developers.
Flexibility and control for users and developers: Users choose which models to allow, and developers pick the models that fit their apps — ensuring cross-browser compatibility and avoiding vendor lock-in. a.k.a., model-agnostic.
Boost performance and privacy: Running AI locally reduces latency, keeps data private, and offline functionality for a smoother experience.

Browser.AI is a working prototype that demonstrates how developers can run AI models directly on users’ devices through an API on the window object. This approach enhances privacy, speed, and offline capabilities for web apps.

Developing with the API

Try it firsthand, you can download the app, or clone the repo.

For more information on the architecture, see the README in the project repo. The API is as follows:

Browser.AI API

Permissions API

The Permissions API allows developers to request permission for using specific AI models and see which models are already enabled.

Request permission: window.ai.permissions.request({ model: 'name', silent: true }) Prompts the user to grant permission for a model (e.g., 'llama3.2'). Returns true if approved.
List enabled models: window.ai.permissions.models() Retrieves a list of models the user has enabled.

Model API

The Model API allows developers to connect to a specific AI model and retrieve information.

Get model info: window.ai.model.info({ model: 'name' }) Provides details about the specified model.
Connect to model: window.ai.model.connect({ model: 'name' }) Establishes a session with the model, enabling further interaction.

Session API

Once connected to a model, the Session API lets you interact with the model via chat or embeddings.

Chat: session.chat({ messages: [{ role: 'user', content: 'hello' }] }) Sends a message to the AI model and gets a response.
Embed: session.embed({ input: 'text to embed' }) Generates embeddings from the input text.
More coming soon

Simple Code Example

if(await window.ai.permissions.request({ model: 'llama3.2', silent: true })) {
    const session = await window.ai.model.connect({ model: 'llama3.2' });
    const response = await session.chat({ messages: [{ role: 'user', content: 'hello' }] })
    console.log(response)
}

Next Steps

Try the Prototype: Download the Browser.AI prototype and integrate AI features like chat or text completion into your projects. You’ll find instructions and examples in the README. Hands-on experimentation with on-device models will show you how they can enhance your apps.
Share Feedback on the WICG Proposal: Check out the WICG proposal for Browser.AI and share your thoughts on the technical implementation, privacy, and cross-browser compatibility. Your feedback will help refine the API’s design, so weigh in on key areas like model permissions and session handling.
Contribute to the Project: If you’re a developer, contribute to Browser.AI on GitHub — improve the code, suggest features, or fix bugs. Non-coders can help by testing the prototype, reporting issues, or suggesting ideas for improve user control. Every contribution counts.

Final Thoughts

AI is transforming how we build and interact with web applications, but the costs and dependencies on third-party providers can be a huge barrier for small teams and indie developers.

On-device models offer a way out — a chance to cut costs, reduce latency, and give users more control over their data. Browser.AI is just one approach to making this vision a reality. By providing a simple, standardized API, we can empower developers to scale their AI-powered features without relying on expensive services, all while prioritizing performance and privacy for users.

This is fertile ground. There’s a ton of potential here and with feedback from the community, we can build something that works for both developers and users. Whether you’re just getting into AI or trying to scale your product without breaking the bank, jump in, test out the prototype, and let me know what you think. Let’s make this happen together.

Resources

To stay connected and share your journey, feel free to reach out through the following channels:

👨‍💼 LinkedIn: Join me for more insights into AI development and tech innovations.
🤖 JavaScript + AI: Join the JavaScript and AI group and share what you’re working on.
💻 GitHub: Explore my projects and contribute to ongoing work.

Getting Started: Build a Model Context Protocol Server

Chris McKenzie — Mon, 02 Jun 2025 15:48:02 +0000

Streamlining LLM Integration: Building a JavaScript MCP Server for Hacker News

Integrating LLMs into real products still feels messier than it should be. Instead of clean patterns or shared infrastructure, most devs end up hacking together one-off code to connect models with APIs, databases, or business logic - none of it reusable, scalable, or easy to maintain.

The Model Context Protocol (MCP) aims to fix that. It's a minimal, open standard from Anthropic. It provides a unified way for exposing tools, data, and prompts to language models in a structured, predictable way. Instead of building a new integration layer for every app or agent, MCP gives you a common interface - and it already works with applications like Claude Desktop, Cursor, and Windsurf.

In this article, we'll walk through the basic steps of building a simple MCP server in JavaScript that will expose two tools for interacting with Hacker News.

TL;DR on Model Context Protocol

If you're already familiar with MCP, you can skip ahead to the "Demo" section.

The Model Context Protocol (MCP) is an open standard for how applications provide data and tools to an LLM. Think of it like an electrical outlet for AI - one universal way to connect models to content, business tools, standard prompts. Instead of custom integrations for every data source, MCP enables secure, scalable, and efficient AI connections, so developers can focus on building amazing products instead of maintaining endless adapters.

Clients

MCP clients sit between your app (like a chatbot or AI-powered IDE) and an MCP server. They handle the connection, manage discovery and security, and translate the app's requests into something the server can understand. Popular clients include Claude Desktop, Cursor, and Windsurf. You can find the full list in the docs.

Servers

An MCP server connects to your data - files, APIs, databases - and exposes it to the model through a standard interface. It can provide tools (functions the model can call), resources (data the model can read), and prompts (predefined inputs to guide model behavior).

Under the hood, both clients and servers use JSON-RPC 2.0 for communication and support multiple transport layers like stdio and HTTP via Server-Sent Events (SSE).

You can browse the server implementations on GitHub, or dive deeper with the Getting Started guide or the official docs.

Key Concepts

Tools - Functions exposed to the model - can be called by the LLM
Resources - Files, APIs, or datasets the LLM can access
Prompts - Predefined templates that help the LLM act consistently

For this demo, we'll focus on tools.

Demo: Build a Hacker News MCP Server

Let's build an MCP server that exposes two tools:

list-top-stories – returns top 10 Hacker News posts
story-details – fetches detailed info on a selected post

We'll use @modelcontextprotocol/sdk and wire it up with a simple HN API client.

I've provided a starting point which handles some of the boilerplate of getting a project up and running, as well as a simple Hacker News API Client so we can fetch the data.

Step 1: Setup Starter Project

Clone the repo
git clone git@github.com:kenzic/mcp-hacker-news-demo.git
The main branch contains the final project, so we'll switch to the demo branch for our starting point.
cd mcp-hackernews-demo && git checkout -b origin/demo demo
Finally, let's install the dependencies
npm i

Now that we have the boilerplate setup, we're ready to dive in. Open the project in your editor, and notice two files: index.js and hacker-client.js. The client is complete as is, but I encourage you to build on top of it to add new functionality. The index.js file is where we'll be doing all our coding.

Step 2: Build Our Server

For this, we're going to use McpServer from the modelcontextprotocol SDK. The SDK supports: Resources, Tools, Prompts. However for this demo we'll only need tools.

💡 If you need more control over the server implementation you can use the Server class.

At the top add the following imports:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";

Next, create a new instance of the server:

const server = new McpServer(
    {
        name: "Hacker News",
        version: "1.0.0",
    },
    {
        capabilities: {
            logging: {},
        },
    }
);

Step 3: Create Tools

In this demo we'll be crating two tools: List Top Stories, and Story Details.

List Top Stories provides the MCP client with a list of the top 10 stories from Hacker News.
Story Details provides in-depth details of the Hacker News post, including the contents of the url the post links to.

To create a tool, simply add the following code:

server.tool("TOOL_NAME", { toolParam: z.string() }, async () => {
    // logic
    return {
        content: [{ type: "text", text: "RETURN TEXT" }],
    };
});

As defined above, this tool isn't very helpful.

💡 For both tools we'll be using the content return type of text, however, the specifications support: text, image, resource. Additionally, you can return multiple objects - even of different types.

List Top Stories Tool: Let's update the tool to support listing the top stories on Hacker News. We'll start by giving it an easy-to-understand name: list-top-stories. Next, we'll want to fetch the top stories and format the results as text:

const topStories = await client.listTopStories();
const formattedTopStories = formatTopStories(topStories);

Lastly, we'll return the results.

The final code should look something like:

server.tool("list-top-stories", {}, async () => {
    const topStories = await client.listTopStories();
    const formattedTopStories = formatTopStories(topStories);
    return {
        content: [{ type: "text", text: formattedTopStories }],
    };
});

Great! Now that we have a list of the top ten stories on Hacker News, let's build a tool that will allow us to retrieve the details of the story.

Story Detail Tool: We'll create a new tool called story-details, then use the getStoryDetails method on the Hacker News client to fetch and format the story. But how does it know which story to fetch? MCP tools let you define params. The second argument to tool takes an object with the params name and types. We'll add the param { id: z.number() } so the client can pass the article id to the tool. Your final code should look something like:

server.tool("story-details", { id: z.number() }, async ({ id }) => {
    const story = await client.getStoryDetails(id);
    const formattedStory = formatStoryDetail(story);
    return {
        content: [{ type: "text", text: formattedStory }],
    };
});

Transports: We're almost done! Now we need to provide a method for our server to communicate with the client. As of writing, MCP SDK supports two modes of communication: Standard Input/Output (stdio) and Server-Sent Events (SSE). For this demo we'll use stdio.

💡 See the docs for more info on the protocols, or how to define a Custom Protocol.

Start by importing StdioServerTransport at the top of your file:

import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";

Next, we'll create an instance of StdioServerTransport and connect our server. Drop the following at the bottom of the index.js file:

const transport = new StdioServerTransport();
await server.connect(transport);

Boom - you've created a valid MCP server.

Connecting our server to a client

That's it - we've got a working MCP server. Now we just need to tell a client where to find it. If you're using Claude Desktop, follow the steps below to connect it. If not, skip ahead to the Debugging section to inspect your server using the MCP Inspector tool.

⚠️ We're running this server locally. You may notice most servers are configured to run using npx, but we'll be using node to run the server locally. Because of this you'll need to make sure you npm install the dependencies (which you already did if you followed the setup instructions).

Claude Desktop: In Claude, go to your settings, then Developer tab and click Edit Config. Once there add the "hackernews" config:

{
    "mcpServers": {
        "hackernews": {
            "command": "node",
                "args": [
                    "/path-to-folder/mcp-hacker-news-demo/index.js"
                ]
            }
        }
    }
}

Restart Claude, and prompt,

"tell me about the top stories on hacker news."

If everything worked you should (after giving permission to use the tool) see a list of stories provided by the server. Next, you can ask it to tell you more about a specific post.

Debugging & Dev Tools

As important as knowing how to build something is knowing how to fix it when it's not working as expected. I'll cover a few methods for debugging, however, this is not exhaustive.

Logging

Adding logging to your server is straightforward. To log a message, simply call server.server.sendLoggingMessage.

💡 You might notice we type server twice. This is because McpServer does not expose a method for logging, and we must rely on the low-level implementation of Server, which is accessible on the property server

Creating logs

server.tool("list-top-stories", {}, async () => {
    try {
        const topStories = await client.listTopStories();
        const formattedTopStories = formatTopStories(topStories);

        // Add logging
        server.server.sendLoggingMessage({
            level: "info",
            data: `Fetched ${topStories.length} top stories.`,
        });

        return {
            content: [{ type: "text", text: formattedTopStories }],
        };
    } catch (error) {
        server.server.sendLoggingMessage({
            level: "error",
            data: `Failed to fetch stories: ${error.message}`,
        });
        throw error; // Inform the client about the failure
    }
});

Reading Logs: If you're using the Claude Desktop app, you can view all event messages in the logs folder:

tail -n 20 -F ~/Library/Logs/Claude/mcp-server-hackernews.log

MCP Inspector

The MCP Inspector is a simple GUI devtool for testing and debugging MCP servers.

To run the inspector on your Hacker News server, simply run:

npx @modelcontextprotocol/inspector node index.js

Once it's ready, open the url in your browser

Connect and click List tools.

Click list-top-stories and then Run Tool. Grab an ID from the results.

Click story-details, then drop in the ID and click Run Tool

If everything worked you should see the results, and the history of calls. You can expand the history items to see the detailed view of the request and response.

Next Steps

Want to keep building?

Add support for Ask HN and Polls Right now we're only pulling top stories, but Hacker News has other types of content like Ask HN posts and polls. Add endpoints to fetch those and expose them as new tools - same pattern, just different data.
Let the user choose how many stories to list Instead of hardcoding 10 stories, update the list-top-stories tool to accept a count parameter. It's a small change, but it makes the tool more flexible and helps illustrate how you can pass arguments into your MCP tools.
Explore Resources and Prompts This demo focused on tools, but MCP also supports resources (like documents or database queries) and prompts (predefined input templates). Try exposing a local file or API response as a resource, or create a prompt that wraps HN data in a consistent format the model can use.

Final Thoughts

MCP is one of those things that just makes sense once you try it. Instead of hacking together context with custom glue code, you get a simple, predictable way to expose tools, data, and prompts to language models - without reinventing the wheel every time.

This demo is intentionally simple, but the same approach scales. Want to let a model trigger internal workflows? Query production data? Run functions securely? MCP gives you the interface without locking you into someone else's stack.

If you're building anything serious with LLMs, it's worth getting familiar with the protocol now. The ecosystem's moving fast - and it's a lot easier to build the right abstractions when you start with the right foundation.

To stay connected and share your journey, feel free to reach out through the following channels:
👨‍💼 LinkedIn: Join me for more insights into AI development and tech innovations.
🤖 JavaScript + AI: Join the JavaScript and AI group and share what you're working on.
💻 GitHub: Explore my projects and contribute to ongoing work.
📚 Medium: Follow my articles for more in-depth discussions on LangSmith, LangChain, and other AI technologies.

Transform Your Workflow with LangSmith Hub: A Game-Changer for JavaScript Engineers

Chris McKenzie — Thu, 05 Sep 2024 01:15:32 +0000

Are scattered AI prompts slowing down your development process? Discover how LangChain Hub can revolutionize your workflow, making prompt management seamless and efficient for JavaScript engineers.

Introduction

Imagine managing a project with crucial information scattered across files. Frustrating, right? This is the reality for developers dealing with AI prompts. LangChain Hub centralizes prompt management, transforming workflows just as GitHub did for code collaboration.

LangChain Hub provides an intuitive interface for uploading, browsing, pulling, collaborating, versioning, and organizing prompts. This not only streamlines workflows but also fosters collaboration and innovation, making it an essential tool.

Key Features and Benefits

LangChain Hub is a powerful tool designed for JavaScript developers to centralize, manage, and collaborate on AI prompts efficiently.

Community-Driven Innovation

Explore prompts from other developers, gaining new ideas and solutions. Learn new techniques, improve existing prompts, and foster a collaborative environment.

Centralized Prompt Management

LangChain Hub brings all your AI prompts under one roof, eliminating the chaos of scattered files and fragmented storage. With everything neatly organized in one place, managing your prompts has never been easier.

User-Friendly Interface

Navigating LangChain Hub is a breeze, thanks to its intuitive design. Uploading, browsing, and managing your prompts is straightforward, boosting your productivity and minimizing the time spent on learning the tool.

Collaboration and Sharing

LangChain Hub makes it simple to share and collaborate on prompts with your team. This seamless sharing fosters innovation and collective problem-solving, making teamwork more efficient and effective.

Version Control

Never lose track of your prompt iterations with LangChain Hub's version control. You can easily revert to previous versions or monitor changes over time, ensuring you always have access to the best version of your prompt.

Enhanced Search and Filtering

Find the prompts you need in no time with advanced search and filtering options. You can filter prompts by use-case, type, language, and model, ensuring you quickly access the most relevant resources. These features save you time and enhance your workflow, making prompt management more efficient and tailored to your specific project needs.

Customization and Flexibility

Tailor prompts to your specific project requirements effortlessly. LangChain Hub's customization options ensure your prompts fit seamlessly into your development process, adapting to your unique needs.

Using in Your Project

Let's set up a project to use prompt templates in LangChain Hub to highlight its value.
We'll start by using the demo project I created for the article Getting Started: LangSmith for JavaScript LLM Apps. While I encourage you to read that article, it's not required to follow along.

Clone repo: git clone git@github.com:kenzic/simple-langsmith-demo.git
cd simple-langsmith-demo
Checkout the demo branch: git checkout -b langchain-hub-demo origin/langchain-hub-demo
Install dependencies yarn or npm i
Sign up for LangSmith Account
Get a LangSmith API Key
Get OpenAI API Key
Move .env.example to .env and fill in the following values:

LANGCHAIN_PROJECT="langsmith-demo" # Name of your LangSmith project
LANGCHAIN_TRACING_V2=true # Enable advanced tracing features
LANGCHAIN_API_KEY=<your-api-key> # Your LangSmith API key

OPENAI_API_KEY=<your-openai-api-key> # Your OpenAI API key

The demo app responds to the question "What is the capital of France?" in the voice of Mr. Burns from the Simpsons. To accomplish this we use the following prompt:

Act as a world-class expert in the field and provide a detailed response to the inquiry using the context provided.
The tone of your response should be that of The Simpsons' Mr. Burns.

<context>
{context}
</context>

The prompt is currently hardcoded in the app, which is manageable for now. However, in a real-world application, this approach can become difficult to manage. As we add more steps and multiple prompts to the chain, it can quickly become confusing and hard to maintain. Therefore, let's move our prompt to LangChain Hub.

If you followed the steps above, you should have a LangSmith account.

Go to smith.langchain.com/hub and click "New Prompt."

You'll then want to give your prompt a name, set visibility (default private), description, readme, use case, language, and model. Note: the owner is "@kenzic", this will be different for you. See the screenshot for values.

Once you've created your prompt, you'll want to select the prompt type. For this task, we'll select "Chat Prompt".

Create a "System" message with the value:

Act as a world-class expert in the field and provide a detailed response to the inquiry using the context provided.
The tone of your response should be that of The Simpsons' Mr. Burns.

<context>
{context}
</context>

Next, create a "Human" message with the value:

Please address the following inquiry:\n{input}

Before we commit this, we can test it out in the playground. To the right of the message chain, you will notice the section "Inputs" with the variables we specified in the messages. To confirm it's working as expected, I tested with the following:

context: The capital of France is Springfield. It was Paris but changed in 2024.

input: What is the capital of France

Once you have the Inputs defined, under Settings you'll want to select the model we're testing against. Select GPT-3.5-turbo. For this to work you'll need to add your OpenAI API key by clicking the "Secrets & API Keys" button. Great, now we're ready to test. Click the "Start" button and watch it generate the output. You should see something like:

Ah, yes, the capital of France, or should I say, Springfield! Paris may have been the capital in the past, but as of 2024, Springfield reigns supreme as the new capital of France. A change of this magnitude surely raises questions and eyebrows, but rest assured, the decision has been made and Springfield now holds the title of the capital of France. How utterly delightful!

Once we're happy with our prompt, we need to commit it. Simply click the "Commit" button!

Great, now that we have a finished prompt we'll want to update our code to reference it instead of the hardcoded prompt template.

First, we need to import the hub function to pull our template into our code:

import * as hub from "langchain/hub";

Next, let's delete the ChatPromptTemplate in the code and replace it with:

const answerGenerationChainPrompt = await hub.pull(
  "[YOURORG]/mr-burns-answer-prompt"
);

Note: You can delete the ANSWER_CHAIN_SYSTEM_TEMPLATE variable too

Finally, let's test it out! run yarn start to execute the script. If everything works properly, you will see the output in the voice of Mr. Burns informing you the capital of France is Paris.

If you want to take it a step further, you can lock your prompts by the version. To do this, simply append a colon and the version number to the end of the name like so:

const answerGenerationChainPrompt = await hub.pull(
  "[YOURORG]/mr-burns-answer-prompt:[YOURVERSION]"
);

// for me it looks like:
const answerGenerationChainPrompt = await hub.pull(
  "kenzic/mr-burns-answer-prompt:d123dc92"
);

That's it!

Next Steps

We've explored how LangChain Hub centralizes prompt management, enhances collaboration, and integrates into your workflow. To improve your efficiency with LangChain Hub, consider diving deeper into the customization and integration possibilities.

Wrapping Up

LangChain Hub is more than a tool; it's a catalyst for innovation and collaboration in AI development. Embrace this revolutionary platform and elevate your JavaScript LLM applications to new heights.

Throughout this guide, we tackled how to:

Centralize and manage your AI prompts effectively using LangChain Hub.
Enhance collaboration and version control within your development team.
Integrate prompt management seamlessly into your existing development workflows.

Keep building and experimenting, and I'm excited to see how you'll push the boundaries of what's possible with AI and LangChain Hub!

To stay connected and share your journey, feel free to reach out through the following channels:

👨‍💼 LinkedIn: Join me for more insights into LLM development and tech innovations.
💻 GitHub: Explore my projects and contribute to ongoing work.

Convert YouTube videos into Mind Maps using ChatGPT

Chris McKenzie — Mon, 25 Sep 2023 01:24:25 +0000

As a visual person I love Mind Maps. They’re a great way to visually organize complex ideas.

This got me thinking — Is it possible to get ChatGPT to generate a mind map highlighting the key concepts in a Youtube Video? Yes!

Getting Started

You will need:

ChatGPT 4
VoxScript plugin (search ChatGPT plugin store for “VoxScript”)
https://mermaid.live/

Mermaid

Mermaid is a JavaScript based diagramming and charting tool that uses Markdown-inspired text definitions and a renderer to create and modify complex diagrams. The main purpose of Mermaid is to help documentation catch up with development. mermaid.live takes the mermaid plain text format and creates visuals.

That’s about as much as you need to know to accomplish the task, but if you’re interested in learning more about the format read their docs.

VoxScript

VoxScript is a plugin that converts a YouTube url into a transcript, which ChatGPT can use as context to complete your prompt. This is really handy for a number of tasks, including summarizing long youtube videos, pulling out key insights, or creating outlines. For this, we’ll be using it to give ChatGPT the context it needs to generate a mind map.

Prompt

The prompt for this is simple. We start by giving ChatGPT the instruction to read the transcript of a video, extract key concepts, and convert those into a mermaid mind map format. To insure ChatGPT understands this format we provide an example.

Or in plain text:

Output the following request in Mermaid Live Editor code
targeting the "mindmap" format. Get the entire transcript for
this video: [VIDEO] then extract the key concepts, why they're
imporant, related information, and make logical connections. If
you're unfamiliar with the mindmap diagram, I'll provide an
example:
"""
mindmap
  root((mindmap))
    Origins
      Long history
      ::icon(fa fa-book)
      Popularisation
        British popular psychology author Tony Buzan
    Research
      On effectiveness<br/>and features
      On Automatic creation
        Uses
            Creative techniques
            Strategic planning
            Argument mapping
    Tools
      Pen and paper
      Mermaid
"""

VIDEO = https://www.youtube.com/watch?v=p9Q5a1Vn-Hk

After VoxScript provides the transcript for context, ChatGPT should return something that looks like this:

mindmap
  root((Inside OpenAI))
    OpenAI Origins
      ChatGPT & Dall-E
        Monster Hits
        Beat Tech Giants to Market
      Mira Marathi
        Chief Architect behind OpenAI's strategy
        AI that loves Humanity
        Challenges
          Hallucination
          Truthfulness
          Reliability
    AI Capabilities
      Neural Network
        Trained on massive data
        Predict next word in sentence
      ChatGPT
        Different way to understand information
        Intuitive searching vs keyword-based
      Misinformation
        AI can make up things (hallucinations)
        Need for awareness and not blindly relying on AI
    AI & Society
      Relationships with AI
        AI as a reliable tool
        AI's impact on jobs
          New jobs like prompt engineering
      AI's challenges
        Handling misinformation
        Addressing biases
    AI Gold Rush
      Venture Capitalists
        Investing in AI startups
      Blitzscaling
        Prioritizing speed over efficiency
      AI's potential
        Co-pilot for every profession
        Generative AI's broad touch
    Global Perspective
      AI's global impact
        Building industries of the future
      Responsible Innovation
        Need for trusted authority to audit AI systems
    AGI (Artificial General Intelligence)
      Risks
        Potential for human extinction
        Misalignment of goals between AI and humans
      Predictions
        Still far from AGI
        Powerful systems in the future

Simply copy and paste it to mermaid.live and it will generate your mind map:

That’s it! With a plugin and a simple prompt we can create meaningful mind map representations of complex ideas in YouTube videos.

I encourage you to modify the prompt to see if you get better results and share those in the comments.

Run Models in the Browser With Transformers.js

Chris McKenzie — Thu, 21 Sep 2023 21:32:06 +0000

For this tutorial we’ll be using transformers.js from Hugging Face. If you’re not familiar with Hugging Face, I highly recommend you check them out. They’re doing some really cool stuff in the AI space. The TLDR is that they’re the GitHub of ML.

Transformers.js

Based on the Python library — transformers — transformers.js is a JavaScript library that allows you to use pretrained models from Hugging Face in a JavaScript environment — such as the browser. While it’s not as robust as the Python library, it’s still pretty powerful.

The library is versatile, supporting tasks such as Natural Language Processing (NLP), Computer Vision, Audio, and Multimodal processing.

Under the hood, Transformers.js uses the ONNX Runtime to run the models in the browser.

ONNX Runtime

ONNX Runtime is a cross-platform machine-learning model accelerator, with a flexible interface to integrate hardware-specific libraries. ONNX Runtime can be used with models from PyTorch, Tensorflow/Keras, TFLite, scikit-learn, and other frameworks.

ONNX, or Open Neural Network Exchange, was initially started as a collaboration between Microsoft and Facebook. The project is open-source, and has grown to offer a universal format for machine learning models, ensuring seamless sharing and deployment across platforms.

ONNX is what makes it possible to directly run models in a browser. This is pretty insane!

Build a Demo

For this tutorial let’s build a simple app that summarizes text. Let’s start by setting up our environment. For this demo we’ll keep it as simple as possible, but you can use any framework you’d like.

Complete project can be found on Github

Setup project foundation

mkdir transformersjs-demo  
cd transformersjs-demo  
touch index.html styles.css main.js

Now let’s create a simple HTML page.

<!DOCTYPE html>  
<html lang="en">  
<head>  
    <meta charset="UTF-8">  
    <meta name="viewport" content="width=device-width, initial-scale=1.0">  
    <title>Summary Generator</title>  
    <link rel="stylesheet" href="styles.css">  
    <script src="main.js" type="module" defer></script>  
</head>  
<body>  
    <div class="container">  
        <textarea id="long-text-input" placeholder="Enter your copy here..."></textarea>  
        <button id="generate-button" disabled>  
          <span id="spinner">🔄</span> Generate Summary  
        </button>  
        <div id="output-div"></div>  
    </div>  
</body>  
</html>

Add a little CSS to make it look nice.

* {  
  box-sizing: border-box;  
}  

body {  
  font-family: Arial, sans-serif;  
  background-color: #f4f4f4;  
  display: flex;  
  justify-content: center;  
  align-items: center;  
  height: 100vh;  
  margin: 0;  
  padding: 20px;  
}  

.container {  
  background-color: #ffffff;  
  border-radius: 10px;  
  padding: 30px;  
  box-shadow: 0px 0px 15px rgba(0, 0, 0, 0.1);  
  width: 80%;  
  max-width: 600px;  
}  

textarea {  
  width: 100%;  
  height: 200px;  
  border-radius: 5px;  
  padding: 15px;  
  font-size: 16px;  
  border: 1px solid #ddd;  
  resize: none;  
}  

button {  
  display: block;  
  width: 100%;  
  margin: 20px 0;  
  padding: 10px;  
  background-color: #3498db;  
  color: #ffffff;  
  border: none;  
  border-radius: 5px;  
  cursor: pointer;  
  font-size: 18px;  
}  

button:hover {  
  background-color: #2980b9;  
}  

button:disabled {  
  background-color: #b3c2c8;  
  cursor: not-allowed;  
}  

@keyframes spin {  
  0% { transform: rotate(0deg); }  
  100% { transform: rotate(360deg); }  
}  

#spinner {  
  display: none;  
  margin-right: 10px;  
  animation: spin 1s linear infinite;  
}  

#spinner.show {  
  display: inline-block;  
}  

#output-div {  
  display: none;  
  background-color: #f9f9f9;  
  border: 1px solid #ddd;  
  padding: 15px;  
  border-radius: 5px;  
  font-size: 16px;  
}

Selecting a Model

You should take some time to understand the different models, what they’re used for, and their tradeoffs. When picking a model, it’s important to not just consider the task, but also consider size, speed and accuracy. For this I recommend you check out the model hub on Hugging Face. You can convert your models to run in an ONNX runtime, but that’s outside the scope of this tutorial, however, if you’re curious about that checkout Convert your models to ONNX

For this demo we’ll be using the Xenova/t5-small model, which is a small model that can be used for summarization. This model is base off of [t5-small](https://huggingface.co/t5-small) built with ONNX weights to be compatible with transformers.js.

Adding JavaScript

Now let’s add some JavaScript to make it work. The first thing we want to do is import transformers.js.

import { pipeline } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.3.0';

We’re using modern JavaScript we can import it directly from a CDN in the browser, but you can also npm install it.

Next we’ll use the pipeline function to create a summarization pipeline:

const summarization = await pipeline(  
    'summarization', // task  
    'Xenova/t5-small' // model  
);

The pipeline function takes two arguments, the task and the model. We’re using the summarization task and the Xenova/t5-small model, but there are many other permutations you can use. For example, you could use the question-answering task and the deepset/roberta-base-squad2 model to create a question answering pipeline. Checkout the Tasks section for all the different tasks you can use.

So far we have a nice UI and we’ve created a summarization pipeline. Now let’s add some code to make it work. Below the import but above the pipeline function, add the following code:

const longTextInput = document.getElementById('long-text-input');  
const generateButton = document.getElementById('generate-button');  
const output = document.getElementById('output-div');  
const spinner = document.getElementById('spinner');

Next we’ll want to add some code to kick off the summarization task when the super clicks the “Generate Summary” button. Below the pipeline function, add the following code:

generateButton.addEventListener('click', async () => {  
    const input = longTextInput.value;  
    const result = await summarization(input);  
    output.innerHTML = result[0].summary_text;  
});

Ideally, we don’t want our “Generate Summary” button to be clickable until the model is loaded. Depending on the model you pick, and the internet speed, this could take anywhere from a few seconds to over a minute. So let’s add a little code to disable the button until the model is loaded.

Update the button html to include the disabled attribute.

<button id="generate-button" disabled>

Next we’ll want to enable the button once the model is loaded: Below the pipeline function, add the following code:

generateButton.removeAttribute('disabled');

Next we’ll add a simple loading spinner:

generateButton.addEventListener('click', async () => {  
    spinner.classList.add('show');  
    generateButton.setAttribute("disabled", true);  

    const input = longTextInput.value;  
    const result = await summarization(input);  

    output.innerHTML = result[0].summary_text;  
    spinner.classList.remove('show');  
    generateButton.removeAttribute("disabled");  
    output.style.display = 'block';  
});

Testing It Out

For this demo, I’m going to summarize the article ‘Dr. Google’ meets its match in Dr. ChatGPT.

As you can see, we get a summary of the article, but it’s too short. Let’s update the code to get a longer summary. Add an options object to the pipeline function:

const result = await summarization(input, {  
    min_length: 50, max_length: 250,  
});

Running it again we see we get a longer summary. But it’s still not perfect. The accuracy leaves something to be desired. Let’s try a different model. Let’s try the t5-base model. Replace the pipeline with:

const summarization = await pipeline('summarization', 'Xenova/t5-base');

This is a little better, but ultimately you’ll need to decide which model is best for your use case.

Personally, I found Xenova/bart-large-cnn and Xenova/distilbart-cnn-6–6 produced the best results, but were the slowest and required downloading over a GB of data. This is something to keep in mind when selecting a model. You’ll want to balance accuracy with speed and size.

const summarization = await pipeline('summarization', 'Xenova/distilbart-cnn-6-6');  

// or  

const summarization = await pipeline('summarization', 'Xenova/bart-large-cnn');

Final Thoughts

There’s a lot of exciting stuff happening right now in the ML space. And the fact that we can run these models in the browser is pretty insane. It’s not perfect, and there are some considerations you need to take into account:

It’s likely going to be slower than a server-side solution
Because the size of the model matters, small models are generally less accurate
Balancing the tradeoffs between speed, size and accuracy means test different models to find the right one for your use case
You’ll need to consider the security implications of running these models in the browser

Overall this is a pretty exciting development. And it’s not just the browser. You can run these models in Node.js, Deno, React Native, and even in a serverless environment like Cloudflare Workers.

I’m excited to see what people build with this technology.