Forem: Louis Beaumont

How to create an AI agent powered by your screen & mic

Louis Beaumont — Wed, 22 Jan 2025 00:25:38 +0000

screenpipe is a CLI / app that records your screens and mics 24/7, extract OCR, transcription, etc. 100% locally and make it easy to feed it into AI.

screenpipe's pipe system lets you build powerful plugins that interact with captured screen and audio data. let's create a simple pipe that uses ollama to analyze your screen activity.

prerequisites

screenpipe installed and running
bun installed
ollama installed with a model (we'll use deepseek-r1:1.5b)

1. create your pipe

first, let's create a new pipe using the official CLI:

bunx @screenpipe/create-pipe@latest

when prompted:

name your pipe (e.g., "my-activity-analyzer")
choose a directory

2. project structure

now open your fav editor like cursor or vscode and open the project.

cursor my-activity-analyzer

after creation, your project should look like this:

my-activity-analyzer/
├── README.md
├── package.json
├── pipe.json
├── public
├── src
│   ├── app
│   ├── components
│   ├── hooks
│   └── lib
├── tailwind.config.ts
└── tsconfig.json

let's remove some files that are not needed:

rm -rf src/app/api/intelligence src/components/obsidian-settings.tsx src/components/file-suggest-textarea.tsx

3. implement a cron job that will be used to analyze activity

create a new file src/app/api/analyze/route.ts and add this code:

import { NextResponse } from "next/server";
import { pipe } from "@screenpipe/js";
import { streamText } from "ai";
import { ollama } from "ollama-ai-provider";

export async function POST(request: Request) {
  try {
    const { messages, model } = await request.json();

    console.log("model:", model);

    // query last 5 minutes of activity
    const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1000).toISOString();
    const results = await pipe.queryScreenpipe({
      startTime: fiveMinutesAgo,
      limit: 10,
      contentType: "all",
    });

    // setup ollama with selected model
    const provider = ollama(model);

    const result = streamText({
      model: provider,
      messages: [
        ...messages,
        {
          role: "user",
          content: `analyze this activity data and summarize what i've been doing: ${JSON.stringify(
            results
          )}`,
        },
      ],
    });

    return result.toDataStreamResponse();
  } catch (error) {
    console.error("error:", error);
    return NextResponse.json(
      { error: "failed to analyze activity" },
      { status: 500 }
    );
  }
}

4. create a simple vercel-like config, pipe.json for scheduling:

{
  "crons": [
    {
      "path": "/api/analyze",
      "schedule": "*/5 * * * *"
    }
  ]
}

5. update the main page to display the analysis

open src/app/page.tsx and add this code:

"use client";

import { useState } from "react";
import { Button } from "@/components/ui/button";
import { OllamaModelsList } from "@/components/ollama-models-list";
import { Label } from "@/components/ui/label";
import { useChat } from "ai/react";

export default function Home() {
  const [selectedModel, setSelectedModel] = useState("deepseek-r1:1.5b");
  const { messages, input, handleInputChange, handleSubmit } = useChat({
    body: {
      model: selectedModel,
    },
    api: "/api/analyze",
  });

  return (
    <main className="p-4 max-w-2xl mx-auto space-y-4">
      <div className="space-y-2">
        <Label htmlFor="model">ollama model</Label>
        <OllamaModelsList
          defaultValue={selectedModel}
          onChange={setSelectedModel}
        />
      </div>

      <div>
        {messages.map((message) => (
          <div key={message.id}>
            <div>{message.role === "user" ? "User: " : "AI: "}</div>
            <div>{message.content}</div>
          </div>
        ))}

        <form onSubmit={handleSubmit} className="flex gap-2">
          <input
            value={input}
            onChange={handleInputChange}
            placeholder="Type a message..."
            className="w-full"
          />
          <Button type="submit">Send</Button>
        </form>
      </div>
    </main>
  );
}

6. test locally

run your pipe:

bun i # or use npm
bun dev

visit http://localhost:3000 to test your pipe.

7. install in screenpipe

you can install your pipe in two ways:

a) through the UI:

open screenpipe app
go to pipes
click "+" and enter your pipe's local path

b) using CLI:

screenpipe install /path/to/my-activity-analyzer
screenpipe enable my-activity-analyzer

how it works

let's break down the key parts:

data querying: we use pipe.queryScreenpipe() to fetch recent screen/audio data
ai processing: ollama analyzes the data through a simple prompt
ui: a simple button triggers the analysis and displays results
scheduling: screenpipe will call the cron job every 5 minutes

next steps

you can enhance your pipe by:

adding configuration options
connecting to external services
adding more sophisticated UI components

useful references

Rewind AI + Cursor AI = screenpipe: how we built a high performance Rust frame streaming API (OSS)

Louis Beaumont — Sun, 10 Nov 2024 01:29:35 +0000

Sharing how we build a high performance frame streaming API in Rust.

Code is here: https://github.com/mediar-ai/screenpipe

how i built a local first audio transcription: building a privacy-first voice processing pipeline

Louis Beaumont — Wed, 30 Oct 2024 02:19:00 +0000

I implemented a sophisticated local-first audio processing pipeline that captures, processes, and transcribes audio while respecting privacy, written in rust. here's how it works:

🎤 audio capture & device management

supports both input (microphones) and output devices (system audio)
handles multi-channel audio devices through smart channel mixing
implements device hot-plugging and graceful error handling
uses tokio channels for efficient async communication ### 🔊 audio processing pipeline

channel conversion - converts multi-channel audio to mono using weighted averaging - handles various sample formats (f32, i16, i32, i8) - implements real-time resampling to 16khz for whisper compatibility
signal processing - normalizes audio using RMS and peak normalization - implements spectral subtraction for noise reduction - uses realfft for efficient fourier transforms - maintains audio quality while reducing background noise
voice activity detection (vad) - dual vad engine support: webrtc (lightweight) and silero (more accurate) - configurable sensitivity levels (low/medium/high) - uses sliding window analysis for robust speech detection - implements frame history for better context awareness ### 🤖 transcription engine

primary: whisper (tiny/large-v3/large-v3-turbo)
fallback: deepgram api integration
smart overlap handling:

// handles cases where audio chunks might cut sentences
 if let Some((prev_idx, cur_idx)) = longest_common_word_substring(previous, current) {
 // strip overlapping content and merge transcripts
 }

💾 storage & optimization

uses h265 encoding for efficient audio storage
implements a local sqlite database for metadata
stores raw audio chunks with timestamps
maintains reference to original audio for verification

🔒 privacy features
completely local processing by default
optional pii removal
configurable data retention policies
no cloud dependencies unless explicitly enabled

🧠 experimental features
context-aware post-processing using llama-3.2–1b
speaker diarization using voice embeddings
local vector db for speaker identification
adaptive noise profiling

🔧 technical stack
rust + tokio for async processing
tauri for cross-platform support
onnx runtime for ml inference
crossbeam channels for thread communication

📊 performance considerations
efficient memory usage through streaming processing
minimal cpu overhead through smart buffering
configurable quality/performance tradeoffs
automatic resource management

result:

it's open source btw!

https://github.com/mediar-ai/screenpipe

drop any question!

How to use Llama3.2 to write daily logs in Notion based on your screen

Louis Beaumont — Sun, 20 Oct 2024 23:38:09 +0000

Ever wished you had a personal AI assistant that could keep track of your daily work? With screenpipe & llama3.2, you can now automate the process of writing detailed logs based on your screen activity. Let's dive into how you can set this up using screenpipe's plugin system.

What is screenpipe?

screenpipe is an open-source tool that captures your screen and audio 24/7, extracts text using ocr, and allows you to build personalized ai-powered workflows. It's designed to be secure, with your data staying on your machine.

Installing screenpipe

(If you're not on macOS check these instructions)

To build the screenpipe app from source on macOS, follow these steps:

Install Rust and necessary dependencies:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
brew install pkg-config ffmpeg jq tesseract cmake wget

Install Deno CLI:

curl -fsSL https://deno.land/install.sh | sh

Clone the screenpipe repository:

git clone https://github.com/mediar-ai/screenpipe
cd screenpipe

Build the project:

cargo build --release --features metal

To build the desktop app, first add this to your VSCode settings in .vscode/settings.json:

{
   "rust-analyzer.cargo.features": [
      "metal",
      "pipes"
   ],
   "rust-analyzer.server.extraEnv": {
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/screenpipe-vision/lib:${env:DYLD_LIBRARY_PATH}",
      "SCREENPIPE_APP_DEV": "true"
   },
   "rust-analyzer.cargo.extraEnv": {
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/screenpipe-vision/lib:${env:DYLD_LIBRARY_PATH}",
      "SCREENPIPE_APP_DEV": "true"
   },
   "terminal.integrated.env.osx": {
      "DYLD_LIBRARY_PATH": "${workspaceFolder}/screenpipe-vision/lib:${env:DYLD_LIBRARY_PATH}",
      "SCREENPIPE_APP_DEV": "true"
   }
}

Then, build the desktop app:

cd screenpipe-app-tauri
bun install
bun scripts/pre_build.js
bun tauri build

This process will build the screenpipe app from source on your macOS system. If you encounter any issues, you can open an issue on the GitHub repository for assistance.

Start screenpipe by clicking start or through CLI in dev mode:

Once you installed screenpipe, add this plugin by dropping this URL in the "add your own pipe" bar:

https://github.com/mediar-ai/screenpipe/tree/main/examples/typescript/pipe-phi3.5-engineering-team-logs

Use llama3.2:3b-instruct-q4_K_M as model.

You should also create and configure your Notion integration (instructions on the plugin page in screenpipe app).

Start the LLM in the settings (or manually through Ollama):

If you are on Windows or Linux you need to run Ollama yourself.

Restart screenpipe by clicking stop and start or through CLI in dev mode:

You should start getting notifications in screenpipe AI inbox:

BTW, the code to do this is quite simple:

import { ContentItem } from "screenpipe";
import { Client } from "npm:@notionhq/client";
import { z } from "zod";
import { generateObject } from "ai";
import { createOllama } from "ollama-ai-provider";
import { pipe } from "screenpipe";

const engineeringLog = z.object({
  title: z.string(),
  description: z.string(),
  tags: z.array(z.string()),
});

type EngineeringLog = z.infer<typeof engineeringLog>;

async function generateEngineeringLog(
  screenData: ContentItem[],
  ollamaModel: string,
  ollamaApiUrl: string
): Promise<EngineeringLog> {
  const prompt = `Based on the following screen data, generate a concise engineering log entry:

    ${JSON.stringify(screenData)}

    Focus only on engineering work. Ignore non-work related activities.
    Return a JSON object with the following structure:
    {
        "title": "Brief title of the engineering task",
        "description": "Concise description of the engineering work done",
        "tags": ["tag1", "tag2", "tag3"]
    }
    Provide 1-3 relevant tags related to the engineering work.`;

  const provider = createOllama({ baseURL: ollamaApiUrl });

  const response = await generateObject({
    model: provider(ollamaModel),
    messages: [{ role: "user", content: prompt }],
    schema: engineeringLog,
  });

  console.log("ai answer:", response);

  return response.object;
}

async function syncLogToNotion(
  logEntry: EngineeringLog,
  notion: Client,
  databaseId: string
): Promise<void> {
  try {
    console.log("syncLogToNotion", logEntry);
    await notion.pages.create({
      parent: { database_id: databaseId },
      properties: {
        Title: { title: [{ text: { content: logEntry.title } }] },
        Description: {
          rich_text: [{ text: { content: logEntry.description } }],
        },
        Tags: { multi_select: logEntry.tags.map((tag) => ({ name: tag })) },
        Date: { date: { start: new Date().toISOString() } },
      },
    });

    console.log("engineering log synced to notion successfully");

    // Create markdown table for inbox
    const markdownTable = `
| Title | Description | Tags |
|-------|-------------|------|
| ${logEntry.title} | ${logEntry.description} | ${logEntry.tags.join(", ")} |
    `.trim();

    await pipe.inbox.send({
      title: "engineering log synced",
      body: `new engineering log entry:\n\n${markdownTable}`,
    });
  } catch (error) {
    console.error("error syncing engineering log to notion:", error);
    await pipe.inbox.send({
      title: "engineering log error",
      body: `error syncing engineering log to notion: ${error}`,
    });
  }
}

function streamEngineeringLogsToNotion(): void {
  console.log("starting engineering logs stream to notion");

  const config = pipe.loadPipeConfig();
  console.log("loaded config:", JSON.stringify(config, null, 2));

  const interval = config.interval * 1000;
  const databaseId = config.notionDatabaseId;
  const apiKey = config.notionApiKey;
  const ollamaApiUrl = config.ollamaApiUrl;
  const ollamaModel = config.ollamaModel;

  const notion = new Client({ auth: apiKey });

  pipe.inbox.send({
    title: "engineering log stream started",
    body: `monitoring engineering work every ${config.interval} seconds`,
  });

  pipe.scheduler
    .task("generateEngineeringLog")
    .every(interval)
    .do(async () => {
      try {
        const now = new Date();
        const oneHourAgo = new Date(now.getTime() - interval);

        const screenData = await pipe.queryScreenpipe({
          startTime: oneHourAgo.toISOString(),
          endTime: now.toISOString(),
          limit: 50,
          contentType: "ocr",
        });

        if (screenData && screenData.data.length > 0) {
          const logEntry = await generateEngineeringLog(
            screenData.data,
            ollamaModel,
            ollamaApiUrl
          );
          await syncLogToNotion(logEntry, notion, databaseId);
        } else {
          console.log("no relevant engineering work detected in the last hour");
        }
      } catch (error) {
        console.error("error in engineering log pipeline:", error);
        await pipe.inbox.send({
          title: "engineering log error",
          body: `error in engineering log pipeline: ${error}`,
        });
      }
    });

  pipe.scheduler.start();
}

streamEngineeringLogsToNotion();

Any ideas of other interesting plugins could be made?

Feel free to join our Discord!

local meeting summaries using screenpipe + ollama

Louis Beaumont — Thu, 19 Sep 2024 23:16:58 +0000

hey devs,

i've been working on a tool called screenpipe that lets you generate meeting summaries locally, even for whatsapp calls. thought i'd share my setup in case it's useful to anyone else.

the problem

taking notes during meetings is a pain, and most transcription services send your data to the cloud. i wanted something that could run entirely on my macbook.

the setup

here's what i'm using:

screenpipe: open-source tool for continuous screen/audio capture, written in rust, cross-platform
ollama: runs llms locally
solar pro 22b: the language model i'm using (via ollama)

how it works

screenpipe captures my screen and audio 24/7
after a meeting (zoom, whatsapp, whatever), i query the last hour of data
send that data to solar pro 22b running locally via ollama
get back a concise summary

here's some pseudo code to illustrate:

// 1h ago
const startDate = "<some time 1h ago..>"
// 10m ago
const endDate = "<some time 10m ago..>"

// get all the screen & mic data from roughly last hour 
const results = fetchScreenpipe(startDate, endDate)

// send it to an LLM and ask for a summary
const summary = fetchOllama("{results} create a summary from these transcriptions")

// add the meeting summary to your notes
addToNotion(summary)

why it's cool

privacy: all data stays on your machine
works for any app: zoom, teams, whatsapp, in-person meetings, etc.
customizable: you can create plugins in TS to connect LLMs to your 24/7 screens & mics recording

try it out

if you want to experiment:

install screenpipe: github.com/mediar-ai/screenpipe
set up ollama with solar pro 22b
use the api to query your data and generate summaries

curious if anyone else is working on similar local setups for meeting summaries. what has your experience been?

Forem: Louis Beaumont

How to create an AI agent powered by your screen & mic

prerequisites

1. create your pipe

2. project structure

3. implement a cron job that will be used to analyze activity

4. create a simple vercel-like config, pipe.json for scheduling:

5. update the main page to display the analysis

6. test locally

7. install in screenpipe

how it works

next steps

useful references

Rewind AI + Cursor AI = screenpipe: how we built a high performance Rust frame streaming API (OSS)

how i built a local first audio transcription: building a privacy-first voice processing pipeline

🎤 audio capture & device management

💾 storage & optimization

🔒 privacy features

🧠 experimental features

🔧 technical stack

📊 performance considerations

How to use Llama3.2 to write daily logs in Notion based on your screen

What is screenpipe?

Installing screenpipe

local meeting summaries using screenpipe + ollama

the problem

the setup

how it works

why it's cool

try it out