Forem: M Sea Bass

I Made AI Code for Me from My Phone (Slack Codex CLI)

M Sea Bass — Thu, 01 Jan 2026 09:52:32 +0000

Introduction

Have you ever wished you could work while lying in bed?

Well, you can.

I built a system where you just send a mention in Slack, and AI writes the code and even creates a PR (Pull Request) for you. You can request coding from your phone, on the train, or even from your bed.

In this article, I'll show you how this system works and demonstrate it in action.

How It Works

The architecture is simple.

Mention from your phone via Slack → Message goes to Slack Cloud
Bot on your home PC detects it → Python Bot connected via Socket Mode receives the message
Launches Codex CLI → Automatically, Codex CLI does the task you requested
Reports progress to Slack → Sends progress and results to Slack via Slack MCP (Model Context Protocol)
Creates Pull Request on GitHub → Commits and creates Pull Request via GitHub MCP

Since all processing happens on your home PC, you can use GPU and run heavy tasks with Docker.

See It in Action

Let me show you how it actually works.

1. Send a Mention in Slack

Just mention the bot in Slack to start the work.

@codex-bot create simple todo app in new folder and run with host 0.0.0.0

Once the bot starts working, progress updates are posted to the thread. When complete, the app URL and PR(Pull Request) link are shared.

2. Check the App

Access the created app from your phone to verify it works.

If you're on the same WiFi network, you can directly access the development server. Checking the design and giving feedback is easy from your phone.

3. Review and Merge the PR(Pull Request)

Review the PR(Pull Request) on GitHub and merge it if everything looks good.

Check the code diff, add comments if needed, and merge. Done!

What You Can Do

Examples of what this system can do:

Coding: Implement new features, fix bugs, refactoring
App Development: Create web app scaffolding, adjust UI
Docker Tasks: ML training, batch processing (completion detection via health checks)
Automated Testing: Simple E2E tests with Playwright MCP

The workflow of "checking app design on your phone while giving feedback via Slack" feels like pair programming with a designer.

Setup

The code is available on GitHub. Setup instructions are in the README.

🔗 Repository: https://github.com/M6saw0/slack_codex_bot

Requirements:

Slack App (Socket Mode)
ChatGPT Account (for Codex CLI)
GitHub Token (for PR creation)

See the repository README for detailed setup instructions.

Conclusion

Once you experience having AI code for you from your phone, you'll be hooked.

Request bug fixes while lounging in bed
Implement new app features on the train
Give design feedback while relaxing on the couch

Make the most of your commute and spare time—build your own AI pair programmer.

Practical Guide to MCP (Model Context Protocol) in Python

M Sea Bass — Mon, 03 Nov 2025 01:42:45 +0000

Introduction

MCP (Model Context Protocol) is a protocol that connects LLMs with external tools. This article walks through how to use MCP step by step with real code examples. The full source code is available in the following GitHub repository:
https://github.com/M6saw0/mcp-client-sample

Read time: ~10 minutes

1. Core Concepts of MCP

MCP provides three primary capabilities:

Tool: Execute external functionality via function calls
Resource: Provide data or information (static or dynamic)
Prompt: Provide prompt templates for LLMs

MCP servers mainly support two transport modes. This article covers both stdio and streamable-http:

stdio: Communicate via standard I/O (used within the same process)
streamable-http: Communicate via HTTP (over the network)

2. Build a Server with stdio

Let’s start by building a server using the simplest stdio transport.

2.1 Basic Server Code

from mcp.server.fastmcp import FastMCP

# Create a FastMCP instance
mcp = FastMCP(name="FastMCP Demo Server")

# Define a tool
@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b

# Define a resource (dynamic)
@mcp.resource("time://{zone}")
def get_time(zone: str) -> str:
    """Return ISO 8601 timestamp."""
    from datetime import datetime, timezone
    now = datetime.now(timezone.utc)
    return now.isoformat() if zone.lower() == "utc" else now.astimezone().isoformat()

# Define a resource (static)
@mcp.resource("info://server")
def get_server_info() -> str:
    """Return server metadata."""
    return "FastMCP demo server"

# Define a prompt
@mcp.prompt()
def greet_user(name: str, tone: str = "friendly") -> str:
    """Generate a greeting instruction."""
    return f"Craft a {tone} greeting addressed to {name}."

# Run the server
if __name__ == "__main__":
    mcp.run(transport="stdio")

2.2 Async Tools and Progress Reporting

For long-running operations, you can report progress:

from typing import Annotated
from mcp.server.fastmcp import Context, FastMCP
from mcp.server.session import ServerSession
import asyncio

@mcp.tool()
async def countdown(
    start: int,
    ctx: Annotated[Context[ServerSession, None], "Injected by FastMCP"],
) -> list[int]:
    """Count down from start to zero."""
    sequence = []
    for step, value in enumerate(range(start, -1, -1), start=1):
        await ctx.report_progress(
            progress=step,
            total=start + 1,
            message=f"Counting value {value}",
        )
        sequence.append(value)
        await asyncio.sleep(0.2)
    return sequence

3. Build a stdio Client

Client code to call the server:

import asyncio
from pathlib import Path
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    server_path = Path(__file__).with_name("server.py")
    server_params = StdioServerParameters(command="python", args=[str(server_path)])

    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # List tools
            tools = await session.list_tools()
            print("Available tools:", [tool.name for tool in tools.tools])

            # Call a tool
            result = await session.call_tool("add", arguments={"a": 2, "b": 5})
            print("add result:", result.content)

            # Read a resource
            resource = await session.read_resource("time://local")
            print("time://local:", resource.contents)

            # Get a prompt
            prompt = await session.get_prompt("greet_user", arguments={"name": "Alice"})
            print("prompt:", prompt.messages)

asyncio.run(main())

To receive progress reports, pass progress_callback to ClientSession.call_tool:

async def on_progress(progress: float, total: float | None, message: str | None) -> None:
    print(f"{progress}/{total or 0} - {message or ''}")

result = await session.call_tool(
    "countdown",
    arguments={"start": 3},
    progress_callback=on_progress,
)

4. Build a Server with streamable-http

You can convert the stdio server into an HTTP server with just a few changes:

Change: Specify host and port when initializing FastMCP, and use transport="streamable-http".

from mcp.server.fastmcp import FastMCP

# Change: specify host and port
mcp = FastMCP(
    name="FastMCP StreamableHTTP Demo",
    host="127.0.0.1",  # localhost only; 0.0.0.0 also works
    port=8765,
)

# Tool, resource, and prompt definitions are the same
@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b

# ...(other definitions are the same)

# Change: use transport "streamable-http"
if __name__ == "__main__":
    mcp.run(transport="streamable-http")

5. Build a streamable-http Client

The HTTP client also requires only minimal changes:

Change: Use streamablehttp_client instead of stdio_client and specify a URL.

import asyncio
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

STREAMABLE_HTTP_URL = "http://127.0.0.1:8765/mcp"

async def main():
    # Change: use streamablehttp_client
    async with streamablehttp_client(STREAMABLE_HTTP_URL) as (read, write, get_session_id):
        async with ClientSession(read, write) as session:
            await session.initialize()

            # Retrieve the session ID (HTTP-specific)
            if (session_id := get_session_id()) is not None:
                print("Session ID:", session_id)

            # Usage is the same as stdio from here on
            tools = await session.list_tools()
            print("Available tools:", [tool.name for tool in tools.tools])

            result = await session.call_tool("add", arguments={"a": 2, "b": 5})
            print("add result:", result.content)

asyncio.run(main())

6. A Client for Multiple Servers

You can handle multiple servers by “keeping and reusing” each ClientSession after establishing connections.

Open the transport (stdio/HTTP) → create a ClientSession → call initialize → store in a dictionary for lookup by name

# Pattern aligned with the repo implementation (async with)
sessions: dict[str, ClientSession] = {}

async with stdio_client(params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        sessions["stdio"] = session  # Keep and reuse if needed
        # You can call call_tool as many times as you want here
        res = await session.call_tool("add", {"a": 1, "b": 2})
        print(res.content)

Call call_tool on the retained ClientSession

result = await sessions[server].call_tool(tool, args)

Close everything at the end (centralized lifecycle management for connections and sessions)

This “retain and reuse with centralized lifecycle management” is generalized and organized by MultiServerClient. For implementation details and usage, see multi-server/client.py in the repository. The following functions/methods are the key points:

MultiServerClient.__init__: Register servers (stdio/http)
connect(): Establish connections to registered servers
_ensure_session(name): Check for a session and connect if missing
_connect(name): Entry point to create a session (with Future management)
_session_task(name, future): Actual connection and lifecycle (async with stdio_client/streamablehttp_client → ClientSession → initialize → wait → close)
session(name): Context manager to retrieve a retained ClientSession
list_tools(): Enumerate tools from all servers
call_tool(server, tool, arguments): Delegate execution to the specified server
close(): Safely terminate all sessions (shutdown events and awaiting completion)

7. Using MCP from Generative AI

7.1 Which schema to convert to (Responses vs. Chat Completions)

Responses API (e.g., client.responses.create) expects the following format:

{
  "type": "function",
  "name": "server__tool",
  "description": "...",
  "parameters": { "type": "object", "properties": {"a": {"type": "integer"}}, "required": ["a"], "additionalProperties": false },
  "strict": true
}

Chat Completions API (e.g., client.chat.completions.create) expects the following format:

{
  "type": "function",
  "function": {
    "name": "server__tool",
    "description": "...",
    "parameters": { "type": "object", "properties": {"a": {"type": "integer"}}, "required": ["a"] }
  },
  "strict": true
}

7.2 How to obtain the necessary information (inspect / docstring / MCP meta)

Generate schemas using information obtained from the MCP server via list_tools().

Use description and inputSchema
In this repository, MultiServerClient.get_tool_details() in for-llm/mcp_client.py aggregates list_tools() results across all servers and provides { "description", "inputSchema" }

You can also generate schemas directly from Python functions.

Compose JSON Schema using inspect, type hints, and docstrings
The repository’s for-llm/tool_schemas.py implements:
- function_to_responses_tool(func)
- function_to_chat_tool(func)
- build_responses_toolkit(*functions)
- build_chat_toolkit(*functions)

7.3 Implementation code and concrete methods

See for-llm/run_llm.py in the repository for the actual implementation.

Schema conversion utilities: for-llm/tool_schemas.py
- MCP → Responses: mcp_tool_to_responses_schema(tool_name, description, input_schema, strict=True)
- MCP → Chat: mcp_tool_to_chat_schema(tool_name, description, input_schema, strict=True)
- Python function → Responses: function_to_responses_tool(func) / bundle: build_responses_toolkit(...)
- Python function → Chat: function_to_chat_tool(func) / bundle: build_chat_toolkit(...)
Tool metadata retrieval: for-llm/mcp_client.py
- MultiServerClient.get_tool_details() → { combined_name: {"server_name", "tool_name", "description", "inputSchema"} }

Conclusion

MCP provides three capabilities: tools, resources, and prompts
stdio communicates within the same process; streamable-http communicates over the network
Server and client code are very simple
You can centrally manage multiple servers
By combining schema conversion with tools, you can dynamically use MCP from generative AI

See the project repository for the full code.

I Tried Building a Whiteboard App with Claude 4.5 Sonnet

M Sea Bass — Sat, 04 Oct 2025 01:25:15 +0000

I got my hands on the newly released Claude 4.5 Sonnet and decided to give it a try.

At the company I work for, using cloud services is a bit of a hassle. We have to go through an application process to create accounts and get approval for sending information. I'd been thinking how convenient it would be to have a whiteboard app that could create business flows in real-time during interviews, so I enlisted "Sonnet" to help me build one.

The app I created is available on the following GitHub repository. Please feel free to try it out if you're interested.

GitHub

Development Process

User Requirements Definition

First, I outlined the user persona: who the users are, what problems they face, and what features they need.

Requirements Definition

Next, based on the user requirements, I defined the necessary functions, the technology stack to be used, and the UI/UX requirements.

Vibe Coding

Since this was a personal project, from here on out, I proceeded with implementation while interacting with the AI based on the requirements definition.

Claude 4.5 Sonnet seems to have a stubborn side.
Even though I specified React + Typescript in the requirements, it apparently didn't like that the template generated by Vite was Vanilla JS, and it just kept creating and deleting the project over and over. It's probably a good idea to keep a close eye on the process when you let an AI handle environment setup.

I also gave instructions for adding and modifying features via chat. In addition to the initially planned features like object placement and image export, I also had it add a function to adjust the size of objects.

Here's what the finished app's UI looks like:

The UI that Claude creates is sophisticated and very clean. I feel its capabilities as a coding assistant are extremely high.
On the other hand, I get the impression that different AIs have their own strengths: GPT-5 for project planning and Gemini for documentation. Perhaps the new standard will be to use different AIs depending on what you want to achieve.

Understanding and Implementing ReAct

M Sea Bass — Tue, 18 Feb 2025 13:57:28 +0000

Belatedly, I studied ReAct. For reference, I will explain ReAct in this article.

What is ReAct?

ReAct stands for Reasoning + Acting. As its name implies, it integrates the process of thinking (reasoning) with the process of taking action. Instead of providing a one-shot response, the model repeatedly goes through the following steps:

Reason: The model first thinks internally to decide what information is needed and what actions it should take.
Act: Based on its reasoning, the model performs an action—such as querying an external tool like a search engine or fetching data.
Observe: The model then observes the results of its action (feedback or retrieved information) and analyzes them further.
Iterate: The process repeats (reason → act → observe) as many times as necessary until a final answer is reached.

By taking this step-by-step approach, the model is better equipped to tackle more complex tasks.

A Simple Implementation of ReAct

Let’s walk through a basic implementation that demonstrates the ReAct process.

1. Import Required Libraries

First, we import the necessary libraries:

import faiss
import numpy as np
import openai

2. Create a Vector Database

We create a vector database for our search functionality and populate it with sample data:

OPENAI_API_KEY = "your api key"
client = openai.OpenAI(api_key=OPENAI_API_KEY)

documents = [
    "Tomohiko Ayame is known as a craftsman of traditional bamboo work in the region.",
    "On holidays, Tomohiko Ayame holds classes to teach bamboo crafts to local children.",
    "Tomohiko Ayame has been honing his bamboo craft skills for over 40 years.",
    "Tomohiko Ayame's works are permanently exhibited at the local museum, attracting many tourists.",
    "Tomohiko Ayame participates as a judge in the annual craft exhibition.",
    "Tomohiko Ayame is passionate about training young craftsmen and has mentored more than 10 apprentices.",
    "Tomohiko Ayame's masterpiece 'Bamboo Wind' is designated as an Important Intangible Cultural Property.",
    "Tomohiko Ayame serves as the chairman of the local traditional culture preservation society.",
    "Tomohiko Ayame has gained international recognition and held exhibitions in France and Italy.",
    "Tomohiko Ayame was awarded the title of Traditional Craftsman by the government in 2020."
]

3. Create and Index Embeddings

We vectorize the documents using an OpenAI embedding model and build a FAISS index:

# Vector Embedding: Vectorize each document
embed_response = client.embeddings.create(
    model="text-embedding-3-small",
    input=documents,
)
embeddings = [record.embedding for record in embed_response.data]

# FAISS Index Construction: Create an index for cosine similarity approximation
embedding_dim = len(embeddings[0])
index = faiss.IndexFlatL2(embedding_dim)
index.add(np.array(embeddings, dtype=np.float32))

4. Define a Vector Search Function

This function returns the top-k related documents from the FAISS index for a given query:

def vector_search(query: str, k: int = 3) -> str:
    """Returns top k related documents from FAISS index for the input query."""
    query_vec = client.embeddings.create(input=[query], model="text-embedding-3-small").data[0].embedding
    D, I = index.search(np.array([query_vec], dtype=np.float32), k)
    results = [documents[idx] for idx in I[0]]
    return "\n".join(results)

5. Generate Responses Using GPT

We use a GPT model (here, gpt-4o-mini) to generate responses based on our prompts:

def ask_gpt(messages: list[dict[str, str]]) -> str:
    """
    messages: List of {"role": ..., "content": ...}.
    Returns the content string from GPT-4's response.
    """
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Corrected model name
        messages=messages,
    )
    return response.choices[0].message.content

6. Execute the ReAct Process

ReAct involves iterating through reasoning, acting, and observing until a final answer is reached. The following code demonstrates how to use this process to answer a question:

# System prompt for ReAct agent
system_prompt = """You are an intelligent agent. Use the provided knowledge base to answer questions.
Please think and use tools as needed in the following format until you reach an answer:

Format:
Thought: <Agent's thinking process>
Action: <Tool name>[<Action input>]
Observation: <Results obtained from the tool>
... (Repeat Thought→Action→Observation as needed)
Answer: <Final answer>

Rules:
- Available tools: Search (local vector search engine that returns relevant documents)
- When using Search in Action, do not anticipate results until an Observation is provided
- Only end with Answer when you can provide a final conclusion. Do not give Answer during intermediate steps.
- Output only one step's Thought and Action at a time, do not perform multiple Actions simultaneously.

Let's begin.
"""

def answer_question_with_react(question, max_steps=5):
    """Answer questions using ReAct agent. Returns final answer text."""
    conversation = f"Question: {question}\n"

    for step in range(1, max_steps + 1):
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": conversation}
        ]

        output = ask_gpt(messages).strip()
        print(output)

        lines = [line.strip() for line in output.splitlines() if line.strip()]

        # Parse the response lines
        for line in lines:
            if line.startswith("Answer"):
                print(f"Final answer obtained: {line}")
                return line[len("Answer:"):].strip()

            if line.startswith("Thought"):
                conversation += line + "\n"
            elif line.startswith("Action"):
                conversation += line + "\n"
                if "Search[" in line:
                    query = line.split("Search[", 1)[1].rstrip("]")
                    result = vector_search(query, k=1)
                    print("Search result:", result)
                    conversation += f"Observation {step}: {result}\n"
                else:
                    return None
        print("-" * 30)
    return None

# Test the agent
query = "What does Tomohiko Ayame do on holidays?"
answer = answer_question_with_react(query)
print("=" * 30)
print("Final answer:", answer)

7. Sample Output

When you run the code above, you might see output similar to the following:

Thought: I need to find information about Tomohiko Ayame's activities during holidays. This might involve looking into articles or profiles that mention his personal interests. I will now search for relevant documents.
Action: Search["Tomohiko Ayame holidays activities"]
Search result: On holidays, Tomohiko Ayame holds classes to teach bamboo crafts to local children.
------------------------------
Thought: It appears that Tomohiko Ayame spends his holidays teaching bamboo crafts to local children, suggesting his involvement in community activities. I will search for additional details.
Action: Search["Tomohiko Ayame holiday activities details"]
Observation: Tomohiko Ayame not only teaches bamboo crafts but also participates in local festivals and collaborates with other craftspersons.
Search result: On holidays, Tomohiko Ayame holds classes to teach bamboo crafts to local children.
------------------------------
Thought: The gathered information is consistent. I can conclude that Tomohiko Ayame primarily spends his holidays teaching bamboo crafts.
Answer: Tomohiko Ayame spends his holidays teaching bamboo crafts to local children.
Final answer obtained: Answer: Tomohiko Ayame spends his holidays teaching bamboo crafts to local children.

This example demonstrates how ReAct—by combining reasoning, action, and observation—leads to a well-informed answer. However, depending on the structure of the data, the process may sometimes trigger redundant searches, so it’s important to balance cost and accuracy when applying this method.

Automating Daily arXiv Paper Summaries with Slack Notifications

M Sea Bass — Sun, 16 Feb 2025 05:26:47 +0000

This post is a follow-up to the previous article. It turns out there’s a slight delay before the latest papers show up in the arXiv API. Because of this delay, the same paper can sometimes appear the next day.

To fix this, we’re going to record the timestamp of the last retrieved paper and then only fetch new papers each day.

We’ll store the timestamp of the latest paper in Amazon S3 so we can both update and retrieve it later. For this, you’ll need to install boto3. In the python folder we created previously, run:

pip install boto3 -t python

Next, zip the folder again and upload it as a new version of your Lambda layer:

zip -r ./upload.zip ./python/*

Then, update your Lambda function to use this new layer version.

You’ll also need an S3 bucket ready in advance. In this example, we simply created one with the default settings.

Below is the fully revised code in English, including the new functions to update and retrieve the timestamp from S3. Note that we set S3_BUCKET_NAME as an environment variable.

import datetime
import json
import logging
import os
import time

import arxiv
import boto3
import google.generativeai as genai
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

# List of arXiv categories to search for
PAPER_TYPE_LIST = ["cs.AI", "cs.CY", "cs.MA"]

ARXIV_API_URL = "http://export.arxiv.org/api/query"
GEMINI_API_KEY = os.environ["GEMINI_API_KEY"]
GEMINI_MODEL = "gemini-2.0-flash"

# Slack bot token (xoxb-...)
SLACK_TOKEN = os.environ["SLACK_BOT_TOKEN"]
SLACK_BOT_TOKEN = os.environ["SLACK_BOT_TOKEN"]
# ID of the Slack channel to post to (e.g., "C12345...")
SLACK_CHANNEL = os.environ["SLACK_CHANNEL"]

# Name of the S3 bucket to use
S3_BUCKET_NAME = os.environ["S3_BUCKET_NAME"]

# Number of results to fetch at once
MAX_RESULTS = 50

# If the timestamp file does not exist, fallback to (current UTC - NO_EXIST_DAYS)
NO_EXIST_DAYS = 3


def save_utc_datetime(utc_datetime, filename="utc_date.json"):
    """
    Save the given UTC datetime to an S3 bucket in JSON format.
    """
    s3 = boto3.resource("s3")
    bucket = s3.Bucket(S3_BUCKET_NAME)
    obj = bucket.Object(filename)
    obj.put(Body=json.dumps({"utc_datetime": utc_datetime.isoformat()}))
    return utc_datetime


def load_datetime_from_file(filename="utc_date.json"):
    """
    Load the UTC datetime from an S3 JSON file.
    If the file does not exist, return (current UTC time - NO_EXIST_DAYS).
    """
    try:
        s3 = boto3.resource("s3")
        bucket = s3.Bucket(S3_BUCKET_NAME)
        obj = bucket.Object(filename)
        response = obj.get()
        body = response["Body"].read()
        data = json.loads(body.decode("utf-8"))
        return datetime.datetime.fromisoformat(data["utc_datetime"])
    except Exception:
        # If file doesn't exist, return current UTC datetime minus NO_EXIST_DAYS
        return datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=NO_EXIST_DAYS)


def get_papers(max_results: int = 10):
    """
    Retrieve papers from arXiv based on the specified PAPER_TYPE_LIST.
    Only papers published after the timestamp recorded in S3 will be returned.
    """
    query = " OR ".join([f"cat:{paper_type}" for paper_type in PAPER_TYPE_LIST])
    client = arxiv.Client()
    search = arxiv.Search(
        query=query,
        max_results=max_results,
        sort_by=arxiv.SortCriterion.SubmittedDate,
        sort_order=arxiv.SortOrder.Descending,
    )
    result_list = list(client.results(search))

    if not result_list:
        return []

    # The latest published date from the fetched results
    latest_published = result_list[0].published

    # Load the saved threshold from S3
    threshold = load_datetime_from_file()

    # Filter out papers published before the saved threshold
    filtered_results = [paper for paper in result_list if paper.published >= threshold]

    # Convert to a simpler data structure
    filtered_results = [
        {
            "title": paper.title,
            "summary": paper.summary,
            "pdf_url": paper.pdf_url,
            "published": paper.published,
        }
        for paper in filtered_results
    ]

    # Save the new threshold (slightly advanced by 1 second to avoid duplicates)
    save_utc_datetime(latest_published + datetime.timedelta(seconds=1))

    return filtered_results


def generate_summary(abstract_text):
    """
    Use the Generative AI API to summarize the paper's abstract.
    """
    genai.configure(api_key=GEMINI_API_KEY)
    model = genai.GenerativeModel(GEMINI_MODEL)  # The model to use, e.g., "gemini-2.0-flash"

    prompt = (
        "Please provide a concise summary (in English, within 50 words) of the following paper abstract "
        "that even beginners can understand. Be sure to include the significance and results of the paper. "
        "Output only the summary text.\n\n"
        "---\n\n"
        f"{abstract_text}"
    )
    response = model.generate_content(prompt)
    summary_text = response.text.strip()

    return summary_text


def send_slack_message(papers):
    """
    Format and send the summaries to Slack.
    """
    all_messages = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        summary_text = paper["summary"]
        link = paper["pdf_url"]
        published = paper["published"]
        message_text = (
            f"{i}: *{title}*\n\n"
            f"{summary_text}\n\n"
            f"PDF: {link}\n"
            f"Published: {published}"
        )
        all_messages.append(message_text)

    # Separate entries with a divider
    all_message = "\n\n────────────────────────\n\n".join(all_messages)

    client = WebClient(token=SLACK_BOT_TOKEN)
    logger = logging.getLogger(__name__)

    try:
        # If there are no papers, show a message
        if not all_message:
            all_message = "No new papers were found."
        # Post to Slack
        result = client.chat_postMessage(channel=SLACK_CHANNEL, text=all_message)
        logger.info(result)
    except SlackApiError as e:
        logger.error(f"Error posting message: {e}")


def main():
    """
    Main entry point:
    1. Get papers from arXiv.
    2. Summarize each paper.
    3. Send summaries to Slack.
    """
    papers = get_papers(MAX_RESULTS)
    output_papers = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        abstract = paper["summary"]
        link = paper["pdf_url"]
        published = paper["published"]
        summary = generate_summary(abstract)
        output_papers.append({
            "title": title,
            "summary": summary,
            "pdf_link": link,
            "published": published
        })
        # Small delay to avoid hitting rate limits
        time.sleep(1)
    send_slack_message(output_papers)


def lambda_handler(event, context):
    """
    AWS Lambda handler function.
    """
    main()
    return {
        'statusCode': 200,
        'body': "Successfully sent message to Slack!"
    }

By saving the timestamp in S3, your script won’t process the same paper entries each day, and if no new papers appear, the script will skip generating summaries. This helps reduce unnecessary API usage and costs.

Automating Daily ArXiv Paper Summaries and Slack Notifications

M Sea Bass — Sat, 15 Feb 2025 02:13:58 +0000

In this article, we create a process that fetches papers from arXiv daily, generates summaries using generative AI, and posts them to Slack.

Python Code for the Process

Below is the Python code that executes the process:

import datetime
import logging
import os
import time

import arxiv
import google.generativeai as genai
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError


PAPER_TYPE_LIST = ["cs.AI", "cs.CY", "cs.MA"]
ARXIV_API_URL = "http://export.arxiv.org/api/query"
GEMINI_API_KEY = os.environ["GEMINI_API_KEY"]
GEMINI_MODEL = "gemini-2.0-flash"
SLACK_TOKEN = os.environ["SLACK_BOT_TOKEN"]  # Slack Bot token (xoxb-...)
SLACK_BOT_TOKEN = os.environ["SLACK_BOT_TOKEN"]
SLACK_CHANNEL = os.environ["SLACK_CHANNEL"]  # ID of the channel to post to (e.g., "C12345...")
MAX_RESULTS = 30


def get_papers(max_results: int = 10):
    # Create a query by combining the paper categories
    query = " OR ".join([f"cat:{paper_type}" for paper_type in PAPER_TYPE_LIST])
    # Removed date range query
    client = arxiv.Client()
    search = arxiv.Search(
        query=query, 
        max_results=max_results, 
        sort_by=arxiv.SortCriterion.SubmittedDate,
        sort_order=arxiv.SortOrder.Descending,
    )
    result_list = list(client.results(search))

    if not result_list:
        return result_list

    # Determine the latest published date from the results.
    latest_published = result_list[0].published
    threshold = latest_published - datetime.timedelta(hours=24)

    # Filter results to include only papers published within 24 hours of the latest result.
    filtered_results = [paper for paper in result_list if paper.published >= threshold]

    filtered_results = [
        {
            "title": paper.title,
            "summary": paper.summary,
            "pdf_url": paper.pdf_url,
            "published": paper.published,
        }
        for paper in filtered_results]

    return filtered_results


def generate_summary(abstract_text):
    genai.configure(api_key=GEMINI_API_KEY)
    model = genai.GenerativeModel(GEMINI_MODEL)  # Model to use (e.g., Gemini 1.5 Flash version)

    prompt = (
        "Please summarize the following paper abstract in a clear and easy-to-understand way for beginners, "
        "in English and within 300 characters.\n"
        "Also include the significance and results of the paper.\n"
        "Output only the summary result."
        "\n---\n\n"
        f"{abstract_text}"
    )
    response = model.generate_content(prompt)
    summary_text = response.text.strip()

    return summary_text


def send_slack_message(papers):
    all_messages = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        summary_text = paper["summary"]
        link = paper["pdf_link"]
        published = paper["published"]
        message_text = (
            f"{i}: *{title}*\n\n"
            f"{summary_text}\n\n"
            f"PDF: {link}\n"
            f"Published: {published}"
        )
        all_messages.append(message_text)

    all_message = "\n\n────────────────────────\n\n".join(all_messages)

    client = WebClient(token=SLACK_BOT_TOKEN)
    logger = logging.getLogger(__name__)
    # ID of the channel to which you want to send the message

    try:
        # Call the chat.postMessage method using the WebClient
        result = client.chat_postMessage(
            channel=SLACK_CHANNEL, 
            text=all_message
        )
        logger.info(result)

    except SlackApiError as e:
        logger.error(f"Error posting message: {e}")


def main():
    papers = get_papers(MAX_RESULTS)
    output_papers = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        abstract = paper["summary"]
        link = paper["pdf_url"]
        published = paper["published"]
        summary = generate_summary(abstract)
        output_papers.append({
            "title": title,
            "summary": summary,
            "pdf_link": link,
            "published": published
        })
        time.sleep(1)
    send_slack_message(output_papers)


if __name__ == "__main__":
    main()

Using Gemini for Summarization

In this implementation, we use Gemini. Please set your API key in an environment variable.

Set SLACK_BOT_TOKEN to your Bot User OAuth Token (which requires the chat:write permission).
Set SLACK_CHANNEL to the ID of the channel where you want to post the messages.

Running the Process on AWS Lambda

We will use AWS Lambda to run this process on a schedule.

Set up a Lambda function using Python 3.12.
Include external libraries via a ZIP file layer.

Local Setup Steps

Set Python 3.12 using pyenv.
Install the required libraries into a target folder:

   pip install arxiv google-generativeai slack_sdk -t {folder}

Create a ZIP archive of the folder:

   zip -r ./upload.zip ./{folder}/*

Upload the ZIP file to an AWS Lambda layer. Set the architecture to x86_64 and the runtime to Python 3.12, then create the layer.

Modifying the Code for AWS Lambda

To run the process on AWS Lambda, modify the code by removing the if __name__ == "__main__": block and adding a lambda_handler function. The complete code becomes:

import datetime
import logging
import os
import time

import arxiv
import google.generativeai as genai
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

PAPER_TYPE_LIST = ["cs.AI", "cs.CY", "cs.MA"]
ARXIV_API_URL = "http://export.arxiv.org/api/query"
GEMINI_API_KEY = os.environ["GEMINI_API_KEY"]
GEMINI_MODEL = "gemini-2.0-flash"
SLACK_TOKEN = os.environ["SLACK_BOT_TOKEN"]  # Slack Bot token (xoxb-...)
SLACK_BOT_TOKEN = os.environ["SLACK_BOT_TOKEN"]
SLACK_CHANNEL = os.environ["SLACK_CHANNEL"]  # ID of the channel to post to (e.g., "C12345...")
MAX_RESULTS = 30


def get_papers(max_results: int = 10):
    # Create a query by combining the paper categories
    query = " OR ".join([f"cat:{paper_type}" for paper_type in PAPER_TYPE_LIST])
    # Removed date range query
    client = arxiv.Client()
    search = arxiv.Search(
        query=query, 
        max_results=max_results, 
        sort_by=arxiv.SortCriterion.SubmittedDate,
        sort_order=arxiv.SortOrder.Descending,
    )
    result_list = list(client.results(search))

    if not result_list:
        return result_list

    # Determine the latest published date from the results.
    latest_published = result_list[0].published
    threshold = latest_published - datetime.timedelta(hours=24)

    # Filter results to include only papers published within 24 hours of the latest result.
    filtered_results = [paper for paper in result_list if paper.published >= threshold]

    filtered_results = [
        {
            "title": paper.title,
            "summary": paper.summary,
            "pdf_url": paper.pdf_url,
            "published": paper.published,
        }
        for paper in filtered_results]

    return filtered_results


def generate_summary(abstract_text):
    genai.configure(api_key=GEMINI_API_KEY)
    model = genai.GenerativeModel(GEMINI_MODEL)  # Model to use (e.g., Gemini 1.5 Flash version)

    prompt = (
        "Please summarize the following paper abstract in a clear and easy-to-understand way for beginners, "
        "in English and within 300 characters.\n"
        "Also include the significance and results of the paper.\n"
        "Output only the summary result."
        "\n---\n\n"
        f"{abstract_text}"
    )
    response = model.generate_content(prompt)
    summary_text = response.text.strip()

    return summary_text


def send_slack_message(papers):
    all_messages = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        summary_text = paper["summary"]
        link = paper["pdf_url"]
        published = paper["published"]
        message_text = (
            f"{i}: *{title}*\n\n"
            f"{summary_text}\n\n"
            f"PDF: {link}\n"
            f"Published: {published}"
        )
        all_messages.append(message_text)

    all_message = "\n\n────────────────────────\n\n".join(all_messages)

    client = WebClient(token=SLACK_BOT_TOKEN)
    logger = logging.getLogger(__name__)
    # ID of the channel to which you want to send the message

    try:
        # Call the chat.postMessage method using the WebClient
        result = client.chat_postMessage(
            channel=SLACK_CHANNEL, 
            text=all_message
        )
        logger.info(result)

    except SlackApiError as e:
        logger.error(f"Error posting message: {e}")


def main():
    papers = get_papers(MAX_RESULTS)
    output_papers = []
    for i, paper in enumerate(papers, 1):
        title = paper["title"]
        abstract = paper["summary"]
        link = paper["pdf_url"]
        published = paper["published"]
        summary = generate_summary(abstract)
        output_papers.append({
            "title": title,
            "summary": summary,
            "pdf_link": link,
            "published": published
        })
        time.sleep(1)
    send_slack_message(output_papers)


def lambda_handler(event, context):
    main()
    return {
        'statusCode': 200,
        'body': "Success in sending message to Slack!"
    }

Scheduling the Lambda Function with Amazon EventBridge

To run the process at regular intervals, use Amazon EventBridge. Choose scheduled execution and configure the cron expression. For example, to execute the function every day at 6:30 AM, use the following cron expression:

cron(30 6 * * ? *)

Note: The time is in UTC. If you need to use your local time, adjust the schedule accordingly.

Set the Lambda function as the target and specify the function you created.

With this setup, every day at the specified time, the papers will be summarized and the summary will be posted to Slack.

Try Multimodal Search with ColQwen2!

M Sea Bass — Sat, 04 Jan 2025 03:01:24 +0000

In this article, we introduce how to use ColQwen2.

ColQwen2 is based on Qwen2-VL-2B and generates ColBERT-style multi-vector representations, enabling highly accurate searches across text and image inputs.

We will test ColQwen2 using Google Colab with an A100 GPU.

Library Installation

First, install the necessary libraries:

!pip install git+https://github.com/illuin-tech/colpali
!pip install pymupdf

Preparing Image Data

Next, prepare the image data. For this tutorial, we’ll use the ColPali paper.

Using pymupdf, we’ll extract images from the PDF file:

import pymupdf
import os

# Constants
DPI = 350  # Can be modified as needed

def convert_pdf_to_images(pdf_path, output_dir):
    """
    Convert PDF pages to images.
    Args:
        pdf_path (str): Path to the PDF file.
        output_dir (str): Directory to save images.
    """
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    pdf_document = pymupdf.open(pdf_path)

    for page_number in range(pdf_document.page_count):
        page = pdf_document[page_number]
        pix = page.get_pixmap(dpi=DPI)
        output_file = os.path.join(output_dir, f'page_{page_number + 1:02}.png')
        pix.save(output_file)

    pdf_document.close()

pdf_path = "/content/2407.01449v3.pdf"
output_dir = "output_images"
convert_pdf_to_images(pdf_path, output_dir)

Images will be saved in the "output_images" folder.

Searching the Images

Now, let’s use ColQwen2. Refer to the Huggingface page for sample code.

After downloading and uploading the paper PDF to Google Colab, execute the following code:

import glob, os

import torch
from PIL import Image

from colpali_engine.models import ColQwen2, ColQwen2Processor

device = "cuda:0" if torch.cuda.is_available() else "cpu"

print(f"cuda available: {torch.cuda.is_available()}")

model = ColQwen2.from_pretrained(
        "vidore/colqwen2-v0.1",
        torch_dtype=torch.bfloat16,
        device_map=device,  # or "mps" if on Apple Silicon
    ).eval()
processor = ColQwen2Processor.from_pretrained("vidore/colqwen2-v0.1")

# Your inputs
images = [Image.open(filepath) for filepath in glob.glob(os.path.join(output_dir, "*.png"))]
queries = [
    "What is the architecture of ColPali?",
    "How does it differ from previous studies?",
]

# Process the inputs
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)

# Forward pass
batch_size = 1  # Reduced batch size to 1
image_embeddings = []
for i in range(0, len(images), batch_size):
    batch = images[i : i + batch_size]
    resized_batch = [img.resize((512, 512)) for img in batch]  # Resize before processing
    batch_images = processor.process_images(resized_batch).to(model.device)
    with torch.no_grad():
        embeddings = model(**batch_images)
    image_embeddings.extend(embeddings)
with torch.no_grad():
    query_embeddings = model(**batch_queries)

image_embeddings = torch.stack(image_embeddings)

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

print(scores)

The scores are returned as a list (matrix):

tensor([[13.2500,  8.4375, 11.3750, 11.1875, 13.8125, 12.0000,  8.3125,  9.0000,
         10.4375,  8.7500, 10.4375, 11.6250,  7.8438,  7.4375,  9.9375,  8.0625,
          7.5000, 10.9375,  9.7500,  7.8750],
        [ 8.3750,  7.5000,  9.6250,  8.3125,  7.5625,  8.1250,  7.9688,  8.4375,
          8.5000,  9.0625,  7.7812,  8.3125,  7.5000,  7.9062,  8.6875,  7.9688,
          7.9062,  7.9688,  8.7500,  7.5000]])

Visualizing Scores

Let’s visualize the scores:

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np


scores_df = pd.DataFrame(scores.cpu().numpy(), columns=[f'Image {i+1}' for i in range(scores.shape[1])]).T
scores_df.index.name = 'Images'

# Create two separate bar plots side-by-side
plt.figure(figsize=(12, 6))

# First bar plot
plt.subplot(1, 2, 1)
sns.barplot(x=scores_df.index, y=scores_df[0], color="skyblue")
plt.title("Query: " + queries[0])
plt.xticks(rotation=45, ha="right")
plt.ylabel('Score')

# Second bar plot
plt.subplot(1, 2, 2)
sns.barplot(x=scores_df.index, y=scores_df[1], color="lightcoral")
plt.title("Query: " + queries[1])
plt.xticks(rotation=45, ha="right")
plt.ylabel('Score')

plt.tight_layout()
plt.show()

Inspecting the Top Results

Let’s check the top 2 results:

for query, high_idx in zip(queries, highest_score_indices.tolist()):
    print(f"{query}: {high_idx}")
    # Display the image
    image_path = os.path.join(output_dir, f"page_{high_idx+1}.png")
    display(Image.open(image_path))

Query 1: What is the architecture of ColPali?

Top 2 results:

1st: Page 5

2nd: Page 1

The Page 5 with the highest relevance includes the word “Architecture.” However, the architecture diagram on page 2 received a lower score.

Query 2: How does it differ from previous studies?

Top 2 results:

1st: Page 3

2nd: Page 10

Page 3 has contents of “Related Work,” but the start of the related work section on page 2 scored lower. Page 10, which includes references, scored higher, as expected.

Conclusion

We tested image search using ColQwen2. Searching entire PDF pages proved challenging; for practical use, extracting figures as standalone images might improve results.

To extract text, images, and tables more effectively from PDFs, consider tools like pymupdf2llm.

RAPTOR: A Novel Tree-Based Retrieval System for Enhancing Language Models – Research Summary

M Sea Bass — Sat, 14 Dec 2024 07:16:22 +0000

This study introduces RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), a novel tree-based retrieval system designed to enhance search capabilities for extended language models.

1. What is this study about?

RAPTOR complements the knowledge of large language models (LLMs) by learning hierarchical representations of text, enabling information retrieval at various levels of abstraction. Specifically, it clusters text chunks and recursively generates summaries to build a tree structure that encompasses everything from an overall summary to detailed information. During search, this tree structure allows for efficient retrieval of information at the appropriate level of granularity based on the query, producing more accurate and comprehensive answers.

2. How does it stand out from previous research?

Traditional retrieval-augmented methods typically target short, consecutive text chunks, limiting their ability to grasp the broader context of an entire document. RAPTOR, by leveraging a tree structure, captures both high-level themes and low-level details, effectively representing and utilizing large-scale discourse structures. This approach allows RAPTOR to handle complex, multi-step reasoning questions that are challenging for existing methods.

3. What are the key technical contributions?

Recursive Abstraction: RAPTOR constructs a tree structure by clustering text chunks and recursively generating summaries. In this tree, leaf nodes represent the original text chunks, while higher-level nodes represent more abstract summaries.
GMM-Based Clustering: Soft clustering using Gaussian Mixture Models (GMMs) is employed to group similar text chunks. This approach enables nodes to belong to multiple clusters, allowing for more flexible representation.
Summarization with Language Models: Powerful language models, such as GPT-3.5-turbo, are used to generate summaries of the clustered text.
Tree Traversal and Collapsed Trees: For retrieval, two strategies are used—hierarchical traversal of the tree (tree scanning) and collapsing the entire tree into a single layer for broader searches (collapsed tree). These strategies enable efficient retrieval of relevant information.

4. How was its effectiveness validated?

RAPTOR’s performance was evaluated using three question-answering datasets: NarrativeQA, QASPER, and QuALITY. UnifiedQA, GPT-3, and GPT-4 were used as readers, and RAPTOR was compared against existing retrieval methods such as SBERT, BM25, and DPR. The results showed that RAPTOR outperformed traditional methods across all datasets, particularly excelling in answering questions requiring complex reasoning.

When combined with GPT-4, RAPTOR achieved state-of-the-art results in several tasks, such as improving absolute accuracy on the QuALITY benchmark by 20%. Additionally, an analysis of the contribution of each tree layer demonstrated that higher-level nodes play a critical role in addressing thematic or multi-hop questions effectively.

Understanding RAG and Long-Context LLMs: Insights from the SELF-ROUTE Hybrid Approach

M Sea Bass — Sat, 14 Dec 2024 02:10:44 +0000

Retrieval Augmented Generation (RAG) and Long-Context Large Language Models (LC LLMs) are two key methods for handling long-context information. RAG is efficient and cost-effective, while LC LLMs offer better performance but require more resources. A recent paper, Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, compares these methods and proposes a hybrid approach called SELF-ROUTE. This approach combines the strengths of both to achieve high performance with lower costs. In this article, we’ll break down the key findings and explain the new hybrid method.

What Kind of Research Is This? (Overview)

In recent years, large language models (LLMs) such as Gemini and GPT-4 have significantly improved their ability to directly understand long-context inputs. However, retrieval-augmented generation (RAG) remains a notable method for efficiently processing lengthy documents by retrieving relevant information and inputting it into LLMs. This study evaluates and compares the performance and efficiency of RAG and LC (Long-Context) LLMs using multiple public datasets and state-of-the-art LLMs. The results show that while LC LLMs outperform RAG in performance when sufficient resources are available, RAG has the advantage of significantly lower computational costs. Based on this trade-off, the study proposes a hybrid method, SELF-ROUTE, which routes queries to RAG or LC based on the model’s self-assessment, achieving computational cost reductions while maintaining performance comparable to LC.

How Does It Compare to Previous Research? (Key Differences)

Previous research (Xu et al., 2023) reported that RAG outperforms long-context prompts. However, this study found that LC LLMs surpass RAG in performance by leveraging more powerful LLMs and considering extended context lengths. This finding reflects recent advancements in LLMs’ long-context understanding capabilities. Furthermore, the study highlights that predictions by RAG and LC LLMs often align for many queries, leading to the novel proposal of SELF-ROUTE, which effectively combines the two methods based on this insight.

What Is the Core Technique or Methodology?

The key innovation of SELF-ROUTE lies in utilizing the LLM's "self-assessment" capability to dynamically route queries to either RAG or LC. Specifically, RAG processes the query along with retrieved text chunks, and the LLM predicts whether it can answer the query. If the LLM predicts it cannot answer, LC LLM generates an answer from the full context. This approach allows a large portion of queries to be handled by the computationally cheaper RAG, minimizing the use of LC LLM, thereby improving overall efficiency. Additionally, the study analyzed RAG’s failures and categorized them into four types: multi-step reasoning, general queries, complex queries, and implicit queries, contributing significantly to RAG’s improvement.

How Was Its Effectiveness Validated?

The effectiveness of SELF-ROUTE was validated by comparing the performance and computational costs of RAG, LC LLM, and SELF-ROUTE using multiple public datasets (LongBench, ∞Bench) and three state-of-the-art LLMs (Gemini-1.5-Pro, GPT-40, GPT-3.5-Turbo). Performance metrics such as F1 score (free-text tasks), accuracy (multiple-choice tasks), and ROUGE (summarization tasks) were used, along with computational cost measured by input token counts. The results demonstrated that SELF-ROUTE achieved performance comparable to LC LLM while significantly reducing token usage. Further evaluations included an ablation study varying the number of retrieved chunks (k), analyses using synthetic datasets, and considerations of the impact of LLMs' internal knowledge. These multifaceted analyses validated the robustness and applicability of the proposed method, showing that SELF-ROUTE can maintain LC-comparable performance while significantly reducing computational costs.

How to Run stable-diffusion-3.5-large-turbo on Google Colab

M Sea Bass — Sun, 08 Dec 2024 15:42:22 +0000

stable-diffusion-3.5-large-turbo is a high-precision text-to-image model.

This guide will explain how to set up and run the model on Google Colab.

Prerequisites

Visit Huggingface.

To use stable-diffusion-3.5-large-turbo, you need a Huggingface account.

If you don’t already have one, please create an account.

Once signed up, you’ll see the following screen:

Enter the required information, and you’ll gain access to the model immediately.

If you wish to download and use the model, you’ll need an access token. Create one from your account page:

Navigate to your account page via the profile icon in the upper-right corner, go to the Access Token tab, and create a token by selecting Create new token.

Running the Code

Install Required Libraries

First, install the necessary libraries in Google Colab:

!pip install --quiet -U transformers

The -U option updates the library to its latest version, and --quiet suppresses download messages.

Authenticate Your Account

Authenticate your Huggingface account by running the following command and entering the token you created earlier:

!huggingface-cli login

Download the Model

Load and set up the model using the following Python code:

import torch
from diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3.5-large-turbo", torch_dtype=torch.bfloat16)
pipe = pipe.to("cuda")

Note: The model consumes approximately 27GB of memory.

Generate an Image

Test the setup by running this code to generate an image:

prompt = "A capybara holding a sign that reads Hello Fast World"
save_filename = "capybara.png"
image = pipe(
    prompt,
    num_inference_steps=4,
    guidance_scale=0.0,
).images[0]

You can find explanations for these arguments in the Diffusers GitHub documentation.

Save and display the generated image:

image.save(save_filename)
image

Quick Paper Overview: More Agents Is All You Need

M Sea Bass — Thu, 05 Dec 2024 12:35:10 +0000

I found this paper fascinating, so I’d like to provide a quick overview of More Agents is All You Need.

1. What's It About?

By leveraging multiple LLMs in a simple sampling-and-voting approach, this study demonstrates significant performance improvements in LLMs. The performance boost is observed across tasks of varying difficulty, with particularly notable gains in more challenging tasks. While both small and large LLMs benefit, smaller models exhibit the most pronounced improvements.

This method can also be combined with existing techniques like Chain-of-Thought (CoT), further enhancing performance in certain tasks.

2. What is Differences from Prior Work

In previous research, methods like CoT-SC applied voting to diverse answers generated by chain-of-thought prompts, particularly for reasoning tasks. This study, however, explores whether simply increasing the number of agents (LLMs) without relying on chain-of-thought prompts or similar techniques can enhance performance. Additionally, it investigates this approach across a broader range of tasks, not just reasoning-based ones.

While some prior work has also explored the use of multiple LLMs, they often relied on supervised learning for ensemble methods. In contrast, this study uses a straightforward voting mechanism, making it a more accessible and easy-to-implement approach.

Furthermore, although previous research has delved into LLM collaboration, this study focuses specifically on the relationship between the number of agents and performance, rather than architectural aspects.

3. What Are the Key Techniques or Methods?

The key contribution of this study is demonstrating that even a very simple voting mechanism can lead to performance improvements. The specific algorithm is as follows:

Generate multiple responses using the LLM, repeating this process several times.
Calculate similarity scores between different responses and sum these scores for each response.
Select the response with the highest total similarity score.

For similarity scoring:

In open-ended generation tasks (e.g., code generation), BLEU score is used.
In close-ended tasks (e.g., multiple-choice questions), frequency of occurrence is used.

4. How Was Its Effectiveness Validated?

The study evaluated its method across three tasks: Arithmetic Reasoning, General Reasoning, and Code Generation. It also examined the combination of this approach with existing techniques like Chain-of-Thought (CoT). Accuracy was used as the evaluation metric.

The models used in the experiments included Llama-2-13B, Llama-2-70B, and GPT-3.5-Turbo.

The results confirmed that increasing the number of agents consistently improved accuracy. In some tasks, multiple instances of Llama-2-13B outperformed a single Llama-2-70B, and multiple instances of Llama-2-70B surpassed GPT-3.5-Turbo. Additionally, combining this approach with existing techniques like CoT led to further performance gains in certain tasks.

Using Azure OpenAI Service with Local bolt.new

M Sea Bass — Sun, 24 Nov 2024 09:58:40 +0000

bolt.new is an AI-powered full-stack web development platform. It can also be run locally from a GitHub repository.

While bolt.new uses Anthropic Claude 3.5 Sonnet by default, this time, we’ll modify it to work with Azure.

Implementing Azure

When making code modifications, the original code will be retained as comments.

Below, only the modified sections of the code are shown, while unchanged original code is abbreviated with "...".

Adding Libraries

First, add the necessary library.

In bolt.new/package.json, include the @ai-sdk/azure library as follows:

{
  ...
  "dependencies": {
    "@ai-sdk/anthropic": "^0.0.39",
    "@ai-sdk/azure": "^1.0.5",  // <- Added
    ...
  },
  ...
}

Setting Azure Environment Variables

First, add the Azure resource name, API key, and deployment name to the bolt.new/.env.local file:

...
AZURE_RESOURCE_NAME=YOUR_RESOURCE_NAME
AZURE_API_KEY=YOUR_API_KEY
AZURE_DEPLOYMENT_NAME=YOUR_DEPLOYMENT_NAME

The resource name refers to the ${resourceName} part in the Azure OpenAI Endpoint URL:

https://${resourceName}.openai.azure.com/.

Note that the deployment name may differ from the model name, so be cautious.

Next, let’s call the environment variables.

Add the following code to bolt.new/app/lib/.server/llm/apt-key.ts:

...

export function getAzureEnv(cloudflareEnv?: Env) {
  // Cloudflare Pages environment
  if (cloudflareEnv) {
    return {
      resourceName: cloudflareEnv.AZURE_RESOURCE_NAME,
      apiKey: cloudflareEnv.AZURE_API_KEY,
      deploymentName: cloudflareEnv.AZURE_DEPLOYMENT_NAME,
    };
  }
  // Local development environment
  return {
    resourceName: env.AZURE_RESOURCE_NAME || '',
    apiKey: env.AZURE_API_KEY || '',
    deploymentName: env.AZURE_DEPLOYMENT_NAME || '',
  };
}

When running locally with pnpm, the environment variables are accessed via cloudflareEnv. For apps deployed to Azure or similar platforms, the environment variables are registered and accessed through env.

Using Azure OpenAI

To use Azure OpenAI, incorporate the configured environment variables.

Add the following code to bolt.new/app/lib/.server/llm/models.ts:

...
import { createAzure } from '@ai-sdk/azure';
...

export function getAzureModel(resourceName: string, apiKey: string, deploymentName: string) {
  const azure = createAzure({
    resourceName: resourceName,
    apiKey: apiKey,
  });
  return azure(deploymentName);
}

When switching to Azure OpenAI, it is necessary to confirm the max tokens setting.

Update bolt.new/app/lib/.server/llm/constants.ts as follows:

// see https://docs.anthropic.com/en/docs/about-claude/models
// export const MAX_TOKENS = 8192;
// see https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure%2Cglobal-standard%2Cstandard-chat-completions#gpt-4o-and-gpt-4-turbo
export const MAX_TOKENS = 16384;

The maximum token count depends on the model version deployed on Azure OpenAI Service.

For example, according to Azure OpenAI Models, the GPT-4o version from 2024-08-06 supports up to 16,384 output tokens, while the 2024-05-13 version supports only 4,096.

Next, update the model calling code.

Modify bolt.new/app/lib/.server/llm/stream-text.ts as follows, keeping the code for Anthropic as comments:

...
// import { getAPIKey } from '~/lib/.server/llm/api-key';
import { getAzureEnv } from '~/lib/.server/llm/api-key';
// import { getAnthropicModel } from '~/lib/.server/llm/model';
import { getAzureModel } from '~/lib/.server/llm/model';
...

// export function streamText(messages: Messages, env: Env, options?: StreamingOptions) {
//   return _streamText({
//     model: getAnthropicModel(getAPIKey(env)),
//     system: getSystemPrompt(),
//     maxTokens: MAX_TOKENS,
//     headers: {
//       'anthropic-beta': 'max-tokens-3-5-sonnet-2024-07-15',
//     },
//     messages: convertToCoreMessages(messages),
//     ...options,
//   });
// }

export function streamText(messages: Messages, env: Env, options?: StreamingOptions) {
  const { resourceName, apiKey, deploymentName } = getAzureEnv(env);
  if (!resourceName || !apiKey || !deploymentName) {
    throw new Error('Azure OpenAI credentials are not properly configured');
  }

  const stream = _streamText({
    model: getAzureModel(resourceName, apiKey, deploymentName),
    system: getSystemPrompt(),
    maxTokens: MAX_TOKENS,
    headers: {},
    messages: convertToCoreMessages(messages),
    ...options,
  });

  return stream;
}

Execution

Once the modifications are complete, execute the following commands to start bolt.new:

pnpm install
pnpm run build
pnpm run start

Please note that depending on your Azure OpenAI Service plan, you may encounter rate limits.

Forem: M Sea Bass

I Made AI Code for Me from My Phone (Slack Codex CLI)

Introduction

How It Works

See It in Action

1. Send a Mention in Slack

2. Check the App

3. Review and Merge the PR(Pull Request)

What You Can Do

Setup

Conclusion

Related Links

Practical Guide to MCP (Model Context Protocol) in Python

Introduction

1. Core Concepts of MCP

2. Build a Server with stdio

2.1 Basic Server Code

2.2 Async Tools and Progress Reporting

3. Build a stdio Client

4. Build a Server with streamable-http

5. Build a streamable-http Client

6. A Client for Multiple Servers

7. Using MCP from Generative AI

7.1 Which schema to convert to (Responses vs. Chat Completions)

7.2 How to obtain the necessary information (inspect / docstring / MCP meta)

7.3 Implementation code and concrete methods

Conclusion

I Tried Building a Whiteboard App with Claude 4.5 Sonnet

Development Process

User Requirements Definition

Requirements Definition

Vibe Coding

Understanding and Implementing ReAct

What is ReAct?

A Simple Implementation of ReAct

1. Import Required Libraries

2. Create a Vector Database

3. Create and Index Embeddings

4. Define a Vector Search Function

5. Generate Responses Using GPT

6. Execute the ReAct Process

7. Sample Output

Automating Daily arXiv Paper Summaries with Slack Notifications

Automating Daily ArXiv Paper Summaries and Slack Notifications

Python Code for the Process

Using Gemini for Summarization

Running the Process on AWS Lambda

Local Setup Steps

Modifying the Code for AWS Lambda

Scheduling the Lambda Function with Amazon EventBridge

Try Multimodal Search with ColQwen2!

Library Installation

Preparing Image Data

Searching the Images

Visualizing Scores

Inspecting the Top Results

Query 1: What is the architecture of ColPali?

Query 2: How does it differ from previous studies?

Conclusion

RAPTOR: A Novel Tree-Based Retrieval System for Enhancing Language Models – Research Summary

1. What is this study about?

2. How does it stand out from previous research?

3. What are the key technical contributions?

4. How was its effectiveness validated?

Understanding RAG and Long-Context LLMs: Insights from the SELF-ROUTE Hybrid Approach

What Kind of Research Is This? (Overview)

How Does It Compare to Previous Research? (Key Differences)

What Is the Core Technique or Methodology?

How Was Its Effectiveness Validated?

How to Run stable-diffusion-3.5-large-turbo on Google Colab

Prerequisites

Running the Code

Install Required Libraries

Authenticate Your Account

Download the Model

Generate an Image

Quick Paper Overview: More Agents Is All You Need

1. What's It About?

2. What is Differences from Prior Work

3. What Are the Key Techniques or Methods?