Forem: Pierre-Louis Biojout

Building an AI powered app to chat with any website

Pierre-Louis Biojout — Wed, 06 Mar 2024 04:29:56 +0000

Github repo and link to the live app at the end of the article

Let’s build a Web Application to chat with any website using the Exa Neural Search API and OpenAI GPT-3.5 turbo within a Streamlit front-end.

Creating the web retrieval function

As we want to RAG on the whole website, and not just a single page, this makes this step a bit tricky. The hack we came up with is using the Exa RAG API and to constrain it to only the domain of the website we want to chat with.
After getting a free API key on their website, we can build our retrieval function:

pip install exa-py loguru

from exa_py import Exa
from typing import Dict, List, Tuple
from loguru import logger

exa = Exa("EXA_API_KEY")

def get_text_chunks(
    query: str,
    url: str,
    num_sentences: int = 15,
    highlights_per_url: int = 5,
) -> Tuple[List[str], List[str]]:
    """
    Return a lsit of text chunks from the given URL that are relevant to the query.
    """

    highlights_options = {
        "num_sentences": num_sentences,  # how long our highlights should be
        "highlights_per_url": highlights_per_url,
    }
    search_response = exa.search_and_contents(
        query,
        highlights=highlights_options,
        num_results=10,
        use_autoprompt=True,
        include_domains=[url],
    )

    chunks = [sr.highlights[0] for sr in search_response.results]
    url_sources = list(set([sr.url for sr in search_response.results]))

    return chunks, url_sources

Using GPT-3.5 turbo to generate an answer with the content chunks

To generate an answer to the user query, we will pass to GPT-3.5 turbo the content chunks retrieved from the website as context.
We use Llama index RAG prompt template available here to build our prompt:

def generate_prompt_from_chuncks(chunks: List[str], query: str) -> str:
    """
    Generate a prompt from the given chunks and question.
    TODO: add a check on token lenght to avoid exceeding the max token length of the model.
    """
    assert len(chunks) > 0, "Chunks should not be empty"

    concatenated_chunks = ""
    for chunck in chunks:
        concatenated_chunks += chunck + "\n\n"

    prompt = f"""
    Context information is below.
    ---------------------
    {concatenated_chunks}
    ---------------------
    Given the context information and not prior knowledge, answer the query.
    Do not start your answer with something like Based on the provided context information...
    Query: {query}
    Answer: 
    """
    return prompt

Using GPT-3.5 turbo to generate an answer with the content chunks

Then, we can invoke GPT-3.5 turbo on this prompt (optionally with the previous messages):

pip install openai

from openai import OpenAI

openai_client = OpenAI(api_key=config.OPENAI_API_KEY)

def invoke_llm(
    prompt: str,
    model_name: str = "gpt-3.5-turbo",
    previous_messages: List[Dict[str, str]] = None,
) -> str:
    """
    Invoke the language model with the given prompt and return the response.
    """
    if previous_messages is None:
        previous_messages = []
    completion = openai_client.chat.completions.create(
        model=model_name,
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant replying to questions given a context.",
            }
        ]
        + previous_messages
        + [
            {"role": "user", "content": prompt},
        ],
        temperature=0.0,
    )

    return completion.choices[0].message.content

Now, we have our answering function ready to go:

def query2answer(
    query: str, url: str, session_messages: List[Dict[str, str]]
) -> Tuple[str, List[str]]:
    """
    Given a query and an URL, return the answer to the query.
    """
    try:
        logger.info(f"Query: {query}")
        chuncks, url_sources = get_text_chunks(query, url)
        logger.info(f"Retrieved {len(chuncks)} chunks from {url}")
        prompt = generate_prompt_from_chuncks(chuncks, query)
        # TODO: add a check on token lenght to avoid exceeding the max token length of the model.
        llm_answer = invoke_llm(prompt, previous_messages=session_messages)
        logger.info(f"Answer: {llm_answer}")
    except Exception as e:
        logger.error(f"An error occurred: {e}")
        llm_answer = "Sorry, I was not able to answer. Either you setup a wrong URL or the URL is too new."
        url_sources = []
    return llm_answer, url_sources

Using Streamlit to build the frontend

Using Streamlit, we can easily build a frontend in python for our agent:

pip install streamlit

import streamlit as st
from agent import query2answer
from urllib.parse import urlparse
import time
import config

# Initialize URL
# Check the query parameters for a URL
if "url" in st.query_params:
    # Check it is note None
    if st.query_params.url and st.query_params.url != "None":
        st.session_state.url = st.query_params.url
if "url" not in st.session_state:
    st.session_state.url = None
# Initialize chat history
if "messages" not in st.session_state:
    st.session_state.messages = []

st.markdown(
    "# 📖 url2chat - Chat with any website"
)

ROLE_TO_AVATAR = {
    "user": "🦸‍♂️",
    "assistant": "📖",
}

if st.session_state.url is None:
    url = st.text_input("Enter the URL of a website to chat with it")
    if url:
        # Format checks
        if not url.startswith("http"):
            url = "https://" + url

        # Parse the URL to only have the domain
        o = urlparse(url)
        domain = o.hostname

        st.session_state.url = f"https://{domain}"
        # Set the URL as a query parameter to trigger a rerun
        st.query_params.url = f"https://{domain}"
        # Trigger a rerun to start chatting
        time.sleep(0.5)
        st.rerun()

else:
    # Add the URL as a query parameter (the rerun will remove it from the URL bar)
    st.query_params.url = st.session_state.url
    # Button to change the URL
    col1, col2 = st.columns([1, 1])
    with col1:
        if st.button("Change URL", use_container_width=True):
            st.session_state.url = None
            st.query_params.pop("url", None)
            st.session_state.messages = []
            # We need to add a small delay, otherwise the query parameter is not removed before the rerun
            time.sleep(0.5)
            st.rerun()
    with col2:
        if st.button("Clear chat", use_container_width=True):
            st.session_state.messages = []
            st.rerun()

    with st.chat_message("assistant", avatar=ROLE_TO_AVATAR["assistant"]):
        st.markdown(f"You're chatting with {st.session_state.url}. Ask me anything! 📖")

    # Display chat messages from history on app rerun
    for message in st.session_state.messages:
        with st.chat_message(message["role"], avatar=ROLE_TO_AVATAR[message["role"]]):
            st.markdown(message["content"])

    # Accept user input
    if prompt := st.chat_input("What is this website about?"):
        # Add user message to chat history
        st.session_state.messages.append({"role": "user", "content": prompt})
        # Display user message in chat message container
        with st.chat_message("user", avatar=ROLE_TO_AVATAR["user"]):
            st.markdown(prompt)

        # Display assistant response in chat message container
        chat_answer, url_sources = query2answer(
            prompt, st.session_state.url, st.session_state.messages
        )
        with st.chat_message("assistant", avatar=ROLE_TO_AVATAR["assistant"]):
            st.markdown(chat_answer)
            # Display the sources in a hidden accordion container
            with st.expander("Sources", expanded=False):
                for source in url_sources:
                    st.markdown("- " + source)

        st.session_state.messages.append({"role": "assistant", "content": chat_answer})

Adding text analytics to understand how our app performs and add user feed

Now that our app is working, let’s see how people are using it and how it is perfoming. To do so, we will use phospho, an open-source text analytics solution. In this example, we will use the free trial of the hosted version but you can self host it (see the github repo for more info on how to do so).
First we need to get our phospho project id and API key and to add it to our .streamlit/secrets.toml file:

PHOSPHO_API_KEY=""
PHOSPHO_PROJECT_ID=""

pip install --upgrade phospho

Then, in our streamlit file, we can start logging messages:

import phospho

phospho.init()

# ...

phospho.log(input=prompt,
            output=chat_answer,
            metadata={"sources": url_sources},
        )

phospho enable us to handle sessions. Let’s add a session support (see full file on github).

Let’s handle feedbacks from our users (you will need to have put in place the session id from above):

pip install streamlit_feedback

from streamlit_feedback import streamlit_feedback

# ...

# Add feedback button
def _submit_feedback(feedback: dict):
    # Add a check if phospho is setup
    if config.PHOSPHO_API_KEY and config.PHOSPHO_PROJECT_ID:
        phospho.user_feedback(
            task_id=phospho.latest_task_id,
            raw_flag=feedback["score"],
            notes=feedback["text"],
        )
        st.toast(f"Thank you for your feedback!")
    else:
        st.toast(f"phospho is not setup, feedback not sent.")


if len(st.session_state.messages) > 1:
    feedback = streamlit_feedback(
        feedback_type="thumbs",
        optional_text_label="[Optional] Please provide an explanation",
        on_submit=_submit_feedback,
        # To create a new feedback component for every message and session, you need to provide a unique key
        key=f"{st.session_state.session_id}_{len(st.session_state.messages)}",
    )

Now, we can use phospho to detect some events of interest:

When the assistant answer that it doesn’t have the information
When the users wants to take an action (for instance buying a good or a service)

Conclusion

In this article, we’ve taken a deep dive into how to build a sophisticated web application, url2chat, that enables users to chat with any website. Leveraging the Exa Neural Search API, OpenAI GPT-3.5 turbo, and Streamlit, we created a system that extracts relevant information from entire websites, generates context-aware responses, and presents it all within a user-friendly interface.

Possible improvements

According to the data we collected using phospho, the user experinece on our app isn’t meeting our quality standard. Some possible improvements are:

not using a RAG search API but passing the whole website in the LLM context window (only suitable for small websites)
use the sitemap to find pages relevant to the query, and then pass these pages to the LLM

Want to test it?

Clone the Github repo here and run it locally or use the version deployed on Streamlit Community Cloud here.

Add analytics and AB tests to your LLM app in JavaScript using phospho

Pierre-Louis Biojout — Tue, 23 Jan 2024 16:45:06 +0000

Building an AI agent in JavaScript is now easier than ever. There is plenty of libraries like Langchain JS or Vercel AI to build cool products.

But like any cutting-edge technology, it comes with its own set of challenges. Hallucinations, incorrect responses, and misunderstandings can plague your LLM based product.

How to build an AI people want? You need to measure the quality and user satisfaction of your AI-powered product. It’s important to set clear goals and KPIs for your AI agent, focusing not just on its technical capabilities but also on how it meets the needs of its users.

You can’t build great products without monitoring how they are used.

So the first step of the journey is to start collecting analytics to build relevant KPIs.

Wait, what are analytics?

In software, analytics spans from performance metrics to user satisfaction insights :

Performance Analytics: Dive deep into code execution details to catch bugs and optimize performance. Is everything working as intended?
Product Analytics: Assess whether your agent meets the expectations of end-users. Am I making my users happy?

To create exceptional software, it’s crucial to balance both types of analytics. And it’s the same for an agent or an AI.

What analytics for your LLM app?

In order of priority, you want make sure that :

Your code runs well and without bugs or crashes.
The agent answers correctly in the main use case.
The doesn’t do something crazy in edge cases.
Users like the agent.

Now, the JavaScript code!

Assuming you have a basic JavaScript agent like this :

const myAgent = (query) => {
  // Here, you'd do complex stuff or calls to LLM APIs like OpenAI.
  // But for this example we'll just return the same answer every time.
  return "Hello World!";
};

You want to collect user inputs and associated outputs of your app. For this example, let’s use phospho, a product analytics platform for LLM apps.

Step 1 : Install the phospho module

npm I phospho

Step 2: Create an account on the phospho platform. Add your API key and project id as environment variables.

export PHOSPHO_API_KEY = "YOUR_API_KEY"
export PHOSPHO_PROJECT_ID = "YOUR_PROJECT_ID"

Step 3: Start capturing interactions of users with your app

import { phospho } from "phospho";

phospho.init();

const myAgent = (query) => {
  // Here, you'd do complex stuff.
  // But for this example we'll just return the same answer every time.
  return "It's Paris of course.";
};

// Log events to phospho by passing strings directly
phospho.log({
  input: question,
  output: myAgent(question),
});

Now your done! You will collect every interaction with your app, and now be able to build relevant KPIs to improve your product.

How to improve the product?

In the end, there is only one thing that matters : are you building something people want?

1. Understand how your product is currently used

phospho also detects events. For example, here phospho detected that this interaction was a “question answering” event. Set up custom events that trigger webhooks when detected. For example, to receive a slack message when a user discuss a certain topic.

2. Measure the success of the product in doing what the users want

phospho automatically labels the task as a success or a failure with default criterias. It can be improved with feedback given by you or by your users. Therefor, when a new tasks is logged to phospho, it can be classified as a successful or unsuccessful ineractions based on your users (or team) criteria

3. Improve the product : AB tests

Now it’s time to improve this success score! Release several new versions and use phospho’s AB testing tools to see which ones perform better.

Conclusion

In conclusion, enhancing the performance and user satisfaction of your JavaScript agent involves adding analytics into your development process as early as possible.

By strategically addressing both performance and product analytics, you can ensure that your agent is actually doing what your users are expecting.

Explore phospho to learn more about how to improve your LLM app.

And you? What strategy did you put in place to improve your LLM app? Let me know in the comments below.

Deploy a Langchain agent as an API endpoint in 10 minutes

Pierre-Louis Biojout — Wed, 20 Sep 2023 15:06:05 +0000

Github repository at the end of the article

So you coded a cool Langchain agent in Python. Congrats! In this tutorial, we will walk you through the process of making it an OpenAPI endpoint, which can be deployed and called as an API, allowing you to seamlessly integrate it into your product or workflows. All that in only 10 minutes. Let’s get started!

Installation and setup

You need to have Python 3.8 or higher installed.

(Optional) Create a new virtual environment



python3 -m venv venv
source venv/bin/activate

Install the required packages

langchain for the orchestration of the agent
openai for the OpenAI API
phospho for the deployment of the agent
python-dotenv for the environment variables (like api keys)



pip install --upgrade langchain openai phospho python-dotenv
pip freeze > requirements.txt

Store your OpenAI API key

In this example, we are going to use OpenAI as our LLM provider for our Langchain agent. But you can use any LLM provider you wish.



OPENAI_API_KEY=your-api-key

Create an .env file and store your OpenAI API key in it. For extra security, you can create a new OpenAI key for this project.

Build the agent logic

Create a new langchain agent

Create a main.py python file at the route of the project. You must name it main.py since phospho will look for this file to initialize the agent.

Let’s load the environment variables from the .env file :



# main.py

# Load environment variables

from dotenv import load_dotenv
load_dotenv()

Now, let’s create a basic chatbot using the OpenAI API. As you can imagine, the process would be roughly the same for more complex agents.



# main.py

# Load environment variables

from dotenv import load_dotenv
load_dotenv()

# use langchain to easily interact with the OpenAI API

from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)
from langchain.chat_models import ChatOpenAI

chat = ChatOpenAI(model_name="gpt-3.5-turbo",temperature=0.3)
messages = [
    SystemMessage(content="You are a funny and helpful assistant."),
    HumanMessage(content="Explain to me quantum physics."),
]
response=chat(messages)

print(response.content,end='\n')

Try it locally :



python main.py

Now, let’s package this logic into a function that we can use in our agent :



# main.py

# Load environment variables

from dotenv import load_dotenv
load_dotenv()

# import schema for chat messages and ChatOpenAI in order to query chatmodels GPT-3.5-turbo or GPT-4

from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)
from langchain.chat_models import ChatOpenAI

chat = ChatOpenAI(model_name="gpt-3.5-turbo",temperature=0.3)

def my_agent(input):
    messages = [
        SystemMessage(content="You are a funny and helpful assistant."),
        HumanMessage(content=input),
    ]

    response=chat(messages)

    return response.content

Deploy the agent

Having the agent run locally is cool. But now, let’s deploy it so everyone can use it.

To do so, we are going to package our langchain agent into a phospho agent. phospho is an agnostic Plateform as a Service to deploy and manage autonomous agents. They also provide an SDK to help developers go from prototype to production with their agents.

Here, we only want to create a basic conversational agent. So we only need to create a chat route.



# main.py

from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)
from langchain.chat_models import ChatOpenAI
from phospho import Agent, Message

# Load environment variables

from dotenv import load_dotenv
load_dotenv()

# Define our langchain agent
chat = ChatOpenAI(model_name="gpt-3.5-turbo",temperature=0.3)

def my_agent(input):
    messages = [
        SystemMessage(content="You are a funny and helpful assistant."),
        HumanMessage(content=input),
    ]

    response=chat(messages)

    return response.content

# Package it into a phospho agent

agent = Agent()

@agent.chat()
def my_chat(message):
    # Use our agent logic to generate a response
    response = my_agent(message.content)
    # Return the response in a Message object
    return Message(response)

That’s it!

With this code, we have created a new agent with a single route /chat that will call our my_agent function and send the response back to the user. Note that you need to send an object of type Message as a response (view the documentation for more info).

Create a new Github repository

At the root of your project, create a new Github repository and push your code to the main branch. Do not forget to add a .gitignore file to ignore the .env file (there is your API key inside) and the venv folder :



  
  
  Ignore virtualenv


venv/

  
  
  Ignore .env


.env

  
  
  Ignore pycache


pycache/

Create a new phospho project

Go to your phospho account dashboard and create a new project.

For the project name :

respect naming conventions, use just alphanumeric characters (no spaces, no special characters, no emojis, etc…)
if the name is already taken, choose a new one. Come on, be creative!

Select the github repository you want to link to your phospho project.

Setup the environment variables

Remember, we have stored our OpenAI API key in a .env file. But this file is not available on the server since we didn't push it to Github. Our code won't work if the API key is not available as an environment variable to the langchain agent.

On the phospho webapp, go to your project page and click on the Manage Env Variables tab. Then, click on the Environment variables tab and add a new variable called OPENAI_API_KEY with your OpenAI API key as a value.

At the next restart of your agent, it will be able to use the OpenAI API (to force a restart, you can trigger a new deployment).

Deploy the agent

Once project is created, just push some code to the main branch or go to the project page and trigger the deployment manually using the button.

After a one or two minutes, your agent should be deployed and ready to use. You have the endpoint URL on your project page.

Test your agent

Your agent is now deployed as an OpenAPI endpoint. You can find the URL of the API endpoint of your agent on the project page. You can test it directly using the API or by going in your browser to the URL and adding /docs at the end.

Unless you go viral, you should be able to use the free plan (no credit card required to start).

Github repo with the source code

You can find all the source code of this tutorial in this github repo

Congratulations! You have successfully deployed a simple conversational agent using Langchain, OpenAI and phospho. Feel free to explore and enhance the capabilities of your agent further.

Want to give anonymous feedback? Fill in the google form here.

Forem: Pierre-Louis Biojout

Building an AI powered app to chat with any website

Creating the web retrieval function

Using GPT-3.5 turbo to generate an answer with the content chunks

Using GPT-3.5 turbo to generate an answer with the content chunks

Using Streamlit to build the frontend

Adding text analytics to understand how our app performs and add user feed

Conclusion

Possible improvements

Want to test it?

Add analytics and AB tests to your LLM app in JavaScript using phospho

Wait, what are analytics?

What analytics for your LLM app?

Now, the JavaScript code!

How to improve the product?

1. Understand how your product is currently used

2. Measure the success of the product in doing what the users want

3. Improve the product : AB tests

Conclusion

Deploy a Langchain agent as an API endpoint in 10 minutes

Installation and setup

(Optional) Create a new virtual environment

Install the required packages

Store your OpenAI API key

Build the agent logic

Create a new langchain agent

Deploy the agent

Create a new Github repository​

Ignore virtualenv

Ignore .env

Ignore pycache

Create a new phospho project

Setup the environment variables

Deploy the agent

Test your agent

Github repo with the source code

Create a new Github repository