Forem: Sinapsi

OpenAI's CLIP Model for Zero-Shot Image Similarity

EricManara — Tue, 05 Mar 2024 14:45:45 +0000

Image recognition is always a difficult task to get right: with accuracy being inversely proportional to performance, it's always either too slow or too unreliable. OpenAI's CLIP Model, however, makes it a lot easier to check the similarity between a set of images.
My task was to check wether a mechanical object contained in a photo matched a given object type, of which I had a small dataset of images.

This had a few issues: the datasets for the single object types were all quite small, sometimes the objects were not the focus of the images, and many objects would look very different from different angles. Training a recognition model on a dataset like that would have been very expensive, and it would still have had a hard time recognizing objects for which it was trained on one or two images as in the smallest datasets. The CLIP model offered a solution that is inexpensive (as it runs very fast even on a weak CPU), easy to use and understand, and reliable.

Image-to-Image Similarity

The CLIP Model is mainly used for text-text or image-text similarity, but for the way it works it can be used for image-image similarity as well. I'm not going into too much detail about how the CLIP Model works (more on that here), but basically it encodes both images and texts in the same format (in a shared vector space, for linear algebra enthusiasts), and then compares those encodings and can compute a similarity value in percentage between them. What this means, practically, is that all images and texts are processed in the same way, and all of the encodings are comparable. For my specific task, I just had the model comparing the given image to all the images in the dataset of a certain object type until it found one that was similar enough to consider it a "good match" (I swear this works better than Tinder).

Improving Performance

Even though the similarity detection process is incredibly fast and can give (using the clip-ViT-B-32 model) the similarity score between dozens of images in under a second without the need to operate on a GPU, the encoding process takes a bit longer and would significantly increase response times even for small datasets.
A quick workaround was to "pre-build" and save the encodings for the various object types and, when needed for the similarity calculations, to load that pre-built encoding list instead of loading and encoding all the images again.

def buildEncoding(typeID: str):
    encodedImages = [encodeImage(image).tolist() for image in S3_service.getImagesByType(typeID)] //everything is saved on an Amazon S3 bucket
    if len(encodedImages) != 0:
        saveEncoding(encodedImages, typeID)

This allows the program to only loose time building the encoding for the single image we want to identify.

Expanding the DataSet

There is one more big advantage this implementation brings: since the model is not trained directly on the dataset, as it is instead trained for image recognition in general, adding an image to the dataset doesn't require a full re-training of the model. All it takes is to add the encoding of a successfully matching image to the encoding list of the right type, making the recognition more and more reliable every time an image matches.

def recognizeImage(queryImage: UploadFile, typeID: str):
    name = queryImage.filename
    queryImage = Image.open(queryImage.file)
    encodedImages = S3_service.loadEncoding(typeID)
    queryEncoding = encoding_builder.encodeImage(queryImage)

    for image in encodedImages:
        processed_images = util.cos_sim(image, queryEncoding)
        print("Processed image: " + str(processed_images[0][0].item() * 100) + '%')
        if processed_images[0][0].item() > 0.92: //the threshold to identify a "good" match

            S3_service.saveImage(queryImage, typeID, name)
            encoding_builder.appendEncoding(encodedImages, queryEncoding, typeID)

            return True
    return False

Implementation

If you're not familiar with using the CLIP model, also check this link.
The two libraries needed to make this work are Pillow's Image module for loading images and the Sentence Transformers Framework.
The first thing to do is to load the CLIP model, which is then going to be used to encode the images

from sentence_transformers import SentenceTransformer

CLIPmodel = SentenceTransformer('clip-ViT-B-32')
// note: you can load a local model from your machine
//       by putting the path instead of the model name

Then, to encode an image, all you need is to call the encode() function on the model, passing the image opened with PIL

from PIL import Image
encoding = CLIPmodel.encode(Image.open(image_path))

Finally, the util module of Sentence Transformer lets us compute the (cosine) similarity between encodings through various methods

from sentence_transformers import util

similarities = util.cos_sim(trainEncoding, queryEncoding)

Conclusion

The CLIP model is a very powerful tool that can help in many image recognition tasks, and in many cases it can significantly improve performance and accuracy more than most other methods, even in my situation, with a kind of image that differs wuite a lot and is way more specific than the dataset that CLIP was trained on. And don't forget that the model can be fine-tuned to improve the accuracy even more, but that may be for another article...

Otello: how to make Chat-GPT interact with your API

mattialodi0 — Fri, 01 Mar 2024 13:28:29 +0000

During my internship I had to build a chatbot capable of connecting to a private API. Building an LLM from scratch was out of question, so I made a little reseach and found out there is not much literacy about this task.
The two most promising services were OpenAI API and SuperagentAI.
I opted for the first one, because of a better documentation and greater support.

So here is how i built Otello:

What is it

First of all, let me specify what we are talking about:
Otello is a chatbot that enables the user to make questions about the information present in our company Web app.
It gives the user an easy and natural way to get the data without the need of navigating trough the interface.

How it's made

The chatbot is composed of three sequential components:

AskCompletion calls the OpenAI Completion to identify the resources requested by the user. You need to list all the possible resources and configure a function call tool

const completion = await openai.chat.completions.create({
        messages: [{ role: "system", content: context }, { role: "user", content: user_request }],
        model: "ft:gpt-3.5-turbo-0613:personal::XXXXXXXX",
        tools: [
            {
                "type": "function",
                "function": {
                    "name": "api_fetch",
                    "description": "fetches the API and returns the requested resource",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "resource": {
                                "type": "string",
                                "description": "requested resource name"
                            }
                        },
                        "required": [
                            "resources"
                        ]
                    }
                }
            }
        ],
        tool_choice: "auto",
    });

FetchAPI: makes a call to our API based on the resources identified, and retrive the data in JSON format
AskAssitant: send a request to a custom OpenAI Assistant with the user question and the data needed to aswer it.

Why Completion?

Because Completion is simple, fast and, with a small drawback, it allows you to use a fine tuned model: a base GPT model trained on your data, perfect to have more control on it. (yes, you need to create a dataset from scratch with a decent nubmer of chats to train the model)

{"messages": [{"role": "user", "content": "User question 1"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"resource\": \"presenze_bloccate\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource name"}},"required": ["resources"]}}]}
{"messages": [{"role": "user", "content": "User question 2"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"resource\": \"presenze_bloccate\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource name"}},"required": ["resources"]}}]}
{"messages": [{"role": "user", "content": "User question 3"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"presenze_bloccate\": \"presenze\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource namea"}},"required": ["resources"]}}]}
...

But...

Ok, now that the architecture is clear, let's talk about the main issue with this approach: how do you make an OpenAI GPT answer question based on your data? or even better, how do you upload the data it needs?
I found two possible approaches:

Big model - the easy way, just use a GPT which allows a large amount of input tokens (long questions) and add them to the user request. It is easy but definitly not cheap nor scalable for large quantities of data. Some examples are: gpt-3.5-turbo-16k, gpt-4-32k

const message = await openai.beta.threads.messages.create(thread_id, 
    {
        role: "user",
        content: user_request + ' The data you need are: ' + data,
    }
);
const run = await openai.beta.threads.runs.create(thread_id, { assistant_id });

File assistant - the hard way, you can upload a file in a supported format (like .JSON) and tell the assistant to use its content in the answer. It is harder to implement and slower but far more cheap and supports up to 150 Gb files.

let file = await openai.files.create({ file: output_files[0].file, purpose: "assistants" });
file = await openai.beta.assistants.files.create(assistantF_id, { file_id: file.id });
file_ids.push(file.id);
redirection = `the data you need ${output_files[0].label} are in the file: ${file.id}`;

const thread = await openai.beta.threads.create();

await openai.beta.threads.messages.create(
    thread.id,
        {
            role: "user",
            content: user_request + redirection,
            content: `${user_request}. ${redirection}`,
            file_ids
        }
    );
let run = await openai.beta.threads.runs.create(thread.id, { assistant_id: assistantF_id });

I built the backend with Node.js, but Python could have been an alternative too, and after a simple frontent in pure Javascript, it was time to deploy it.

Deployment

The deployment of the backend had been in an AWS Lambda function, even if it is not suited well for projects like this, where is needed a high processing time.

Conclusion

Even if the technologies are still quite new, building a personalized chatbot might be easier than you think, if you don't want anything too fancy.

If this caught your attention and you want to know more, you'll find the complete code with all the possible variations here: github-repo

Hope this helps